Posts

Comments

Comment by Archimedes on Anthropic release Claude 3, claims >GPT-4 Performance · 2024-03-04T23:23:46.692Z · LW · GW

AI Explained already has a video out on it:

The New, Smartest AI: Claude 3 – Tested vs Gemini 1.5 + GPT-4

Comment by Archimedes on Why you, personally, should want a larger human population · 2024-02-24T03:02:49.143Z · LW · GW

Supposing humanity is limited to Earth, I can see arguments for ideal population levels ranging from maybe 100 million to 100 billion with values between 1 and 10 billion being the most realistic. However, within this range, I'd guess that maximal value is more dependent on things like culture and technology than on the raw population count, just like a sperm whale's brain being ~1000x the mass of an African grey parrot's brain doesn't make it three orders of magnitude more intelligent.

Size matters (as do the dynamic effects of growing/shrinking) but it's not a metric I'd want to maximize unless everything else is optimized already. If you want more geniuses and more options/progress/creativity, then working toward more opportunities for existing humans to truly thrive seems far more Pareto-optimal to me.

Comment by Archimedes on When do "brains beat brawn" in Chess? An experiment · 2024-02-23T22:04:00.126Z · LW · GW

Leela now has a contempt implementation that makes odds games much more interesting. See this Lc0 blog post (and the prior two) for more details on how it works and how to easily play odds games against Leela on Lichess using this feature.

GM Matthew Sadler also has some recent videos about using WDL contempt to find new opening ideas to maximize chances of winning versus a much weaker opponent.

I'd bet money you can't beat LeelaQueenOdds at anything close to a 90% win rate.

Comment by Archimedes on The case for training frontier AIs on Sumerian-only corpus · 2024-01-16T02:23:05.336Z · LW · GW

Why not go all the way and use a constructed language (like Lojban or Ithkuil) that's specifically designed for the purpose?

Comment by Archimedes on Nonlinear’s Evidence: Debunking False and Misleading Claims · 2023-12-26T21:18:55.624Z · LW · GW

All the pics and bragging about how wonderful their adventures were really rub me the wrong way. It comes across as incredibly tone deaf to the allegations and focusing on irrelevant things. Hot tubs, beaches, and sunsets are not so important if you’re suffering from deeper issues. Good relationship dynamics are way more important than scenery and perks, especially in a small group setting.

Comment by Archimedes on When will GPT-5 come out? Prediction markets vs. Extrapolation · 2023-12-12T03:29:00.159Z · LW · GW

The markets you linked to roughly align with my intuition that GPT-5 before 2025 is likely, although maybe not fully released publicly.

Things to keep in mind:

  1. We know that GPT-5 is officially in development.
  2. The hype since ChatGPT went public is huge and competition is substantial.
  3. Because of hype and competition, investment will be large in terms of dollars and talent.
  4. There will be strong incentive to show a return on investment rather than holding back and waiting for something worthy of calling "GPT-5".

Other than the amount of investment in next-gen models, most of my intuition is related to the human and marketing factors involved. OpenAI won't want to lose its lead by waiting many years.

Comment by Archimedes on So you want to save the world? An account in paladinhood · 2023-11-23T02:16:03.197Z · LW · GW

That sounds rather tautological.

Assuming ratfic represents LessWrong-style rationality well and assuming LW-style rationality is a good approximation of truly useful instrumental reasoning, then the claim should hold. There’s room for error in both assumptions.

Comment by Archimedes on My AI Predictions 2023 - 2026 · 2023-10-16T22:52:53.524Z · LW · GW

My hunch is that there's sufficient text already if an AI processes it more reflectively. For example, each chunk of text can be fed through a series of LLM prompts intended to enrich it, and then the model trains on the enriched/expanded text.

Comment by Archimedes on A quick remark on so-called “hallucinations” in LLMs and humans · 2023-09-26T02:26:26.089Z · LW · GW

I definitely like the term "confabulate" more than "hallucinate". It's more accurate and similar to what humans do. My favorite confabulation examples in humans are split-brain experiments.

"split-brain" patients, whose left and right brain hemispheres have been surgically disconnected for medical treatment. Neuroscientists have devised clever experiments in which information is provided to the right hemisphere (for instance, pictures of naked people), causing a change in behavior (embarrassed giggling). Split-brain individuals are then asked to explain their behavior verbally, which relies on the left hemisphere. Realizing that their body is laughing, but unaware of the nude images, the left hemisphere will confabulate an excuse for the body's behavior ("I keep laughing because you ask such funny questions, Doc!").

https://www.edge.org/response-detail/11513

Comment by Archimedes on Paper: LLMs trained on “A is B” fail to learn “B is A” · 2023-09-26T01:52:19.204Z · LW · GW

I found your thread insightful, so I hope you don't mind me pasting it below to make it easier for other readers.

Neel Nanda ✅ @NeelNanda5 - Sep 24

The core intuition is that "When you see 'A is', output B" is implemented as an asymmetric look-up table, with an entry for A->B. B->A would be a separate entry

The key question to ask with a mystery like this about models is what algorithms are needed to get the correct answer, and how these can be implemented in transformer weights. These are what get reinforced when fine-tuning.

The two hard parts of "A is B" are recognising the input tokens A (out of all possible input tokens) and connecting this to the action to output tokens B (out of all possible output tokens). These are both hard! Further, the A -> B look-up must happen on a single token position

Intuitively, the algorithm here has early attention heads attend to the prev token to create a previous token subspace on the Cruise token. Then an MLP neuron activates on "Current==Cruise & Prev==Tom" and outputs "Output=Mary", "Next Output=Lee" and "Next Next Output=Pfeiffer"

"Output=Mary" directly connects to the unembed, and "Next Output=Lee" etc gets moved by late attention heads to subsequent tokens once Mary is output.

Crucially, there's an asymmetry between "input A" and "output A". Inputs are around at early layers, come from input embeddings, and touch the input weights of MLP neurons. Outputs are around more at late layers, compose with the unembedding, and come from output weights of MLPs

This is especially true with multi-token A and B. Detecting "Tom Cruise" is saying "the current token embedding is Cruise, and the prev token space says Tom", while output "Tom Cruise" means to output the token Tom, and then a late attn head move "output Cruise" to the next token

Thus, when given a gradient signal to output B given "A is" it reinforces/creates a lookup "A -> B", but doesn't create "B->A", these are different lookups, in different parameters, and there's no gradient signal from one to the other.

How can you fix this? Honestly, I can't think of anything. I broadly think of this as LLMs working as intended. They have a 1 way flow from inputs to outputs, and a fundamental asymmetry between inputs and outputs. It's wild to me to expect symmetry/flow reversing to be possible

Why is this surprising at all then? My guess is that symmetry is intuitive to us, and we're used to LLMs being capable of surprising and impressive things, so it's weird to see something seemingly basic missing.

LLMs are not human! Certain things are easy for us and not for them, and vice versa. My guess is that the key difference here is that when detecting/outputting specific tokens, the LLM has no notion of a variable that can take on arbitrary values - a direction has fixed meaning

A better analogy might be in-context learning, where LLMs CAN use "variables". The text "Tom Cruise is the son of Mary Lee Pfeiffer. Mary Lee Pfeiffer is the mother of..." has the algorithmic solution "Attend to the subject of sentence 1 (Tom Cruise), and copy to the output"

Unsurprisingly, the model has no issue with reversing facts in context! Intuitively, when I remember a fact A is B, it's closer to a mix of retrieving it into my "context window" and then doing in-context learning, rather than pure memorised recall.

Comment by Archimedes on Lack of Social Grace Is an Epistemic Virtue · 2023-08-01T02:42:22.474Z · LW · GW

Adversarial gaming doesn't match my experience much at all and suggesting options doesn't feel imposing either. For me at least, it's largely about the responsibility and mental exertion of planning.

In my experience, mutual "where do you want to go" is most often when neither party has a strong preference and neither feels like taking on the cognitive burden of weighing options to come to a decision. Making decisions takes effort especially when there isn't a clearly articulated set of options and tradeoffs to consider.

For practical purposes, one person should provide 2-4 options they're OK with and the other person can pick one option or veto some option(s). If they veto all given options, they must provide their own set of options the first person can choose or veto. Repeat as needed but rarely is more than one round needed unless participants are picky or disagreeable.

Comment by Archimedes on An upcoming US Supreme Court case may impede AI governance efforts · 2023-07-20T03:24:32.050Z · LW · GW

Why do you think that the same federal bureaucrats who incompetently overregulate other industries will do a better job regulating AI?

Chevron deference means that judges defer to federal agencies instead of interpreting the laws themselves where the statute is ambiguous. It's not so much a question of overregulation vs underregulation as it is about who is doing the interpretation. For example, would you rather the career bureaucrats in the Environmental Protection Agency determine what regulations are appropriate to protect drinking water or random judges without any relevant expertise?

One consequence of blowing up Chevron deference is that one activist judge in Texas can unilaterally invalidate FDA approval of a drug like mifepristone for the entire country that's been safe, effective, and available on the markets for decades by substituting his own idiosyncratic opinion instead of deferring to the regulatory agency whose entire purpose is to make those kinds of determinations.

Government agencies aren't always competent but the alternative is a patchwork of potentially conflicting decisions from judges ruling outside of their area of expertise.

Comment by Archimedes on When do "brains beat brawn" in Chess? An experiment · 2023-06-30T02:04:49.284Z · LW · GW

If you're open to more experimentation, I'd recommend trying playing against Leela Chess Zero using some of the newer contempt parameters introduced in this PR and available in the latest pre-release version. I'm really curious if you'd notice significant style differences with different contempt settings.

Update: The official v0.30.0 release is out now and there is a blog post detailing the contempt settings. Additionally, there is a Lichess bot set up specifically for knight odds games.

Further update: There are now three Lichess bots set up to play odds games on Lichess: LeelaKightOdds, LeelaQueenForKnight, and LeelaQueenOdds. They are currently featured bots on https://lichess.org/player/bots

Comment by Archimedes on Why "AI alignment" would better be renamed into "Artificial Intention research" · 2023-06-16T01:56:00.442Z · LW · GW

"Artificial Intention" doesn't sound catchy at all to me, but that's just my opinion.

Personally, I prefer to think of the "Alignment Problem" more generally rather than "AI Alignment". Regardless of who has the most power (humans, AI, cyborgs, aliens, etc.) and who has superior ethics, conflict arises when participants in a system are not all aligned.

Comment by Archimedes on UFO Betting: Put Up or Shut Up · 2023-06-14T23:08:34.675Z · LW · GW

This is the most similar question that I could find that already exists.

https://www.metaculus.com/questions/7384/alien-tech-in-solar-system-before-2030/

Comment by Archimedes on Transformative AGI by 2043 is <1% likely · 2023-06-08T01:46:53.481Z · LW · GW

There are still HR and legal overhead costs involved if you have human operators.

I think part of the answer is also that the space of things low-wage workers can physically do remotely via teleoperation isn't that much larger than the space of things that can be fully automated but still much smaller than the space of things a local human can do. It's a fairly narrow band to exploit, IMO, and the labor cost arbitrage spread is rarely worth the complexity of the extra logistics, capital investment, and maintenance.

Comment by Archimedes on Open Thread With Experimental Feature: Reactions · 2023-05-27T14:30:57.560Z · LW · GW

As @kave pointed out already, the right side being thicker would make it more clear that it's specifically a delta.

Comment by Archimedes on Open Thread With Experimental Feature: Reactions · 2023-05-27T14:08:30.508Z · LW · GW

Something like one of these?

https://thenounproject.com/icon/bored-251902/ https://thenounproject.com/icon/annoyed-4979573/

Comment by Archimedes on I bet $500 on AI winning the IMO gold medal by 2026 · 2023-05-13T02:37:11.045Z · LW · GW

Embodied tasks just aren't an area where comparison makes much sense yet. What kind of perception tasks did you have in mind?

Comment by Archimedes on I bet $500 on AI winning the IMO gold medal by 2026 · 2023-05-12T03:10:13.684Z · LW · GW

Does this "paradox" still hold in the era of recent multimodal AI? In particular, what are some things that are easy for humans but hard for AI, other than things requiring embodiment? What areas are human mechanical Turks still much better at? (I believe there are areas but pretty fuzzy about what they are.)

Comment by Archimedes on Clarifying and predicting AGI · 2023-05-05T03:51:54.234Z · LW · GW

I really like the t-AGI concept for a specific task or benchmark but find it difficult to generalize since capabilities are not uniform. How do you classify an AGI that is 1-second AGI at some tasks but 1-day AGI at others (e.g. due to serial processing speed advantages)?

Comment by Archimedes on Accuracy of arguments that are seen as ridiculous and intuitively false but don't have good counter-arguments · 2023-04-30T17:25:06.332Z · LW · GW

There is indeed the Myth of the flat Earth that is a misconception about the beliefs of scholars in the Middle Ages and some scholars certainly understood the concept of a spherical Earth since at least Eratosthenes. I'm referring to earlier history like ancient Egyptian, Chinese, and pre-Socratic Greek cosmologies. Admittedly, it's not a great example since most of the debates about it are lost to history, and such debates wouldn't involve the same kind of reasoning and evidential standards we use today.

Comment by Archimedes on Contra Yudkowsky on Doom from Foom #2 · 2023-04-30T17:04:25.057Z · LW · GW

Yes, clearly the less time the human has, the better Leela will do relatively. One thing to note though is that Lichess Elo isn't completely comparable across different time controls. If you look at the player leaderboard, you can see that the top scores for bullet are ~600 greater than for classical, so scores need to be interpreted in context.

Self-Elo inflation is a fair point to bring up and I don't have information on how well it translates.

Comment by Archimedes on Contra Yudkowsky on Doom from Foom #2 · 2023-04-30T05:20:51.941Z · LW · GW

791556 is nowhere near the strongest network available. It's packaged with lc0 as a nice small net. The BT2 net currently playing at tcec-chess.com is several hundreds of Elo stronger than T79 and likely close to superhuman level, depending on the time control. It's not the very latest and greatest, but it is publicly available for download and should work with the 0.30.0-rc1 pre-release version of lc0 that supports the newer transformer architecture if you want to try it yourself. If you only want completely "official" nets, at least grab one of the latest networks from the main T80 run.

I'm not confident that BT2 is strictly superhuman using pure policy but I'm pretty sure it's at least close. LazyBot is a Lichess bot that plays pure policy but uses a T80 net that is likely at least 100 Elo weaker than BT2.

Comment by Archimedes on Accuracy of arguments that are seen as ridiculous and intuitively false but don't have good counter-arguments · 2023-04-30T04:12:50.698Z · LW · GW

Historical examples of things that once sounded ridiculous but turned out to be true:

  1. Spherical Earth (vs Flat Earth)
  2. Heliocentrism (vs Geocentrism)
  3. Germ theory (vs e.g. Miasmatic theory)
  4. Evolution via natural selection
  5. Quantum mechanics
  6. Relativity
  7. Plate tectonics
  8. Black holes

It's harder to know what qualifies as false examples since they do (now) have good counterarguments, but maybe something like these:

  1. Phlogiston theory
  2. Vitalism
  3. Luminiferous aether
  4. Lamarckian evolution
  5. Cold fusion
  6. Steady State cosmology (vs Big Bang)
  7. Caloric theory
  8. Spontaneous generation

Examples of ideas with less certain status:

  1. String theory / quantum gravity / unified physics
  2. Multiverse hypothesis / simulation hypothesis
  3. Existence and nature of extraterrestrial life
  4. Nature of dark matter & dark energy
  5. Epigenetic roles in disease and inheritance
  6. Origins of life / abiogenesis / panspermia
  7. Nature of consciousness and reality
Comment by Archimedes on Will GPT-5 be able to self-improve? · 2023-04-30T03:05:36.918Z · LW · GW

LLMs are truly generating novel ideas, they are just interpolating between existing ideas via memorized statistical patterns.

Is this intended to be "LLMs are not truly generating novel ideas"?

Comment by Archimedes on Contra Yudkowsky on Doom from Foom #2 · 2023-04-28T04:04:10.542Z · LW · GW

I don’t know much about Leela Zero and Katago but I do know that Leela Chess Zero (lc0) without search (pure policy) is near superhuman levels. I’ll see if I can dig up more precise specifics.

Comment by Archimedes on $250 prize for checking Jake Cannell's Brain Efficiency · 2023-04-27T23:19:44.524Z · LW · GW

I'm not interested in the prize, but as long as we're spot-checking, this paragraph bothered me:

It turns out that spreading out the communication flow rate budget over a huge memory store with a slow clock rate is fundamentally more powerful than a fast clock rate over a small memory store. One obvious reason: learning machines have a need to at least store their observational history. A human experiences a sensory input stream at a bitrate of about 10^6 bps (assuming maximal near-lossless compression) for about 10^9 seconds over typical historical lifespan, for a total of about 10^15 bits. The brain has about 2∗10^14 synapses that store roughly 5 bits each, for about 10^15 bits of storage. This is probably not a coincidence.

The idea of bitrate * lifespan = storage capacity makes sense for a VHS or DVD but human memory is completely different. Only a tiny percentage of all our sensory input stream makes it through memory consolidation into long-term memory. Sensory memory is also only one type of memory alongside others like semantic, episodic, autobiographical, and procedural (this list is neither mutually exclusive nor collectively exhaustive). Brains are very good at filtering the sensory stream to focus on salient information and forgetting the vast majority of the rest. This results in memory that is highly compressed (and highly lossy).

Because brain memory is so different than simple storage, this napkin math is roughly analogous to computing body mass from daily intake of air, water, and food mass and neglecting how much actually gets stored versus ultimately discarded. You can do the multiplication but it's not very meaningful without additional information like retention ratios.

Comment by Archimedes on Fast Minds and Slow Computers · 2023-04-26T03:38:45.525Z · LW · GW

Have the distributed architecture trends and memristor applications followed the rough path you expected when you wrote this 12 years ago? Is this or this the sort of thing you were gesturing at? Do you have other links or keywords I could search for?

Comment by Archimedes on Contra Yudkowsky on AI Doom · 2023-04-24T01:44:26.528Z · LW · GW

Biology is incredibly efficient at certain things that happen at the cell level. To me, it seems like OP is extrapolating this observation rather too broadly. Human brains are quite inefficient at things they haven't faced selective pressure to be good at, like matrix multiplication.

Claiming that human brains are near Pareto-optimal efficiency for general intelligence seems like a huge stretch to me. Even assuming that's true, I'm much more worried about absolute levels of general intelligence rather than intelligence per Watt. Conventional nuclear bombs are dangerous even though they aren't anywhere near the efficiency of a theoretical antimatter bomb. AI "brains" need not be constrained by the size and energy constraints of a human brain.

Comment by Archimedes on The basic reasons I expect AGI ruin · 2023-04-19T04:16:37.857Z · LW · GW

Similar to humans, LLMs can do 6-digit multiplication with sufficient prompting/structure!

https://www.lesswrong.com/posts/XvorpDSu3dwjdyT4f/gpt-4-multiplication-competition

Comment by Archimedes on GPTs are Predictors, not Imitators · 2023-04-09T03:44:19.605Z · LW · GW

The Bailey-Borwein-Plouffe formula is a nice one.

Comment by Archimedes on All AGI Safety questions welcome (especially basic ones) [April 2023] · 2023-04-09T03:17:14.392Z · LW · GW

The big one is obviously "make long time scale plans to navigate a complicated 3D environment, while controlling a floppy robot."

This is like judging a dolphin on its tree-climbing ability and concluding it's not as smart as a squirrel. That's not what it was built for. In a large number of historically human domains, GPT-4 will dominate the village idiot and most other humans too.

Can you think of examples where it actually makes sense to compare GPT and the village idiot and the latter easily dominates? Language input/output is still a pretty large domain.

Comment by Archimedes on Risks from GPT-4 Byproduct of Recursively Optimizing AIs · 2023-04-07T16:49:49.445Z · LW · GW

Readers of this may also be interested in this post from 2015:

Should AI Be Open?

Comment by Archimedes on Is "Recursive Self-Improvement" Relevant in the Deep Learning Paradigm? · 2023-04-07T03:59:04.363Z · LW · GW

I don't have an inside view. If I did, that would be pretty powerful capabilities information.

I'm pointing at the possibility that we already have more than sufficient resources for AGI and we're only separated from it by a few insights (a la transformers) and clever system architecture. I'm not predicting this is true just that it's plausible based on existing intelligent systems (humans).

Epistemic status: pondering aloud to coalsce my own fuzzy thoughts a bit

I'd speculate that the missing pieces are conceptually tricky things like self-referential "strange loops", continual learning with updateable memory, and agentic interactions with an environment. These are only vague ideas in my mind but, for some reason, feel difficult to solve but don't feel like things that require massive data and training resources so much as useful connections to reality and itself.

Comment by Archimedes on Is "Recursive Self-Improvement" Relevant in the Deep Learning Paradigm? · 2023-04-06T23:52:24.515Z · LW · GW

Resource accumulation certainly can't grow exponentially indefinitely and I agree that RSI can't improve exponentially forever either, but it doesn't need to for AI to take over.

An AI doesn't have to get far beyond human-level intelligence to control the future. If there's sufficient algorithmic overhang, current resources might even be enough. FOOM would certainly be easier if no new hardware were necessary. This would look less like an explosion and more like a quantum leap followed by slower growth as physical reality constrains rapid progress.

Comment by Archimedes on Is "Recursive Self-Improvement" Relevant in the Deep Learning Paradigm? · 2023-04-06T23:24:45.567Z · LW · GW

I agree that humans are not drastically more intelligent than all other animals. This makes the prospect of AI even scarier, in my opinion, since it shows how powerful accumulated progress is.

I believe that human-level intelligence is sufficient for an AI to be extremely dangerous if it can scale while maintaining self-alignment in the form of "synchronized behavior and collective action". Imagine what a tech company could achieve if all employees had the same company-aligned goals, efficient coordination, in silico processing speeds, high-bandwidth communication of knowledge, etc. With these sorts of advantages, it's likely game over before it hits human-level intelligence across the board.

Comment by Archimedes on Is "Recursive Self-Improvement" Relevant in the Deep Learning Paradigm? · 2023-04-06T22:55:16.269Z · LW · GW

Until AI is at least as sample-efficient and energy-efficient as humans are at learning, there are significant algorithmic gains that are possible. This may not be possible under the current deep-learning paradigm but we know it's possible under some paradigm since evolution has already accomplished it blindly.

I do share your skepticism that something like an LLM alone could recursively improve itself quickly. Assuming FOOM, my model of how it happened has deep learning as only part of the answer. It's part of the recursive loop but is used mostly as a general heuristic module, much like the neural net of a chess engine is only a piece of the puzzle; you still need a fast search algorithm that uses the heuristics efficiently.

Comment by Archimedes on Orthogonality is expensive · 2023-04-03T21:56:25.129Z · LW · GW

Agreed. Only a very weak form of orthogonality is necessary to have dangerously unaligned AI be the default.

Comment by Archimedes on Why I Think the Current Trajectory of AI Research has Low P(doom) - LLMs · 2023-04-03T04:12:41.889Z · LW · GW

Since you are checking your thoughts, here are a couple of posts that argue the contrary to your 1.2:

https://www.lesswrong.com/posts/MmmPyJicaaJRk4Eg2/the-limit-of-language-models

https://www.lesswrong.com/posts/mEbYbekPXoWvrHraW/human-level-ai-can-plausibly-take-over-the-world

Comment by Archimedes on Some thought experiments on digital consciousness · 2023-04-03T03:09:32.473Z · LW · GW

So as a rough analogy, if you were a computer program, the conscious part of the execution would be kind of like log output from a thread monitoring certain internal states?

Comment by Archimedes on Some thought experiments on digital consciousness · 2023-04-02T21:06:31.018Z · LW · GW

Funny that you should mention élan vital. The more I read about it, the more "consciousness" seems to me to be similarly incoherent and pseudoscientific as vitalism. This isn't a fringe view and I'd recommend skimming the Rejection of the Problem section of the Hard problem of consciousness page on Wikipedia for additional context. It's hard not to be confused about a term that isn't coherent to begin with.

Supposing each scenario could be definitively classified as conscious or not, would that help you make any predictions about the world?

Comment by Archimedes on Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky · 2023-04-01T04:15:36.661Z · LW · GW

One of the simplest ways for AI to have power over the physical world is via humans as pawns. A reasonably savvy AI could persuade/manipulate/coerce/extort/blackmail real-life people to carry out the things it needs help with. Imagine a powerful mob boss who is superintelligent, never sleeps, and continuously monitors everyone in their network.

Comment by Archimedes on Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky · 2023-04-01T03:50:48.126Z · LW · GW

Not many people consider GPT-4 extremely dangerous on its own. Hooking up something at that level of intelligence into a larger system with memory storage and other modules is a bit more threatening, and probably sufficient to do great harm already if wielded by malevolent actors unleashing it onto social media platforms, for example.

The real danger is that GPT-4 is a mile marker we've blown by on the road to ever more capable AI. At some point, likely before climate change becomes an existential threat, we lose control and that's when things get really weird, unpredictable, and dangerous.

Eliezer has near-zero hope for humanity's survival. I think we'd all agree that the universal agreement he suggests is not something plausible in the current world. He's not advocating for it because he believes it might happen but rather it's the only thing he thinks might be enough to give us a shot at survival.

Comment by Archimedes on High Status Eschews Quantification of Performance · 2023-03-25T23:36:18.913Z · LW · GW

Interesting. I wonder how much selection bias there was in responses.

Comment by Archimedes on High Status Eschews Quantification of Performance · 2023-03-25T03:38:53.150Z · LW · GW

The SAT in-contrast is kind of useful anyways, because kind of everyone saturates on practicing for the SAT, at least in the US, but that test sadly also maxes out at an average that is below the average IQ test result in the LessWrong community

Am I correct in interpreting this as you implying that the average member of the LessWrong community got perfect SAT scores (or would have had they taken it)?

Comment by Archimedes on Microsoft Research Paper Claims Sparks of Artificial Intelligence in GPT-4 · 2023-03-25T00:25:14.404Z · LW · GW

In my opinion, GPT-4 is intelligent but needs extra infrastructure to allow memory and continuous learning to become an agentic AGI. It might not be that hard to add that infrastructure though and people are already trying: https://github.com/noahshinn024/reflexion-human-eval

Comment by Archimedes on Retrospective on ‘GPT-4 Predictions’ After the Release of GPT-4 · 2023-03-18T20:51:08.842Z · LW · GW

OK. Let's make it even more extreme. Suppose you take a commercial flight. The likelihood of dying in a crash is on the order of 1 in 10 million. From a percent error or absolute error perspective, 99.99999% isn't that different from 99% but that is the difference between one plane crash per year globally and a couple of dozen plane crashes per hour on average. These are wildly different in terms of acceptable safety.

There's a backup link in the comments: https://www.thejach.com/public/log-probability.pdf

Comment by Archimedes on Wittgenstein's Language Games and the Critique of the Natural Abstraction Hypothesis · 2023-03-18T18:48:14.180Z · LW · GW

14 is certainly the most likely continuation but it could also be

  • 16 if it's a list of numbers k where k^2 + 7 is prime
  • 18 if it's a list of numbers of the form 3^i +/- 3^j

These continuations are unlikely in general but are the kind of thing that might show up in an academic mathematics paper.

Comment by Archimedes on Retrospective on ‘GPT-4 Predictions’ After the Release of GPT-4 · 2023-03-18T18:23:25.223Z · LW · GW

Suppose you predicted 91% but the actual value was 99%. The percent error may only be about 8% but the likelihood of a wrong answer is 1/100 instead of your predicted 9/100, which is a huge difference.

You may be interested in the links in this post: https://www.lesswrong.com/posts/6Ltniokkr3qt7bzWw/log-odds-or-logits