LessWrong 2.0 Reader

OK, thanks. So, I totally buy that e.g. pretrained GPT-4 has a pretty good understanding of concepts like honesty and morality and niceness, somewhere in its giant library of concepts it understands. The difficulty is getting it to both (a) be a powerful agent and (b) have its goals be all and only the concepts we want them to be, arranged in the structure we want them to be arranged in.

An obvious strategy is to use clever prompting and maybe some other techniques like W2SG to get a reward model that looks at stuff and classifies it by concepts like honesty, niceness, etc. and then use that reward model to train the agent. (See Appendix G of the W2SG paper for more on this plan)

However, for various reasons I don't expect this to work. But it might.

Are you basically saying: This'll probably work? 

(In general I like the builder-breaker dialectic in which someone proposes a fairly concrete AGI alignment scheme and someone else says how they think it might go wrong)

If your text generation algorithm is "repeatedly sample randomly (at a given temperature) from a probability distribution over tokens", that means you control a stream of bits which don't matter for output quality but which will be baked into the text you create (recoverably baked in if you have access to the "given a prefix, what is the probability distribution over next tokens" engine).

So at that point, you're looking for "is there some cryptographic trickery which allows someone in possession of a secret key to determine whether a stream of bits has a small edit distance from a stream of bits they could create, but where that stream of bits would look random to any outside observer?" I suspect the answer is "yes".

That said, this technique is definitely not robust to e.g. "translate English text to French and then back to English" and probably not even robust to "change a few tokens here and there".

Alternatively, there's the inelegant but effective approach of "maintain an index of all the text you have ever created and search against that index, as the cost to generate the text is at least an order of magnitude higher^[1] than the cost to store it for a year or two".

1. ^{^}

   I see $0.3 / million tokens generated  on OpenRouter for llama-3.1-70b-instruct, which is just about the smallest model size I'd imagine wanting to watermark the output for. A raw token is about 2 bytes, but let's bump that up by a factor of 50 - 100 to account for things like "redundancy" and "backups" and "searchable text indexes take up more space than the raw text". So spending $1 on generating tokens will result in something like 0.5 GB of data you need to store.

   Quickly-accessible data storage costs something like $0.070 / GB / year, so "generate tokens and store them in a searchable place for 5 years" would be about 25-50% more expensive than "generate tokens and throw them away immediately".

Good points. I probably should have said "the midas problem" (quoting Cold Takes) instead of "outer alignment." Idk. I didn't choose my terms carefully.

Eli Lifland asked:

I'm curious to what extent you guys buy this argument: we should be thankful that LLMs are SOTA and might scale to AGI, because they are more oracle-y than agent-y compared to other methods we could have imagined using, and this is likely to continue in relative terms

I answered:

I think about it like this: AGI was always going to be an agent. (Well, we can dream about CAIS-world but it was never that likely.) The question is which capabilities come in which order. Ten years ago people saw lots of RL on video games and guessed that agency would come first, followed by real-world-understanding. Instead it's the opposite, thanks to the miracle of pretraining on internet text.In some sense, we got our oracle world, it's just that our oracles (Claude, GPT4...) are not that useful because they aren't agents... as some would have predicted...Is it better to have agency first, or world-understanding first? I'm not sure. The argument for the latter is that you can maybe do schemes like W2SG and scalable oversight more easily; you can hopefully just summon the concepts you need without having to worry about deceptive alignment so much.The argument for the former is that it might be better for wakeup/governance/etc. if we had bloodthirsty vicious video-game-winning bots that behaved like some sort of alien tiger, and then tech companies were trying to train them to speak english and obey human commands... possibilities like deceptive alignment would seem obviously so much more salient and scary. So people would be taking said possibilities seriously even when the AIs lack sufficient common sense to deceive humans successfully -- by contrast with today's world where the AIs totally know how to pretend to be aligned and the conversation focuses on how there is No Evidence that they are doing that in practice.

As the person who requested of MIRI to release the Sequences as paper books in the first place, I have asked MIRI to release the rest of them, and credibly promised to donate thousands of dollars if they did so. Given the current situation vis-a-vis AI, I'm not that surprised that it still does not appear to be a priority to them, although I am disappointed.

MIRI, if you see this, yet another vote for finishing the series! And my offer still stands!

Curating content is effectively a battle against advertising, and simultaneously a form of advertising -- the difference is that you recommend stuff you like, as opposed to recommending stuff someone paid you to recommend. (And there are ways to blur the line, such as "I will only recommend the stuff I like, but only if you also pay me" or "I will recommended 90% the stuff I like, and 10% the stuff you pay me for, with or without disclosing which is which". Even these two are difficult to distinguish from each other; I may be legally required to disclose whether someone has paid me, but not whether I genuinely liked the thing regardless.)

Do we then get the next layer of people recommending the recommenders? "This guy's recommendations are always solid; with that guy it's hit and miss."

Possible use of AI: keep writing your random thoughts in a private diary. Then tell an AI to find the repeated topics and arrange them into a coherent article. (I wonder whether an AI could go through my comments written on LessWrong, weigh them by karma, and turn them into a book.)

[note that this is not what Mitchell_Porter and I are disagreeing over in this related comment thread: https://www.lesswrong.com/posts/uPi2YppTEnzKG3nXD/nathan-helm-burger-s-shortform?commentId=AKEmBeXXnDdmp7zD6 [LW(p) · GW(p)] ]

Contra Roger Penrose on estimates of brain compute

[numeric convention used here is that <number1>e<number2> means number1 * 10 ^ number2. Examples: 1e2 = 100, 2e3 = 2000, 5.3e2 = 530, 5.3e-2 = 0.053]

Mouse

The cerebral cortex of a mouse has around 8–14 million neurons while in those humans there are more than 10–15 billion - https://en.wikipedia.org/wiki/Mouse_brain

A mouse brain is approximately 3 orders of magnitude smaller than a human brain. The neurons that make it up are quite similar, with the exception of a few changes such as human cortical neurons tending to have more synapses.

In his paper, '', Penrose discusses the hypothesis that the pattern of protein subunits in microtubules allows for additional processing in the brain. His estimate of per neuron compute is 1e16 FLOPs, for a total human brain compute of 1e27 FLOPs.

This would mean that a mouse brain has about 1e24 FLOPs of compute.

Fruit Fly

In 2023, using the data from 2017 (above), the full brain connectome (for a female) was made available, containing roughly 5x10^7 chemical synapses between ~130,000 neurons. - https://en.wikipedia.org/wiki/Drosophila_connectome

https://www.biorxiv.org/content/10.1101/2023.06.27.546656v2

1.3e5 neurons * 1e16 FLOPs/neuron = 1.3e11 FLOPs

A more extreme case is that of the fruit fly. A fruit fly has around 1e5 neurons. With Roger Penrose's 10^16 FLOP/s/neuron compute estimate, this works out to a fruit fly brain having around 1.3e11 FLOP/s.

GPUs of 2023

A back-of-the-envelope calculation based on market size, price-performance, hardware lifespan estimates, and the sizes of Google’s data centers estimates that there is around 3.98 * 1e21 FLOP/s of computing capacity on GPUs and TPUs as of Q1 2023.

source: https://wiki.aiimpacts.org/doku.php?id=ai_timelines:hardware_and_ai_timelines:computing_capacity_of_all_gpus_and_tpus

So, Roger Penrose's estimate claims that a single mouse brain has 1000 times more computing power of all the GPUs on Earth in Q1 2023. 

Training a Large Language Model

"... H100’s TF32 precision to achieve one petaflop of throughput ..."

https://www.nvidia.com/en-us/data-center/h100/

A petaflop is a unit of computing speed equal to one thousand million million (10¹⁵) floating-point operations per second.

1e15 * 60 seconds/min * 60 min/hour  = 3.6e17 FLOP/s per H100 GPU hour.

Training Llama 3.1 405B utilized 3.08e7 GPU hours of computation on H100-80GB GPUs. So about 1e25 FLOP/s.

https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md

A single inference instance of LLama 405B can be run on an 8 H100 GPU server. So, about 1e15 FLOP/s * 8 = 8e15 FLOP/s

Nathan's Estimate

My upper estimate of the compute of the human brain is about 1e16 FLOPs. My 'best guess' Fermi estimate is about 1e15. In other words, my best Fermi estimate is that the human brain is doing about the same amount of computation as an H100 at max throughput. My upper estimate is about 10x that, so 10 H100 GPUs. If I weren't restricting myself to nearest order of magnitude, I'd guess somewhere in between 1e15 and 1e16.

Training

So for a human brain to do the compute that was used to train Llama 3.1 405B, it would require 3e6 - 3e7 hours of waking thought. So somewhere between 500 years and 5000 years. 

A mouse brain therefore, would take around 5e5 to 5e6 years.

Inference

My estimate is that a human brain should be able to run a single LLama 405B inference instance somewhere between 1/10th speed or about normal speed. Even under my worst case assumptions, a human brain should be more than sufficient to run a LLama 70B instance.

A mouse brain would need to run LLama 405B at 1/1e3 to 1/1e4 speed (and I'd be concerned about insufficient RAM equivalent to even hold enough of it to enable this! That's a whole different discussion). 

Roger's Estimate

Training

By Roger Penrose's estimate, a mouse brain should be able to train an LLM equivalent to LLama 405B in 10 seconds. A human brain should be able to train 100 such models every second.

Inference

Roger's estimate suggests a single mouse brain should be able to run over 100 million (~1.2e8) copies of LLama 405B simultaneously at full speed.

A human brain could run 1000x that many, so about 1e11 copies simultaneously.

You'd need to network 1e4 fruit fly brains together to run a single copy of LLama 405B. Or a mere 200 fruit flies to run Llama 8B. What a bargain!

Baboons

From an ethological perspective, we can analyze the behaviors of animals in the wild in respect to computationally difficult problems.  For example, baboons solving the traveling salesman problem of choosing which resource areas to visit over time. 

https://www.youtube.com/watch?v=vp7EbO-IXtg&t=198s

If the baboons had 10^26 flops (ungenerously estimating them at 1/10th of humans) of compute at their disposal, it would be trivial for them to calculate a better solution to their resource acquisition problem. Why don't they?

Empirical resolution

So, there is limited flexibility of compute in biological brains. Most of the neurons in the human brain are wired up in relatively strict motifs to accomplish certain particular types of computation. So you can't actually test any of these theories as stated. The exception to this general rule is that the cortex of infant mammals (and birds, etc) is relatively flexible, and about as close to pure 'general compute' that biological brains get. So if you wired up an infant mouse with a Brain-Computer-Interface connecting its cortex to a computer, and removed the other sensory inputs and outputs, it's theoretically possible to get some estimates. It would also be necessary to the compute estimates to be just for the cortex instead of the whole brain.

If Roger Penrose's ideas were right, then a single long-lived rodent (e.g. naked mole rat) could be worth hundreds of billions of dollars of compute. Seems like if the scientific community suspected this were true, then at least a few scientists would be trying to develop such a BCI system?

Starting from now? I agree that that's true in some worlds that I consider plausible, at least, and I agree that worlds whose survival-probabilities are sensitive to my choices are the ones that render my choices meaningful (regardless of how determinisic they are).

Conditional on Earth being utterly doomed, are we (today) fewer than 75 qbitflips from being in a good state? I'm not sure, it probably varies across the doomed worlds where I have decent amounts of subjective probability. It depends how much time we have on the clock, depends where the points of no-return are. I haven't thought about this a ton. My best guess is it would take more than 75 qbitflips to save us now, but maybe I'm not thinking creatively enough about how to spend them, and I haven't thought about it in detail and expect I'd be sensitive to argument about it /shrug.

(If you start from 50 years ago? Very likely! 75 bits is a lot of population rerolls. If you start after people hear the thunder of the self-replicating factories barrelling towards them, and wait until the very last moments that they would consider becoming a distinct person who is about to die from AI, and who wishes to draw upon your reassurance that they will be saved? Very likely not! Those people look very, very dead.)

One possible point of miscommunication is that when I said something like "obviously it's worse than 2^-75 at the extreme where it's actually them who is supposed to survive" was intended to apply to the sort of person who has seen the skies darken and has heard the thunder, rather than the version of them that exists here in 2024. This was not intended to be some bold or suprising claim. It was an attempt to establish an obvious basepoint at one very extreme end of a spectrum, that we could start interpolating from (asking questions like "how far back from there are the points of no return?" and "how much more entropy would they have than god, if people from that branchpoint spent stars trying to figure out what happened after those points?").

(The 2^-75 was not intended to be even an esitmate of how dead the people on the one end of the extreme are. It is the "can you buy a star" threshold. I was trying to say something like "the individuals who actually die obviously can't buy themselves a star just because they inhabit Tegmark III, now let's drag the cursor backwards and talk about whether, at any point, we cross the a-star-for-everyone threshold".)

If that doesn't clear things up and you really want to argue that, conditional on Earth being as doomed as it superficially looks to me, most of those worlds are obviously <100 quantum bitflips from victory today, I'm willing to field those arguments; maybe you see some clever use of qbitflips I don't and that would be kinda cool. But I caveat that this doesn't seem like a crux to me and that I acknowledge that the other worlds (where Earth merely looks unsavlageable) are the ones motivating action.

I have not followed this thread in all of its detail, but it sounds like it might be getting caught up on the difference between the underlying ratio of different quantum worlds (which can be expressed as a probability over one's future) and one's probabilistic uncertainty over the underlying ratio of different quantum worlds (which can also be expressed as a probability over the future but does not seem to me to have the same implications for behavior).

Insofar as it seems to readers like a bad idea to optimize for different outcomes in a deterministic universe, I recommend reading the Free Will (Solution) [? · GW] sequence by Eliezer Yudkowsky, which I found fairly convincing on the matter of why it's still right to optimize in a fully deterministic universe, as well as in a universe running on quantum mechanics (interpreted to have many worlds).

I propose we wrap this particular thread up for now (with another reply from you as you wish).

I will say that for this bit

Because there's a big assumption that biology is more or less what's at the bottom of this human-stack. 

Well it isn't. 

There isn't a "the bottom."

It's like a banana tree. 

When you peel everything away, what is actually left? 

Well it turns out if you were able to PEEL SOMETHING AWAY, it wasn't the Truth of You. So discard it. And you keep going.

And that's the path. 

Being asked "So what's the answer? What's the path?" feels more like answering a riddle than being asked "The capital city of England is London. Please repeat back to me the capital city of England?". 

Direct speech is clear and unambiguous. Direct speech is like "Please can you close the door?" and indirect speech is like "Oh I guess it's chilly in here" or "Perhaps we should get people's temperature preferences", which may be a sincere attempt to communicate that you want the door closed but isn't direct. What you wrote was not especially unambiguous or non-metaphorical. I think it's a sincere attempt at communication but it's not direct. Being asked to just answer "can anyone repeat back what I'm saying without adding or subtracting anything?" seems hard when you wrote in a rather metaphorical and roundabout way.