LessWrong 2.0 Reader

"I still think there's something unnecessary in worry about value drift, and I could probably make a better argument for that"

Do you have any plans on writing a post about why we shouldn't worry about AI value drift? This topic is my number 1 current interest, so I'd really appreciate even a short outline of the argument that you have in mind.

A first candidate abstraction is the neuron doctrine: the idea that the mind is governed purely by patterns of neuron spiking. This abstraction seems simple enough (it could be captured with an artificial neural network of the same size: requiring around ${10}^{20}$ parameters).

Human brain only holds about 200-300 trillion synapses, getting to 1e20 asks for almost a million parameters per synapse.

The conclusion does not follow from the argument.

The argument suggests that it is unlikely that a perfect replica of the functioning of a specific human brain can be emulated on a practical computer. The conclusion generalizes that out to no conscious emulation of a human brain, at all.

These are enormously different claims, and neither follows from the other.

For people asking about ‘noticing 5 things happening within a second or so’, you can see at least one (fictional but representative) example in Scaffolding for "Noticing Metacognition" [LW · GW] 

Huh, thanks, nevermind.

Before the internet?

I think the idea is that there would be clues, particularly talented engineers not doing public work. Of course, I don't know who's carefully tracking that for Chinese scientists.

Scientists did guess the Manhattan project existed because several top physicists had ceased publishing. I don't know if that made it to the relevant governments. That would be vastly easier to do with the internet - if anyone is bothering.

The first reasoning trace in the QwQ blog post seems impressive in how it manages to eventually stumble on the correct answer despite the 32B model clearly having no clue throughout, so it manages to effectively explore while almost blind. If this is sufficient to get o1-preview level results on reasoning benchmarks, it's plausible that RL in such post-training is mostly unhobbling the base models rather than making them smarter.

So some of these recipes might fail to have any way of scaling far, in the same sense that preference tuning doesn't scale far (unlike AlphaZero). QwQ post doesn't include a scaling plot, and the scaling plot in the DeepSeek-R1 post doesn't show improvement with further training, only with thinking for more tokens. The o1 post shows improvement with more training, but it might plateau in the uninteresting way instruction/preference post-training plateaus, making the model reliably do the thing its base model is already capable of in some sense. The similarity between o1, R1, and QwQ is superficial enough that the potential to scale with more post-training might be present in some of them and not others, or in none of them.

It is what it is in the original: 

(I left all typos unedited, IMO the presence of typos is an interesting data point about the attention that went to stuff like typos in these high stakes negotiations)

We had originally just wanted space cycles donated

I think this is a mistake, and it should be "spare cycles" instead.

Seems helpful for understanding how believing-ins [LW · GW] get formed by groups, sometimes.