LessWrong 2.0 Reader

Which was not terribly secret. The details of the Project were indeed super-secret, to the point where most of the politicians hadn't known anything, but many had observed the signs of a major project of some sort and some got as far as a nuclear bomb specifically. Also, commercial satellites with meter resolution did not exist which could quantify major facilities or new cities like Los Alamos or Hanover (but overflights, and then satellites, now exist and have helped reveal later nuclear bomb programs). I'm sure you can find plenty more about secrecy breaches in Rhodes.

This was not necessarily all that useful in the context of WWII - of course America had some big secret projects going, everyone did. It was a total world war. Everyone was aware there was a war on. The devil was in the details of what the program was - a failure like the V2-s, or a success like Enigma decrypts and Manhattan? But a binary exists/does-not-exist is useful in a peacetime context and the current discussion. (If nothing else, the fact that DeepSeek keeps publishing is a signal.)

this could have been noise, but i noticed an uptick in text fearing spies, in the text i've seen in the past few days^[1]. i actually don't know how much this concern is shared by LW users, so i think it might be worth writing that, in my view:

• (AFAIK) both governments are currently reacting inadequately to unaligned optimization risk. as a starting prior, there's not strong reason to fear more one government^[2] observing ML gatherings over the other, absent evidence that one or the other will start taking unaligned optimization risks very seriously, or that one or the other is prone to race towards ASI.
  • (AFAIK, we have more evidence that the U.S. government may try to race, e.g. this [LW · GW], but i could have easily missed evidence as i don't usually focus on this)
  • tangentially, a more-authoritarian government could be better situated to react fastly to optimization risks (in particular), if it started taking them seriously
    • (Bostrom made a similar argument in 'The Vulnerable World Hypothesis', about a hypothetical world with easy-to-individually-make nuclear bombs)
• much of the text i see about this seems nationalistic to me. i guess i'd warn that being wary of nationalistic biases is important for people who have them, you're not actually fulfilling your values by helping your preferred country/government/delineation-of-earth's-surface accidentally kill everyone first.

that said i don't typically focus on governance or international-AI-politics, and have not put much thought into this.

1. ^{^}

   examples: yesterday, saw this twitter/x post (via this quoting post)

   today, opened lesswrong and saw this shortform about two uses of the word spy [LW(p) · GW(p)] and this shortform about how it's hard to have evidence against the existence of manhattan projects [LW(p) · GW(p)]

   this was more than usual, and i sense that it's not a coincidence / is part of a pattern

2. ^{^}

   of those of US/china

Llama-3-405B is an important anchor for compute of other models. With 4e25 FLOPs and conservative training techniques it's about as capable, so the other models probably don't use much more. If they have better techniques, they need less compute to get similar performance, not more. And they probably didn't train for more than 6 months. At $2 per H100-hour^[1], $3 billion buys 6 months of time on 300K H100s. There are no publicly known training systems this large, the first 100K H100s systems started appearing in the later part of this year. Thus the training cost figures must include smaller experiments that in aggregate eat more compute than the largest training runs, with the now-ubiquitous smaller clusters also used for inference.

So anchoring to total number of GPUs is misleading about frontier model training because most GPUs are used for inference and smaller experiments, and the above estimate shows that figures like $3 billion for training are also poor anchors. If instead we look at 20K H100s as the typical scale of largest clusters in mid 2023 to early 2024, and 4 months as a typical duration of frontier model training, we get $120 million at $2 per H100-hour or 8e25 dense BF16 FLOPs at 40% compute utilization, only about 2x Llama-3-405B compute. This agrees with how Dario Amodei claimed that in Jun 2024 the scale of deployed models is about $100 million.

"I still think there's something unnecessary in worry about value drift, and I could probably make a better argument for that"

Do you have any plans on writing a post about why we shouldn't worry about AI value drift? This topic is my number 1 current interest, so I'd really appreciate even a short outline of the argument that you have in mind.

A first candidate abstraction is the neuron doctrine: the idea that the mind is governed purely by patterns of neuron spiking. This abstraction seems simple enough (it could be captured with an artificial neural network of the same size: requiring around ${10}^{20}$ parameters).

Human brain only holds about 200-300 trillion synapses, getting to 1e20 asks for almost a million parameters per synapse.

The conclusion does not follow from the argument.

The argument suggests that it is unlikely that a perfect replica of the functioning of a specific human brain can be emulated on a practical computer. The conclusion generalizes that out to no conscious emulation of a human brain, at all.

These are enormously different claims, and neither follows from the other.

For people asking about ‘noticing 5 things happening within a second or so’, you can see at least one (fictional but representative) example in Scaffolding for "Noticing Metacognition" [LW · GW] 

Huh, thanks, nevermind.

Before the internet?

I think the idea is that there would be clues, particularly talented engineers not doing public work. Of course, I don't know who's carefully tracking that for Chinese scientists.

Scientists did guess the Manhattan project existed because several top physicists had ceased publishing. I don't know if that made it to the relevant governments. That would be vastly easier to do with the internet - if anyone is bothering.

The first reasoning trace in the QwQ blog post seems impressive in how it manages to eventually stumble on the correct answer despite the 32B model clearly having no clue throughout, so it manages to effectively explore while almost blind. If this is sufficient to get o1-preview level results on reasoning benchmarks, it's plausible that RL in such post-training is mostly unhobbling the base models rather than making them smarter.

So some of these recipes might fail to have any way of scaling far, in the same sense that preference tuning doesn't scale far (unlike AlphaZero). QwQ post doesn't include a scaling plot, and the scaling plot in the DeepSeek-R1 post doesn't show improvement with further training, only with thinking for more tokens. The o1 post shows improvement with more training, but it might plateau in the uninteresting way instruction/preference post-training plateaus, making the model reliably do the thing its base model is already capable of in some sense. The similarity between o1, R1, and QwQ is superficial enough that the potential to scale with more post-training might be present in some of them and not others, or in none of them.