LessWrong 2.0 Reader

I think my points argue more that control research might have higher expected value than some other approaches, that don't address delegation at all or are much less tractable. But I agree, if slop is the major problem, then most current control research doesn't adress it, though it's nice to see that this might change if Buck is right.

And my point about formal verification was to work around the slop problem by verifying the safety approach to a high degree of certainty. I don't know if it's feasible, though, but some seem to think so. Why do you think it's a bad idea?

Ok, so why not just train a model on fake anomaly detection/interp research papers? Fake stories about 'the bad AI that got caught', 'the little AI that overstepped', etc. I don't know how to word it, but this seems like something closer to intimidation than alignment, which I don't think makes much sense as a strategy intended to keep us all alive.

Which other people have described the situation otherwise and where? Genuine question, I'm pretty much learning about all of this here.

Correspondingly the importance I assign to increasing the intelligence of humans has drastically increased.

I feel like human intelligence enhancement would increase capabilities development faster than alignment development, maybe unless you've got a lot of discrimination in favor of only increasing the intelligence of those involved with alignment.

I can't bring myself to read it properly. The author has an ax to grind, he wants interplanetary civilization and technological progress for humanity, and it's inconvenient to that vision if progress in one form of technology (AI) has the natural consequence of replacing humanity, or at the very least removing it from the driver's seat. So he simply declares "There is No Reason to Think Superintelligence is Coming Soon", and the one doomer strategy he does approve of - the enhancement of human biological intelligence - happens to be one that once again involves promoting a form of technological progress. 

If there is a significant single failure behind getting to where we are now, perhaps it is the dissociation between "progress in AI" and "humanity being surpassed and replaced by AI" that has occurred. It should be common sense that the latter is the natural outcome of creating superhuman AI. 

I know you're not endorsing the quoted claim, but just to make this super duper extra stupid explicit: running terrorist organizations is illegal, so this is the type of thing you would also say if Ziz was leading a terrorist organization, and you didn't want to see Ziz arrested.

I think it was my error as I realise now the first paragraph was a confusing setup. I've trimmed it a bit so hopefully it won't be so any more!

DeepSeek-R1 ... Run RL to convergence

Not to convergence, the graphs in the paper keep going up. Which across the analogy might explain some of the change from o1 to o3 (the graphs in the o1 post also keep going up), though new graders coded for additional verifiable problems are no doubt a large part of it as well.

o3-mini has the same knowledge cutoff date as 4o and o1 (late 2023)

It seems like o1-mini is its own thing, might even start with a base model that's unrelated to GPT-4o-mini (it might be using its own specialized pretraining data mix). So a clue about o3-mini data doesn't obviously transfer to o3.

if it used GPT-5 as a base model

The numbering in GPT-N series advances with roughly 100x in raw compute at a time. If original GPT-4 is 2e25 FLOPs, then a GPT-5 would need 2e27 FLOPs, and a 100K H100s training system (like the Microsoft/OpenAI system at the site near the Goodyear airport) can only get you 3e26 FLOPs or so (in BF16 in 3 months). The initial Stargate training system at Abilene site, after it gets 300K B200s, will be 7x stronger than that, so will be able to get 2e27 FLOPs. Thus I expect GPT-5 in 2026 if OpenAI keeps following the naming convention, while the new 100K H100s model this year will be GPT-4.5o or something like that.

I feel like intelligence enhancement being pretty solidly in the near-term technological horizon provides strong argument for future governance being much better. There are also maybe 3-5 other technologies that seem likely to be achieved in the next 30 years bar AGI that would all hugely improve future AGI governance. 

And then a lot of the post seems to make really quite bad arguments against forecasting AI timelines and other technologies, doing so with... I really don't know, a rejection of bayesianism? A random invocation of an asymmetric burden of proof? If anyone learned anything useful fromits section on timelines or technological forecasting, please tell me, since it really is among the worst things I have heard Ben Landau Taylor write, who I respect a lot. The stuff as written really makes no sense. I am personally on the longer end of timelines, but none of my reasoning looks anything like that.

Seriously, what are the technological forecasts in this essay: 

While there is no firm ground for any prediction as to how long it will take before any technological breakthrough [to substantial intelligence enhancement], if ever, it seems more likely that such a regime would have to last worldwide for a century or several centuries before such technology were created.

I will very gladly take all your bets that intelligence augmentation will not take "several centuries". What is the basis of this claim? Like, IDK, I see no methodology that suggests anything remotely as long as this, and so many forms of trend extrapolation, first principles argument, reference class forecasting and so many other things that suggest things happen faster than that. 

I really don't get the worldview that writes this essay. A worldview in which not-even-particularly-sci-fi technologies should by default be assumed to take centuries (centuries!!!) to be developed. A worldview in which even as AI systems destroy every single benchmark anyone has ever come up with, the hypothesis that AI might be soon gets dismissed because... I really don't know. Because the author wants to maintain authority over reference classes and therefore vaguely implied it can't happen soon. 

There is no obvious prior over technological developments or progress. There not being proofs around doesn't support that things won't happen soon. And I would so gladly take this worldview's money if it's willing to actually draw some probability distributions that are spread out enough to put large fractions of its probability mass on centuries away.

I wrote that this "is the best sociological account of the AI x-risk reduction efforts of the last ~decade that I've seen." The line has some disagree reacts inline; I expect this is primarily an expression that the disagree-ers have a low quality assessment of the article, but I would be curious to see links to any other articles or posts that attempt something similar to this one, in order to compare whether they do better/worse/different. I actually can't easily think of any (which is why I felt it was not that bold to say this was the best).

Edit: I've expanded the opening paragraph, to not confuse my comment for me agreeing with the object level assessment of the article..