LessWrong 2.0 Reader

Seems like that guy has failed to grasp the fact that some things are naturally more predictable than others.  Estimating how much concrete you need to build a house is just way easier than estimating how much time you need to design and code a large novel piece of software (even if the requirements don't change mid-project).

It's nicely done, and I think it will be helpful for anyone in refining their thinking. Thanks for doing this! And for making it easy to edit. That makes it a more general tool for scenario evaluation.

Bravo and big upvote! Spinning out concrete scenarios like this is going to sharpen our collective thinking. Everyone should do this. I'll take a shot at it soon.

I find the timeline highly plausible. In this world, though what happened to the rest that were months behind? Since it took a while to reach ASI, now we have to hope those are all aligned and well-used too - unless somebody halts those projects. Leading to the big question:

What is the government doing once AGI is achieved? Surely Trump isn't keeping his little fingers off it.

These questions are for everyone as much as for Dave. After we've got some good scenarios-to-AGI in our collective minds, we should be better able to push past them to scenarios for impacts.

Can someone elaborate on how the risk of military attacks to nuclear powerplants is usually accounted for?

During the invasion of Ukraine in 2022, I remember a great fuss around the shelling of the Zaporizhzhia Nuclear Power Plan. It was argued a disaster on the magnitude of chernobyl would be impossible, but I'm unaware of the technical aspects involved and some people were still very afraid.

If nuclear (be it large or SMR) is to become commonplace, what kind of risks are involved in these attacks?

I largely agree that ASI will follow AGI faster, but with a couple caveats.

The road from AGI to superintelligence will very likely be fairly continuous. You could slap the term "superintelligence" almost wherever you want after it passes human level.

I do see some reasons that the road will go a little slower than we might think. Scaling laws are logarithmic; making more and better chips requires physical technology that the AGI can help with but can't do until it gets better with robotics, possibly including new hardware (although humanoid robotics will be close to adequate for most things by then, with new control networks rapidly trained by the AGI).

If the architecture is similar to current LLMs, it's enough like human thought that I expect the progression to remain logarithmic; you're still using the same clumsy basic algorithm of using your knowledge to come up with ideas, then going through long chains of thought and ultimately experiments to test the validity of different ideas.

It's completely dependent on what we mean by superintelligence, but creating new technologies in a day will take maybe five years after the first clearly human-level general real AGI on this path, in my rough estimate.

Of course that's scaled by how hard people are actually trying for it.

I think they meant that as an analogy to how developed/sophisticated it was (ie they're saying that it's still early days for reasoning models and to expect rapid improvement), not that the underlying model size is similar.

There have been comments from OAI staff that o1 is "GPT-2 level" so I wonder if it's a similar size?

I've updated future posts to have start time at 6:30 and doors open at 6pm.

Well that escalated quickly (at the very end).

That was true this week, but the first time I attended (the 12th) I believe it wasn't, I arrived at what I think was 6:20-6:25 and found everything had already started.