LessWrong 2.0 Reader

a lot of skill ceilings are much higher than you might think, and worth investing in

The former doesn't necessarily imply the latter in general, because even if we are systematically underestimating the realistic upper bound for our skill level in these areas, we would still have to deal with diminishing marginal returns to investing in any particular one. As a result, I am much more confident of the former claim being correct for the average LW reader than of the latter. In practice, my experience tells me that you often have "phase changes" of sorts, where there's a rather binary instead of continuous response to a skill level increase: either you've hit the activation energy level, and thus unlock the self-reinforcing loop of benefits that flow from the skill (once you can apply it properly and iterate on it or use it recursively), or you haven't, in which case any measurable improvement is minimal. It's thus often more important to get past the critical point than to make marginal improvements either before or after hitting it.

On the other hand, many of the skills you mentioned afterwards in your comment seem relatively general-purpose, so I could totally be off-base in these specific cases.

Yes! I've always been a huge believer in this idea that the ease of eating a food is important and underrated. Very underrated.

I'm reminded of this clip of Anthony Bourdain talking about burgers and how people often put slices of bacon on a burger, but that in doing so it makes the burger difficult to eat. Presumably because when you go to take a bite you the whole slice of bacon often ends up sliding off the burger.

Am I making this more enjoyable by adding bacon? Maybe. How should that bacon be introduced into the question? It's an engineer and structural problem as much as it is a flavor experience. You really have to consider all of those things. One of the greatest sins in "burgerdom" I think is making a burger that's just difficult to eat.

The document seems to try to argue that Uber cannot possibly become profitable. I would be happy to take a bet that Uber will become profitable within the next 5 years. 

And indeed it already has.

Baltimore, MD, USA 

January 4. Time is TBD. The location is in Rockville. You can message me for the details. 

Fundamentally, the story was about the failure cases of trying to make capable systems that don't share your values safe by preventing specific means by which its problem solving capabilities express themselves in scary ways, instead of the way you are getting at here, which is having those systems actually operationally share your values. A well aligned system, in the traditional ‘Friendly AI’ sense of alignment, simply won't make the choices that the one in the story did.

Seems like that guy has failed to grasp the fact that some things are naturally more predictable than others.  Estimating how much concrete you need to build a house is just way easier than estimating how much time you need to design and code a large novel piece of software (even if the requirements don't change mid-project).

It's nicely done, and I think it will be helpful for anyone in refining their thinking. Thanks for doing this! And for making it easy to edit. That makes it a more general tool for scenario evaluation.

Bravo and big upvote! Spinning out concrete scenarios like this is going to sharpen our collective thinking. Everyone should do this. I'll take a shot at it soon.

I find the timeline highly plausible. In this world, though what happened to the rest that were months behind? Since it took a while to reach ASI, now we have to hope those are all aligned and well-used too - unless somebody halts those projects. Leading to the big question:

What is the government doing once AGI is achieved? Surely Trump isn't keeping his little fingers off it.

These questions are for everyone as much as for Dave. After we've got some good scenarios-to-AGI in our collective minds, we should be better able to push past them to scenarios for impacts.

Can someone elaborate on how the risk of military attacks to nuclear powerplants is usually accounted for?

During the invasion of Ukraine in 2022, I remember a great fuss around the shelling of the Zaporizhzhia Nuclear Power Plan. It was argued a disaster on the magnitude of chernobyl would be impossible, but I'm unaware of the technical aspects involved and some people were still very afraid.

If nuclear (be it large or SMR) is to become commonplace, what kind of risks are involved in these attacks?

I largely agree that ASI will follow AGI faster, but with a couple caveats.

The road from AGI to superintelligence will very likely be fairly continuous. You could slap the term "superintelligence" almost wherever you want after it passes human level.

I do see some reasons that the road will go a little slower than we might think. Scaling laws are logarithmic; making more and better chips requires physical technology that the AGI can help with but can't do until it gets better with robotics, possibly including new hardware (although humanoid robotics will be close to adequate for most things by then, with new control networks rapidly trained by the AGI).

If the architecture is similar to current LLMs, it's enough like human thought that I expect the progression to remain logarithmic; you're still using the same clumsy basic algorithm of using your knowledge to come up with ideas, then going through long chains of thought and ultimately experiments to test the validity of different ideas.

It's completely dependent on what we mean by superintelligence, but creating new technologies in a day will take maybe five years after the first clearly human-level general real AGI on this path, in my rough estimate.

Of course that's scaled by how hard people are actually trying for it.