LessWrong 2.0 Reader

No worries; thanks!

@gwern [LW · GW]

My guess is:

• AI pause: no observation on what safety issue to address, work on capabilities anyways, then may lead to only capability improves. (Assumption is that AI pausing means no releasing of models.)
• RSP: observed O, shift more resources to work on mitigating O and less on capabilities, and when protection P is done, publish the model, then shift back to capabilities. (Ideally.)

The bitter lesson says that there are many things you don't need to understand, but it doesn't say you don't need to understand anything.

I think you're doing a "we just need X" with recursive self-improvement. The improvement may be iterable and self-applicable... but is it general? Is it on a bounded trajectory or an unbounded trajectory? Very different outcomes.

Humans are capable of solving conceptually difficult problems, so they do. An easier path might be possible that doesn't depend on such capabilities, and doesn't stall for their lack, like evolution doesn't stall for lack of any mind at all. If there is more potential [LW(p) · GW(p)] for making models smarter alien tigers [LW(p) · GW(p)] by scaling RL in o1-like post-training, and the scaling proceeds to 1 gigawatt and then 35 gigawatt training systems [LW(p) · GW(p)], it might well be sufficient to get an engineer AI that can improve such systems further, at 400x and then 10,000x the compute of GPT-4.

Before o1, there was a significant gap, the mysterious absence of System 2 capabilities, with only vague expectation that they might emerge or become easier to elicit from scaled up base models. This uncertainty no longer gates engineering capabilities of AIs. I'm still unsure that scaling directly can make AIs capabile of novel conceptual thought, but AIs becoming able to experimentally iterate on AI designs seems likely, and that in turn seems sufficient to eventually mutate these designs towards remaining missing capabilities.

(It's useful to frame most ideas as exploratory engineering rather than forecasting. The question of whether something can happen, or can be done, doesn't need to be contextualized [LW · GW] within the question of whether it will happen or will be done. Physical experiments are done under highly contrived conditions, and similarly we can conduct thought experiments or conceptual arguments under fantastical or even physically impossible conditions. Thus I think Carl Shulman's human level AGI world is a valid exploration of the future of AI, even though I don't believe that most of what he describes happens in actuality before superintelligence changes the premise. It serves as a strong argument for industrial and economic growth driven by AGI, even though it almost entirely consists of describing events that can't possibly happen.)

I'd imagine that RSP proponents think that if we execute them properly, we will simply not build dangerous models beyond our control, period. If progress was faster than what labs can handle after pausing, RSPs should imply that you'd just pause again. On the other hand, there's not a clear criteria for when we would pause again after, say, a six month pause in scaling.

Now whether this would happen in practice is perhaps a different question.

I still don't know what you mean.

I think it's not an unreasonable point to take into account when talking price, but also a lot of the time it's serves as a BS talking point for people who don't really care about the subtleties.

Okay, I talked more on what conclusions we can draw from LLMs that actually generalize to superhuman AI here, so go check that out, but the really short summary is human values are less complicated and more dependent on data than people thought, and we can specify our values rather easily without it going drastically wrong:

This is not a property of LLMs, but of us:

https://www.lesswrong.com/posts/tDkYdyJSqe3DddtK4/alexander-gietelink-oldenziel-s-shortform#mPaBbsfpwgdvoK2Z2 [LW(p) · GW(p)]

I'm not sure I'm understanding your setup (I only skimmed the post). Are you using takeoff to mean something like "takeoff from now" or ("takeoff from [some specific event that is now in the past]")? If I look at your graph at the end, it looks to me like "Paul Slow" is a faster timeline but a longer takeoff (Paul Slow's takeoff beginning near the beginning of the graph, and Fast takeoff beginning around the intersection of the two blue lines).