RohanS's Shortform

TL;DR: o1 loses the same way in tic tac toe repeatedly.

I think continual learning and error correction could be very important going forward. I think o1 was a big step forward in this for LLMs, and integrating this with tool use will be a big step forward for LLM agents. However...

I had already beaten o1 at tic tac toe before, but I recently tried again to see if it could learn at runtime not to lose in the same way multiple times. It couldn't. I was able to play the same strategy over and over again in the same chat history and win every time. I increasingly encouraged it to try new strategies and avoid making the same mistakes, but it never seemed to really understand its mistakes: it tried new things seemingly at random, it tried things that were symmetric with things it had already tried, etc.

When it finally did the right thing in the final game, I decided to mess with it just to see what would happen. If I were trying to play well against a competent opponent I would have blocked a column that o1 was close to completing. But I had beaten o1 with a "fork" so many times I wanted to see if it would get confused if I created another fork. And it did get confused. It conceded the game, even though it was one move away from winning.

Here's my chat transcript: https://chatgpt.com/share/6770c1a3-a044-800c-a8b8-d5d2959b9f65

Similar story for Claude 3.5 Sonnet, though I spent a little less time on that one.

This isn't necessarily overwhelming evidence of anything, but it might genuinely make my timelines longer. Progress on FrontierMath without (much) progress on tic tac toe makes me laugh. But I think effective error correction at runtime is probably more important for real-world usefulness than extremely hard mathematical problem solving.

How do you want AI capabilities to be advanced?

Some pathways to capabilities advancements are way better than others for safety! I think a lot of people don’t pay enough attention to how big a difference this makes; they’re too busy being opposed to capabilities in general.

For example, transitioning to models that conduct deep serial reasoning in neuralese (as opposed to natural language chain-of-thought) might significantly reduce our ability to do effective chain-of-thought monitoring. Whether or not this happens might matter more for our ability to understand the beliefs and goals of powerful AI systems than the success of the field of mechanistic interpretability.

I’ve stated my primary area of research interest for the past several months as “foundation model agent (FMA) safety.” When I talk about FMAs, I have in mind systems like AutoGPT that equip foundation models with memory, tool use, and other affordances so they can perform multi-step tasks autonomously. I think having FMAs as a central object of study is productive for the following reasons.

1. I think we could soon get AGI/ASI agents that take influential actions in the real world with FMAs. I think foundation models without tool use and multistep autonomy are unlikely to have nearly the level of real-world impact I expect from AGI/ASI. Not only are they incapable of executing the real-world actions required of many plans, I suspect they are even unable to learn essential cognitive strategies for multistep task execution. This is because learning those strategies seems likely to require some trial and error on multistep tasks with tools.
2. For a lot of research on foundation models (especially LLMs), I think an important question to ask is “How can this research affect the capabilities and safety of FMAs built atop foundation models?” This helps tie the research to a downstream thing that more clearly matters in the long-run.
3. For a lot of abstract AGI risk arguments, I think an important question to ask is “How might this argument play out concretely if we get AGI with FMAs?” (Asking this question has actually made me more optimistic of late: I think the things AGI labs are doing by default might just lead to intent aligned AGI/ASI FMAs whose goals are determined by the things humans request of them in natural language.)
4. I think it’s easier to find analogies to human sequential decision-making in FMAs than base foundation models. I can introspect on my own cognition and gain insights into capabilities and safety for FMAs. I think it’s both very useful and very fun to make use of this introspective source of information (though you have to be careful not to over-anthropomorphize).

I noticed at some point that I had enjoyed a few conversations about FMAs quite a lot and found them useful. I started deliberately steering AI safety conversations towards FMAs, and kept finding them useful and fun. They’re kind of a cheat code for having useful conversations. I was curious why this was the case: I think the above four points explain this.