[Link] Sarah Constantin: "Why I am Not An AI Doomer"

I feel like a lot of the objections around agency are answered by the Clippy scenario, and gwern's other essay on the topic, Tool AIs want to be Agent AIs. The AGI need not start with any specific goal or agency. However, the moment it starts executing a prompt that requires it to exhibit agency or goal directed behavior, it will. And at that point, unless the goal is set up such that the agent pursues its goal in a manner that is compatible with the continued existence of humanity over the long term, humanity is doomed. Crafting a goal in this manner is very difficult, and making sure that the AGI pursues this goal and no others are both very difficult tasks individually. Together, they are nigh impossible. Thus, with a very strong likelihood, the moment the AGI either receives a prompt or discovers a prompt that requires it to behave like an agent, humanity is doomed.

I agree that AGIs need to possess a world model, but I disagree that we will be able to distinguish an AI that possesses a world model from an AI that "merely" knows word associations. The internals of an AI are opaque, despite the best efforts of interpretability research to shine light on the giant inscrutable matrices. An AI with a world model, I predict, won't look much different from an AI without a world model. Maybe some weights will be different, and some update functions will have changed. Will we be able to point to any specific weight or combination of weights and say, "Aha, the AI has developed a world model!" Probably not, no more than we can look at any specific set of neurons in the human brain and say, "Aha, there lies the seat of consciousness!"

Given the two points above, we may not be able to tell when any given AI passes the threshold to becoming an AGI. And once an AI has passed the threshold, we won't necessarily be able to control which prompt causes the AI to begin simulating an agent. Given those two, I fail to see why we shouldn't behave as if AGI is on a short timeline. After all, if one is approaching a cliff from an unknown distance in the darkness, the wise thing to do is not to assume that the cliff is still miles away and stride boldly into the unknown. Instead it behooves us to probe carefully, trying to determine whether there's solid ground or empty space ahead.

This post made me pretty sad, because I think it focuses on precisely the wrong parts of the AI doom case (AGI is near), and concedes all of the points she shouldn't concede (alignment is hard).

If I were like Sarah and thought that alignment is super hard but AGI is a long way off, I would probably still consider myself a "doomer" and would be very worried.

Agency requires reasoning about the consequences of one’s actions. "I need to do such-and-such, to get to my goal." This requires counterfactual, causal reasoning.

Have you ever tried to explain the difference between correlation and causation to someone who didn't understand it? I'm not convinced that this is even something humans innately have, rather than some higher-level correction by systems that do that.

A computer chess engine trained exclusively on one format for representing the game would generally not be able to transfer its knowledge to a different format.

You can hook a chess-playing network up to a vision network and have it play chess using images of boards - it's not difficult. Perhaps a better example is that language models can be easily coupled to image models to get prompted image generation. You can also translate between language pairs that didn't have direct translations in the training data.

thus we do not know how to build machines that can pursue goals coherently and persistently

This post seems rather specific to LLMs for how much it's trying to generalize; I think there's been more progress on that than Sarah seems to realize.

Really great post.

The two parts that stand out most to me are the Causality and Ontology Change sections.

Regarding Causality, I agree that there will be little investment into robotics as a mechanism for intervening on the world and building causal models. However, I don't see why practicing on videogames doesn't produce this sort of interventionist data and why AIs wouldn't learn causal models from that. And it doesn't seem that expensive to create the data. It's already happened a bunch with AIs trained on mutltiplayer videogames, and later on it will get cheaper, and overall investment will continue to increase by orders of magnitude. So I expect this will happen.

Regarding Ontology Change, I currently think of this argument as a variant of "The AI needs to understand and model itself well enough to be able understand when its concepts are mistaken. To build this sort of self-aware AGI requires lots of new insights about how agents work that we do not have. Therefore we cannot build one."

From an x-risk reduction perspective, I try to think of the worst case. Even if I believe that machine learning systems will not find programs that have the sorts of self-reflective properties that an agent needs in order to be able to notice that its concepts are mistaken (which I am honestly uncertain about), I also think that 15 years from now the world would still be massively overrun with Machine Learning, and the smartest minds in the world would be excited about AGI, and the top researchers would work on things related to this question. This is my main counterargument to a lot of arguments that we're not there yet theoretically — I expect the investment into AI research over the next decade will increase massively with respect to today.

However the last four paragraphs of the post are a direct response to this, and feel notably hopeful to me.

So, the kind of advance we’re worried about must come from the rare maverick dreamer types who have their sights fixed on a distant vision of “true” AGI and are willing to spend years scribbling in the wilderness to get there.

Such an advance is of course not impossible -- but it’s a very different threat model from the armies of machine learning researchers and engineers making rapid incremental progress on deep neural nets because they are immediately rewarded with professional success for doing so.

You could probably find all the intellectually generative “AI dreamer” types and talk to them individually -- those sorts of people tend to share their ideas in writing.

If the lines of communication remain open -- if the current AI debate doesn’t tribalize to the point that “pro-AI” and “anti-AI” factions hate each other and can’t have friendly discussions -- then it might be remarkably tractable to just, y’know, persuade a handful of individuals that they should maybe not work too hard to get the world to take notice of their theoretical ideas.

This is easily the best anti near-term AGI article I have read so far. Especially cross-ontology goal robustness is a great point.