I want to credit Rohin Shah as the person that I got this success story from, but can't find the post or comment where he talked about it.

It might be from Following human norms [AF · GW]?

With a norm-following AI system, the success story is primarily around accelerating our rate of progress. Humans remain in charge of the overall trajectory of the future, and we use AI systems as tools that enable us to make better decisions and create better technologies, which looks like “superhuman intelligence” from our vantage point today.

If we still want an AI system that colonizes space and optimizes it according to our values without our supervision, we can figure out what our values are over a period of reflection, solve the alignment problem for goal-directed AI systems, and then create such an AI system.

Which was referenced again in Learning preferences by looking at the world [AF · GW]:

If I had to point towards a particular concrete path to a good future, it would be the one that I outlined in Following human norms [? · GW]. We build AI systems that have a good understanding of “common sense” or “how to behave normally in human society”; they accelerate technological development and improve decision-making; if we really want to have a goal-directed AI that is not under our control but that optimizes for our values then we solve the full alignment problem in the future. Inferring preferences or norms from the world state could be a crucial part of helping our AI systems understand “common sense”.

It's not the same as your Interim Quality-of-Life Improver, but it's got similar aspects.

It's also related to the concept of a "Great Deliberation" where we stabilize the world and then figure out what we want to do. (I don't have a reference for that though.)

If it wasn't these (but it was me), it was probably something earlier; I think I was thinking along the lines of Interim Quality-of-Life Improver in early-to-mid 2018.

I'm surprised this post didn't get more comments and spark more further research. Rereading it, I think it's both an excellent overview/distillation, and also a piece of strategy research in its own right. I wish there were more things like this. I think this post deserves to be expanded into a book or website and continually updated and refined.

Seems like a good starting point for discussion. Researchers need to have some picture of what AI alignment is "for," in order to think about what research directions look most promising.

Corrigible Contender

A semi-autonomous AGI that does not have long-term preferences of its own but acts according to (its understanding of) the short-term preferences of some human or group of humans

In light of recent discussion [LW(p) · GW(p)], it seems like this part should be clarified to say "actual preferences" or "short-term preferences-on-reflection".

Also in the table, for Corrigible Contender should the reliance on human safety be changed from "High" to "Medium"? (My feeling is that since the AI isn't relying on the current humans' elicited preferences, the reliance on human safety would be somewhere between that of Sovereign Singleton and Pivotal Tool.)

(I'm making these suggestions mainly because I expect people will continue to refer to this post in the future.)

I really enjoyed the alliterations in some of the names. They reminded me of “series of unfortunate events” book titles.

Regarding pivotal acts. You linked the arbital article that defines it as any action that causes a large change in solving alignment. One of those examples was uploading 50 safety researchers at 1000x speed, so the pivotal act does the hard work of creating the safe environment to solve alignment.

Being handed a safe oracle would also allow sped up, safe progress in alignment.

Do you have in mind a pivotal act that requires years/decades of more work in a non-safe environment? (Or not guaranteed to be highly safe)

I don't understand what you mean by "Reliance on human safety". Can you clarify/elaborate? Is this like... relying on humans' (meta-)philosophical competence? Relying on not having bad actors? etc...

Other possible success stories are semi-success stories, where the outcome is not very good, but some humans survive and significant part of human values is preserved.

One case of the semi-success story is that many sovereign AIs control different countries or territories and implement different values in them. In some of these territories AIs' values will be very close to the best possible implementation of aligned AI. Other AIs could be completely inhuman. Slow takeoff could end in such world of many AIs.

Another case is that unfriendly AI decides not to kill humans for some instrumental reasons (research, acausal trade with other AI, just not bother killing them). It could even run many simulations of human history including simulations of friendly AIs-sovereigns and their human civilizations. In that case, many people will live very happy lives despite being controlled by unfriendly AI. Like some people were happy under Saddam Hussein rule.

Semi-success stories could be seen as a more natural outcome as we typically don't have perfect things in life.

Does an "AI safety success story" encapsulate just a certain trajectory in AI (safety) development?

Or does it also include a story about how AI is deployed (and by who, etc.)?

I like this post a lot, but I think it ends up being a bit unclear because I don't think everyone has the same use cases in mind for the different technologies underlying these scenarios, and/or I don't think everyone agrees with the way in which safety research is viewed as contributing to success in these different scenarios... Maybe fleshing out the success stories, or referencing some more in-depth elaborations of them would make this clearer?

I think that our research is at a sufficiently early stage that most technical work could contribute to most success stories. We are still mostly understanding the rules of the game and building the building blocks. I would say that we work on AI safety in general until we find anything that can be used at all. (There is some current work on things like satisficers that seem less relevant to sovereigns. I am not discouraging working on areas that seem more likely to help some success stories, just saying that those areas seem rare.)

Aside: If you want all alliteration, “Pivotal Performer/Predictor” (depending on whether tool or oracle) and “Rapid Researcher” might be alternative names for types 2 and 5.

very clear and simple. tempting to dismiss this as not significant/novel, but there is a place for presenting basic things well.

And it's positively framed. We could all use a little hope right now.

The noise in my model of what AI safety research is supposed to do and I had learned to ignore it. It surprises me how big a difference it makes, how comparatively calm and settled I feel to have the typical success narratives in front of me, disambiguated from each other. There's much more confusion to tackle, but it seems more manageable.

The next time I stumble upon an AI discussion, I expect I will look up this post for a refresher and to organize my thoughts on what model each person is using.

Assuming what Wei_Dai wrote is accurate, of course. I can tell that this post is greatly approachable, but I'm not in a place to assess whether the approach is *correct*. David Krugeur disputed a few cells on the AI forum. Nor am I certain whether it's a useful communication block for anyone already in the safety field. i see a few pingback posts, 105 votes. Would it see use in the office or is it strictly inferior to other models for communicating about AI futures?

Could attempt reading some AI papers and posts, and judge whether this post helps me contextualize the research in a meaningful way. If it help, that would be evidence for it serving an introductory purpose well. Humans like their stories.

**i am so tired i don't want to run the experiment**

Improve:
- Clarify what CAIS stands for, acronym is not expanded on the page
- Could use some Ethos. This is a widely accessible post, it makes sense to establish who the author his and why anyone would trust his opaque assessments
- Flesh out description of autonomy better
- Remove parentheticals asking for comments
- Survey which orgs/researchers are considering which scenarios. Including this information gives a next action to take if a reader wishes to engage further. As Donald Hobson pointed out, the most common category of work might be "technical work [that] could contribute to most success stories". this would also be important information worth knowing.
- could follow up with a similarly basic overview of semi-success. I don't recommend doing it with failure stories. there are too many and it would be a major bummer.

I'm going to dispute a few cells in your grid.

• I think pivotal tool story has low reliance on human safety (although I'm confused by that row in general).
• Whether sovereigns would require restricted access is unclear. This is basically the question of whether single-agent, single-user alignment will likely produce a solution to multi-agent, multi-user alignment (in a timely manner).
• ETA: the "interim quality of life improver" seems to roughly be talking about episodic RL, which I would classify as "medium" autonomy.

For the alignment newsletter:

Planned summary: It is difficult to measure the usefulness of various alignment approaches without clearly understanding what type of future they end up being useful for. This post collects "Success Stories" for AI -- disjunctive scenarios in which alignment approaches are leveraged to ensure a positive future. Whether these scenarios come to pass will depend critically on background assumptions, such as whether we can achieve global coordination, or solve the most ambitious safety issues. Mapping these success stories can help us prioritize research.

Planned opinion: This post does not exhaust the possible success stories, but it gets us a lot closer to being able to look at a particular approach and ask, "Where exactly does this help us?" My guess is that most research ends up being only minimally helpful for the long run, and so I consider inquiry like this to be very useful for cause prioritization.