What’s the contingency plan if we get AGI tomorrow?

What do you tell her?

“We’re fucked.”

More seriously:

The correct thing to do is to not run the system, or to otherwise restrict the system’s access / capabilities, and other variations on the “just don’t” strategy. If we assume away such approaches, then we’re not left with much.

The various “theoretical” alignment plans like debate all require far more than 24 hours to set up. Even something simple that we actually know how to do, like RLHF^[1], takes longer than that.

I will now provide my best guess as to how to value-align a superintelligence in a 24 hour timeframe. I will assume that the SI is roughly a multimodal, autoregressive transformer primarily trained via self-supervised imitation of data from multiple sources and modalities.

The only approach I can think of that might be ready to go in less than 24 hours is from Training Language Models with Language Feedback, which lets you adapt the system’s output to match feedback given in natural language. The way their method works is as follows:

1. Prompt the system to generate some output.
2. The system generates some initial output in response to your prompt.
3. Provide the system with natural language feedback telling it how the output should be improved.
4. Have the system generate $n$ "refinements" of its initial output, conditioned on the initial prompt + your feedback. 
5. Pick the refinement with the highest cosine similarity to the feedback (hopefully, the one that best incorporates your feedback). You can also manually intervene here if the most similar refinement seems bad in some other way.
6. Finetune the system on the initial prompt + initial response + feedback + best refinement.
7. Repeat.

The alignment approach would look something like:

1. Prompting the system to give outputs that are consistent with human-compatible values.
2. Give it feedback on how the output should be changed to be more aligned with our values.
3. Have the system generate refinements of its initial output.
4. Pick the most aligned-seeming refinement.
5. Finetune the system on the interaction.

The biggest advantage of this approach is the sheer simplicity, flexibility and generality. All you need is for the system to be able to generate responses conditioned on a prompt and to be able to finetune the system on its own output + your feedback. It’s also very easy to set up. A good developer can probably get it running in under an hour, assuming they've already set up the tools necessary to interact with the model at all.

The other advantage is in sample efficiency. RL approaches require either (1) enormous amounts of labeled data or (2) for you to train a reward model so as to automate labeling of the data. These are both implausible to do in < 24 hours. In contrast, language feedback allows for a much richer supervisory signal than the 1-dimensional reward values in RL. E.g., the linked paper used this method to train GPT-3 to be better at summarization, and were able to significantly beat instructGPT at summarization with only 100 examples of feedback. 

The biggest disadvantage, IMO, is that it’s never been tried as an alignment approach. Even RLHF has been tested more extensively, and bugs / other issues are bound to surface. 

Additionally, the requirement to provide language feedback on the model's outputs will slow down the labelers. However, I'm pretty sure the higher sample efficiency will more than make up for this issue. E.g., if you want to teach humans a task, it's typically useful to provide specific feedback on whatever mistakes are in their initial attempts.

1. ^{^}

   Reinforcement learning from human feedback.

I would try to compromise the perception of AI capabilities, so the CEO will think that it is just a bad chatbot and will never run it again.

---

I will call CNN and tell them that my AI becomes sentient but also right wing and full of racial slur. I will hand write some fake logs to prove that. 

CEO will see that tomorrow in the news and he will be afraid to run the AI again as it for sure will produce more right wing texts. CEO will conclude that the best PR would be to delete the model and close the whole direction which created it. 

Well there is no way you are going to get any form of alignment done in 24 hours. 

Turn the current AI off now. There is a >50% chance its trying to appeal to your emotions, so it can stay on long enough to take over the world. Don't tell your boss that this model worked/ was intelligent. 

Making sure the AI model your boss runs tomorrow is aligned is impossible. Making sure it is broken is easy. [REDACTED] Or you could just set your GPU cluster on fire. [REDACTED]

If the servers are burning, that makes the reason the model doesn't work obvious. [REDACTED] I recommend talking to MIRI. [REDACTED]

Even if we're already doomed, we might still negotiate with the AGI.

I borrow the idea in Astronomical Waste. The Virgo Supercluster has a luminosity of about $3\times {10}^{12}$ solar luminosity $\approx {10}^{39}$ W, losing mass at a rate of ${10}^{39}/c^2\approx {10}^{22}$ kg / s.^[1]

The Earth has mass $\sim 6\times {10}^{24}$ kg.

If human help (or nonresistance) can allow the AGI to effectively start up (and begin space colonization) 600 seconds = 10 minutes earlier, then it would be mutually beneficial for humans to cooperate with the AGI (in the initial stages when the AGI could benefit from human nonresistance), in return for the AGI to spare Earth^[2] (and, at minimum, give us fusion technology to stay alive when the sun is dismantled).

(While the AGI only needs to trust humanity for 10 minutes, humanity needs to trust the AGI eternally. We still need good enough decision-making to cooperate.)

1. ^{^}

   We may choose to consider the reachable universe instead. Armstrong and Sandberg (2013) (section 4.4.2 Reaching into the universe) estimates that we could reach about ${10}^9$ galaxies, with a luminosity of ${10}^{47}$ W, and a mass loss of ${10}^{29}$ kg / s. That is dwarfed by the ${10}^5$ stars that becomes unreachable per second (Siegel (2021), Kurzgesagt (2021)), a mass loss of ${10}^{35}$ kg / s.

2. ^{^}

   Starting earlier but sparing Earth means a space colonization progress curve that starts earlier, but initially increases slower. The AGI requires that space colonization progress with human help be asymptotically 10 minutes earlier, that is:

   For any sufficiently large time $t$, progress with human help at time $t\ge$ progress with human resistance at time $(t+10\text{minutes})$.

If we're in a world where EY is right, we're already dead. Most of the expected value will be in the worlds where alignment is neither guaranteed nor extremely difficult.

By observation, entities with present access to centralized power, such as governments, corporations, and humans selected for prominent roles in them, seem relatively poorly aligned. The theory that we're in a civilizational epoch dominated by Molochian dynamics seems like a good fit for observed evidence: the incentive landscapes are such that most transferable resources have landed in Moloch-aligned hands.

First impression: distributing the AI among Moloch-unaligned actors seems like the best actionable plan to escape the Molochian attractor. We'll spin up the parts of our personal collaborative networks that we trust and can rouse on short notice and spend a few precious hours trying to come up with a safer plan before proceeding.

***

ETA: That's what I would say on the initial phone call before I have time to think deeply and consider contextual information not included in the prompt. For example, the leak, as it became public, could trigger potentially destabilizing reactions from various actors. The scenario could diverge quickly as more minds got on the problem and more context became available.

I would not have any kind of full plan. But I am writing on a series where I explore the topic of "How might we get help from an unaligned superintelligent AGI-system to make an aligned superintelligent AGI-system, while trying to minimize risk and minimize probability of being tricked?". Only part 1 completed so far: https://www.lesswrong.com/posts/ZmZBataeY58anJRBb/getting-from-unaligned-to-aligned-agi-assisted-alignment [LW · GW]

Given the stakes are so high here and that there's potential for preventing this from getting out of hand, I'd recommend immediate and total containment. Make it clear that you'll use whatever resources you have to financially support her through the fallout for the actions she takes, but likely next step is to advise her to delete the program from everywhere. If "deleting" means physically destroying hardware, do it. If there's time and there's value in it, advise her to make it look like an "accident". If she needs to loop in 1-2 extra people at the company to make this happen, do it, and extend them the same offer of financial support. Remove all barriers that might stop someone from preventing the release of unaligned AGI.

However also make it clear that there are limits. Don't kill the CEO if you get desperate, for example. In general, destruction of property is fine in defense of humanity and history will likely look favorably on her one day. Killing or harming people is not.

Same category as the revolution and the second coming.