Labs should be explicit about why they are building AGI

This seems like a good thing for labs to do^[1]. I'd go one step earlier and propose that labs make a clear and explicit page (on their website or similar) stating their views on the risk from powerful AI systems. The proposal given in this post seems somewhat more ambitious and costly than the thing I'm proposing in this comment, though the proposal in the post is correspondingly somewhat better.

As far as what a "page stating their views on risk" looks like, I'm imagining something like (numbers are made up):

Views within [AI LAB] vary, but leadership thinks that the risk of AI killing over 1 billion people is around 10%. We think the probability of an AI originating from [AI LAB] killing over 1 billion people in the next 5 years is around 3%. We specifically think that the probability of an AI system mostly autonomously taking over a large chunk of the world is around 10%. The risk of an AI system assisting with terrorism which kills over 1 billion people is around 3%. ...

AI labs often use terms like "AI safety" and "catastrophe". It's probably unclear what problem these terms are pointing at. I'd like it if whenever they said "catastrophe" they say something like:

We do XYZ to reduce the probability of AI caused catastrophe (e.g. the deaths of over 1 billion people, see here for our views on AI risk)

Where here links to the page discussed above.

And similar for using the terms AI safety:

by AI safety, we primarily mean the problem of avoiding AI caused catastrophe (e.g. the deaths of over 1 billion people, see here for our views on AI risk)

I'd consider this ask fulfilled even if this page stated quite optimistic views. At that point, there would be a clear disagreement to highlight.

I'm not sure about how costly these sorts of proposals are (e.g. because it makes customers think you're crazy). Possibly, labs could coordinate to release things like this simultaneously to avoid tragedy of the commons (there might be anti-trust issues with this).

OpenAI's Planning for AGI and beyond already writes about why they are building AGI:

Our mission is to ensure that artificial general intelligence—AI systems that are generally smarter than humans—benefits all of humanity.

If AGI is successfully created, this technology could help us elevate humanity by increasing abundance, turbocharging the global economy, and aiding in the discovery of new scientific knowledge that changes the limits of possibility.

AGI has the potential to give everyone incredible new capabilities; we can imagine a world where all of us have access to help with almost any cognitive task, providing a great force multiplier for human ingenuity and creativity.

On the other hand, AGI would also come with serious risk of misuse, drastic accidents, and societal disruption. Because the upside of AGI is so great, we do not believe it is possible or desirable for society to stop its development forever; instead, society and the developers of AGI have to figure out how to get it right. ^A

^A We seem to have been given lots of gifts relative to what we expected earlier: for example, it seems like creating AGI will require huge amounts of compute and thus the world will know who is working on it, it seems like the original conception of hyper-evolved RL agents competing with each other and evolving intelligence in a way we can’t really observe is less likely than it originally seemed, almost no one predicted we’d make this much progress on pre-trained language models that can learn from the collective preferences and output of humanity, etc.

AGI could happen soon or far in the future; the takeoff speed from the initial AGI to more powerful successor systems could be slow or fast. Many of us think the safest quadrant in this two-by-two matrix is short timelines and slow takeoff speeds; shorter timelines seem more amenable to coordination and more likely to lead to a slower takeoff due to less of a compute overhang, and a slower takeoff gives us more time to figure out empirically how to solve the safety problem and how to adapt.

This doesn't include discussion of what would make them decide to stop building AGI, but would you be happy if other labs wrote a similar statement? I'm not sure that AI labs actually have an attitude of "we wish we didn't have to build AGI."

Slightly superhuman AGIs (that are very productive and have serial speed advantage, but not qualitatively more intelligent) are not directly uncontainably dangerous, if they don't get an opportunity to build improved versions. The current mode of scaling seems unlikely to suddenly jump from weak systems to superintelligent systems without visibly passing through this stage, if AGIs at this stage don't immediately escape and improve themselves outside the standard process.

An AGI is an enormous risk, primarily because it can trigger building of misaligned superintelligence, either accidentally or through misuse. Any use of first AGIs that isn't targeting improvement of alignment and security is a kind of misuse in the sense of opportunity cost, since it's burning the timeline to misaligned superintelligence to do something other than addressing its risks.

So one problem with potential regulation (which in its origin and application gets to interact with public statements that companies make about their plans and motives) is that it might be permissive enough to build weak AGIs, without being permissive enough to apply them for developing safety measures against emergence of misaligned superintelligence.

in this talk, @Connor Leahy [LW · GW] says that he's talked to the head of "all of these labs" and they all say approximately this.

(i agree it should be more public and clearly legible than being-reported-about-through-connor)

Agree, and in addition to this, I think labs should be really explicit about which part of their motives are supposedly altruistic and which parts are explicitly about avoiding the scenario in which they, personally, don't benefit from AGI. Because "we need to build AGI because it's the only way to advance humanity's technology from now on" is one thing (debatable, IMO, but at least an argument one can make) and "we need to build AGI because if we don't there's a risk that immortality tech won't be here soon enough to apply to our CEO" is another. Never mind the talk about "capturing all value" that Sam Altman has done; "I need to ensure absolute power for myself because I don't feel like trusting anyone else with it" is supervillain talk. And these arguments should be substantiated: if they think there's risks, we want to see real numbers, estimates, and processes and data by which those numbers were calculated. It's one thing to argue something is worth risking the Earth for, which can in some extreme cases be true, but to do so unilaterally without even being transparent about the precise entity and nature of the risks is indefensible.

They are worried about competition from other nations, specifically from China

The People's Republic of China are probably closer than the United Arab Emirates, and definitely closer than the Russians, or the North Koreans, or random Somali warlords. However, as the cost of compute comes down and algorithmic efficiency improvements continue, the number of players with the needed resources will increase (especially if significant parts of the necessary technology are open-sourced). The PRC are also arguably perhaps less likely than some of the others on that list to unintentionally kill us all (they are pretty familiar with using technological means to control the thinking of intelligent agents, after all), and thus more likely to instead actually succeed in enshrining the permanent global rule of the Chinese Communist Party into the terminal goal of a sovereign superintelligence. However, judging by their actions (as opposed to their public ideology), I think they are fairly unlikely to have an AI implement anything resembling actual CEV.

If we (the West, the G20, or the UN) decided to try to enforce an actual extended pause, then access to sufficient compute, talent, and technical know-how would need to be strictly prevented for everyone, I would imagine using the intelligence and military resources of most of the leading nations in the world. Moore's Law, academic publishing, and the open-source community would each make this increasingly difficult. (Bear in mind that the Chinese are currently buying high-end gaming graphics cards and doing some resoldering work on them to turn them into usable AI training compute.) We would thus be building up a large overhang.

I'd really love to hear some serious, practical proposals for how we, as a civilization or international community, could implement an extended pause (preferably without requiring anything in "pivotal act" territory.)