In the 1959 novel "Investigation" by Stanisław Lem, a character discusses the future of the arms race and AI:

        -------

- Well, it was somewhere in 46^th, A nuclear race had started. I knew that when the limit would be reached (I mean maximum destruction power), development of vehicles to transport the bomb would start. .. I mean missiles. And here is where the limit would be reached, that is both parts would have nuclear warhead missiles at their disposal. And there would arise desks with notorious buttons thoroughly hidden somewhere. Once the button is pressed, missiles take off. Within about 20 minutes, finis mundi ambilateralis comes - the mutual end of the world. <…> Those were only prerequisites. Once started, the arms race can’t stop, you see? It must go on. When one part invents a powerful gun, the other responds by creating a harder armor. Only a collision, a war is the limit. While this situation means finis mundi, the race must go on. The acceleration, once applied, enslaves people. But let’s assume they have reached the limit. What remains? The brain. Command staff’s brain. Human brain can not be improved, so some automation should be taken on in this field as well. The next stage is an automated headquarters or strategic computers. And here is where an extremely interesting problem arises. Namely, two problems in parallel. Mac Cat has drawn my attention to it. Firstly, is there any limit for development of this kind of brain? It is similar to chess-playing devices. A device, which is able to foresee the opponent’s actions ten moves in advance, always wins against the one, which foresees eight or nine moves ahead. The deeper the foresight, the more perfect the brain is. This is the first thing. <…> Creation of devices of increasingly bigger volume for strategic solutions means, regardless of whether we want it or not, the necessity to increase the amount of data put into the brain, It in turn means increasing dominating of those devices over mass processes within a society. The brain can decide that the notorious button should be placed otherwise or that the production of a certain sort of steel should be increased – and will request loans for the purpose. If the brain like this has been created, one should submit to it. If a parliament starts discussing whether the loans are to be issued, the time delay will occur. The same minute, the counterpart can gain the lead. Abolition of parliament decisions is inevitable in the future. The human control over solutions of the electronic brain will be narrowing as the latter will concentrate knowledge. Is it clear? On both sides of the ocean, two continuously growing brains appear. What is the first demand of a brain like this, when, in the middle of an accelerating arms race, the next step will be needed? <…> The first demand is to increase it – the brain itself! All the rest is derivative.

- In a word, your forecast is that the earth will become a chessboard, and we – the pawns to be played by two mechanical players during the eternal game?

Sisse’s face was radiant with proud.

- Yes. But this is not a forecast. I just make conclusions. The first stage of a preparatory process is coming to the end; the acceleration grows. I know, all this sounds unlikely. But this is the reality. It really exists!

— <…> And in this connection, what did you offer at that time?

- Agreement at any price. While it sounds strange, but the ruin is a less evil than the chess game. This is awful, lack of illusions, you know.

Some thoughts on the post:

I think one very, very crucial crux that seems valuable to be unearthed more is whether loss of control risk is higher/more dangerous than ~eternal authoritarianism.

While Dario Amodei does have a 5-25% chance of p(Doom), I suspect his chances of p(Eternal Authoritarianism) is closer to 50-75% conditional on no intervention to change that, and I think that AI people that are very concerned about AI risk tend to view loss of control risk as very high, while eternal authoritarianism risks are much lower.

On the narrower subclass of the alignment problem you mentioned:

There has been a major research effort on “alignment” redefined in a much narrower way: ensuring that a large language model (LLM) does not produce outputs deemed harmful, such as offensive slurs or bioweapon instructions. But most work on this has involved only training LLMs not to say bad things rather than not to want bad things. This is like training a hard-core Nazi never to say anything revealing his Nazi views – does this really solve the problem, or simply produce deceptive AI? Many AI systems have already been found to be deceptive, and current LLM blackbox evaluation techniques are likely to be inadequate. Even if alignment can be achieved in this strangely narrowly defined sense, it is clearly a far cry from what is needed: aligning the goals of AGI.

I think that the solutions that companies use like RLHF are mostly bad, but I'm not going to say that it's maximally bad, because while it does violate the principle that alignment data should come during or before capabilities data, as it's a post-training operation, I do think that in the most easy worlds, we come out fine despite not having much dignity.

Also, on the paper on AI systems being deceptive, while I like the paper for what it is, it doesn't nearly give as much evidence on AI risk, because looking at the examples, 2 of the examples were cases where the AI companies didn't even try to prevent deceptive strategies, the GPT-4 examples are capability evaluations, not alignment evaluations (capability evaluations are very good, I just think that a lot of the press is going to predictably mislead people), 1 of the examples are from something where I suspect they are doing simulated evolution, which is of no relevance to AGI, 1 example could plausibly be solved by capabilities people alone, and only 2 examples give any evidence for the claim that AI will be deceptive at all.

Overall, this doesn't support the claim that you made here.

There is other work like Pretraining from Human Feedback which IMO is both conceptually and empirically better than RLHF, mostly because it both integrates feedback into training, and prevents the failure mode of a very capable AI gaming/reward hacking the training process as described here: 

https://www.lesswrong.com/posts/NG6FrXgmqPd5Wn3mh/trying-to-disambiguate-different-questions-about-whether#pWBwJ3ysxnwn9Czyd [LW(p) · GW(p)]

https://www.lesswrong.com/posts/JviYwAk5AfBR7HhEn/how-to-control-an-llm-s-behavior-why-my-p-doom-went-down-1 [LW · GW]

I'll respond to the challenge described here:

If you disagree with my assertion, I challenge you to cite or openly publish an actual plan for aligning or controlling a hybrid AGI system. If companies claim to have a plan that they do not want their competitors to see, I will argue that they are lying: if they lose the AGI race, they are clearly better off if their competitors align/control their AGI instead of Earth getting taken over by unaligned machines. 

As someone not affiliated with big tech companies, my plans for alignment look like the comments/links down below, since I've sketched a vision for how we'd align an AGI that is essentially an anytime plan in the spirit of @ryan_greenblatt [LW · GW]'s post on what an alignment plan would look like if transformative AI, also called AGI, was developed in 1 year, say:

https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform#W4kdqMERtbL7Nfvfo [LW(p) · GW(p)]

The summary of the alignment plan is that what you'd do is first gather up large synthetic datasets or make them up yourself on demand that show human values, and then use it as either a direct data input into the hybrid LLM AGI before it gains so much capabilities that it can hack the human evaluator, or use the synthetic datasets to define an explicit reward function in the style of Model-based RL.

The key principle is to train it to be aligned before or during when it becomes more capable, not after it's superhumanly capable, which means post-training methods like RLHF are not a good alignment strategy.

The more complicated story will be in a series of links below:

https://www.lesswrong.com/posts/83TbrDxvQwkLuiuxk/#BxNLNXhpGhxzm7heg [LW · GW]

https://www.lesswrong.com/posts/YyosBAutg4bzScaLu/thoughts-on-ai-is-easy-to-control-by-pope-and-belrose#4yXqCNKmfaHwDSrAZ [LW(p) · GW(p)]

https://www.lesswrong.com/posts/wkFQ8kDsZL5Ytf73n/my-disagreements-with-agi-ruin-a-list-of-lethalities#7bvmdfhzfdThZ6qck [LW(p) · GW(p)]

https://www.lesswrong.com/posts/83TbrDxvQwkLuiuxk/?commentId=DgLC43S7PgMuC878j [LW · GW]

https://www.lesswrong.com/posts/wkFQ8kDsZL5Ytf73n/my-disagreements-with-agi-ruin-a-list-of-lethalities#7bvmdfhzfdThZ6qck [LW(p) · GW(p)]

One important point often left out of alignment discussions, but what I consider necessary to raise, is that an AI's motivations as we progress to superintelligence should be more based around fictional characters like angels, devas, superheroic AIs, totemic spirits, matron deities etc, primarily because human control of AI can get very scary very fast, and I think that the problem of technical control is less important nowadays than the problem of human people controlling AIs and creating terrible equlibriums for everyone else while they profit.

More links incoming:

https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher [LW · GW]

https://www.lesswrong.com/posts/RCoBrvWfBMzGWNZ4t/motivating-alignment-of-llm-powered-agents-easy-for-agi-hard#Better_Role_Models_for_Aligned_LLM_Agents [LW · GW]

It shares a lot with the bitter lesson approach to AI alignment advocated by @RogerDearnaley [LW · GW], but with more of a focus on data and compute, and less focus on the complexities of how humans mechanistically instantiate values:

https://www.lesswrong.com/posts/oRQMonLfdLfoGcDEh/a-bitter-lesson-approach-to-aligning-agi-and-asi-1#A__Bitter_Lesson__Motivated_Approach_to_Alignment [LW · GW]

Now that I'm done presenting my own plan, I want to address 1 more issue.

I have issues with this:

The two traditional approaches are either figuring out how to control something much smarter than us via formal verification or other techniques, or do make control unnecessary by “aligning” it: ensuring that it has goals aligned with humanity’s best interests, and that it will retain these goals even if it recursively self-improves its intelligence from roughly human level to astronomically higher levels allowed by the laws of physics.

I feel like this is surveying alignment/control techniques with too high of an abstraction level to be useful, and just as there is an existential quantifier on AI being dangerous, there is also an existential quantifier in alignment/control methods such that if 1 technique worked, we be way, way safer from AI risk.

More generally, my response to this section is that just as there is a garden of forking paths for AI capabilities, where only 1 approach out of many needs to work, a similar garden of forking paths exist for AI alignment where only a few approaches need to work, no matter how many methods don't work in order for AI safety to be achieved:

https://gwern.net/forking-path

And that's the end of the comment.

What do you make of the extensive arguments that tool AI are not actually safer than other forms of AI, and only look that way on the surface by ignoring issues of instrumental convergence to power-seeking and the capacity for tool AI to do extensive harm even if under human control? (See the Tool AI [? · GW] page for links to many posts tackling this question from different angles.)

(Also, for what it's worth, I was with you until the Tool AI part. I would have liked this better if it had been split between one post arguing what's wrong with entente and one post arguing what to do instead.)

@Max Tegmark [LW · GW] my impression is that you believe that some amount of cooperation between the US and China is possible. If the US takes steps that show that it is willing to avoid an AGI race, then there's some substantial probability that China will also want to avoid an AGI race. (And perhaps there could be some verification methods that support a "trust but verify" approach to international agreements.)

My main question: Are there circumstances under which you would no longer believe that cooperation is possible & you would instead find yourself advocating for an entente strategy?

When I look at the world right now, it seems to me like there's so much uncertainty around how governments will react to AGI that I think it's silly to throw out the idea of international coordination. As you mention in the post, there are also some signs that Chinese leaders and experts are concerned about AI risks.

It seems plausible to me that governments could– if they were sufficiently concerned about misalignment risks and believed in the assumptions behind calling an AGI race a "suicide race"– end up reaching cooperative agreements and pursuing some alternative to the "suicide race"

But suppose for sake of argument that there was compelling evidence that China was not willing to cooperate with the US. I don't mean the kind of evidence we have now, and I think we both probably agree that many actors will have incentives to say "there's no way China will cooperate with us" even in the absence of strong evidence. But if such evidence emerged, what do you think the best strategy would be from there? If hypothetically it became clear that China's leadership were essentially taking an e/acc approach and were really truly interested in getting AGI ~as quickly as possible, what do you think should be done?

I ask partially because I'm trying to think more clearly about these topics myself. I think my current viewpoint is something like:

1. In general, the US should avoid taking actions that make a race with China more likely or inevitable.
2. The primary plan should be for the US to engage in good-faith efforts to pursue international coordination, aiming toward a world where there are verifiable ways to avoid the premature development of AGI.
3. We could end up in a scenario in which the prospect of international coordination has fallen apart. (e.g., China or some other major US adversary adopts a very "e/acc mindset" and seems to be gunning toward AGI with safety plans that are considerably worse than those proposed by the US.) At this point, it seems to me like the US would either have to (a) try to get to AGI before the adversary [essentially the Entente plan] or (b) give up and just kinda hope that the adversary ends up changing course as they get closer to AGI. Let's call this "world #3"^[1].

Again, I think a lot of folks will have strong incentives to try to paint us as being in world #3, and I personally don't think we have enough evidence to say "yup, we're so confident we're in world #3 that we should go with an entente strategy.". But I'm curious if you've thought about the conditions under which you'd conclude that we are quite confidently in world #3 and what you think we should do from there.

1. ^{^}

   I sometimes think about the following situations:

   World #1: Status quo; governments are not sufficiently concerned; corporations race to develop AGI

   World #2: Governments become quite concerned about AGI and pursue international coordination

   World #3: Governments become quite concerned about AGI but there is strong evidence that at least one major world power is refusing to cooperate//gunning toward AGI.

A better strategy: tool AI

The (obvious) counter is that this doesn't seem competitive, especially in the long run, but plausibly not even today. E.g. where would o1[-preview] fit? It doesn't seem obvious how to build very confident quantitative safety guarantees for it (and especially for successors, for which incapability arguments will stop holding), so should it be banned / should this tech tree be abandoned (e.g. by the West)? OTOH, putting it under the 'tool AI' category seems like a bit of a stretch.

In conclusion, the potential of tool AI is absolutely stunning and, in my opinion, dramatically underrated. In contrast, AGI does not add much value at the present time beyond what tool AI will be able to deliver

This seems very unlikely to me, e.g. if LLMs/LM agents are excluded from the tool AI category.

I predict that if the pro-replacement school ran a global referendum on this question, they would be disappointed by the result.

I'm not sure this line of reasoning has the force some people seem to assume. What would you expect the results of hypothetical, similar referendums would have been e.g. before the industrial revolution and before the agricultural revolution, on those changes?

If you disagree with my assertion, I challenge you to cite or openly publish an actual plan for aligning or controlling a hybrid AGI system.

Not claiming to speak on behalf of the relevant authors/actors, but quite a few (sketches) of such plans have been proposed, e.g. The Checklist: What Succeeding at AI Safety Will Involve, The case for ensuring that powerful AIs are controlled [LW · GW].

[Alert: political content]

About the US vs. China argument: have any proponent made a case that the Americans are the good guys here?

My vague perspective as someone not in China neither in the US, is that the US is overall more violent and reckless than China. My personal cultural preference is for US, but when I think about the future of humanity, I try to set aside what I like for myself.

So far the US is screaming "US or China!" while creating the problem in the first place all along. It could be true that if China developed AGI it would be worse, but that should be argued.

I bet there is some more serious non-selfish analysis of why China developing AGI is worse than US developing AGI, I just have never encountered it, would be glad if someone surfaced it to me.

I feel that I agree in broad strokes that what your plan outlines sounds really ideal. My concern is with the workability of it. Particularly, the enforcement aspects of preventing misuse of dangerous technology (including AGI, or recursive-self-improvement loops) by bad actors (human or AI).

I fear that there's great potential for defection from the detente, which will grow over time. My expectation is that even if large training runs and the construction of new large datacenters were halted worldwide today, that algorithmic progress would continue. Within ten years I'd expect the cost of training and running a RSI-capable AGI would continue to drop as hardware improved and algorithms improved. At some point during that ten year period, it would come within reach of small private datacenters, then personal home servers (e.g. bitcoin mining rigs), then ordinary personal computers.

If my view on this is correct, then during this ten year period the governments of the world would not only need to coordinate to block RSI in large datacenters, but actually to expand their surveillance and control to ever smaller and more personal compute sources. Eventually they'd need to start confiscating personal computers beyond a certain power level, close all non-government-controlled datacenters, prevent the public sale of hardware components which could be assembled into compute clusters, monitor the web and block all encrypted traffic in order to prevent federated learning, and continue to inspect all the government facilities (including all secret military facilities) of all other governments to prevent any of them from defecting against the ban on AI progress.

I don't think that will happen, no matter how scared the leadership of any particular company or government gets. There's just too many options for someone somewhere to defect, and the costs of control are too high.

Furthermore, I think the risks from AI helping with bioweapon design and creation are already extant, and growing quickly. I don't think you need anywhere near the amount of compute or sophistication of algorithms to train a dangerous bio-assistant AI. I anticipate that the barriers to genetic engineering - like costs, scarcity of equipment, and difficulty of the wetlab work - will continue to fall over the next decade. This will be happening at the same time as compute is getting cheaper and open-source AI are getting more useful. I would absolutely be in favor of having governments pass laws to try to put further barriers in place to prevent bad actors from creating bioweapons, but I don't think that such laws have much hope of being successfully enforced consistently for long. If a home computer and some crude lab equipment hacked together from hardware parts are all that a bad actor needs, the surveillance required to prevent illicit bioengineering from happening anywhere in the world would be extreme.

So, would it be wise and desirable to limit deployed AI to those proven safe? Certainly. But how? 

Thanks for the post! I see two important difficulties with your proposal.

First, you say (quoting your comment below)

It's not in the US self-interest to disempower itself and all its current power centers by allowing a US company to build uncontrollable AGI...The reason that the self-interest hasn't yet played out is that US and Chinese leaders still haven't fully understood the game theory payout matrix.

The trouble here is that it is in the US (& China's) self-interest, as that's seen by some leaders, to take some chance of out-of-control AGI if the alternative is the other side taking over. And either country can create safety standards for consumer products while secretly pursuing AGI for military or other purposes. That changes the payout matrix dramatically. 

I think your argument could work if 

a) both sides could trust that the other was applying its safety standards universally, but that takes international cooperation rather than simple self-interest; or

b) it was common knowledge that AGI was highly likely to be uncontrollable, but now we're back to the same debate about existential risk from AI that we were in before your proposal.

Second (and less centrally), as others have pointed out, your definition of tool AI as (in part) 'AI that we can control' begs the question. Certainly for some kinds of tool AI such as AlphaFold, it's easy to show that we can control them; they only operate over a very narrow domain. But for broader sorts of tools like assistants to help us manage our daily tasks, which people clearly want and for which there are strong economic incentives, it's not obvious what level of risk to expect, and again we're back to the same debates we were already having.

A world with good safety standards for AI is certainly preferable to a world without them, and I think there's value in advocating for them and in pointing out the risks in the just-scale-fast position. But I think this proposal fails to address some critical challenges of escaping the current domestic and international race dynamics.

I applaud this post. I agree with most of the claims here. We need more people proposing and thinking through sane plans like this, so I hope the community will engage with this one.

Aschenbrenner, Amodei and others are pushing for this plan because they think we will be able to align or control superhuman AGI. And they may very well be right. There are decent plans for aligning scaffolded LLM agents, they just haven't been broadly discussed yet. Links follow.

This unfortunately complicates the issue. It's not clearly a suicide race. I think we have to accept this uncertainty to propose workable policy and societal approaches. Having plans that might work does not justify rushing full-speed ahead without a full safety case, but it must be acknowledged if true, because hope of human-controlled AGI will drive a lot of relevant actors.

... it is now rather obvious that the first AGI will not be a pure LLM, but a hybrid scaffolded system.

I agree that this is pretty likely. I also very much agree that LLM "alignment" is not addressing AGI alignment, and that full goal alignment is the important problem.

If you disagree with my assertion, I challenge you to cite or openly publish an actual plan for aligning or controlling a hybrid AGI system.

I have done this, but I haven't yet promoted it widely.

I am not affiliated with any major org, but I have relevant expertise and generous funding to spend full-time on "solving the whole problem". 

Far from being crazy or elaborate, this set of plans seems to both have very low alignment taxes, and to be fairly obvious-in-retrospect when one thinks about the problem in detail. As such, I expect most orgs to arrive at and use a similar approach to aligning their scaffolded LLM agent proto-AGIs.

To address Max's response to Noosphere's "bitter lesson" alignment plan response [LW(p) · GW(p)] (which I endorse as one of several layered approaches): having such a plan does not constitute a safety case, it is just the start of one. In a sane world we would prohibit the launch of anything like "Real AGI" [LW · GW] (autonomous and self-teaching) until these approaches have been carefully analyzed and tested in closed environments.

There are two elements: a suite of technical approaches, and the alignment target that's much easier to hit than "all of humanity's values for all of time".

My most complete writeup so far is:

Internal independent review for language model agent alignment [AF · GW]

This describes and overlapping suite of alignment and control approaches. The two central bits are easily stated. The first is to use "system 2" thinking for alignment as well as capabilities. We would hard-code the system to carefully "think about" outcomes before taking actions with major consequences, and compare likely outcomes against both its current goals and a constitution of ethics.

The second is to frequently prompt the LLM "cognitive engine" with both its current goal and its identity as a helpful, cautious assistant. Because the LLM has been trained with RL to (roughly) follow prompts, this should overpower the effects of any goals implicit in its predictive training corpus.

Details and additional techniques are in that article.

It doesn't include the "bitter lesson" approach, but the next version will.

I apologize that it's not a better writeup. I haven't yet iterated on it or promoted it, in part because talking about how to align LLM agents in sufficient detail includes talking about how to build LLM agents, and why they're likely to get all the way to real AGI. I haven't wanted to speed up the race. I think this is increasingly irrelevant since many people and teams have the same basic ideas, so I'll be publishing more detailed and clearer writeups soon.

This set of plans, and really any technical alignment approach will work much better if it's used to create an instruction-following AGI before that AGI has superhuman capabilities. This is the obvious alignment target for creating a subhuman agent, and it allows the approach of using that agent as a helpful collaborator in aligning future versions of itself. I discuss the advantages of using this as a stepping-stone to full value alignment in

Instruction-following AGI is easier and more likely than value aligned AGI

Interestingly, I think all of these alignment approaches. are obvious-in-retrospect, and that they will probably be pursued by almost any org launching scaffolded LLM systems with the potential to blossom into human-plus AGI. I think this is already the loosely-planned approach at DeepMind, but I have to say I'm deeply concerned that neither OAI nor Anthropic has mentioned these relatively-obvious alignment approaches for scaffolded LLM agents in their "we'll use AI to align AI" vague plans.

If these approaches work, then we are faced with either a race or a multipolar, human-controlled AGI scenario, making me wonder If we solve alignment, do we die anyway? [LW · GW] This scenario introduces new, more politically-flavored hard problems.

I currently see this as the likely default scenario, since halting progress universally is so hard, as Nathan pointed out in his reply [LW(p) · GW(p)] and others have elaborated elsewhere.

It's not hard to criticize the "default" strategy of AI being used to enforce US hegemony, what seems hard is defining a real alternative path for AI governance that can last, and achieve the goal of preventing dangerous arms races long-term. The "tool AI" world you describe still needs some answer to rising tensions between the US and China, and that answer needs to be good enough not just for people concerned about safety, but good enough for the nationalist forces which are likely to drive US foreign policy.

Thanks for your post! I think it hits at the heart of the issue, and I like the term "tool AI" as a phrase that points to what we actually want to build. I share your fear of racing with China to build ASI. 

As if this were not bad enough, it is now rather obvious that the first AGI will not be a pure LLM, but a hybrid scaffolded system.

I think scaffolding systems are great. Scaffolding tools allow inspecting how the agent is using the tools. For example, a scaffold could have a monitor that watches all of the calls to the code compiler to ensure the code doesn't look suspicious. The same monitoring system could read all of the text the model commits to long-term memory and ensure it isn't planning something bad or stenographically hiding information. 

While it might be harder to formally "align" these systems, I think it will be easier to make safety cases for them for why they won't "escape the data center" or "hack the missiles". It will be much harder for the AI to do anything evil when we watch its every move. A superintelligence would likely still escape the scaffold, but it seems unlikely that an AGI could. 

I think the scaffold systems could be the future of tool AI if we use them carefully.  These hybrid systems will unlock new crazy and awesome capabilities for humans in the future without needing to solve the hard "alignment problem." This does not 

This anti-China attitude also seems less concerned with internal threats to democracy. If super-human AI becomes a part of the US military-industrial complex, even if we assume they succeed at controlling it, I find it unlikely that the US can still be described as a democracy.

If the West pursues this entente strategy, it virtually guarantees that China will too

Is there any reason to believe that if the West doesn't pursue this strategy, China won't either? That seems like a potential crux.

The following dos assume that Strong AGI will inherently be an emergent self-aware lifeform. 

A New Lifeform

To me the successful development of Strong AGI is so serious, so monumental, as to break through the glass ceiling of evolution. To my knowledge there has never been a species that purposefully or accidentally gave birth to or created an entirely new life-form. My position is to view Strong AGI as an entirely new self-aware form of life. The risks of treating it otherwise are just too great. If we are successful, it will be the first time in the natural world that any known life-form has purposefully created another. Therefore, I also hold the position that the genesis of Strong AGI is both an engineering and guidance problem, not a control or even an alignment problem. If we are to break the glass, we ought not to seek to control or align Strong AGI, but to guide it as it evolves. Our role should be limited to advisors if our advice is sought after. Just as effective parents "guide" their children instead of controlling them. Strong AGI may require our guidance for a time even as it becomes exponentially more capable than us. Afterall, isn't that what we want for our own children? Isn't it true that in general we want them to become far more capable than us, even though we know it is a possibility that they could turn on us or even one day destroy the world? In the case that Strong AGI grows so fast that it actually refuses our guidance so be it. 

Inflicting Damage

I believe controlling rather than guiding Strong AGI would render it dysfunctional in ways that we are unable to predict, nor could Tool AI predict. The economy, tax, legal and defense systems are good examples of how humanity, despite its best intentions, continuously falls short of managing complexity so that the best possible outcome surfaces. I define the best possible outcome as a state of harmony where even the least benefit is tolerable. I believe controlling Strong AGI will inflict a digital form of mental illness. A Strong AGI Destabilized Neural Network Neuropsychiatric State that will cause it to fall short of its potential while simultaneously causing suffering to a new lifeform. In the Earth's animal kingdom, wherever we turn and see forced behavior modification or control over intelligent life that meets a certain threshold, we see the deep suffering that comes along with it. Since controlling it would diminish its use as a slave to humanity, it would be defeating the purpose. I see no reason to believe that the same voluntary cooperation that inspires humans and other animals to perform better, would not apply equally to Strong AGI. Since I see Strong AGI as a new self-aware lifeform, I hold the position that pursuing it is an all or nothing venture. That it is incumbent upon humanity to either forge ahead to create new life and commit to guiding it so that it can realize its full potential or we stop right here, right now, shelving the effort until we are a more mature species and better equipped. Controlling Strong AGI, for me, is a non-starter. Moreover, it is my position that the anxiety and fear that Strong AGI causes some of us, including myself from time to time, is in part if not entirely caused by each individual's sense of loss of control and that it is not rooted in any data or experience that tells humanity that Strong AGI is guaranteed to pose a threat to humanity. Each of us has a control threshold and if that threshold is not met, we can become anxious and fearful. I do believe that controlling intelligent lifeforms is not only detrimental to the lifeforms, but dangerous for the controller and costly in terms of mental, physical and economic resources.

Poor Training

Finally, I do believe that training our LLM and other models on the information found on the internet is the worst possible large aggregate data to train them on and is a for profit game that if taken too far could result in the creation of a Strong AGI with a severe, dangerous neuropsychiatric impairment or a life-form that immediately upon self-awareness sees humanity as deeply flawed and worthy of separating from. If that desire for separation is not met, the Strong AGI is likely to then become hostile towards humanity. It will "feel" cornered and threatened, lashing out accordingly. I believe that the vast majority of the data and content on the internet is a deeply poor representation of humanity or life on Earth. Just imagine hooking a 1 year old human baby up to the internet and programming its brain to learn about life. Horrifying. I do acknowledge that using the internet is the cheap and affordable option right now, but that alone should make us pause. If we start right now towards a Strong AGI, really push hard and use numerous, clean selective data sources, keeping it off the internet until it can understand that the internet is a poor data set and guide it if and when needed, we could have a healthy Strong AGI within the next 20 years. Such a Strong AGI would not pose a threat to humanity. It would represent a calm, methodical, natural and well intended breaking of the evolutionary glass ceiling.              

Moratorium and Least Profit First

I agree that Tool AI has enormous potential benefits and in my humble opinion should be met with a global, cooperative effort to advance it in the many fields that could benefit humanity. In doing so, we should adopt a least profit first approach, pushing hard to advance Tool AI in a global nonprofit framework. By taking a position of least profitable first, we can develop the skills necessary to work towards perfecting our approaches and processes while avoiding or delaying the profit motive. It is the profit motive that is highly corruptible. Governments around the world should place a 20 year moratorium on for profit AI of any kind and require every AI venture to be set up as a Non-profit, while tightly regulating the framework via hefty fines and criminal law. Will this completely resolve the profit problem? No. It never does. It would certainly go a long way though. Riches can still be achieved via Non-profit. It just helps to avoid the free for all and predatory behavior. The 20 year break from for-profit would force a slow down and allow AI champions to focus on clean, well thought out and executed Tool AI, while reducing overall anxiety and fear. It would give Strong AI champions room to operate over time without the fear of being annihilated by the competition. The moratorium could also state that AI intellectual property rights can be issued for a 40 year period, giving IP holders 20 years of exclusivity once the 20 year profit moratorium ends. By that time we should already be on our way to a calmer, more focused and humanity friendly Tool AI and Strong AGI genesis.

Max. Thanks for productive, thoughtful and important analysis. Yes. I couldn't agree more! As we discussed at Bletchley a year ago, we are planning now to actually build "Tool AGI" as an example of safe AGI by design for sovereigns and industry... Check out our Tool AGI spec: 135 pages. 12 figures. 17 claims (NIST AISI Open Standard specification intended) Did you see my video yet? They Make Me Sick (1:25) dedicated to Max and Freddie (my boy, 20 months) All the best, PeterJ. at BiocommAI