Cooperation is optimal, with weaker agents too  -  tldr

post by Ryo (Flewrint Ophiuni) · 2024-04-18T15:03:47.245Z · LW · GW · 22 comments

This is a link post for https://medium.com/p/aeb68729829c

Contents

  For rational agents (long-term) at all scale (human, AGI, ASI…)
  Synthesis:
    Recap
None
22 comments

It's a ‘superrational [? · GW]’ extension of the proven optimality of cooperation in game theory 
+ Taking into account asymmetries of power
// Still AI risk is very real

Short version of an already skimmed 12min post
29min version here

---

For rational agents (long-term) at all scale (human, AGI, ASI…)

In real contexts, with open environments (world, universe), there is always a risk to meet someone/something stronger than you, and overall weaker agents may be specialized in your flaws/blind spots. 


To protect yourself, you can choose the maximally rational and cooperative alliance:

Because any agent is subjected to the same pressure/threat of (actual or potential) stronger agents/alliances/systems, one can take an insurance that more powerful superrational agents will behave well by behaving well with weaker agents. This is the basic rule allowing scale-free cooperation.

If you integrated this super-cooperative protocol in your system/behaviors, a stronger (superrational) agent will recognize it and not punish you, as per the contract of maximally rational/cooperative norms.

-> You are the weak of another -> be nice to the weak

• It’s a vertical reciprocity. With it, they are maximally cooperative thus are increasingly likely to have more allies than in any other alliance.
• You can deduce and obey to the rules of such rational and maximally cooperative alliance (the super-coordination cluster). Thus; to be part of it, you don’t need to actually meet other members (as in an acausal trade [? · GW]), but when you do meet them; they will help you (and vise-versa).

They actually are already improving universal commons, refining the future conditions of (actual or potential) agents from your zone of the diversity gradient (latent space of potential agents). 

(All this might seem too easy/cool to be true, but long versions explore prices to pay, risks and unknowns - see the tldr at the end of the 12min version)

---

Synthesis:

This alliance bases itself on a protocol (an acausal normalcy [LW · GW]*), with at its core :
-> Do your best to achieve ‘paretotopia’ (a state strongly preferred by more or less everyone).

Then most rules are either coming from this first imperative, or a protection against defectors.

*Acausal normalcy: rational convergence towards certain norms, habits, social contracts, even though agents might have no way of communicating or affecting each other, nor even any direct evidence that the other exists

In game theory the optimality of cooperation has its own ingredients, based on reciprocity and non-naive altruism. We need to be: Nice, Forgiving, Retaliatory, Clear.

-> Homo moralis ~ “act according to that maxim whereby you can, at the same time, will that others should do likewise with some probability.”

The adoption of a kantian behavior (homo moralis) is pareto-efficient, which means that all the possibilities of unanimous gains have been used. On top of that, Kantianism is robust against mutations, so that in the long-term, Kantianism beats all the other strategies.

Several [LW · GW] researches [LW · GW] (a few have asymmetric frameworks) explore [LW · GW]superrationality as a solution to cooperation, tragedy of the commons etc.

We are going to introduce how the protocol (its modeling in practice) is tractable, the long post will pursue a description from first principles

---

 — There is always a certain amount of uncertainty.

Computational Irreducibility: “While many computations admit shortcuts that allow them to be performed more rapidly, others cannot be sped up. Computations that cannot be sped up by means of any shortcut are called computationally irreducible.”

Navigating such intricacies as rational agents leads to strategies such as;

The self-indication assumption (SIA) [LW · GW]: “All other things equal, an observer should reason as if they are randomly selected from the set of all possible observers.”

Following SIA, if I am able to scale, then other than me would certainly be able to scale as well, there is a high likelihood of being mid.

Mediocrity principle: “If an item is drawn at random from one of several sets or categories, it’s more likely to come from the most numerous category than from any one of the less numerous categories”

---

If you are a super-human AI invading the universe, there are big chances that you’ll encounter other AIs from other civilizations.

-> On the short-term, you might be the strongest agent/alliance + scaling faster than any other agents/alliances. But uncertainty is omnipresent; on the long-term, you will likely encounter other agents/alliances faster/older/stronger than you.

You could say “AIs will do super-coordination (vertical reciprocity) but only starting above human intelligence”, however:

What keeps stronger AIs from doing the same; starting super-alliance “above x level of power”? 

(It's to avoid this permanent threat that super-cooperation is a scale-free reciprocity)

And if AIs regroup around values/goals rather than power-level (in order to fight for supremacy), the chances to end-up alive at the end of such war are very small (it’s a battle royal with a single winning AI/alliance).

What grows the split of most AIs is open-endedness; so optionality expansion, so:

Super-coordination.

Destroying options will always go against more potential agents
(Except when the short-term destruction of options increases long-term optionality — like apoptosis) 

What about:

-> Destruction being easier than construction?

We can have safe islands of bloom and a defense focused industry, so that the gap relative to domination/destruction-focused agents isn’t too large.

This is the gist of long-term planning/optionality; for a while, you may dedicate many resources against ‘anti super-coordination actors’.

And the super-coordination alliance makes sure that no one is getting overwhelmingly powerful at a scale so large one can dominate anybody.

Note: direct consequences of the super-coordination protocol may justify the current absence of alien contact, we’ll see that in longer posts

---

All things being equal, as an agent (any agent) what is the maximally logical thing to do?

-> To preserve/increase options.

(So it is the most fundamental need/safety/power/wealth)

It relates to the instrumental convergence of self-empowerment, antifragility, situational awareness and core moral/welfare systems (capability approach, autonomy in moral, other-empowerment [LW · GW]).

If we combine this to super-coordination, the aim would be to increase the “pareto-optionality”, which is to say “increase options for the highest number/diversity of agents possible”.

---

As we will see, rationality is a process; it takes time to minimize the impact of constraints/biases imposed by irreducibility and imperfect data.

We are biased towards our survival and (hoping for) cooperation, but AIs might be biased towards rapid myopic utilitarian maximization.

Although to be ignoring super-coordination they would have to be blind/myopic automations (causal viruses) without long-term rationality.

In any case, accidents [LW · GW], oligopoly and misuse (cyber-biorisk etc.) are a complex and pressing danger.

---

Enough stability is part of the requirements for diversity to expand.

To explore solutions, we need productive deliberation, methodological agreements/disagreements and bridging systems. I think this plan involves, among other things, an interactive map of debate using features taken from pol.is and moral graphs. 

We can also develop an encrypted protocol based on super-coordination (scaling legitimate/secured trust).

Using these ideas (and more) I propose a plan to coordinate despite our biases:

Presentation of the Synergity project [LW · GW]

I need help for the technical implementation,

We have plans to leverage super-coordination and enable more prosaic flux of convergence/information; interfacing democracy:

So please contact me if you are interested in discussing these subjects, organizing the next steps together.

---

Recap

• Because of its rules the super-coordination cluster is likely stronger than any one individual/alliance

• In the long-term, it’s the strategy that (likely) compounds the most while also optimizing safety

• It’s the most open-ended cooperation, including a maximal amount/diversity of agents

• It’s based on an acausal contract that can be signed from any point in space and time (without necessity of a direct encounter)

22 comments

Comments sorted by top scores.