Organisation-Level Lock-In Risk Interventions

post by alamerton · 2025-04-01T12:42:21.588Z · LW · GW · 0 comments

Contents

  TL;DR
      Decision-making power being concentrated into the hands of a small group of individuals (AI agents or humans) causing a monopoly, oligopoly, or plutocracy.
  How Lock-In Could Manifest Through This Threat Model
  How the Organisation Structure Lends itself to this
  What Might Happen if There Was a Different Structure
  Organisation Structures
    Alternative Organisation Structures
    External Partnerships and Oversight Systems
  How They Could Effectively Target Organisations
None
No comments

Epistemic status [EA · GW]: my own thoughts and opinions after thinking about organisation structures and dynamics as an intervention point for lock-in risk for about 5 hours. My thoughts here represent the opinion of Formation Research at the time of writing.

TL;DR

We believe lock-in risks are a pressing problem, and that existing organisation structures lend themselves to lock-in via power concentration. This post outlines how lock-in could result from organisation structures and dynamics, and what interventions might be implemented to reduce the risk.

We have identified 4 key threat models [LW · GW] for lock-in; ways we believe undesirable lock-ins could manifest in the future. This post focuses on one specific threat model:

Decision-making power being concentrated into the hands of a small group of individuals (AI agents or humans) causing a monopoly, oligopoly, or plutocracy.

One of the key features of this threat model is organisations. Most global decision-making is done by organisations of some kind, be they governments, corporations, NGOs, or companies of some other kind.

Therefore, we claim that a key intervention point for the manifestation of this threat model lies in the dynamics and structures of these organisations, specifically those related to their decision-making. Decisions made by organisations rely on the configuration of their components: the individuals, dynamics, and organisation structure. We are using this terms to roughly mean:

The types of organisation we believe to be most relevant to AI-relevant power concentration are AI foundation model companies, cloud computing and infrastructure providers, semiconductor and chip manufacturing companies, regulatory and governance bodies, academic and research institutions, venture capital companies, and AI application companies. In this post, we make the claim that interventions targeting organisation structures and dynamics show promise for mitigating undesirable lock-in scenarios via this threat model.

How Lock-In Could Manifest Through This Threat Model

A lock-in could manifest through this threat model if an organisation achieves a decisive strategic advantage in its domain.[1] That is to say the organisation reaches sufficient decision-making power to become the chief decision-making agency of that domain. This is Nick Bostrom’s formation of a Singleton via decisive strategic advantage. It would follow something like these key steps:

  1. The organisation making decisions that allow it to gain decisive strategic advantage
  2. Them using that advantage to become the chief decision maker in their domain, e.g., AI foundation models, cloud compute, etc
  3. They use their domain-level power to make the rules
  4. Monopoly/plutocracy/etc. Power is now concentrated
  5. This can go on indefinitely within the bounds of the domain. If the domain is the world, then potentially indefinitely. This would be more robust if AIs are in charge of it because of digital error-correction

For example, let’s say Anthropenmic is a competitive AI foundation model company, and in their race through the minefield of capability advancement, they were lucky enough to win out over their competition (and natural selection – we’re imagining a bunch of other concerning things go well in this hypothetical). 

Picture of 5 icons with arrows representing racing companies
AI foundation model companies compete in an AI arms race

They are now in a position where their foundation model is the best. The model has been widely distributed and is most users’ top choice. They have achieved the decisive strategic advantage.

The same diagram, with one company going further than the rest
Anthropenmic gains a decisive strategic advantage

 

They can now make the rules for the rest of the domain. They have sufficient power in terms of investment and technological capability such that their products are the most popular choice for the consumer.

Diagram showing a company winning the race
Anthropenmic becomes the main provider, getting to set the norms and stabilise its position

How they have chosen to shape their product is now the default way the consumer’s product is shaped. That is to say Anthropenmic gets to choose how the product works for everyone.

This can now go on indefinitely within the bounds of the domain of AI foundation model production. If it ends up being the case that Antropenmic decides to employ an AI C-suite, or a whole brain emulation of a C-suite, they could potentially keep this power indefinitely.[2]

How the Organisation Structure Lends itself to this

Anthropenmic, other AI foundation companies, and other organisations, typically have a hierarchical structure of individuals which remains mostly stable across time. Yes, individuals come and go, but not to a degree where decision-making can be shaken to much of a degree (and usually, people are replaced more often when their decision-making contribution is misaligned with the rest of the C-suite).

This gives rise to a rigid decision-making trajectory. Because the same group of people are making decisions on behalf of the organisation together, with the same vision in mind, long-term. If it’s approximately the same C-suite making the decisions that put the company at the head of the arms race, those same executives are not going to stop making decisions that help it go in that direction when it wins.

That’s why the competitive company is the one that concentrates power and holds it stable once it achieves the decisive strategic advantage, because it was the competitive team that got it to that place in the first place and that same team won’t want to lose that position once it has won it.

Therefore, the structure of the organisation, while beneficial for achieving the organisation’s goals, has ended up contributing to the power concentration and subsequent lock-in as a consequence of its competitive success.

What Might Happen if There Was a Different Structure

We claim there are two processes, alluded to above, that lend to this:

  1. The competitive striving of the decision-makers to achieve the decisive strategic advantage
  2. The consequential desire to sustain the poll position once the decisive strategic advantage has been achieved

These can be viewed as two phases in the organisation’s race to the top. Each result from trying to win, the first is when there is competition present and decisive strategic advantage has not been achieved, and the second is when the competition has been beaten and the decisive strategic advantage is achieved. So the elements of these two phases are:

We claim that if the arrangement of these individuals, dynamics, and structures is modified, lock-in through this threat model could be effectively mitigated.

Organisation Structures

While organisation structures vary significantly depending on domain, purpose, and size, three archetypal categories are apparent:

  1. Smaller organisations with flat hierarchies and lean/agile teams and principles: AI foundation model companies, venture capital and investment ecosystems, cybersecurity and AI safety companies, and AI application companies.
  2. Larger hierarchical organisations: cloud computing and AI infrastructure companies, semiconductor and chip manufacturing companies
  3. Bureaucratic, centralised organisations: regulatory and governance bodies, geopolitical AI development programmes, academic and research institutions

To combat this lock-in risk, we identify two main potential intervention strategies: alternative organisation structures and dynamics, and external partnerships and oversight systems. These can also be framed as internal and external intervention strategies respectively.

Alternative Organisation Structures

We believe decentralisation is a promising concept for mitigating against lock-in, and especially against power concentration. Decentralised organisations could benefit from group decision-making and distributed decision-making power. This is a technical structural change that lends to the prevention of power concentration.

Generally, democratic decision-making frameworks could offer a strong intervention against power concentration through the rotation of leadership.

This could intercept the rigidity of the decision-making team during competition, letting the organisation continually pivot in a direction that is more representative of the values, goals, mission, and vision of the organisation as a whole, rather than of a stable C-suite.

Furthermore, rotating leadership programmes could offer an advantage as an intervention for hierarchical organisations because the top level of organisations might be all that's left soon.

Rotating leadership could preserve dynamic decision-making

Other promising alternative structures include community-owned digital platforms or computational infrastructure. Holacratic digital organisations could avert the power concentration problem by distributing decision-making power across its users, like in open source software, platform cooperativism, and open value networks.`

Distributed ledgers and blockchain technology also offers some promise in the intervention space for this threat model. Integrating a blockchain into organisations, especially around decision-making, could promote transparency and organisation purpose-aligned decision-making. Decentralised autonomous organisations (DAOs) are the best example of an organisation structure like this. In DAOs, token holders get to vote in organisation decisions. Implementing distributed stakeholders in decision-making may again promote more representative decision-making that mitigates lock-in through power concentration.

Lastly, hybrid organisation structures, which leverage these structural changes while remaining uninvasive, offer another promising intervention strategy, especially one that is potentially more actionable due to its hybridity. Federated networks and polycentric organisations offer examples of hybrid organisation structure alternatives that could help mitigate power concentration.

External Partnerships and Oversight Systems

We also believe that external interventions through partnered organisations could have promise in mitigating lock-in through this threat model. The key idea with this intervention is the conception of an auditing party which monitors organisational ethics, or purpose alignment.

For example, a watchdog group could be formed which monitors AI foundation model companies. The group would target all the relevant companies and only operate when a minimum proportion of companies agree. It could then monitor all the companies according to the same rules, and sanction companies for unethical or purpose-misaligned decision-making.[3]

Monitoring all or most of the relevant target organisations by the same rules would not put any one organisation at a disadvantage in the AI arms race. The entire arms race would be slowed down, normalising the competitive disadvantage for everyone.

A picture of organisations racing forwards, with arrows to show that they are all slowed down by the same amount, preserving their position in the race
External partnerships and oversight systems could be position-preserving

The group might only operate when all companies agree to audits and all audits are being properly processed. If this stops, it stops for all companies, maintaining companies’ progress in the race, keeping the arms race steady. Of course, this does nothing to remove the superintelligence landmine, but that’s not the purpose of this post and is out of scope. The group performing the audits could also rotate over time to prevent any meta-power concentration/entrenchment (one way in which these intervention strategies could be synergistic if combined).

There probably exist other configurations of external auditing and oversight mechanisms that would help to prevent power concentration in a similar way. The key idea here is that an external party can be employed by the organisation to monitor the decision-making dynamics and organisation structure and prevent the manifestation of power concentration in the organisation’s domain, by providing direct sanctions, enforcing changes, or reporting to a regulatory body.

A combination of the above two strategies, as suggested just now, could be synergistic. Implementing an auditing body which rotates democratically, or implementing a rotating leadership protocol and external oversight in one organisation, could provide a more robust intervention strategy than either one alone.

Governance safeguards and regulations are also promising interventions for this threat model, but are not the primary focus of this post. Advocating for regulatory bodies to enforce these interventions, as well as a number of other interventions less related to organisation structures, is a meta-intervention strategy that is implied here but will not be referenced any more than that in this post.

How They Could Effectively Target Organisations

So far we have outlined some possible intervention strategies for this threat model, what they might look like, and how they could be combined. Effectively targeting and deploying these interventions is also important. Here we outline some ways the interventions may be more readily adopted by organisations.

First, interventions that are minimally invasive may increase the likelihood they are adopted by organisations. We mentioned that hybrid decentralisation approaches might be effective, as they don’t change the entire organisation’s structure. We believe the less that needs to be changed for an intervention to be implemented (the easier it is), the more effective it is. Therefore, composing intervention strategies that leverage impact while minimising invasiveness might be a promising way of actually affecting change.

Next, workshopping ideas with organisations could help them get implemented. Working with members of an organisation, who understand the existing individuals, structure, and dynamics, may illuminate the ways the original intervention approach will fail. Approaching the implementation of the intervention like a semi-structured interview – having a loose structure and main goal in mind, but being agile enough to let the implementation take its own shape, provided that the goal is still achieved – may be effective.

Last, in dealing with organisations in which market dynamics and competition play a crucial role, position-preserving interventions – that is, approaches that don’t disrupt any one organisation’s position in the AI arms race – might be more willingly adopted by those with skin in the game and conflicting demands. These may be intervention strategies that affect all the organisations equally, don’t disrupt positions in the arms race at all, or only act as a lever on reducing risks (because competition isn’t the problem, the risks are). A regulation for pausing or slowing AI development is an example of a position-preserving intervention, or all competing organisations signing up for the same auditing agreement.

In this post, we outline the lock-in threat model of decision-making power being concentrated into the hands of a small group of individuals (AI agents or humans) causing a monopoly, oligopoly, or plutocracy, and illustrate how current organisation structures and dynamics lend themselves to the manifestation of a lock-in through this threat model. We then outline two key intervention strategies for this threat model, internal (alternative organisation structures) and external (external partnerships and oversight systems), and discuss how they could be combined and effectively deployed.

  1. ^

    Concept introduced by Nick Bostrom in Superintelligence: Paths, Dangers, Strategies meaning a position of strategic superiority that gives an actor the ability to achieve complete world domination.

  2. ^

    An upcoming post will outline the role of digital error-correction in this process, and interventions therein. For now, AGI and Lock-In [EA · GW], Section 4, goes into detail on this.

  3. ^

    If this came from a governance or regulatory body, something might need to be implemented to ensure that body also adheres, or else they could sneak past the auditing group, get a strategic advantage, and win the arms race.

0 comments

Comments sorted by top scores.