List #1: Why stopping the development of AGI is hard but doable

Contents

  Why stopping the development of AGI is hard but doable
None
11 comments

A friend in technical AI Safety shared a list of cruxes for their next career step. 

The first crux was that they did not believe that progress on AI can be expected to stop. 

Copy-pasting a list that I compiled in response (with light edits):

Why stopping the development of AGI is hard but doable

• History of technological restraint by institutions:
  • From Matthijs Maas' essay [EA · GW]:
    • “In short: some expect that we could not meaningfully slow or halt the development of A(G)I even if we expect extreme risks. 
    • “Yet there is a surprisingly diverse historical track record of technological delay and restraint, even for strategically promising technologies that were seen as 'obvious' and near-inevitable in their time. 
    • “Epistemic hurdles around studying 'undeployed' technologies make it likely that we underestimate the frequency of restraint decisions, or misinterpret their causes. From an outside view, this should lead us to be slightly more optimistic about the viability of restraint for future technologies. 
                                          
       
• A lot of tech does not get developed:
  • From Katja Grace's Slow Down AI post [LW · GW]: 
    • “There seems to be a common thought that technology is a kind of inevitable path along which the world must tread, and that trying to slow down or avoid any part of it would be both futile and extreme.
    • “But empirically, the world doesn’t pursue every technology—it barely pursues any technologies.
      • Sucky technologies
        • For a start, there are many machines that there is no pressure to make, because they have no value. Consider a machine that sprays shit in your eyes. We can technologically do that, but probably nobody has ever built that machine. 
      • Extremely valuable technologies
        • It doesn’t look like it to me. Here are a few technologies which I’d guess have substantial economic value, where research progress or uptake appears to be drastically slower than it could be, for reasons of concern about safety or ethics: ...
      • Coordination is not miraculous world government, usually
        • The common image of coordination seems to be explicit, centralized, involving of every party in the world, and something like cooperating on a prisoners’ dilemma: incentives push every rational party toward defection at all times, yet maybe through deontological virtues or sophisticated decision theories or strong international treaties, everyone manages to not defect for enough teetering moments to find another solution.
        • That is a possible way coordination could be. (And I think one that shouldn’t be seen as so hopeless—the world has actually coordinated on some impressive things, e.g. nuclear non-proliferation.) But if what you want is for lots of people to coincide in doing one thing when they might have done another, then there are quite a few ways of achieving that. 
        • Consider some other case studies of coordinated behavior: ...
           
• Founder's effects and techno-utopian bias:
  • The two key founders of the AI-Alignment community, Nick Bostrom and Eliezer Yudkowsky, each founded a techno-utopian organisation before (World Transhumanist Association and Singularity Institute, respectively).
    • Each with a bias toward:
      • there being an 'inevitable march of progress' of technology,
      • that more 'advanced' technology is good,
      • that technology can be controlled such to not have hazardous effects on human society and the wider ecosystem that we're all part of.
    • The people who selected in to join this community on average resonated more with these assumptions about technological "progress" and got incululated over time to hold these assumptions more strongly.
    • Frankly, we are acting exactly like the AGI researchers here – "progress" toward AGI is inevitable anyway, so that excuses me to focus all of my work on these interesting geeky technical puzzles of how to build (safe) AGI.
  • From Zoe Cremer and Luke Kemp on Democratising Risk:
    • “The historically dominant techno-utopian approach (henceforth the “TUA”) played an important role in establishing the field and drawing attention to the significance of studying human extinction. It is time to examine this approach with a critical eye. We do this to identify weaknesses, areas for further investigation and the need to also explore alternative approaches...
    • “We understand it to be primarily based on three main pillars of belief:
      • transhumanism, total utilitarianism and strong longtermism.
    • “More precisely:
      • (1) the belief that a maximally technologically developed future could contain (and is defined in terms of) enormous quantities of utilitarian intrinsic value, particularly due to more fulfilling posthuman modes of living
      • (2) the failure to fully realise or have capacity to realise this potential value would constitute an existential catastrophe
      • (3) we have an overwhelming moral obligation to ensure that such value is realised by avoiding an existential catastrophe, including through exceptional actions.
    • “[T]he TUA often appears to assume an exogenous threat model in which existential hazards naturally and apolitically arise from inevitable and near-autonomous technological progress. The TUA rarely examines the drivers of risk generation. Instead, key texts contend that regulating or stopping technological progress is either deeply difficult, undesirable, or outright impossible...
    • “Risk assessment has evolved dramatically in past decades. 
      • Scholars now commonly analyse systemic risk (the ability for a single disruption to cascade into systems failures), how risks can cascade across borders and sectors, and how failures in critical systems can synchronise and reinforce each other. 
      • This has led to new forms of complex risk assessment, particularly in climate science and disaster risk reduction. The Intergovernmental Panel on Climate Change (IPCC) sees risk as composed of vulnerabilities, hazards, and exposures, as well as response risks. 
      • Similarly, others have suggested that a complex risk assessment needs to consider four determinants of risk (hazard, vulnerability, exposure, response) as well as how risks link and cascade. Understanding the common drivers across each of these determinants is critical to mitigation efforts...
    • “Most existential risk texts take a simpler, hazard-centric approach.
      • They tend to focus on a few selected hazards: biologically engineered pandemics, Artificial General Intelligence (AGI), nuclear war, climate change, and asteroid strikes. 
      • As currently framed, TUA equates risk with hazard and ignores the wider literature on risk assessment in fields such as disaster risk reduction. 
    • “The choice to structure risk assessment this way has not been explained or defended.
    • “It may have been chosen due to an implicit techno-determinist threat-model: 
      • the TUA often appears to assume an exogenous threat model in which existential hazards naturally and apolitically arise from inevitable and near-autonomous technological progress.
      • The TUA rarely examines the drivers of risk generation. Instead, key texts contend that regulating or stopping technological progress is either deeply difficult, undesirable, or outright impossible...
    • “Bostrom proposed a “Technological Completion Conjecture”: 
      • if technological developments do not cease, then all important, basic technological capabilities will be obtained in the long run. Others offer a more sophisticated view, in which military-economic competition exerts a powerful selection pressure on technological development. This “military-economic adaptionism” constrains sociotechnical change to deterministic paths. Technologies that gift a strong strategic advantage will almost certainly be built. Many in the related Effective Altruism community disregard controlling technology on the grounds of a perceived lack of tractability.
    • “From a techno-utopian perspective, a failure to build these dangerous, powerful technologies is an existential risk. Bostrom, aware of the tension arising from recommending the (albeit careful) development of technologies, warns:
      • “We should not blame civilization or technology for imposing big existential risks. Because of the way we have defined existential risks, a failure to develop technological civilization would imply that we had fallen victims of an existential disaster. […] Without technology, our chances of avoiding existential risk would therefore be nil”.
    • “Whether it is technological determinism, the more nuanced military-economic adaptationism model, or concerns around tractability, the result is the same: regressing, relinquishing, or stopping the development of many technologies is often disregarded as a feasible option. The proposed alternative is “differential technological development”: speeding up and slowing down different technologies to ensure they occur in the safest order possible. Why this is more tractable or effective than bans, moratoriums and other measures has not been fully explained and defended.
  • From Timnit Gebru:
    • Twitter:
      • “Apparently they need to build "AGI," whatever that is, but its also the single thing that poses the biggest "existential risk" to humanity.
      • “Then why are you building it?
      • “Argument seems to be if I build it, it will be the good kind.
    • DAIR:
      • “AI needs to be brought back down to earth... 
      • “It has been elevated to a superhuman level that leads us to believe it is both inevitable and beyond our control. When AI research, development and deployment is rooted in people and communities from the start, we can get in front of these harms and create a future that values equity and humanity.

11 comments

Comments sorted by top scores.

  > people who believe false premises tend to take bad actions.

  Argument 3:.

 - 1; That AGI can very easily be hyped so that even smart people
 can be made to falsely/incorrectly believe that there "might be"
 _any_chance_at_all_ that AGI will "bring vastly positive changes".
   - ie, strongly motivated marketing will always be stronger than truth,
   especially when VC investors can be made to think (falsely)
   that they could maybe get 10000X return on investment.
   - that the nature of AGI, being unknown and largely artificial,
   futuristic, saturated with modernism and tech optimism,
   high geekery, and is also very highly funded, means that AGI
   capabilities development has arbitrary intelligent marketing support.

 - 2; People (and nearly all other animals) are mostly self-oriented.
    - that altruism is usually essentially social signalling,
    and is actually of very little value as benefit to anything
    other than maybe some temporary social prestige building.
    - ie, each possibly participating person will see the possibility
    that maybe they could ride "up to riches" on the research bandwagon,
    and/or on any major shift in the marketing dynamics;
    that in any change there will be winners and loosers,
    and they want a chance to be "on the winning side", since
    everyone has bio-builtin social/market game addiction tendencies
    (biases) and they think that they can use their high intelligence
    to gain some personal strategic advantage.

 - 3; People are *selectively* rational.
   - Ie, that we should not expect deviations from rational agent models,
   because our selective notion of rationality will likely match
   our *also* self-selected models of 'rational actors'.
   - as such, we can expect that there will be all sorts of
   seemingly rational "arguments" that suggest that individual
   selfish and self supporting action (favoring tech development)
   is maybe "mostly harmless", and that at least some of the risks
   are maybe over emphasized, and that "therefore" we should
   maybe shift our actions towards the more (manufactured) "consensus"
   that the "robustly good" action is "keep doing AGI capability
   development" and also "increase safety work" -- and to be assuming
   that anything else is either impossible or maybe "robustly bad",
   or that at the very least, that the things that seem obvious
   are probably not at all obvious, for complicated "rational reasons"
   that just happen to align with their motivated preferred view.

 - 4; thus the false belief that there "might be" some non-zero
 small chance that AGI can be "aligned" so as to bring about
 whatever positive changes (hype the huge return on investment!)
 is so strong/motivating that it dominates all other considerations.
   - as that selective motivated reasoning in the possibility that
   someone can be part of the winning team and make history is
   so strong that even the suggestion that the very notion that
   *any* AGI persistently existing is inherently contradictory
   with the notion of the continuing survival of life on this planet
   is completely rejected without any further examination.