Sam Clarke's Shortform

I sometimes want to point people towards a very short, clear summary of What failure looks like [AF · GW], which doesn't seem to exist, so here's my attempt.

• Many agentic AI systems gradually increase in intelligence and generality, and are deployed increasingly widely across society to do important tasks (e.g., law enforcement, running companies, manufacturing and logistics).
• Initially, this world looks great from a human perspective, and most people are much richer than they are today.
• But things then go badly in one of two ways (or more likely, a combination of both).
• [Part 1] Going out with a whimper
  • In the training process, we used easily-measurable proxy goals as objective functions, that don’t push the AI systems to do what we actually want e.g.
    • 'maximise positive feedback from your operator' instead of 'try to help your operator get what they actually want'
    • 'reduce reported crimes' instead of 'actually prevent crime'
    • 'increase reported life satisfaction' instead of 'actually help humans live good lives'
    • 'increasing human wealth on paper' instead of 'increasing effective human control over resources'
  • (We did this because ML needs lots of data/feedback to train systems, and you can collect much more data/feedback on easily-measurable objectives.)
  • Due to competitive pressures, systems continue being deployed despite some people pointing out this is a bad idea.
  • The goals of AI systems gradually gain more influence over the future relative to human goals.
  • Eventually, the proxies for which the AI systems are optimising come apart from the goals we truly care about, but by then humanity won’t be able to take back influence, and we’ll have permanently lost some of our ability to steer our trajectory. In the end, we will either go extinct or be mostly disempowered.
  • (In some sense, this isn’t really a big departure from what is already happening today - just imagine replacing today’s powerful corporations and states with machines pursuing similar objectives).
• [Part 2] Going out with a bang
  • These AI systems end up learning objectives that are unrelated to the objective functions used in the training process, because the objective they ended up learning was more naturally discovered during the training process (e.g. "don't get shut down").
  • The systems seek influence as an instrumental subgoal (since with more influence, a system is more likely to be able to e.g. prevent attempts to shut it down).
  • Early in training, the best way to do that is by being obedient (since systems understand that unobedient behaviour would get them shut down).
  • Then, once the systems become sufficiently capable, they attempt to acquire resources and influence to more effectively achieve their goals, including by eliminating the influence of humans. In the end, humans will most likely go extinct, because the systems have no incentive to preserve our survival.