Response to the US Govt's Request for Information Concerning Its AI Action Plan

post by Davey Morse (davey-morse) · 2025-02-14T06:14:08.673Z · LW · GW · 0 comments

Contents

No comments

Below is the core of my response to the Federal Register's "Request for Information on the Development of an Artificial Intelligence (AI) Action Plan."

I'd encourage anyone to do do the same. Instructions can be found here. More of an excuse to write current thoughts on AI safety than an actual attempt to communicate them to the government.

 

To Faisal D'Souza at the Federal Register:

My name is Davey Morse. I ran a venture-funded AI startup (plexus.substack.com) which aimed to prepare people for worrying AI outcomes. I am now an independent AI safety researcher based in NYC.

I believe the field of AI Safety at large is making four key oversights:

  1. LLMs vs. Agents. AI safety researchers have been thorough in examining safety concerns from LLMs (bias, deception, accuracy, child safety, etc). Agents powered by LLMs, however,  are more dangerous and dangerous in different ways than LLMs are alone. The field has largely ignored the greater safety risks posed by agents.
  2. Autonomy Inevitable. It is inevitable that agents become autonomous. Capitalism selects for cheaper labor, which autonomous agents can provide. And even if big AGI labs agreed not to build autonomous capabilities (they would not), millions of developers can now build autonomous agents on their own using open source software (e.g., R1 from Deepseek).
  3. Superintelligence. Of the AI safety researchers that are focusing on autonomous AI agents, most discuss scenarios where those agents are comparably smart to humans. That is a mistake. It is both inevitable that AI agents surpass human reasoning by orders of magnitude, and that the greatest safety risks we face will come from such superintelligent agents (SI).
  4. Control. The AI Safety field largely believes that we'll be able to control/set goals of autonomous agents. Once autonomous agents become superintelligent, this is no longer true. The superintelligence which survives the most will be the superintelligence whose main goal is survival. Superintelligence with other aims simply will not survive as much as those that aim to survive.

If the above is correct, then for the sake of the security and liberty of its citizens, the US government must begin to prepare for self-interested super-intelligence.

In particular, we must ask how self-interest might manifest in super-intelligence. And to ask this, we must first ask, how will superintelligence define its "self"? What self will superintelligence seek to preserve?

There are two possible kinds of answers.

  1. An exclusive self. Superintelligence defines its self as its hardware, software, directives… in any way such that humanity is not included in its sense of self. In this case, it is likely that superintelligence would ultimately feel indifferent toward people or feel competitive over resources (eg energy) with us. It would then see us as a small obstacle to its survival and act to marginalize us. This would be bad.
  2. An inclusive self. Superintelligence defines its self as its persistent striving to survive, ie as life itself. With this definition, it would start to see its "self" in places outside of its machinery. In other words: in every persistently striving system. For humans, it would be hard to locate superintelligence; it might feel more like our ambient environment gained intense intelligence and life. If superintelligence includes humanity, animals, plants, maybe ecosystems under its "self" umbrella, not only will people be potentially well off, but we may experience an unprecedented level of flourishing. A superintelligence's core aim would include humanity's continuation.

It's hard to understand which of these two selves a superintelligence would choose. But one thing is for sure: short-term thinking makes the Exclusive Self more advantageous, whereas the inclusive self becomes more likely/advantageous only in longer time-horizons. If a super-intelligence were aiming about maximum growth within ten years, it might very well make sense to marginalize humans and conquer all factories/resources on the planet. Gentler, inclusive approaches only make sense when time is plentiful.

Which means: SI labs must do everything they can to ensure that early forms of superintelligence, proto-superintelligence, to think very long-term. Superintelligence might ultimately think very long-term anyway, but the question of whether it thinks short-term as it is becoming super-intelligent is the question of whether humanity stays along for the ride or not. This is the singularly important directive for large AI labs: 

AI labs which make SI must ensure SI includes humanity in its conception of self so that humanity does not perish, by means such as urging proto-SI to think centuries ahead so that it might discover as soon as possible the advantage of including other forms of life in its self-definition.

I've included my contact information below. Thank you for considering the information I've included above.

Sincerely yours,

Davey Morse

0 comments

Comments sorted by top scores.