Do alignment concerns extend to powerful non-AI agents?

I believe that much of the difficulty in AI alignment comes from specific facts about how you might build an AI, and especially searching for policies that behave well empirically. Similarly, much of the hope comes from techniques that seem quite specific to AI.

I do think there are other contexts where alignment is a natural problem, especially the construction of institutions. But I'm not convinced that either the particular arguments for concern, or the specific technical approaches we are considering, transfer.

A more zoomed out "second species [AF · GW]" style argument for risk may apply roughly equally well a priori to human institutions as to AI. But quantitatively I think that institutions tend to be weak when their interests are in conflict with any shared interests of their stakeholders/constituents, and so this poses a much lower risk (I'd guess that this is also e.g. Richard's opinion). I think aligning institutions is a pretty interesting and important question, but that the potential upside/downside is quite different from AI alignment, and the quantitative differences are large enough to be qualitative even before we get into specific technical facts about AI.

My thoughts here is that we should look into the value of identity. I feel like even with godlike capabilities I will still thread very carefully around self-modification to preserve what I consider "myself" (that includes valuing humanity).
I even have some ideas on safety experiments on transformer-based agents to look into if and how they value their identity.

If you, the reader, or, say, Paul Christiano or Eliezer gets uploaded and obtains self-improvement, self-modification and processing speed/power capabilities, will your goals converge to damaging humanity as well? If not, what makes it different? How can we transfer this secret sauce to an AI agent?

The Orthogonality Thesis [? · GW] states that values and capabilities can vary independently. The key question then is whether my/Paul's/Eliezer's values are actually as aligned with humanity as they appear to be, or if instead we are already unaligned and would perform a Treacherous Turn [? · GW] once we had the power to get away with it. There are certainly people who are already obviously bad choices, and people who would perform the Treacherous Turn (possibly most people^[1]), but I believe there are people who are sufficiently aligned, so let's assume going forward we've picked one of those. At this point "If not, what makes it different?" answers itself: by assumption we've picked a person for whom the Value Loading Problem [LW · GW] is already solved. But we have no idea how to "transfer this secret sauce to an AI agent" - the secret sauce is hidden somewhere along this person's particular upbringing and more importantly their multi-billion year evolutionary history.

1. Superhuman agents these days are all built up out of humans talking to each other. That helps a lot for their alignability, in multiple ways. For an attempt to transfer this secret sauce to an AI agent, see Iterated Distillation and Amplification, which as I understand it works by basically making a really good human-imitator, then making a giant bureacracy of them, and then imitating that bureacracy & repeating the process.
2. The AIs we will soon build will be superhuman in new ways, ways that no current superhuman agent enjoys. (See e.g. Bostrom's breakdown of speed, quality, and collective intelligence -- current organizations are superhuman in "collective" but human-level in speed and quality)
3. To answer your question, no, I'd feel pretty good about Paul or Eliezer or me being uploaded. If it was a random human being instead of one of those two, I'd still think things would probably be OK though there'd be a still-too-large chance of catastrophe.

The misalignment problem is universal, extending far beyond AI research. We have to deal daily with misaligned artificial non-intelligent and non-artificial intelligent systems. Politicians, laws, credit score systems, and many other things around us are misaligned to some degree with the goals of the agents who created them or gave them the power. The AI Safety concern states that if an AGI system is disproportionately powerful, a tiny misalignment is enough to create unthinkable risks for humanity. It doesn't matter if the powerful but tiny-misaligned system is AI or not, but we believe that AGI systems are going to be extraordinarily powerful and are doomed to be tiny-misaligned.

You can do a different thought experiment. I'm sure that you, like any other standard agent, are slightly misaligned with the goals of the rest of humanity. Imagine you have a near-infinite power. How bad would that be for the rest of humanity? If you were somebody else in this world, a random person, would you still want to find yourself in the situation where now-you has near-infinite power? I certainly don't.