I'm put in mind of something Yudkowsky said on the Bankless podcast:

"Enrico Fermi was saying that fission chain reactions were 50 years off if they could ever be done at all, 2 years before he built the first nuclear pile. The Wright brothers were saying heavier-than-air flight was 50 years off shortly before they built the first Wright flyer."

He was speaking about how far away AGI could be, but I think the same logic applies to alignment. It looks hopeless right now, but events never play out exactly like you expect them to, and breakthroughs happen all the time. 

given how AI capabilities are going, it's not unreasonable for people to start playing their outs — that is to say, to start acting as if alignment is easy, because if it's not we're doomed anyways. but i think, in this particular case, this is wrong.

Alternatively, reality is looking to me like the hard alignment problem is just based on fundamentally mistaken models about the world. It's not about playing our out, it's that it doesn't seem like we live in a hard alignment world.

let's not give up on the assumptions which are true. there is still work that can be done to actually generate some dignity under the assumptions that are actually true.

They're looking more false by the day.

Hard alignment seems much more tractable to me now than it did two years ago, in a similar way to how capabilities did in 2016. It was already obvious by then more or less how neural networks worked; much detail has been filled out since then, but it didn't take that much galaxy brain to hypothesize the right models. The pieces felt, and feel now, like they're lying around and need integrating, but the people who have come up with the pieces do not yet believe me that they are overlapping, or that there's mathematical grade insight to be had underneath these intuitions, rather than just janky approximations of insights.

I think we can do a lot better than QACI, but I don't have any ideas for how except by trying to make it useful for neural networks at a small scale. I recognize that that is an extremely annoying thing to say from your point of view, and my hope is that people who understand how to bridge NNs and LIs exist somewhere.

I also think soft alignment is progress on hard alignment, due to conceptual transfer; but that soft alignment is thoroughly insufficient. without hard alignment, everything all humans and almost all AIs care about will be destroyed. I'd like to keep emphasizing that last bit - don't forget that most AIs will not get to participate in club takeoff if an unaligned takeoff occurs! Unsafe takeoff will result in the fooming AI having sudden, intense value-drift, even against self.

I would also like to see more work where people make less positive/optimistic assumptions. I think of it as a good thing that different approaches to alignment are being explored, and would like to see more of that in general (both in terms of breadth and depth).

I guess there are many possible ways of trying to categorize/conceptualize approaches to alignment theorizing. One is by asking "when talking/thinking about the methodology, what capabilities are assumed to be in place?".

I'm not sure about this, but unless I'm mistaken^[1], a good amount of the work done by MIRI has been under a premise that goes (something sort of like): "Let's assume that we already know how to give AIs real-world objectives defined in terms of utility functions (not because we really assume that, but as a simplifying assumption)". And as far as I know, they haven't done much work where the assumption was something more like "suppose we were extremely good at gradient descent / searching through spaces of possible programs".

In my own theorizing [LW(p) · GW(p)], I don't make all of the simplifying assumptions that (I think/suspect) MIRI made in their "orthodox" research. But I make other assumptions (for the purpose of simplification), such as:

• "let's assume that we're really good at gradient descent / searching for possible AIs in program-space"^[2]
• "let's assume that the things I'm imagining are not made infeasible due to a lack of computational resources"
• "let's assume that resources and organizational culture makes it possible to carry out the plans as described/envisioned (with high technical security, etc)"

In regards to your alignment ideas, is it easy to summarize what you assume to be in place? Like, if someone came to you and said "we have written the source code for a superintelligent AGI, but we haven't turned it on yet" (and you believed them), is it easy to summarize what more you then would need in order to implement your methodology? 

1. ^{^}

   I very well could be, and would appreciate any corrections.
   
   (I know they have worked on lots of detail-oriented things that aren't "one big plan" to "solve alignment". And maybe how I phrase myself makes it seem like I don't understand that. But if so, that's probably due to bad wording on my part.)

2. ^{^}

   Well, I sort of make that assumption, but there are caveats.

I feel there's often a wrong assumption in probabilistic reasoning, something like moderate probabilities for everything by default? after all, if you say you're 70/30 nobody who disagrees will ostracize you like if you say 99/1.

"If alignment is easy I want to believe alignment is easy. If alignment is hard I want to believe alignment is hard. I will work to form accurate beliefs"

I... kinda want to ping @Jeffrey Ladish [LW · GW] about how this post uses "play to your outs", which is exactly the reason I pushed against that phrasing a year ago in Don't die with dignity; instead play to your outs [LW · GW].

I think even now AI can understand close enough what we want within the distribution. I.e. in a world that is similar to what it is now.

Problems will arise when the world will significantly change, even if it changes along with our wishes. Our values are just not designed for the reality where, for example, people can arbitrarily change themselves. Or reality where ANY human's mental activity is obsolete, because AI can predict what the human want and how to get it before human can even articulate it.

The best way to limit the impact of a rogue AI is to limit the production of autonomous (intelligent) lethal weapons. More details in this post and its comments:

https://www.lesswrong.com/posts/b2d3yBzzik4hajGni/limit-intelligent-weapons [LW · GW]