I really wish there was an "agree/disagree" button for posts. I'd like to upvote this post (for epistemic virtue / presenting reasonable "contrarian" views and explaining why one holds them), but I also strongly disagree with the conclusions and suggested policies. (I ended up voting neither up nor down.)

EDIT: After reading Akash's comments, and re-reading the post more carefully: I largely agree with Akash (and updated towards thinking that my standards for "epistemic virtue" are/were too low).

How much have you looked into other sources of AI risk [LW · GW]?

In other words, I now believe a significant probability, on the order of 90-99.9%, that alignment is solved by default.

I am on the fence here, and I wonder what specifically pushed you toward this extremely strong update?

KataGo misjudging the safety of certain groups [LW · GW] seems like a pretty significant blow to the Natural Abstractions Hypothesis to me.

It seems that it takes more something to arrive at abstractions that align with intuitive human abstractions, given that even in this case where the human version is unambiguously correct relative to what the AI was actually trained to do, and even though it reached superhuman abilities at Go, it still did not arrive at the same concept.

I don't doubt that a more powerful AI will be able to arrive at the correct concept in the case, given the unambiguous correctness of this abstraction at this level. But we already knew that was the easy case, where we can put the relevant criteria directly in the loss function. In the case where it actually matters, it needs to learn how to be Good even though we don't know how to put that directly in the loss function. And it failing at the easy version of this problem seems like significant evidence that it won't actually converge to the same sort of intuitive concepts we use in the harder cases where humans don't even converge all that well.

Note that lesswrong is also about the philosophy of rationality, or making decisions for our human lives that have the highest probability of success.

While there are various crude hacks, the easiest way to know the highest probability action in your life is to ask an AI you can trust to analyze the situation and do the math.

For about 20 years such an AI was only theoretical with some effort to find what a possible algorithm might look like.

Lesswrong is also about surviving other risks, like plain old aging and being stuck on one planet - both things trustworthy AI makes solvable, and in reality, may have been the only feasible solution. Same with bypassing failed institutions.

Given your probabilities, I think you are under appreciating the magnitude of the downside risks.

I spent some time trying to dig into why we need to worry so much about downside risks of false positives [LW · GW] (thinking we're going to get aligned AI when we're not) and how deep the problem goes, but most of the relevant bits of argumentation I would make to convince you to worry are right at the top of the post.

Your first link appears to be broken. Did you meant to link here [? · GW]? It looks like the last letter of the address got truncated somehow. If so, I'm glad you found it valuable!

For what it's worth, although I think deceptive alignment is very unlikely, I still think work on making AI more robustly beneficial and less risky is a better bet than accelerating capabilities. For example, my posts don't address these [LW · GW] stories [LW · GW]. There are also a lot of other concerns about potential downsides of AI that may not be existential, but are still very important. 

Recalling the 3 subclaims of the Natural Abstraction Hypothesis which I will quote verbatim:

1. Abstractability: for most physical systems, the information relevant “far away” can be represented by a summary much lower-dimensional than the system itself.
2. Human-Compatibility: These summaries are the abstractions used by humans in day-to-day thought/language.
3. Convergence: a wide variety of cognitive architectures learn and use approximately-the-same summaries.

I will note that despite the ordering I think claim 2 is the weakest. I strongly disagree that these claims  being partially, or completely correct means we can expect an AI not to be deceptive.

•  The Natural Abstraction Hypothesis is not a statement about systems converging to valuing certain abstractions the same, merely that they will use similar summaries of data in their decision making processes. Two opposing chess players may use completely identical abstractions in how they view the board (material advantage, king safety etc) but they directly opposed goals.
• Extending that last point, we realise that some understanding of human abstractions is a powerful tool for effective deceptive alignment. If there is a behaviour that is selected for when humans are unaware of it (say, reward hacking) but is strongly selected against when humans are aware of it, it is possible the AI will learn "humans dislike this" but that doesn't mean that it will "dislike this"

In other words, I now believe a significant probability, on the order of 50-70%, that alignment is solved by default.

Let's suppose that you are entirely right about deceptive alignment being unlikely. (So we'll set aside things like "what specific arguments caused you to update?" and tricky questions about modest epistemology/outside views [? · GW]).

I don't see how "alignment is solved by default with 30-50% probability justifies claims like "capabilities progress is net positive" or "AI alignment should change purpose to something else."

If a doctor told me I had a disease that had a 50-70% chance to resolve on its own, otherwise it would kill me, I wouldn't go "oh okay, I should stop trying to fight the disease."

The stakes are also not symmetrical. Getting (aligned) AGI 1 year sooner is great, but it only leads to one extra year of flourishing. Getting unaligned AGI leads to a significant loss over the entire far-future. 

So even if we have a 50-70% chance of alignment by default, I don't see how your central conclusions follow.

I now think the probabilities of AI risk have steeply declined to only 0.1-10%, and all of that probability mass is plausibly reducible to ridiculously low numbers by going to the stars and speeding up technological progress.

I think this is wrong (in that how does speeding up reduce risk? What do you want to speed up?) . I'd be actually interested in the case for this I got promised in the title.

What kind of information would you look out for, that would make you change your mind about alignment-by-default?

What information would cause you to inverse again? What information would cause you to adjust 50% down? 25%?

I know that most probability mass is some measure of gutfeel, and I don't want to introduce nickel-and-diming, more get a feel for what information you're looking for.

Interesting take.

Perhaps there was something I misunderstood, but wouldn't AI alignment work and AI capabilities slowdown still have extreme positive expected value even if the probability of unaligned AI is only 0.1-10%?

Let's say the universe will exist for 15 billion more years until the big rip.

Let's say we could decrease the odds of unaligned AI by 1% by "waiting" 20 years longer before creating AGI, we would lose out 20 years of extreme utility, which is roughly 0.00000001% of the total time (approximation of utility).

 On net we gain 15 billion * 0.01 - 20 * 0.99 ≈ 150 million years of utility.

I do agree that if we start 20 years earlier, we could possibly also populate a little bit more of space, but that should be several orders of magnitudes smaller difference than 1%.

I'm genuinely curios to hear your thoughts on this. 

Your first link here is broken: https://www.lesswrong.com/s/pvoxjtCbkcweBLn7 [? · GW]

Did you mean to include this one? https://www.lesswrong.com/posts/RTkatYxJWvXR4Qbyd/deceptive-alignment-is-less-than-1-likely-by-default [LW · GW]

Thanks for amplifying the post [LW · GW] by that caused your large update. It's pretty fascinating. I haven't thought through it enough yet to know if I find it as compelling as you do.

Let me try to reproduce the argument of that post to see if I've got it:

If an agent already understands the world well (e.g., by extensive predictive training) before you start aligning it (e.g., with RLHF), then the alignment should be easier. The rewards you're giving are probably attaching to the representations of the world that you want them to, because you and the model share a very similar model of the world.

In addition, you really need long term goals for deceptive alignment to happen. Those aren't present in current models, and there's no obvious way for models to develop them if their training signals are local in time.

I agree with both of these points, and I think they're really important - if we make systems with both of those properties, I think they'll be safe.

I'm not sure that AGI will be trained such that it knows a lot before alignment starts. Nor am I sure that it won't have long term goals. I think it will. Let alone ASI

But I think that tool AI might well continue be trained that way. And that will give us a little longer to work on alignment. But we aren't likely to stop with tool AI, even if it is enough to transform the world for the better.

What it will shift to is important, but assuming that this rosy model of alignment is correct, then I'd argue a significant part of the field of AI Alignment should and can change purpose to something else.

Even if you're forecasting is correct, AI alignment is still so pivotal that even the difference between 1% and 0.1% matter. At most, you're post implies that alignment researchers should favor accelerating AI instead of decelerating it.