LessWrong 2.0 Reader

Please, for the love of god, do not keep using a term that people will predictably misread as implying longer timelines. I expect this to have real-world consequences. If someone wants to operationalize a bet about it having significant real-world consequences I would bet money on it.

I would be willing to take the other side of this bet depending on the operationalization.  Certainly I would take the "no" side of the bet for "will Joe Biden (or the next president) use the words 'slow takeoff' in a speech to describe the current trajectory of AI"

I don't think CDT is very common among humans as the natural responses to threats by humans will easily show you (and this has been discussed extensively on LW). 

I think any kind of "fooling" of nascent superintelligences will then probably involve retaliation from adult superintelligences, and this generally means you fucked up. As such "fooling" nascent superintelligences I think is not a good plan (and also, fooling nascent superintelligences is generally really hard and a very bad basis for arguments about what to do).

You said "shouldn't just do what's clearly in his best interests", I was responding to that.

I guess the missile knows where it is

Well, actually the missile knows where it isn't.

I agree in theory, among optimal agents, but I maintain that you all seem weirdly overconfident that the first AI that takes over will be a UDT expected utility maximizer. I think there is a decent chance that it's something like CDT (very common among humans!), and yeah, maybe that's bad and would get mugged by outer gods in the long term, but quite plausibly our simulators is the first and most salient hypothesis it generates, so we get to have our concessions first, and it might not matter what other ill-advised deals the AI gets into later.

Separately, I'm not sure you can circumvent running the sims. I think in an acausal trade story too, it's possible that the way to get reliable predictions on the other's actions is to run sims. I agree that for an expected utility maximizer AI, it doesn't matter that we run million sims and pay ten planets each, instead of running one sim/prediction and paying 10 million planets. However, based on the example of humans, I'm not convinced the AI will be an expected utility maximizer, so it seems better to do the million times ten allocation, so the naive probability of being in a sim is also high.

Taking a second stab at naming the top reasons I expect this to fail (after Ryan pointed out that my first stab was based on a failure of reading comprehension on my part [LW(p) · GW(p)], thanks Ryan):

This proposal seems to me to have the form "the fragments of humanity that survive offer to spend a (larger) fraction of their universe on the AI's goals so long as the AI spends a (smaller) fraction of its universe on their goals, with the ratio in accordance to the degree of magical-reality-fluid-or-whatever that reality allots to each".

(Note that I think this is not at all "bamboozling" an AI; the parts of your proposal that are about bamboozling it seem to me to be either wrong or not doing any work. For instance, I think the fact that you're doing simulations doesn't do any work, and the count of simulations doesn't do any work, for reasons I discuss in my original comment [LW(p) · GW(p)]; I think those parts were argued correctly despite misunderstanding an aspect of your proposal.)

The basic question here is whether the surviving branches of humanity have enough resources to make this deal worth the AI's while.

You touch upon some of these counterarguments in your post -- it seems to me after skimming a bit more, noting that I may still be making reading comprehension failures -- albeit not terribly compellingly, so I'll reiterate a few of them.

The basic obstacles are

• the branches where the same humans survive are probably quite narrow (conditional on them being the sort to flub the alignment challenge). I can't tell whether you agree with this point or not, in your response to point 1 in the "Nate's arguments" section; it seems to me like you either misunderstood what the $2^{-75}$ was doing there or you asserted "I think that alignment will be so close a call that it could go either way according to the minute positioning of ~75 atoms at the last minute", without further argument (seems wacky to me).

• the branches where other humans survive (e.g. a branch that split off a couple generations ago and got particularly lucky with its individuals) have loads and loads of "lost populations" to worry about and don't have a ton of change to spare for us in particular

• there are competing offers we have to beat (e.g., there are other AIs in other doomed Everett branches that are like "I happen to be willing to turn my last two planets into paperclips if you'll turn your last one planet into staples (and my branch is thicker than that one human branch who wants you to save them-in-particular)".

(Note that, contra your "too many simulators" point, the other offers are probably not mostly coming from simulators.)

Once those factors are taken into account, I suspect that, if surviving-branches are able to pay the costs at all, the costs look a lot like paying almost all their resources, and I suspect that those costs aren't worth paying at the given exchange rates.

All that said, I'm fine with stripping out discussion of "bamboozling" and of "simulation" and just flat-out asking: will the surviving branches of humanity (near or distant), or other kind civilizations throughout the multiverse, have enough resources on offer to pay for a human reserve here?

On that topic, I'm skeptical that those trades form a bigger contribution to our anticipations than aliens being sold copies of our brainstates. Even insofar as the distant trade-partners win out over the local ones, my top guess is that the people who win the bid for us are less like our surviving Everett-twins and more like some coalition of kind distant trade partners.

Regardless, I acknowledge that lots of the numbers we're slinging around are uncertain.

I separately think that "the AIs will kill us all (with the caveat that perhaps there's exotic scenarios where aliens pay for our brain-states, and hopefully mostly do nice things with them)" is a fair summary of the situation at hand, and that summarizing "perhaps some kind distant creatures will pay for our souls" as "we can in fact bamboozle an AI into sparing our life" is an unfair summary of the same. We would not be doing any bamboozling. Some kind creatures, upon who we would be a burden, would be paying for us to have some nice things, at commensurate cost to themselves. (My guess is that those kind creatures would mostly not be "us", but even if our Everett twins were the primary trade partners, it feels disingenuous to me to refer to this as any sort of bamboozlement.)

I agree that it is not as strong evidence as if we had access to original CoT, but I think that having deviations in CoT is more likely than summarizer fumbling that hard.

I meant ‘do you think it’s good, bad, or neutral that people use the phrase ‘slow’/‘fast’ takeoff? And, if bad, what do you wish people did instead in those sentences?

I think there is a much simpler argument that would arrive at the same conclusion, but also, I think that much simpler argument kind of shows why I feel frustrated with this critique:

Humanity will not go extinct because we are in a simulation. This is because we really don't like dying, and so I am making sure that after we build aligned AI, I spend a lot of resources making simulations of early-earth to make sure you all have the experience of being alive. This means it's totally invalid to claim that "AI will kill you all". It is the case that AI will kill you in a very small fraction of worlds, which are the small fraction of observer moments of yours located in actual base reality, but because we will spend like 1/100 millionth of our resources simulating early earths surviving, you can basically be guaranteed to survive as well. 

And like... OK, yeah, you can spend your multiverse-fractions this way. Indeed, you could actually win absolutely any argument ever this way: 

I am really frustrated with people saying that takeoff will be fast. Indeed, if we solve AI Alignment I will spend my fraction of the multiverse running early-earth simulations where takeoff was slow, and so no matter what happened in the base-universe, y'alls observer-moments. This means there is very little chance that we will all experience a fast takeoff, because I and others will have made so many early earth simulations that you are virtually guaranteed to experience takeoff as slow. 

I agree that "not dying in a base universe" is a more reasonable thing to care about than "proving people right that takeoff is slow" but I feel like both lines of argument that you bring up here are doing something where you take a perspective on the world that is very computationalist, unituitive and therefore takes you to extremely weird places, makes strong assumptions about what a post-singularity humanity will care about, and then uses that to try to defeat an argument in a weird and twisted way that maybe is technically correct, but I think unless you are really careful with every step, really does not actually communicate what is going on.

It is obviously extremely fucking bad for AI to disempower humanity. I think "literally everyone you know dies" is a much more accurate capture of that, and also a much more valid conclusion from conservative premises than "via multiverse simulation shenanigans maybe you specifically won't die, but like, you have to understand that we had to give up something equally costly, so it's as bad as you dying, but I don't want you to think of it as not-dying", which I am confident is not a reasonable thing to communicate to people. 

Like, yeah, multiverse simulation shenanigans make it hard for any specific statement about what AI will do to humanity to be true. In some sense they are an argument against any specific human-scale bad thing to happen, because if we do win, we could spend a substantial fraction of our resources with future AI systems to prevent that. But I think making that argument before getting people to understand that being in the position to have to do that is an enormous gigantic atrocity, and before you get people to understand the scale of these decisions, is really dumb. Especially if people frame it as "the AI will leave you alone". 

No, the AI will not leave you alone if we lose. The whole universe will be split at its seams and everything you know destroyed and remade and transformed into the most efficient version of itself for whatever goal the AI is pursuing, which yeah, might include trading with some other humanity's in other parts of the multiverse where we won, but you will still be split apart and transformed and completely disempowered (and we have no idea what that will actually look like, and we both know that "dying" is not really a meaningful abstraction in worlds where you can remake brains from scratch). 

I feel like a lot more direct genetic evidence has surfaced: 1, 2, 3, 4.

Those first 4 links, I think, are pretty unconvincing in isolation, but this one is fine.

[Disclaimer 1: I just linked things that I remembered off the top of my head.]

[Disclaimer 2: I think that the case for hereditarianism was quite overwhelming even 14 years ago, so you should consider me biased.]