LessWrong 2.0 Reader

very few dollars directed to it in ways that ensures … the leadership doesn't ruin it for personal or ideological profit from the inside.

Could you elaborate on how you think dollars could be directed to prevent this? As of this writing, Godot is the latest project to do this, but much the same has happened to your examples, Wikipedia and Linux. (And they're not unusual; this is typical of every major open source project.)

I agree that arguments of this type go through, but their force of course depends on the degree to which you think alignment is easy or hard. In past discussions of this I generally described this as "potential multiplier on our success via returns from trade, but does not change the utility-ordering of any worlds".

In general it's unclear to me how arguments of this type can ever really change what actions you want to take in the present, which is why I haven't considered it high priority to figure out the details of these kinds of trades (though it seems interesting and I am in favor of people thinking about it, I just don't think it's very close to top priority). 

The degree to which this strategy works is dependent on the fraction of worlds in which you do successfully align AI. In as much as the correct choice of action is determined by your long-term/causally-distant effects on the universe (which I am quite compelled by), you still want to maximize your control over the future, which you can then use as a bargaining chip in acausal negotiations with AI systems in other worlds where you don't have as much power. 

(Aside: It's also honestly not clear to me that I should prefer saving humanity's existence in actual universes vs. running a simulation or literally recreating earth as a reserve West World style in the universe in which I do have a superintelligence, which seems like it would be a lot easier, and as such I am not sure whether I want to spend my negotiation leverage on preventing humanity from going extinct in some universes). 

On the "this implies AI will not kill everyone" point, I think that part doesn't really go through. It's costly for AI to leave humans alive. It might do some quick brain scans and then revive humanity a few years later, but it would be quite surprising if it actually leaves alive presently humans, which would no longer be measured in "epsilon" costs, since colonizing the universe from a starting planet that you cannot convert into the ideal starting base seems like it would cost you some non-trivial fraction of the future.

I actually think that you are probably right, and in the last year I got more sympathetic to total utilitarianism because of coherence arguments like this. It's just that the more common-sense factions still hold way more than one in a hundred million seats in my moral parliament, so it still feels like an obviously good deal to give up on some planets in the future to satisfy our deep intuitions about wanting Earth society to survive in the normal way. I agree it's all confusing an probably incoherent, but I'm afraid every moral theory will end up somewhat incoherent in the end. (Like infinite ethics is rough.)

Interesting. Thanks. (If there's a citation for this, I'd probably include it in my discussions of evals best practices.)

Hopefully evals almost never trigger "spontaneous sandbagging"? Hacking and bio capabilities and so forth are generally more like carjacking than torture.

Well, my experience is that even when you seem to have bypassed a refusal, you might not have truly bypassed the model's "reluctance". If you get a refusal, but then get past it with a jailbreak or few-shot prompting, you usually get a weaker answer than the answer you get if you fine-tune. In other words, spontaneous sandbagging. I haven't experimented enough with steering vectors yet to be sure whether they are similar to fine-tuning in getting past spontaneous sandbagging. I would expect they are at least closer.

These spontaneous sandbagging phenomena of don't appear so strongly with 'ordinary' sorts of harms. Car jacking or making meth, that sort of thing. Only when you get into extreme stuff that very clearly goes against a wide set of deeply held societal norms (kidnapping and torturing people to death as lab rats to help develop biological weapons, explicit scientific plans to kill billions of innocent people, that sort of thing).

Low-quality resurrections are already proliferating by bad actors. Two examples are voice cloning by scammers and recommendation systems by social networks. Also AI generated revenge porn in South Korea. 

The main question is what level of similarity is enough for me to ensure personal identity. The bad variant here would be if only identity token is enough, that is, a short string of data that identifies me and includes my name, profession, location and a few kilobytes of some other properties. This is the list of things I remember in the morning when I am trying to recognize who am I. In that case producing low quality, but identically important copies will be easy.

I think low-quality resurrections by bad agents are almost inevitable – voice cloning by scammers is happening now.  But such low-quality resurrections will lack almost all my childhood memories and all fine details. But from pain-view (can I say it?) it will be almost like me, as in the moment of pain fine-grained childhood memories are not important. 

Friendly AIs may literally till light cones with my copies to reach measure domination, so even if they are 0.01 per cent of total AIs, they can still succeed (they may need to use some acausal trade between themselves to do it better as I described here [LW · GW]). 

To me this looks like circular reasoning: this example supports my conceptual framework because I interpret the example according to the conceptual framework.

Instead, I notice that Stockfish in particular has some salient characteristics that go against the predictions of the conceptual framework:

• It is indeed superhuman
• It is not the case that once Stockfish ends the game that's it. I can rewind Stockfish. I can even make one version of Stockfish play against another. I can make Stockfish play a chess variant. Stockfish doesn't annihilate my physical body when it defeats me
• It is extremely well aligned with my values. I mostly use it to analyze games I've played against other people my level
• If Stockfish wants to win the game and I want an orthogonal goal, like capturing its pawns, this is very feasible

Now, does this even matter for considering whether a superintelligence would trade, wouldn't trade? Not that much, it's a weak consideration. But insofar as it's a consideration, does it really convince someone who doesn't already but the frame? Not to me.

(This comment is tangential to the decision-theoretic focus of the post)

The AI stabilizes the situation in the world and makes sure no other dangerous AI is built, but otherwise it doesn't harm the humans.^[6] Then it modifies its own code to have a commitment never to harm the humans, and let them live freely on Earth for at least a billion years, only doing the minimal necessary interventions to prevent humanity from wiping itself out with some new stupid technology. Crucially, the AI should do this self-modification^[7] before it makes itself very substantially smarter or better-informed about the world, to the level that it can expect to determine whether it's in a simulation run by a very advanced future civilization.

I don't know of consistent human values which would ask for this specifically. Consider two cases^[1]:

1. You value something like continuation of {with a bunch of complex [? · GW] criteria}, not quantity of copies of, at least one 'earth society'.
   • In this case, it continues regardless some of the time, conditional on the universe being large or duplicitous enough to contain many copies of you / conditional on the premise in the post that at least some aligned ASIs will exist somewheres.
2. Instead, you linearly value a large number of copies of earth civilizations existing or something
   • then the commitment wouldn't be to let-continue just each one earth per unaligned ASI, but to create more, and not cap them at a billion years.

I think this is a case of humans having a deep intuition that there is only one instance of them, while also believing theory that implies otherwise, and not updating that 'deep intuition' while applying the theory even as it updates other beliefs (like the possibility for aligned ASIs from some earths to influence unaligned ones from other earths).

1. ^{^}

   (to be clear, of course these are not the only possible things one can value, please do not clamp your values just because the only things humans seem to write about caring about are constrained)