LessWrong 2.0 Reader

Notably, David is proposing that AIs take different actions prior to making powerful sims: not kill all the humans.

I don't understand why the AI would need to know anything a-priory. In a classical acausal trade situation superintelligence are negotiating with other superintelligences, and they can spend as much time as they want figuring things out. 

IMO, Karl Marx's main errors come down to 2 errors:

1. Overestimating drastically the amount of zero-sum/win-lose interactions in the economy. To be a bit more fair, the notion that positive sum/win-win economics could exist as a supermajority of all trades was likely very undeveloped by Marx's time, and it only got started by at most 20-50 years in the west. This is a common error made across the political spectrum, but Karl Marx's assumption that there had to be conflict between the proletariat and the bourgeoisie probably reflects zero-sum assumptions on economics.

2. Way overestimating how much change/radicalism was necessary to distribute some of the profits of the new capitalism economy, closely related to the zero sum error, but also not realizing that the rich weren't omnipotent and couldn't just arbitrarily change society to fit their preferences.

Yeah, I feel confused that you are misunderstanding me this much, given that I feel like we talked about this a few times. 

Nate is saying that in as much as you are pessimistic about alignment, game theoretic arguments should not make you any more optimistic. It will not cause the AI to care more about you. There are no game theoretic arguments that will cause the AI to give humanity any fraction of the multiverse. We can trade with ourselves across the multiverse, probably with some tolls/taxes from AIs that will be in control of other parts of it, and can ultimately decide which fractions of it to control, but the game-theoretic arguments do not cause us to get any larger fraction of the multiverse. They provide no reason for an AI leaving humanity a few stars/galaxies/whatever. The arguments for why we are going to get good outcomes from AI have to come from somewhere else (like that we will successfully align the AI via some mechanism), they cannot come from game theory, because those arguments only work as force-multipliers, not as outcome changers.

Of course, in as much as you do think that we will solve alignment, then yeah, you might also be able to drag some doomed universes out with you (though it's unclear whether that would be what you want to do in those worlds, as discussed in other comments here). 

I really feel like my point here is not very difficult. The acausal trade arguments do not help you with AI Alignment. Honestly, at the point where you can make convincing simulations that fool nascent superintelligences, it also feels so weird to spend your time on saving doomed earths via acausal trade. Just simulate the earths directly if you really care about having more observer-moments in which earth survives. And like, I don't want to stop you from spending your universe-fraction this way in worlds where we survive, and so yeah, maybe this universe does end up surviving for that reason, but I feel like that's more because you are making a kind of bad decision, not because the game-theoretic arguments here were particularly important.

their force of course depends on the degree to which you think alignment is easy or hard.

I don't think that's true. Even if the alignment problem is hard enough that the AI can be ~100% sure humans would never solve it, reaching such conclusion would require gathering evidence. At the very least, it would require evidence of how intelligent humans are – in other words, it's not something the AI could possibly know a priori. And so passing the simulation would presumably require pre-commiting to spare humans before gathering such evidence.

(1) has AI safety slowed down?

i feel like there haven’t been any big innovations for a while (6-12 months).

(2) if so, when was AI safety’s peak rate of innovation?

(3) what’s caused the slow down?

possible explanations:

• more research is closed-source
• the current ideas are adequate
• ideas are becoming harder to find
• people feel less creative
• i’m simply mistaken
• paul christiano stopped posting
• something else

I think that we should indeed consider what the corvid-god wants at the same point in the future where we're considering building the simulations David describes in this post.

More directly: David isn't proposing we should do particularly different things now, he's just noting an argument that we might take actions later that affect whether unaligned AIs kill us.

Yes, this is a good point. We need a more granular model than a binary 'all the same laws will apply to high priority national defense projects as apply to tech companies' versus 'no laws at all will apply'.

Shout out to Karl Marx for correctly identifying many issues with what he called capitalism, but providing a compeltely inaccurate hypothesis as to the root cause of those issues.

What an undignified way to go.