LessWrong 2.0 Reader

Good point.  The reason AI risk is distinct is simply that it removes the need of those bureaucracies and corporations to keep some humans happy and healthy enough to actually run them.  This doesn't exactly put limits on how much they can disempower humans, but it does tend to provide at least some bargaining power for the humans involved.

Thanks for the detailed objection and the pointers.  I agree there's a chance that solving alignment with designers' intentions might be sufficient.  I think the objection is a good one that "if the AI was really aligned with one agent, it'd figure out a way to help them avoid multipolar traps".

My reply is that I'm worried that avoiding races-to-the-bottom will continue to be hard, especially since competition operates on so many levels.  I think the main question is what's: the tax for coordinating to avoid a multipolar trap?  If it's cheap we might be fine, if it's expensive then we might walk into a trap with eyes wide open.

As for human power grabs, maybe we should have included those in our descriptions.  But the slower things change, the less there's a distinction between "selfishly grab power" and "focus on growth so you don't get outcompeted".  E.g. Is starting a company or a political party a power grab?

As for reading the paper in detail, it's largely just making the case that a sustained period of technological unemployment, without breakthroughs in alignment and cooperation, would tend to make our civilization serve humans' interests more and more poorly over time in a way that'd be hard to resist.  I think arguing that things are likely to move faster would be a good objection to the plausibility of this scenario.  But we still think it's an important point that the misalignment of our civilization is possibly a second alignment problem that we'll have to solve.

This seems mostly right, but, seems plausible to me that the predatory/prey cycle was a necessary prerequisite to get us into a basin where intelligence-sexual-selection was a plausible outcome. 

This does seem different however https://solarfoods.com/ - they are competing with food not fuel which can't be done synthetically (well if at all). Also widely distributed capability like this helps make humanity more resilient e.g. against nuke winter, extreme climate change, space habitats

(I meant the more expansive definition. Plausible that me and Zac talked past each other because of that)

You might like my related essay A Path to Human Autonomy [LW · GW]

FWIW I think of "OpenAI leadership being untrustworthy" (a significant factor in me leaving) as different from "OpenAI having bad safety policies" (not a significant factor in me leaving). Not sure if it matters, I expect that Scott was using "safety policies" more expansively than I do. But just for the sake of clarity:

I am generally pretty sympathetic to the idea that it's really hard to know what safety policies to put in place right now. Many policies pushed by safety people (including me, in the past) have been mostly kayfabe (e.g. being valuable as costly signals, not on the object level). There are a few object-level safety policies that I really wish OpenAI would do right now (most clearly, implementing better security measures) but I didn't leave because of that (if I had, I would have tried harder to check before I left what security measures OpenAI did have, made specific objections internally about them before I left, etc).

This may just be a semantic disagreement, it seems very reasonable to define "don't make employees sign non-disparagements" as a safety policy. But in my mind at least stuff like that is more of a lab governance policy (or maybe a meta-level safety policy).

Yeah, but I'm contrasting this with (IMO more likely) futures where everyone dies, and nothing that's remotely like a human copy goes on. Even if you conceptualize it as "these people died", I think there are much worse possibilities for what sort of entity continues into the future. (i.e. an AI with no human/social/creative/emotional values, that just tiles the universe with simple struggles [? · GW])

I think the most likely outcome is nonsentient AI killing everyone with no morality without anyone having a choice in the matter. (i.e. I'm not contrasting this with "nobody ever builds AI at all" or "we build it but without doing any of the mental-process replacement/augmentation going on in this story", because those seem less likely than the "everyone dies" or "this story happens, but with even less agency and more blatantly dystopian outcomes.")

[of course, the reason I described this as "optimistic" instead of "less pessimistic than I hope for" is that I don't think the characters died, I think if you slowly augment yourself with AI tools, the pattern of you counts as "you" even as it starts to be instantiated in silicon, so I think this is just a pretty good outcome. I also think the world (implies) many people thinking about moral / personhood philosophy before taking the final plunge. I don't think there's anything even plausibly wrong with the first couple chunks, and I think the second half contains a lot of qualifiers (such as integrating his multiple memories into a central node) that make it pretty unobjectionable.

I realize you don't believe that, and, seems fine for you to see it as horror. It's been awhile since I discussed the "does a copy of you count as you" and I might be up for discussing that if you want to argue about it, but also seems fine to leave as-is]

Ah yes, Rational Animations did a great video of that story. That did make superintelligence more graspable, but you know I had watched it and forgotten about it. I think it showed how our human civilization is vulnerable to other intelligences (aliens), but didn't still made the superintelligence concept one that that easy to grok. 

Well, we (humans) categorize our epistemic state largely in propositional terms, e.g. in beliefs and suppositions.

I'm not too confident of this. It seems to me that a lot of human cognition isn't particularly propositional, even if nearly all of it could in principle be translated into that language. For example, I think a lot of cognition is sensory awareness, or imagery, or internal dialogue. We could contort most of that into propositions and propositional attitudes (eg 'I am experiencing a sensation of pain in my big toe', 'I am imagining a picnic table'), but that doesn't particularly seem like the natural lens to view those through.

That said, I do agree that propositions and propositional attitudes would be a more useful language to interpret LLMs through than eg activation vectors of float values.