LessWrong 2.0 Reader

Conjecture's Compendium is now up. It's intended to be a relatively-complete intro to AI risk for nontechnical people who have ~zero background in the subject. I basically endorse the whole thing, and I think it's probably the best first source to link e.g. policymakers to right now.

I might say more about it later, but for now just want to say that I think this should be the go-to source for new nontechnical people right now.

Dog: "Oh ho ho, I've played imaginary fetch before, don't you worry."

Here's an admittedly-uncharitable analogy that might still illuminate what I think is lacking from this picture:

Suppose someone in 1947 wrote a paper called "How we could solve nuclear reactor design," and it focused entirely on remotely detecting nuclear bombs, modeling piles of uranium to predict when they might be nuclear bombs, making modifications to piles of uranium to make them less like nuclear bombs, ultrasound scanning piles of uranium to see if they have bomb-like structures hidden inside, etc.

Then in a subsection, they touch on how to elicit electricity from uranium, and they say that many different designs could work, just boil water and drive a turbine or something, people will work out the details later if they don't blow themselves up with a nuclear bomb. And that the nuclear reactor design solution they mean isn't about building some maximum efficiency product, it's about getting any electricity at all without blowing yourself up.

This paper seems strange because all the stuff people have to figure out later is most of the nuclear reactor design. What's the fuel composition and geometry? What's the moderator material and design? How is radiation managed? Pressure contained? Etc.

Or to leave the analogy, it seems like building safe, useful, and super-clever AI has positive content; if you start from the inverse of an AI that does the bad takeover behavior, you still have most of the work left to go. How to initialize the AI's world model and procedure for picking actions? What to get human feedback on, and how to get that feedback? How to update or modify the AI in response to human feedback? Maybe even how to maintain transparency to other human stakeholders, among other social problems? These decisions are the meat and potatoes of building aligned AI, and they have important consequences even if you've ruled out bad takeover behavior.

If you make a nuclear reactor that's not a bomb, you can still get radiation poisoning, or contaminate the groundwater, or build a big pile of uranium that someone else can use to make a bomb, or just have little impact and get completely eclipsed by the next reactor project.

Sumilarly, an AI that doesn't take over can still take actuons optimized to achieve some goal humans wouldn't like, or have a flawed notion of what it is humans want from it (e.g. generalized sycophancy), or have negative social impacts, or have components that someone will use for more-dangerous AI, or just have low impact.

Your sources are not very clear about that, and it contradicts what I've heard elsewhere, but yes I do admit at the boundaries of where society enforces laws, there do exist people who are forced to do things including prostitution via dominance.

Maybe one could say the essence of our difference is this:

You see the dominance ranking as defined by the backing-off tendency and assume it to be mainly an evolutionary psychological artifact.

Meanwhile, I see the backing-off tendency as being the primary indicator of dominance, but the core interesting aspect of dominance to be the tendency to leverage credible threats, which of course causes but is not equivalent to the psychological tendency to back off.

Under my model, dominance would then be able to cause bargaining power (e.g. robbing someone by threatening to shoot them), but one could also use bargaining power to purchase dominance (e.g. spending money to purchase a gun).

This leaves dominance and bargaining power independent because on the one hand you have the weak-strong axis where both increase but on the other hand you have the merchant-king axis where they directly trade off.

Sure, that's a thing that can happen. I'm moderately confident the other thing is a relatively common thing to happen as well. 

Isn't the counterfactual trolley problem setup backwards? It should be counterfactual Omega giving you the better setup (not tying people to the tracks) if it predicts you'll take the locally "worse" option in the actual case, not the other way around, right?

Because with the current setup you just don't pull and Omega doesn't tie people to tracks.

A posthuman king is not centrally a king (not mortal, very different incentives), and "an AI" is a very vague bag-of-everything that might include things like simulated worlds or bureaucracies with checks and balances as special cases. The reason His Majesty's Democratic Government doesn't really work while the king retains ultimate authority is that the next king can be incompetent or malevolent, or its activities start threatening the king's position and so the king is motivated to restrict them. So even "giving keys to the universe back" is not necessarily that important in the case of a posthuman god-king, but it remains a possibility after the acute risk period passes and it's more clear how to make the next thing work.

LW supports polls? I'm not seeing it in https://www.lesswrong.com/tag/guide-to-the-lesswrong-editor, [? · GW] unless you mean embed a manifold market which.. would work, but adds an extra step between people and voting unless they're registered to manifold already.

Interesting, thanks! My guess is this doesn't include benefits like housing and travel costs? Some of these programs pay for those while others don't, which I think is a non-trivial difference (especially for the bay area)