An argument against indirect normativity

I don't believe that Paul's approach to indirect normativity is on the right track. I also have no idea which of the possible problems you might be talking about. PM me. I'd put something a very high probability that it's either not a problem, or that I thought of it years ago.

Yes, blackmail can enable a remote attacker to root your AI if your AI was not designed with this in mind and does not have a no-blackmail equilibrium (which nobody knows how to describe yet). This is true for any AI that ends up with a logical decision theory, indirectly normative or otherwise, or also CDT AIs which encounter other agents which can make credible precommitments. I figured that out I don't even recall how long ago (remember: I'm the guy who first wrote down an equation for that issue; also I wouldn't be bothering with TDT at all if it wasn't relevant to some sort of existential risk). Didn't talk about it at the time for obvious reasons. The existence of N fiddly little issues like this, any one of which can instantly kill you with zero warning if you haven't reasoned through something moderately complex in advance and without any advance observations to hit you over the head, is why I engage in sardonic laughter whenever someone suggests that the likes of current AGI developers would be able to handle the problem at their current level of caring. Anyway, MIRI workshops are actively working on advancing our understanding of blackmail to the point where we can eventually derive a robust no-blackmail equilibrium, which is all that anyone can or should be doing AFAICT.

This is a reference for indirect normativity and its associated Less Wrong discussion.

For future reference, whenever anybody says "I've got a secret" I strongly desire to learn that secret, especially if it's prefaced with "ooh, and the secret is potentially dangerous" (because "dangerous" translates to "cool"), and I expect that I'm not alone in this.

Surely EY won't mind if you put a trigger warning and rot13 the argument. Or post it on the LW subreddit.

It's really rather confusing and painful to have a discussion without talking about what you are discussion, using only veiled words, indirect hints, and private messages. May I suggest that instead we move the discussion to the uncensored thread if we are going to discuss it at all?

I believe that the whole point of the uncensored thread is to provide a forum for the discussion of topics, such as basilisks, that Eliezer is uncomfortable with us discussing on the main site.

Is this relevant for my variant of "indirect normativity" (i.e. allowing human-designed WBEs, no conceptually confusing tricks in constructing the goal definition)?

(I'm generally skeptical about anything that depends on the concept of "blackmail" distinct from general bargaining, as it seems to be hard to formulate the distinction. It seems to be mostly the affect of being bargained against really unfairly, which might happen with a smart opponent if you are bad at bargaining and are vulnerable to accepting unfair deals, so the solution appears to be to figure out how to bargain well, and avoid bargaining against strong opponents until you do figure that out.)

I think I understand X, and it seems like a legitimate problem, but the comment I think you're referring to here seems to contain (nearly) all of X and not just half of it. So I'm confused and think I don't completely understand X.

Edit: I think I found the missing part of X. Ouch.

I thought of this while reading Paul's original post and assumed he knew about it. I guess I should update in the direction of saying obvious things, but I have trouble distinguishing things that are stated explicitly from inferences that I make quickly. (I also have the opposite problem where if something is obviously wrong and reading it doesn't make me update at all, I don't notice that the point was made and think that the writer is just asserting something.)

Should I make any other updates? Does anyone have experience with reducing the frequency of this sort of error?

If you understand Paul's idea and understand basilisks, figuring out X should take you about five minutes (there's only one obvious way to combine the two ideas), so you might as well do it now.

I agree that X (as well as some related non-basilisk issues) are weaknesses of the indirect normative approach.

In conclusion, I'd like to ask Eliezer to rethink his position on secrecy. If more LWers understood basilisks, somebody might have come up with X earlier.

Unless I am mistaken about what you mean by X they did come up with X earlier. They just couldn't speak of it here at risk of provoking Yudkowskian Hysterics.

If the NBC (No Blackmail Conjecture) is correct, then that shouldn't be a problem.

Hm, I don't see it. Unless you're referring to the case where the humans aren't smart enough to abide by something like UDT. Maybe you estimated it as too obvious :)

EDIT: okay, I see a more general version now. But I think it's speculative enough that it's actually not the biggest problem with Paul's particular proposal. So it's probably still not what you expected me to think of :D