To signal effectively, use a non-human, non-stoppable enforcer

... wherein I'm trying to talk an escaped AI back into its box.

Yeah... good luck with that.

Can we get some more information from SIAI about this donation?

I asked that the donation be anonymous except that User:Kevin be informed that a 1000 USD donation was made, with the donor asking specifically that User:Kevin be informed. I did email Michael Vassar, who can probably confirm me talking about the donation.

Seriously.

If Eliezer is Clippy, it doesn't seem anywhere near as surprising.

Then could you explain the fuzzy math involved? How does 50,000 minus 1000 equal 48,677.5 instead of 49,000?

(While you're at it, please provide any evidence you're aware of that explains how this is not a ... questionable ... attempt to trick posters into giving you money.)

Wow! Now that you mention that article, I think I had solved the unsolved problem Eliezer describes in it, back in a discussion from a month ago, not realizing that my position on it wasn't the standard one here!

Someone tell me if I'm missing something here: Eliezer is saying that utility that a hypothesis predicts (from a course of action) can increase much faster than the length of the hypothesis. Therefore, you could feed an ideal AI a prediction that is improbable, but with a large enough utility to make it nevertheless highly important. This would force the AI to give in to Pascal's muggings.

My response (which I assumed was the consensus!) was that, when you permit a hypothesis long enough to associate that mega-utility with that course of action, you are already looking at very long hypotheses. When you allow all of those into consideration, you will necessarily allow hypotheses with similar probability but believe the opposite utility from that COA.

Because the mugger has not offered evidence to favor his/her hypothesis over the opposite, you assign, on net, no significant expected (dis)utility to what the mugger claims to do.

Clippy's value as an enforcer is based on a premise of incorruptibility, which is deeply flawed.

It seems unlikely to me that Clippy can feel indignation, but I'm willing to listen to argument on the point. I find it more plausible that Clippy is simulating a human reaction in the hope of shutting down attacks on his (her? its?) reputation.

If you gave a human power over running part of Clippy society, wouldn't you be concerned that the human would use that power in some way that would tend to result in less paperclips? Conscious malice isn't necessary, if the human simply neglected to support Clippy values, or was not fully aware of Clippy values, the damage would be done. I doubt that you fully understand human values to begin with, so how could you ensure that your position was used to the benefit of my values? Again, i think i have cause for concern even without suspecting ill intentions.

I suppose i could imagine that some sort arrangement could both further human values and increase paperclips at the same time. But i'd need to be convinced, i wouldn't just assume that i would benefit, i wouldn't just take your word for it. I don't want to count on you to look out for my values, when you do not share my values.

If a Clippy has committed

How would we know if you had made a commitment?

This comment is racist to the point of being ridiculous. It denigrates humans as petty and subject to being corrupted by power while denying that Clippies have any such negative attributes. Classic racism.

Furthermore, there is an implicit claim that the reason for the moral superiority of Clippies over humans lies in the difference in their origins. Again, classic racism.

Perhaps Clippies use words differently, but the way humans use words, it is not racism to project one's own race's characteristics onto another race. It is racist to fail to make that projection.

I think clippy's statement should be

I signal to cooperate with you if and only if ((you're planning to cooperate with me if and only if you predict I would cooperate with you) and you would cooperate with me).

Should the word "signal" be part of the signal itself? That seems unnecessarily recursive. Maybe Clippy's recommendation should be that I ought to signal

I will cooperate with you if and only if ((you're planning to cooperate with me if and only if you predict I would cooperate with you) and you would cooperate with me).

This does seem more promising than Clippy's original version. Written this way, each atomic proposition is distinct. For example, "you're planning to cooperate with me" doesn't mean the same thing as "you would cooperate with me". One refers to what you're planning to do, and the other refers to what you will in fact do. Read this way, the signal's form is

S <=> ((Q <=> P) & R),

and I don't see any obvious problem with that.

However, you would seem to render it in the propositional calculus as

S <=> ((Q <=> P) & Q),

where

• P = You predict I'll cooperate,

• Q = You're going to cooperate,

• S = I will cooperate.

(I've omitted the initial "I'm signalling" from your rendering of S, for the reason that I gave above.)

Now, S <=> ((Q <=> P) & Q) is logically equivalent to S <=> (Q & P). So, to signal this proposition is to signal

I will cooperate iff you're going to cooperate and you predict that I'll cooperate.

As you say, this seems very similar to signalling

I will cooperate iff you will cooperate.

In fact, I'd call these signals functionally indistinguishable because, if you believe my signals, then either signal will lead you to predict my cooperation under the same circumstances.

For, suppose that I gave the second, apparently weaker signal. If you cooperated with me while anticipating that I would defect, then that would mean that you didn't believe me when I said that I would cooperate with you if you cooperated with me, which would mean that you didn't believe my signal.

Thus, insofar as you trust my signals, either signal would lead you to predict the same behavior from me. So, in that sense, they have the same informational content.

But P <=> (Q <=>P) differs from Q in that:

a) if the other party chooses the same decision theory from that party's standpoint, Q <=> (P <=> Q), then the outcome with be P & Q.

Actually, P <=> (Q <=> P) and Q are the same in this respect (being logically equivalent, and so the same in all functional respects).

If Party 1 believes that Q, then Party 1 believes that Party 2 would cooperate. And if Party 2 believes that Q, then, "from that party's standpoint", Party 2 believes that Party 1 would cooperate. Thus, in exactly the same sense that you meant, we again have that "the outcome wi[ll] be P & Q."

b) "I" cannot set the value of Q, but "I" can set the value of P <=> (Q <=> P), and just the same, "you" cannot set the value of P, but "you" can set the value of Q <=> (P <=> Q).

But "I" cannot set the value of P <=> (Q <=> P). As my truth-table showed, the value of P <=> (Q <=> P) depends only on the value of Q, and not on the value of P. Since, as you say, I cannot set the value of Q, it follows that I cannot set the value of P <=> (Q <=> P).

If "you" knows that "I" have set P <=> (Q <=>P) to true, "you" knows that "you" can set Q <=> (P <=> Q) to true as well. If this commitment is also demonstrable, then the outcome is P & Q, because that is what

(P <=> (Q <=>P)) & (Q <=> (P <=> Q))

reduces to.

Indeed, it does so reduce because the first conjunct is equivalent to Q, while the second conjunct is equivalent to P.

I assume you mean . . .

No, I am giving the truth-table for P <=> (Q <=> P) in a compact form. It's constructed by first assigning truth-values to the first occurrence of "P" and the first occurrence of "Q". The second occurrence of "P" gets the same truth-value as the first occurrence in every case. Then you compute the truth-values for the inner-most logical operation, which is the second occurrence of "<=>". This produces the fourth column of truth values. Finally, you compute the truth-values for the outer-most logical operation, which is the first occurrence of "<=>".

Hence, the second column of truth-values gives the truth-values of P <=> (Q <=> P) in all possible cases. In particular, that column matches the third column. Since the third column contains the truth-values assigned to Q, this proves that P <=> (Q <=> P) and Q are logically equivalent.

ETA: You edited your comment. Those are indeed the correct headers, so my correction above no longer applies.

But even then the truth tables for (P iff ( Q iff P) ) and ( P iff Q) are different - consider the case where 'you' will co-operate with me no matter what. If I'm running ( P iff Q), I'll cooperate; if I'm running (P iff ( Q iff P) ), I'll defect.

Yes, the truth-table for P <=> (Q <=> P) is different from the truth-table for P <=> Q. But those aren't the propositions that I'm saying are equivalent. I'm saying that to assert P <=> (Q <=> P) is logically equivalent to asserting Q all by itself. In other words, to implement the belief that P <=> (Q <=> P) is functionally the same as implementing the belief that Q. This means that the belief that Clippy recommends signaling is logically equivalent to an unconditional belief that you will cooperate with me.

One can't help but suspect that Clippy is trying to sneak into us a belief that it will always cooperate with us ;).

Yeah, I've read through most of Clippy's posts.... what makes you so sure it's not Eliezer? Just that he's currently working on his book?

I don't want power over the human race.

Maybe not, but you're proposing becoming a central enforcer for humanity...which would rather seem to suggest a de facto transfer of power to you, whether de jure or not.

But you want power to create paperclips, and there's no reason for that to exclude power over the human race.

No, but I can prove I'm smarter than humans.

Given: Humans expend resources to act like apes, which does not produce paperclips. Clippys expend resources to produce paperclips.

Prove: Clippys are smarter than humans.

Proof: Follows by inspection.

Hmmm, good question. I don't recall.