On self-deception

OP argued that self-deception occurs even if your brain remains unbroken. I would characterize "not breaking my brain" as allowing my prior belief about the book's biasedness to make a difference in my posterior confidence of the book's thesis. In that case the book might be arbitrarily convincing; but I might start with an arbitrarily high confidence that the book is biased, and then it boils down to an ordinary Bayesian tug o' war, and Yvain's comment applies.

On the other hand, I'd view a brain-breaking book as a "press X to self-modify to devout Y-believer" button. If I know the book is such, I decide not to read it. If I'm ignorant of the book's nature, and I read it, then I'm screwed.

True. So in the process of deceiving yourself, you must first become irrational. The problem is then protesting that you are "still a reasonable person."

You seem to be arguing that, since we have strong evidence that the book has strong evidence for Zoroastrianism before we read it, it follows that we already have (the most important part of) our evidence for Zoroastrianism.

Not exactly. I do think this would be a true statement, if the book was a genuine book on Zoroastrianism and not a book which we know was designed to deceive us. But as far as I know it's only tangentially connected to the argument I'm making.

You may want to look at Brandon Fitelson's short paper Evidence of evidence is not (necessarily) evidence.

Thanks for summarizing the paper; I tried to read it but it was written in a way that seemed designed to be as obscure as possible. Your explanation makes more sense.

But I still don't see the problem. Learning a card is black increases the chance it's the ace of spades or clubs, but decreases the chance it's the ace of hearts or diamonds. The chance that it's the ace of spades becomes greater, but the net chance that it's an ace remains exactly the same. Evidence of evidence is still evidence, but evidence of evidence plus evidence of evidence that goes the opposite direction cancel out and make zero evidence.

Again, I'm not sure about the relevance here. It's not the case that, merely by knowing the book exists without reading it, we have new evidence for the existence of some evidence which both supports and, in a different way, opposes Zoroastrianism.

Thank you, that's what I was trying to get at, but didn't know how.

O.K., here's a disanalogy that may be important. In the card case, learning that C is the ace of spades should drastically lower our credence of the card being another x of spades. On the other hand, after reading the Zoroastrianism book, we shouldn't significantly doubt that the other books contain strong evidence, as well, given the known capabilities of the AI.

This isn't a very formal treatment, though.

"Evidence of evidence is more likely to be filtered evidence" is a more accurate phrasing.

The link keeps omitting the colon in the "http://." I don't know why it's doing that.

[Evidence of evidence is not (necessarily) evidence](http://fitelson.org/eee.pdf)

If I read his book, should I take it into account and discount his evidence because of this meta information? Or should I evaluate the evidence?

You should take the meta-information into account, because what you're getting is filtered evidence. See What Evidence Filtered Evidence. If the book only contained very weak arguments, this would suggest that no strong arguments could be found, and would therefore be evidence against what the book was arguing for.

Or should I evaluate the evidence?

The evidence that this was the best book he could give you is evidence.

It's the very fallacy Eliezer argues against where people know about clever arguers and use this fact against everyone else.

Which may very well be an adaptive fallacy that keeps you harder to manipulate by smarter people. It is possible that in the ancestral environment:

Cost of smart people manipulating you > Cost of being somewhat wrong in a hard to spot way

There is some degree to which you should expect to be swayed by empty arguments, and yes, you should subtract that out if you anticipate it. But if the book is a lot more compelling than that, then the book is probably above average both in arguing skill and in actual evidence. You cannot discount it solely as empty anymore, but neither should you assume that all of the "excess" convincing came from evidence - the book could just be unusually well written. You have to balance the improbabilities of evidence vs. writing, and update on the evidence found in that way.

Usually, the uncertainty grows with the size of the thing you're trying to measure. This means that when thinking about super-duper-well-written books, the uncertainty in the writing skill gets really big. And so when balancing the improbabilities of evidence vs. writing, the evidence barely has to do any balancing at all - the writing skill just washes it out.

If the amount of evidence presented is the same, it's better to hear about the truth from a child than from an orator, because the child doesn't have all those orating skills mucking up your signal-to-noise.

Consider the case where a friend says he saw a UFO. There are two possibilities: either the friend is lying/insane/gullible, or UFOs are real (there are probably some other possibilities, but for the sake of argument let's focus on these).

Your friend's statement can have different effects depending on what you already believe. If either probability is already at ~100%, you have no more work to do. IE, if you're already sure your friend is a liar, you dismiss this as yet another lie and don't start believing in UFOs; if you're already sure UFOs exist, you dismiss this as yet another UFO and don't start doubting your friend.

If you're not ~100% sure of either statement, then your observation will increase both the probability that your friend is a liar, and that aliens exist, but in different amounts. If you think your friend usually tells the truth, but you're not sure, it will increase your probability of UFOs quite a bit (your friend wouldn't lie to you!) but as long as you're not going to be sure of UFOs, you also have to leave some room for the case where UFOs aren't real, in which case the statement increases your probability that your friend is a liar.

When you hear a great argument for P, your pre-existing beliefs determine what you do in the same way as in the UFO example. It could mean that your interlocutor is a rhetorical genius so brilliant they can think up great arguments even for false positions. Or it could mean P is true. In real life, the probability of the interlocutor being such a rhetorical genius is always less than ~100%, meaning that it has to increase your probability of P at least a little.

In your example, we already know that the AI is a rhetorical genius who can create an arbitrarily good argument for anything. That totally explains away the brilliant arguments, leaving nothing left to be explained by Zoroastrianism actually being true. It's like when your friend who is a known insane liar says he saw a UFO: the insane liar part already explains away the evidence, so even though you're hearing words that sound like evidence, no probabilities are actually being shifted.

Does the chapter really count as evidence? Normally, X is evidence for Z is P(X|Z) > P(X|not Z). In this case, X = "there are compelling arguments for Z" and you already suppose that X is true whether or not Z. X is therefore not evidence for Z. Of course, after reading the chapter you learn the particular compelling arguments A(1), A(2), ... But those arguments support Z only through X and since you know X is not evidence, A(n) are screened off. Put another way, you know that for each A(n) there is an equally compelling argument B(n) that cancels it out. Knowing what the argument actually says is important only if you want to independently determine its compellingness. But you have assumed that you already know this.

Consider a more realistic scenario: you have a coin and two hypotheses:

• F: the coin is fair
• H: the coin is biased towards heads and it comes up heads twice as frequently as tails

Now you tell your servant: "Toss the coin million times and write the results down. Then, from the record select a subsequence S which has P(S|H) = 1.58 P(S|F) and tell me." The servant follows your instruction and says "HHTH". Now you have learned something new (the servant could for example tell "THHH" instead), but you don't update your odds in favour of H by factor 1.58, because it was almost certain in advance that the servant would be able to locate a subsequence of desired property whether or not H holds.

I think I have the brainware and knowledge to do it, but not as well or reliably as Eliezer, and I'd need to practice both social skills in general for a few years, and this particular task for 10 to 100 attempts before getting good at it. Does that count?

Part of Eliezer's thesis was that converting to a religion doesn't qualify if you would still correctly anticipate experimental results that you'd need to explain away.

As an atheist, your prior was low that the Christianity book would convince you, but as a Zoroastrian, your prior is now high that the Christianity book would convince you? I'm saying that you seem to have changed your opinion about the books.

I can add this to initial conditions: AI must pick a valid argument for any proposition it argues

(Can't find SMBC Delphic comic. Looking.)

in general minimize the number of logical errors / tricks used.

It's arguing for false propositions. You can specify a "Sudden volcanic eruption sufficient to destroy every Island in Indonesia that minimizes harm to humans", but don't be surprised if a few people are inconvenienced by it, considering what the minimum requirements to meet the first conditions are.

Actually, I take back the first part. I have some prior P(zoroastrianism is true). The fact that the P(book about zoroastrianism uses spurious reasoning designed to convinced me it's true and it actually is true) is lower, is irrelevant. I don't care if it's true, I only care about Zoroastrianism itself. Besides, with the initial condition on reasoning used, this second probability is also bounded because if Zoroastrianism is true, the book is in fact perfectly valid. So P(The book is lying) = P(Zoroastrianism is false).

No he expects that if he reads the book, his posterior belief in the proposition is likely going to be high. But his current prior belief in the truth of the proposition is low.

Also, as I made clear in my update, AI is not perfect, merely very good. I only need it to be good enough for the whole episode to go through, i.e. that you don't argue that a rational person will never believe in Z after reading the book and my story is implausible.

So in other words, the person is expecting to be persuaded by something other than the truth. Perhaps on the basis that the last N times he read one of these books, it changed his mind.

In that case, it is no different than if the person were stepping into a brain modification booth, and having his mind altered directly. Because a rational person would simply not be conned by this process. He would see that he currently believes in the existence of the flying spaghetti monster, and that he just read a book on the flying spaghetti monster prepared by a superintelligent AI which he had asked to prepare for him ultra-persuasive but entirely biased collections of evidence, and remember that he didn't formerly believe in the flying spaghetti monster. He would conclude on this basis that his belief probably has no basis in reality, i.e. is inaccurate, and stop believing (with such high probability) in it.

If we are to accept that the AI is good enough to prevent this happening - a necessary premise of the thought experiment - then it must be preventing the person from being rational in this way, perhaps by including statements in the book that in some extraordinary way reprogram his mind via some backdoor vulnerability. Let's say that perhaps the person is an android creating by the AI for its own amusement, which responds to certain phrases with massive anomalous changes in its brain wiring. That is simply the only way I can accept the premises that:

a) the person applies Bayes's theorem properly (if this is not true, then he is simply not “mentally consistent” as you said)

b) he is aware that the books are designed to persuade him with high probability

c) he believes that the propositions to be proven in the books are untrue in general

d) he believes with high probability that the books will persuade him

which, unless I am very much mistaken, are equivalent to your statements of the problem.

If reading a book is not basically equivalent to submitting knowingly to brain modification for belief in something, then one of the above is untrue, i.e. the premises are inconsistent and the thought experiment can tell us nothing.

Remember that you are trying to prove than “one can really intentionally deceive oneself and be in a mentally consistent (although weird) state”. I accept that there is nothing mentally inconsistent about submitting to have one’s beliefs changed by brain surgery in one’s sleep. But accepting the fact that “intentionally deceiving oneself” is just three words that could be applied to any referent, I don’t think that your apparent referent is what Eliezer was talking about in the post you linked to. So you haven’t refuted him.

“Intentionally deceiving oneself” in his discussion means “deciding that one should believe something, and then (using the mundane tools available to us now, like reading books, chanting mantras, going to church, meditating etc.) forcing oneself to believe something else”. This may be possible in the trivial sense that 0 is not a probability, but in a practical sense it is basically “impossible” and that is all that Eliezer was arguing.

I’m sure Eliezer and anyone else would agree that it is possible to be an ideal Bayesian, and step in a booth in order to have oneself modified to believe in the flying spaghetti monster. It does seem to me that in order for the booth to work, it is going to have to turn you into a non-Bayesian irrational person, erase all of your memories about these booths or install false beliefs about the booths and then implant barriers in your mind to prevent the rest of your brain from changing this belief. It seems like a very difficult problem to me – but then we are talking about a superintelligent AI! In fact I expect that you’d need to be altered so much that you couldn’t even expect to be approximately the same person after leaving the booth.

Incidentally, this reminds me of a concept discussed in Greg Egan’s book “Quarantine”, which you might find interesting.

EDIT:

On re-reading, I see that the modification process as I described it doesn't actually uphold the premises of your thought experiment, because only one iteration of book-reading could occur before the person is no longer "mentally consistent" i.e. rational, and he can't ever read more than one of the books either (since his beliefs about or knowledge of the books themselves have been changed - which is not what he asked of the AI). So in order for the premises to be consistent, the book-programming-brain-surgery would have to completely wipe his mind and build a set of experiences from scratch so as to make the Universe seem consistent with evidence of the flying spaghetti monster, without it having to turn him into a non-Bayesian. The person would have to have evidence that the AI is clever enough that he should believe that it will be able to make books that persuade him of anything. And the AI would probably reset his mind at a point where he believes that he has never actually read any of the books yet.

What if the person realises that this exact scenario might already have happened? If the person was aware of the existence of this AI, and that he was in the business of asking it to do things liable to change his mind, I don't suppose that the line of reasoning I have outlined here would be hard for him to arrive at. This would be likely to undermine his belief in the reality of his entire life experiences in general, lowering his degree of belief in any particular deity. I suppose the easiest way around this would be if the AI were to make him sufficiently unintelligent that he doesn't come to suspect this, but is just barely capable of understanding the idea of a really smart being that can make "books" to persuade him of things (bearing in mind that according to the premises of the thought experiment, he has to be mentally consistent, i.e. Bayesian, and cannot have arbitrary barriers erected inside in his mind).

It seems that this thought experiment has turned out to be an example of the hidden complexity of wishes!

I don't think about persuation like: "You have to believe this, under threat of pain, in 3... 2... 1... NOW!"

It's more like this: We have some rationalist tools -- methods of thinking which, when used propertly, can improve our rationality. If some methods of thinking can increase rationality, then avoiding them, or intentionally using some contrary methods of thinking, could decrease rationality... could you agree with that?

Omega could scan your brain, and deliver you an electric shock whenever your "Bayesian reasoning circuit" is activated. So you would be conditioned to stop using it. On the other hand, Omega would reward you for using the "happy death spiral circuit", as long as the happy thought is related to Zoroastrianism. It could make rational reasoning painful, irrational reasoning pleasant, and this way prepare you for believing whatever you have to believe.

In real brainwashing there is no Omega and no brain scans, but a correct approach can trigger some evolutionary built mechanisms that can reduce your rationality. (It is an evolutionary advantage to have a temporary rationality turn-off switch for situations when being rational is a great danger to your life. We are not perfect thinkers, we are social beings.) The correct approach is not based on fear only, but uses a "carrot and stick" strategy. Some people can resist a lot of torture, if in their minds they do not see any possibility to escape. For efficient brainwashing, they must be reminded that there is an escape, that it's kind of super easy, and it only involves going through the "happy death spiral"... which we all have a natural tendency to do, anyway. The correctly broken person is not only happy to have escaped physical pain, but also enjoys the new state of mind.

I think 1984 described this process pretty well, but I don't have it here to quote it. The brainwashed protagonist is not just happy to escape torture (he knows that soon... spoiler avoided), but he is happy to resolve his mental conflict by developing the correct anti-rationalists skills. Now he is able to grok the Party, and in his new mind this is the happy ending -- he is like a wirehead. He could never achieve this while being rational. Being rational was his "original sin".