Circular belief updating

A sorcerer has two ways to manipulate people:

1) Move things around in the world.

2) Directly influence people's minds.

I'm not going to talk about option 2 because it stops people from being perfect reasoners. (If there's a subset of option 2 that still lets people be perfect reasoners, I'd love to hear it - that might be the most interesting part of the puzzle). That leaves option 1.

Here's a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer's existence. We will adopt the usual "Nash equilibrium" assumption that the observer knows the sorcerer's strategy in advance. This seems like a fair idealization of "moving things around in the world". What would the different types of sorcerers do?

Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn't exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.

A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say "either Bright exists or I just saw a very improbable coincidence". A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer's prior. Invisible has the same set of best strategies as Dark. I won't spell out the proofs here, anyone sufficiently interested should be able to work them out.

To summarize: if sorcerers can only move things around in the world and cannot influence people's minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn't do anything like "updating on the strength of their own beliefs". The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.

Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!

Like cousin_it, I'm assuming no mind tampering is involved, only evidence tampering

Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?

I don't think the feedback loops exist. Bob saying "the fact that I believe in bright is evidence of Bright's existence" is double-counting the evidence; deducing "and therefore, Bright exist" doesn't bring in any new information.

It's not that different from saying "I believe it will rain tomorrow, and the fact that I believe that is evidence that it is rain tomorrow, so I'll increase my degree of belief. But wait, that makes the evidence even stronger!".

If Bob and Daisy took the evidence provided by their belief into account already, how does this affect my own evidence updating? Should I take it into account regardless, or not at all, or to a smaller degree?

Just ignore the whole "belief in dark is evidence against dark" thing, Daisy already took that information into account when determining her own belief, you don't want to double count it.

Treat it the same way as you'd treat hearing Bob tell you that in Faery, the sky is Blue, and Daisy telling you that in Faery, the sky is Green.

Daisy isn't in a loop at all. There's apparently evidence for Dark and that is tempered by the fact that its existance indicates a failing on Dark's part.

For Bob, to make an analogy, imagine Bob is wet. For you, that is evidence that it is raining. It could be argued that being wet is evidence that it's raining for Bob as well. But generally speaking Bob will know why Bob is wet. Given the knowedge of why Bob is wet, the wetness itself is masked off and no longer relevant. If Bob has just had a bath, then being wet no longer constitutes any evidence of rain. If Bob was outside and water fell on him from the sky, it probably did rain, but his being wet no longer constitutes any additional evidence in that case either (well, ok, it has some value still as confirmation of his memory, but it's orders of magnitude less relevant).

Similarly Bob should ask "Why do I believe in Bright?". The answer to that question contains all the relevant evidence for Bright's existance, and given that answer Bob's actual belief no longer constitutes evidence either way. With that answer, there is no longer a loop for Bob either.

One final point, you have to consider the likelihood of belief in case 4. If you would expect some level of belief in sorcerors in Faerie even when there are no sorcerors, then case 4 doesn't fall behind as much as you might think. Once you've got both Bob and Daisy, case 4 doesn't just break even, it's actually way ahead.

The apparent paradox is resolved as long as you note that P(Daisy thinks Dark does exist|Dark does exist) > P(Daisy thinks Dark doesn't exist|Dark does exist).

That is, even if Dark does exist and does want to hide his existence, his less-than-100%-effective attempts to hide will produce non-zero evidence for his existence and make the probability that Daisy will believe in Dark go up by a non-zero amount.

Adding some structure to this hypothetical: At time t=0, Bob and Daisy have certain priors for their beliefs on sorcery, which they have not adjusted for this argument. Bob's belief was Position 1, with reduced strength, and Daisy's was Position 3, with greater strength.

I'll call your argument A0.

At time t=1, Bob and Daisy are both made aware of A0 and its implications for adjusting their beliefs. They update; Bob's belief in 1 increases, and Daisy's belief in 3 decreases.

More arguments:

A1: If Position 1 is true, then Bright is likely to cause you to increase your belief in him, therefore increasing your belief in Bright is evidence for Position 1.

A1': Corollary: Decreasing your belief in Bright is evidence against Position 1.

A2:If Position 3 is true, then Dark is likely to cause you to decrease your belief in him, therefore decreasing your belief in Dark is evidence for Position 3.

A2': Corollary: Increasing your belief in Dark is evidence against Position 3.

At time t=2, Bob and Daisy are exposed to A1 and A2, and their converses A1' and A2'. If they believe these, they should both increase credence for Positions 1 and 3, following A1 and A2, then increase credence for Position 1 and decrease it for Position 3, following A1 and A2', then follow A1 and A2 again, etc. This might be difficult to resolve, as you mention in your first question.

However, there is a simple reason to reject A1 and A2: Their influence is totally screened off! Bob and Daisy know why they revised their beliefs, and it was because of the valid argument A0. Unless Bright and Dark can affect the apparent validity of logical arguments (in which case your thoughts can't be trusted anyway), A0 is valid independent of which position is true. This action moves them to begin a feedback loop, but stop after a single iteration.

There is a valid reason they might want to continue a weaker loop.

A3: That you have encountered A0 is evidence for the sorcerers whose goals are served by having you be influenced by A0.

A3': That you have encountered A3 is evidence for the sorcerers whose goals are served by having you be influenced by A3.

A3'': That you have encountered A3'' is evidence for the sorcerers whose goals are served by having you be influenced by A3''.

etc.

But this is only true if they didn't reason out A0 or A3 for themselves, and even then A3', A3'', etc. should be considered obvious implications of A3 for a well-reasoned thinker. (In fact, A3 is properly more like "That you have encountered a valid argument is evidence for the sorcerers whose goals are served by having you be influenced by that argument.") So that adds at most one more layer, barring silly Tortoise-and-Achilles arguments.

Given all that, for your second question, you still should take their beliefs into account, but possibly to a slightly lesser degree.

A point I'm confused on: when you, based on A0, update based on their A0-updated belief, are you double-counting A0? If so, you should update to lesser degree. But is that so?

If people believing in Dark is the only possible evidence against Dark, with what certainty should you believe in Dark?

Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?

It is not sane.

If you use a belief (say A) to change the value of another belief (say B), then depending on how many times you use A, you arrive at different values. That is, if you use A or A,A,A,A as evidence, you get different results.
It would be as if:

P(B|A) <> P(B|A,A,A,A)

But the logic underlying bayesian reasoning is classical, so that A <-> A+A+A+A, and, by Jaynes' requirement IIIc (see page 19 of The logic of science):

P(B|A) = P(B|A,A,A,A)

Please give numeric probabilities for Bob and Daisy's beliefs, and examples of (private) evidence that would justify them. I'm not sure you can construct a scenario where they're rational, truthful, have similar priors, and disagree by all that much about sorcery.

With respect to question 1, Aumann's Agreement Theorem would require that if they are acting rationally as you stated and with common knowledge, they would have to agree. That being the case, according to the formalism, question 2 is ill-posed.

Your proposed state of affairs could hold if they lack common knowledge (including lack of common knowledge of internal utility functions despite common knowledge of otherwise external facts and including differing prior probabilities). To resolve question 2 in that case you would have to assign probabilities to the various forms that the shared and unshared knowledge could take, to determine which state of affairs most probably prevails. For example, you may use your best estimation and determine that the state of affairs that prevails in Faerie is similar to the state of affairs that prevails in your own world/country/whatever, in which case you should weight the evidence provided by each person similar to how you would rate it if you were surveying your fellow countrymen. This is all fairly abstract because approaching such a thing formally is currently well outside our capabilities.