Free Will as Unsolvability by Rivals

If you are dealing with an intelligence vastly more or less intelligent than yourself, there is no contest. One of you can play the other like tic-tac-toe. The stupid party's values are simply irrelevant to the final outcome.

Possible counterexamples: the rabies virus is pretty good at getting mammals to do its bidding. Similarly, the protozoa Toxoplasma gondii, when it infects mice and rats, actually causes the infected rodents to become attracted to, instead of averse to, the scent of cat urine, which assists the parasite in transferring itself to its preferred host.

Consider a specific thought experiment where modeling others is important, for example Prisoner's Dilemma. Even if your opponent is much smarter than you, smart enough to simulate your every thought, you can bargain with them using your access to the outcome-button. You can threaten them with defecting if they think too fast or too difficult for you to follow (in particular, they won't be able to unconditionally simulate you if they comply). And so you'll rob them of this first-mover advantage by having your finger on the trigger. Your decision is still your own.

(This idea comes from the discussion of "unintended simulation" thought experiment variant on SIAI decision theory list.)

We don't want to be understood.

+1. Look at the reaction (of just about anybody) to being reduced to a "type", no matter how well it fits.

"You're not dead. You're as alive as I am."

Then words tumbled from him; more than in all the days since Relay. "True. The illusion of self-awareness. Happy automatons, running on trivial programs. I'll bet you never guess. From the inside, how can you? From the outside, from Old One's view--" He looked away from her, dizzy with a doubled vision.

Ravna drifted closer till her face was just centimeters from his. She floated free, except for one foot tucked into the floor. "Dear Pham, you are wrong. You've been at the Bottom, and at the Top, but never in between.... 'The illusion of self-awareness'? That's a commonplace of any practical philosophy in the Beyond. It has some beautiful consequences, and some scary ones. All you know are the scary ones. Think: the illusion must apply just as surely to the Powers."

"No. He could make devices like you and I."

-- Vernor Vinge, A Fire Upon The Deep (published in April 1992)

Free Will as Unsolvability by Rivals

I thank you for this new insight, besides general rationality training this is the kind of thing I visit LW for!

This matches the sense that facing a terrifyingly powerful intelligence, one that can solve us completely, strips away our free will, which in turn probably explains the common misconception that free will is incompatible with reductionism -- knowing that an explanation exists feels like having the explanation be known by someone. We don't want to be understood.

This might also explain why one needed something as strong as religion to make hundreds of millions of people to stop nominally believing in Libertarian free will. The number of people's who ceased to believe in it because of their belief in a omnipotent omniscient "terrifyingly powerful intelligence" dwarfs the number who have done so on materialist grounds.

I only just now found this pithy, relevant quote:

You have the right to remain cryptic.

...though of course it would be rude to suggest that the meaning I attribute to it was the intended one.

I agree that the feeling of free will is the mind projection bias, but I think we can make things simpler and less arbitrary by applying it to ourselves rather than rivals. It goes like "Why do I think I have free will? Because I cannot predict what I will do next. Therefore I am unpredictable." So "feels like free will" would be a property of any chaotic self-reflecting being with the mind projection bias.

This can be extended back to define free will itself with the additional stipulation that free will is defined as the stuff that makes you feel like you have free will. Some people definitely reject this additional stipulation, but I think the other options are silly :P

Pavitra, I'd love to know what you think about my post on free will:

For example, children and some mentally handicapped people are not responsible in the same way as healthy adults. They cannot give consent or enter into legally binding contracts. One of the reasons for this is that they lack control, are easily influenced by others. Healthy humans exert a higher control than children and handicapped people. You experience, or possess a greater extent of freedom proportional to the amount of influence and effectiveness of control you exert over the environment versus the environment over you.

In other words, I think a paperclip maximizer is dangerous because it has more free will, i.e. is free to (not free from) realize what it wants as its effect on the universe is much larger than that of a human(s). An agent's perception to be free is therefore correlated with the ability to realize its goals, the probability of success.