Rationality and Winning

Note, though, that if you fail to achieve a correct answer, it is futile to protest that you acted with propriety. "Rationalists should win" is still normative, even if "Rationalists always win" isn't true.

Another thing that's pretty crucial here is that rationality is only aimed at expected winning.

Suppose we live on Lottery Planet, where nearly everyone has a miserable life, but you can buy a lottery ticket for a chance of $BIGNUM dollars. Nonetheless, the chances of winning the lottery are so small that the expected value of buying a ticket is negative. So the rational recommendation is to refrain from buying lottery tickets.

Nonetheless, the agents who would be "smiling down from their huge piles of utility" could only be the ones who "irrationally" bought lottery tickets. (Credit for this example goes to someone else, but I can't remember who...)

You shouldn't expect rationality to help you win absolutely. Some people will just get lucky. You should expect it to help you do better than average, however. The rationalist on lottery planet is certainly likely to be doing better than the average lottery-ticket buyer.

I am wary of excluding work ethic and mastery over akrasia from rationality, and I am not sure about intelligence.

Akrasia and work ethic are choices. Aspiring rationalists who find themselves not making the choices they have found to be rational should seek to remedy this situation, not excuse themselves for having akrasia.

Some limitations on how rational you can be might be unfair, but that doesn't stop them from making you irrational.

I'm having trouble calling a person who can rattle out a perfectly rational thing to do in every circumstance but instead spending their life complaining about how they would do this and that if only they didn't have akrasia a rational one.

There's a tale of Naive Agent. When the Naive Agent comes across a string, NA parses it into a hypothesis, and adds this hypothesis into his decision system if the hypothesis is new to NA (it is computationally bounded and doesn't have full list of possible hypotheses); NA tries his best to set the prior for the hypothesis and to adjust the prior in the most rational manner. Then the NA is acting on his beliefs, consistently and rationally. One could say that NA is quite rational.

Exercise for the reader: point out how you can get NAs to give you money by carefully crafting strings.

The flaw in NA is that when NA comes across a string that is parse-able into a hypothesis, NA is doing invalid update, adjusting the probability of something from effectively 0 to a non-zero value. That has to be done to be able to learn new things. At the same time, doing so makes the subsequent 'rational' processing exploitable.

rationality is a normative concept defined by logic, Bayesian probability theory, and Bayesian decision theory

Compare with what Russell said about mathematics:

"Pure Mathematics is the class of all propositions of the form “p implies q,” where p and q are propositions containing one or more variables, the same in the two propositions, and neither p nor q contains any constants except logical constants."

Where human rationality is concerned, simple measures analogous to deductive correctness in mathematics don't capture many important aspects that reflect aesthetic of purpose or quality of understanding. In mathematics, understanding of an argument is often more important than its deductive correctness, for a flawed presentation of a sound idea can be fixed, or given a rigorous foundation long after it's first formulated.

[This distinction (and analogy that carries it over from the similar argument about mathematics) deserves a full-length article, but for now I'll just point it out.]

It would be more correct to say that "Winning as defined by general society norms" is not a strong measure of rationality. "Actual Winning", as defined by the agent's own total values, certainly must be.

Also, while the prior probability of winning is (should be) higher in the rationality group, and lower outside, there are likely still many more winners outside the rationality group, because there are so many more people outside it than within. Making use of the availability heuristic to estimate "winning" and decide whether rationality pays off won't work well.

Rational thinking has helped me overcome my akrasia in the past, so if someone isn't very good at overcoming theirs, I see that as weak evidence of poor rationality.

Thus, one could have highly rational beliefs and make highly rational choices and still fail to win due to akrasia, lack of resources, lack of intelligence, and so on. Like intelligence and money, rationality is only a ceteris paribus predictor of success.

I disagree here. Akrasia, resources, and intelligence are all factors that should be taken into account by a rational agent. The reason rational agents don't always win is the complexity of factors in the real world is too great to predict reliably, no matter how rational and intelligent you are. Rationality provides the best possible means of "balancing the odds", but nothing can guarantee success.

First, it seems to me that this is mainly a debate over the definition of instrumental rationality. And I suspect the reason people want to have this debate is so they can figure out whether they count as especially "instrumentally rational" or not.

The simplest definition of "instrumentally rational" I can think of is "a person is instrumentally rational to the extent they are good at acting to achieve their goals". Thus somebody with akrasia would not qualify as a very instrumentally rational under this simple definition. Your definition amounts to drawing the boundary of agency differently so it doesn't end at the person's body, but slices through their brain between them and their akrasia. I don't much like this definition because it seems as though knowing what the best thing to do (as opposed to doing it) should be in the domain of epistemic rationality, not instrumental rationality.

I would prefer to draw the line at the person's entire brain, so that someone who had a better intuitive understanding of probability theory might qualify as being more instrumentally rational, but an especially wealthy or strong person would not, even if those characteristics made them better at acting to achieve their goals.

Related thread on word usage: http://lesswrong.com/lw/96n/meta_rational_vs_optimized/

It seems to me that the LessWrong rationality does not concern itself with the computational limitations of the agents, using as normative the idealized model that ignores limitations, and lacks extensive discussion of comparative computational complexities of different methods, as well as the security of the agent from deliberate (or semi accidental) subversion by other agents. (See my post about naive agent)

Thus the default hypothesis should be that the teachings of LessWrong for the most part do not increase the efficacy (win-ness) of computationally bounded agents, and likely decrease the efficacy. Most cures do not work, even those that intuitively should; furthermore there is a strong placebo effect when it comes to reporting of the efficacy of the cures.

The burden of proof is not on those who claim it does not work. The expected utility of the LW teachings should start at zero, or small negative value (for the time spent, instead of spending that time e.g. training for a profession, studying math in more conventional way, etc).

As intuition pump for the computationally limited agents, consider a weather simulator that has to predict weather on specific hardware, having to 'outrun' the real weather. If you replace each number in the weather simulator with the probability distribution of the sensor data (with Bayesian updates if you wish), you will obtain a much, much slower weather simulator, which will have to simulate weather on a lower resolution grid, and will perform much worse than original weather simulator, on same hardware. Improving weather prediction within same hardware is a very difficult task with no neat solutions, that will involve a lot of timing of the different approaches.

So, let me explain why Actual Winning isn't a strong measure of rationality.

Your post is basically saying that if you believe that a negative Singularity is likely and that a positive Singularity has lots of expected utility, then if you work to achieve a positive Singularity you are rational (consistency) and therefore winning. And since nobody can disprove your claim that the Singularity is near, until the very end of the universe, you will be winning winning winning....without actually achieving anything ever.

I hope your next post is going to explain why I should care.