[SEQ RERUN] Zut Allais!

One guess, pointed out in the original comments, might be that there is reason to prefer certainty when making deals with untrustworthy agents. For instance, if I promise you a certain $24,000, and then you don't get it, you know for sure that I lied, as does everyone else who was aware of the deal, which is pretty bad for me. If I promise you a 33/34 chance of $27,000 then if you don't get it I can always claim you were just unlucky, giving me at least plausible deniability. Thus there is significant reason for you to prefer the first, since the more I have to lose by betraying you the less likely I am to do it. The same argument does not carry in the case of 33% versus 34%.

I suspect that with infinite computational power on all sides this effect would vanish, and failing to deliver on any deal would decrease my trustworthiness by a certain amount depending on the plausibility of other explanations. However, humans don't have infinite computational power, so we tend to just save time by labelling people as "trustworthy" or "untrustworthy" meaning, creating the incentive to bias towards absolute promises rather than probabilistic promises.

Of course, this is all quite complicated, it's just one thought that springs to mind. It may be better just to favour the null hypothesis of "evolution is stupid, the human brain is a massive kludge that doesn't normally operate on anything resembling expected utility, massive mistakes are to be expected".

We observe two biases simultaneously:

"How silly to treat the difference between 97% and 100% so much differently than the difference between 33% and 34%!"

"How silly to assign 97% probability to things that only happen 70% of the time!"'

You don't get introspective access to the probabilities your brain is implicitly using, so this sort of error is unsurprising. Natural selection isn't going to do anything about it until people start making serious decisions on the basis of explicit expected value calculations.

Note the implicit selection effect here: cognitive biases that did get selected out of the gene pool don't get posts on Less Wrong, so we're left with heuristics that helped more than they hurt (all things considered) in the ancestral environment.

It's a hack. Computation isn't free.

I observe that in my experience, acting confident is highly correlated with both a preference for certainty and sexual attractiveness. That might have something to do with it. Peacock tails aren't especially adaptive, either.