A fun estimation test, is it useful?

I got 8 right. The ones I got wrong were the Great Lakes (upper bound was too small by a factor of 1e8) and the currency in circulation (my range was .75-1.25T, and the right answer was slightly lower than my lower bound).

The problem I have with this is while it's structured well to punish overconfidence, it's not structured well to publish underconfidence. You can do better than 95% of respondents (according to the graph the author posted) if you write down -inf for your lower bound and +inf for your upper bound, since all 10 will be within your range (and so your error magnitude is 1, just as if you got 8 right). The fact that you're twice as bad at estimating as someone who writes down 0 as a lower bound and +inf as a upper bound shows up nowhere.

The best test of estimation ability would probably be: each guess costs you log(upper)-log(lower), and a guess that includes the answer gives you Y points. Y determines the minimum level of knowledge you need to guess, but if you narrow your guess you improve your score. You could then figure out what confidence interval people are using from the width they select, and see how that compares with their reported confidence.

I got eight out of ten. I thought that the Pacific coastline was much shorter (but I was imagining it as basically a smooth curve, and terminating it at the Bering Strait and somewhere down near India -- is that how they measured it, I wonder?), and I thought blue whales were much less heavy. Now I have to go find out how something that weighs 190 tons manages to propel itself!

I was surprised by one particular aspect of the blog comments; the recurring theme of "I don't think it's valuable to make estimates like this, because if I ever gave a project estimate of 'two to twenty weeks' I would be laughed out of the room."

What do you mean, you would be laughed out? If you're 90% confident that it's two to twenty weeks, then you should feel OK about saying so -- that's better than confidently saying "five weeks" and watching it turn out to be twenty! At least it should be better. It's a shame if people feel compelled to whitewash guesses like that to save face.

It's an interesting phenomenon (and yes, my intervals were too small). I was actually expecting the reverse. His explanation reminds me of the conjunction fallacy--we overvalue specificity.

Total length of the coastline of the Pacific Ocean

Isn't this a classic example of a divergent series, though?

I found another test that's more comprehensive. It has lots more questions, lets you give a confidence estimate for each, and tells you how well calibrated you are at 0% to 100% probability. And it notes both underconfidence and overconfidence.

http://www.projectionpoint.com/test1.php

I got a 78 out of 100.

Interesting quote from the comments in the explanation:

"I have even heard that when the country of Italy puts out a work request, they take all of the companies estimates and average them. The company closest to average gets the contract."

This sounds like an excellent idea, and should presumably incentivise contractors to give more accurate estimates than they would normally. Are there other methods of getting estimates from contractors that reward accurate estimates?

The problem being, of course, that it would always be difficult to punish contractors for finishing work early, as this gets incentives wrong in a different way.

Either I am the only irrational person here, or there is a strong publication bias. 4 out of 10. Once I was off nearly 3 orders of magnitude.

I got all 10 right, but I had previously heard of similar results, so I don't know whether I would have otherwise known to be that careful.

Of course, for questions where I really didn't know the answer, I had to give a range spanning more than an order of magnitude to reliably hit the target (in some cases I could have tightened up my range, but in one case I only barely got it as it was); but I still think that's better than giving confident but wrong answers.

I got 8 / 10, and was very close on one of the incorrect guesses (33,000 - 133,000km for the coastline, reported as >135,600km). On the other hand, I used much more than the allotted 10 minutes, and did a Fermi calculation for each value. I'm not sure why that should be disallowed.

How do you measure a coastline? Isn't it a subjective measurement depending on the scale of precision/resolution (since the actual geometry is fractal)

I skipped that question since I couldn't figure out the standard

See this for instance: http://en.wikipedia.org/wiki/Coastline_paradox

8 out of 10, and one of the ones I got wrong was the blue whale one, where I undershot by a tiny margin (if I had given my value in metric tons, I would have been right, but I know I meant short tons.)

I would have gotten about half of them wrong, but then I looked over them and realized I was leaning too heavily on an expectation of actual knowledge, and if I didn't actually know the answers with any precision, I should just be providing myself with vast margins of error.

I call shenanigans on some of those answers - I had the surface temperature of the Sun to two significant figures, and he rounded it to one. :P

To my credit, though, I hit 30% through honest stupidity, rather than misunderstanding calibration. My understanding of geography is clearly worse than Columbus's, for example.

7 of 10. I underestimated Asian (Eurasian?) continent area by factor 4 (safety margin one order of magnitude), quantity of US dollars by factor 10 (safety margin 3 orders of magnitude) and volume of gr. lakes by factor 0.1 (safety margin 3 orders of magnitude). Other safety margins were 3 orders of magnitude for Titanic, Pacific coast (fractal-like curves can be very long), book titles, and 0.5 from mean value for others. Sigh, I thought I'll have 90%.

Hm, I estimated area of Asian continent as area of triangle with 10000km base (12 timezones for 20000 km and factor of 0.5 for pole proximity) and 10000km height (north pole to equator), and lost one order of magnitude in calculation.

I got 8 out of 10 too (underestimated the lakes by ten orders of magnitude >_> and my upper bound for the whale was close (I had revised it from 100 to 150 tons, still wasn't enough)), without gaming the test in obvious ways.

Again, I found there's a good deal of people in the comments who failed the test and instead of noticing how they could improve themselves, start making up excuses for how they andswered the right way anyway. 90% estimate means 90%, not "What you would answer if your pointy-haired boss asked you for a 90% estimate"!

I do hope that if I did fail a test I wouldn't do that. Hopefully ranting in public about the stupidity of people who explain their failures away instead of acknowledging them will make me more likely to be honest :)

I got 7 right. Two I got wrong (I overestimated Asia's area & Titanic's tickets). One I misread (the Great Lakes, I tried to calculate their surface area, instead of their volume).

8 out of 10.

I got area of Asia wrong in an interesting way. My estimated area of Asia to area of Earth was correct. My estimated area of Earth was wrong because I misremembered the formula for computing sphere surface from sphere diameter.

I got Pacific very wrong by essentially estimating length of extremely smoothed coastline, not real one.

Both were clear cases of overconfidence.

Things I was fully aware I knew very little about like Great Lakes or numbers of books published, I gave suitably wide ranges, and hit both correctly.

I got 5 right, and was off by 1 degree of latitude on the Shanghai one. I kinda knew I was nowhere near 90% confidence, because 5 of my estimates spanned 2 or more orders of magnitude and I wanted them to be at least somewhat meaningful.

I think it's an important skill in general to be able to estimate things, though I might just think that because I got a 9/10 on that test.

Good estimation may not always be useful in the real world if you're giving someone else an estimate on how long something will take (wide estimates are perceived as bad estimates by most, as many of the comments on that blog show intentionally or unintentionally), but it is fun, and I've seen it be personally useful before.

8/10, and, as mentioned in the explanation on the second blog post, I did feel like I was giving ridiculously wide ranges for some of them, turns out I wasn't ridiculous enough - as one commenter points out, a more interesting exercise might be to do the same but with 50% ranges - then the simple solution of "make an estimate and put 3 orders of magnitude each side" doesn't help.

Also, the length of the coastline of the Pacific, which is one of the ones I got right is, as HonoreDB and several commentors on the blog post have pointed out, undefined - it depends what length ruler you use.

Damn, I have been correct only 6 times out of 9 (I did understand one of the questions incorrectly). And I knew both that people are overconfident and that even when told about that they don't compensate sufficiently.