Test Your Calibration!

Advice for future creators of tests: There are people who live outside the US. No one outside the US cares about the 3rd person to be the second dead uncle of the fourth president of the US.

For instance, a majority of tommccabe's quiz questions are highly US-specific.

The point here is that non-Americans will end up guessing almost all questions, making the whole exercise painful and useless.

Why isn't there a 33% option for your test? What if I'm pretty certain that 1 of the answers is wrong, but have no clue which of the others is most likely to be right? Then my confidence is exactly 33%, and I have to either overestimate or underestimate it. The 50% and 25% options seem to cover the other two versions of this scenario (I can eliminate either 2 or 0 of the options almost certainly) but this appears to be a gap.

(incidentally, this only occurred to me because it happened to be the case for the first question on the first of your quizzes...)

Part of the output of your quizzes is a line of the form "Your chance of being well calibrated, relative to the null hypothesis, is 50.445538580926 percent." How is this number computed?

I chose "25% confident" for 25 questions and got 6 of them (24%) right. That seems like a pretty good calibration ... but 50.44% chance of being well calibrated relative to null doesn't seem that good. Does that sentence mean that an observer, given my test results, would assign a 50.44% probability to my being well calibrated and a 49.56% probability to my not being well calibrated? (or to my randomly choosing answers?) Or something else?

Just launched my own version of a calibration test here - https://calibration.lazdini.lv/ it is pretty much identical to http://confidence.success-equation.com/ except the questions should be different each time you visit the site, allowing for regular calibration/recalibration. Questions are retrieved from the free API provided by https://opentdb.com/.

I would like to see a calibration test with open-ended questions rather than multiple choice. Multiple choice makes it easier to judge confidence, but I'm afraid the calibrations won't transfer well to other domains.

(The test-taker would have to grade their test, since open ended questions may have multiple answers, and typos and minor variations shouldn't count as errors. But other than that, the test would be pretty much the same.)

I have seen a problem with selection bias in calibration tests, where trick questions are overrepresented. For example, in this PDF article, the authors ask subjects to provide a 90% confidence interval estimating the number of employees IBM has. They find that fewer than 90% of subjects select a suitable range, which they conclude results from overconfidence. However, IBM has almost 400,000 employees, which is atypically high (more than 4x Microsoft). The results of this study have just as much to do with the question asked as with the overconfidence of the subjects.

Similarly, trivia questions are frequently (though not always) designed to have interesting/unintuitive answers, making them problematic for a calibration quiz where people are expecting straightforward questions. I don't know that to be the case for the AcceleratingFuture quizzes, but it is an issue in general.

Wow, hmm I took quiz 1 so far, and all my high confidence answer groups all scored much lower. For now I blame too much experience with easy multiple choice tests. I only got 19 out of 50 overall

Another problem, I think :

You marked your answers to 17 questions as '25% accurate'. Out of these, 1 answers were correct, for a success rate of 5.8823529411765 percent.

Now I thought I was choosing 25% when I didn't know the answer, but this seems to indicate that I had some information, and was biased against playing my (sometimes correct) hunches when marking 25%.

(I believe this is his LW account, but feel free to correct me)

This is my current LW account.

There were sequels to the Aumann game here here and here; these have better questions but probably the lack of auto-scoring makes it not worth the effort.

If anyone is thinking about creating their own, I would suggest questions with numerical answers so you can give upper and lower bounds of varying confidence, rather than trying to pick your confidence on a binary question and try to force binning or do some sort of filtering.

Also, this lets you give several probability estimates for each question.

http://acritch.com/credence-game/

I get 404s on the quizzes from Tom McCabe.

Question 40 of Salvatier's test:

#40: Which city is closer to Los Angeles, Calif.?

• Phoenix, Ariz.
• Phoenix, Ariz.

Wait... What?