Calibration Test with database of 150,000+ questions

I think the problem here is with many trivia questions you either know the answer or you don't; the dominant factor in my results so far is that I either have no answer in mind, assign 0 probability to my being right and am correctly calibrated there, and then all of my answers at other levels of certainty have turned out right so far so my calibration curve looks almost rectangular.

I might just be getting accurate information that I'm drastically underconfident, but I think this might be one of the worse types of questions to calibrate on. I mean, even if the problem is just that I'm drastically underconfident on trivia questions and shouldn't be assigning less than 50% probability to any of my answers when I have an answer, that sounds sufficiently unrepresentative of most areas where you need calibration, and how most people perform on other calibration tests, for this to be a pretty bad measure of calibration.

Perhaps it would be better as a multiple choice test, so one can have possible answers raised to attention that may or may not be right, and assign probabilities to those?

Awesome!

I've been dying for something like this after I zoomed through all the questions in the CFAR calibration app.

Notes so far:

• The highest-available confidence is 99%, so the lowest-available confidence should be 1% rather than 0%. Or even better, you could add 99.9% and 0.1% as additional options.
• So far I've come across one question that was blank. It just said Category: jewelry and then had no other text. Somehow the answer was Ernest Hemingway.
• Would be great to be able to sign up for an account so I could track my calibration across multiple sessions.

Thanks for providing this!

I have a worry about using trivia questions for calibration: there's a substantial selection effect in the construction of trivia questions, so you're much more likely to get an obscure question pointing to a well-known answer than happens by chance. The effect may be to calibrate people for trivia questions in a way that transfers poorly to other questions.

I think this should be copied/moved to Main. A calibration tool certainly deserves wider circulation.

A nice feature would be to mark a question as unanswerable, and if it gets enough flags you could overview and delete them. I just recently came across a question which asked what can be seen in "this image". Without any image attached, of course.

Interesting idea, thanks for doing it, but saddly many questions are very US-centric. It would be nice to have some "tags" on the questions, and let the users select which kind of questions he wants (for example the non-US people could remove the US-specific ones).

Should it be specified (or should the answerer tell) whether we are from the USA or not? A lot of questions seem to be very USA-centric, so the confidence can heavily depend on whether we live in the USA or not.

Note, website seems broken now. Still loads, but the questions don't, and there is only 1 question without an answer.

I made a major update to the interface to make it look prettier. I've tested this in Chrome but please let me know if it doesn't work in any other OS or browser.

I also added Google Analytics so I can see where people are accessing this from.

A link or button to flip your last right/wrong would be nice. I had assigned 0% confidence for one question and accidentally said I got it right. Misclicks aren't the same as poor calibration.

Also, a little more on how to use it would make sense - the first one or two I did, I thought it was, 'how confident are you that this assertion is true' and I thought it was very oddly phrased. Then I realized.

Great tool. Does the API you're using allow unanswerable questions to be flagged at all though? Just got one question that depended on an image that wasn't there, and another with no question body. Also, labeled axes on the graph might be nice for people who don't already know how calibration curves work and/or don't like unlabeled axes.

I noticed such thing that if previous question was estimated to have 60% confidence and the next one is supposed to be 70% I sometimes leave it as 60 due to laziness.

Well, I'm getting a reasonably exciting calibration curve with lots of ups and downs. Cool!

Bug: when I click "Display Calibration Curve" for a second time, the graph is displayed in a larger size. (Doing this sufficiently many times crashed Chrome.) Refreshing the page fixes this behavior.

Feature request: I would like to be able to see if my 50% correctness for 30% confidence is getting 1 out of 2 questions right or 5 out of 10. (Error bars of some sort would also work.)

You rock!