Shut Up And Guess

This reminds me of an infamous chemistry exam that no longer existed at my college by the time I got there but had passed into the student lore. For each question, you would first mark your answer (multiple choice, 4 or 5 choices), and then mark a confidence option. These were "high confidence" (5 points if right, -3 if wrong), "low confidence" (3 points if right, 0 if wrong), and "I don't know, give me a point" (1 point regardless of what answer is marked).

This exam was not popular with the students.

I think the problem is that people tend to conflate intention with effect, often with dire effect, (eg. "Banning drugs == reducing harm from drug use"). Thus when they see a mechanism in place that seems intended to penalise guessing, they assume that its the same as actually penalising guessing, and that anything that shows otherwise must be a mistake.

This may explian the "moral" objection of the one student: The test attempts to penalise guessing, so working against this intention is "cheating" by exploiting a flaw in the test. With the no-penalty multiple choice, theres no such intent so the assumption is that the benefits of guessing are already factored in.

This may not in fact be as silly as it sounds. Suppose that the test is unrelated to mathematics, and that there is no external motive to doing well. Eg. you are taking a test on Elizabethan history with no effect on your final grade, and want to calibrate yourself against the rest of the class. Here, this kind of test is a flaw, because the test isn't measuring solely what it intends to, but will be biased towards those who spot this advantage. If you are interested solely in an accurate result, and you think the rest of the class won't realise the advantage of guessing, taking the extra marks will just introducing noise, so it is not to your advantage to take them.

For a mathematics or logic based test, the extra benefit could be considered an extra, hidden question. For something else, it could be considered as immoral as taking advantage of any other unintentional effect (a printing error that adds a detectable artifact on the right answer for instance). Taking advantage of it means you are getting extra marks for something the test is not supposed to be counting. I don't think I'd consider it immoral (certainly not enough to forgo the extra marks in something important), but Larry's position may not be as inconsistent as you think.

I'm surprised that test-preparation companies haven't picked up on this. Training people to understand calibration and loss aversion could be very helpful on standardized tests like the SATs. I've never taken a Kaplan or Princeton Review course, but those who have tell me this topic isn't covered. I'd be surprised if the people involved didn't know the science, so maybe they just don't know of a reliable way to teach such things?

They were in SAT prep books 25-27 years ago. (I took the SAT's while I was still 15.) The explanation given was something along the lines of, "Most people say that the SAT penalizes you for guessing, but this is wrong. Rather, it simply makes sure that, on average, guessing won't get you any extra points if you don't know anything about the question. If you can eliminate even one wrong answer out of five, you will always come out ahead by guessing. If you can't, then you still won't lose anything by guessing." They then showed math and examples to back it up.

It was actually in a very early part of the book I read, because they wanted you to understand how important it was to be able to identify even one wrong answer, and thus why the methods you were going to learn for doing that were important.

It would be interesting to re-frame the test as "start with 50 points. Get 1/2 for a correct answer, lose 1/2 for don't know, and lose 1 for a wrong answer". I suspect your friends would accept this as equivalent scoring, and would start guessing more.

The loss aversion is probably less strong because they're already taking a loss by not guessing, so losing just a bit more isn't that painful.

Oh my dear lord Cthulhu. Can I ask what level of class this was? If you say it was a postgraduate course at MIT, I may gather the last sane members of the human race and move to Pluto.

Woman + sex => attraction
Man   + sex => squick

A year back, I encountered a this kind of a test: binary multiple choice, one point for right answer, minus half a point for a wrong answer, zero points for no answer. (Multiple-choice exams of any kind are very rare in Finnish universities, so that's pretty much the only time in my life when I've been faced with a test like that.) Looking at the scoring, I came to the same conclusion as you: my expected score would be higher if I'd just try guessing each of the questions I wasn't sure on.

I didn't follow my own advice. I now wish I had, as I failed that exam. I was under a pretty heavy workload at the time, so I never ended up retaking it. I suspect I'd have passed if I'd just shut up and multiplied.

Why didn't I follow my own advice? I did have some kind of a conscious reason, but in retrospect it seems so flimsy that I have difficulty even formulating it here. It went something along the lines of "I might as well take all the questions I have absolutely no clue on and mark them all as 'true', which gives me a 50-50 chance to be right on each one assuming there are as many true as there are false questions. But what if the lecturer, forseeing that somebody would reason this way, wrote the questions in such a way that one alternative is more frequently correct than the other, and there isn't a 50-50 chance for all questions to be 'true'? Then my expected return calculation would be off, possibly costing me points!"

Yes, I'm aware of all the flaws in that line of thought, no need to point them out. I really didn't think it through properly. That implies that the very thing you suggest happened to your friends, happened to me - I instinctively disliked the idea, and then rationalized myself a (bad) reason not to do it.

I suspect that your friends were simply trying to rationalize their previous behavior or avoid admitting they were wrong. I'll bet more of them would have been sympathetic to your arguments if they'd been presented before they'd ever taken a test of that type. In fact, I'll bet a few of them would find your arguments so obvious as to barely be worth mentioning if presented in this context. (E. g. if you'd posed as a brainteaser to your friends: "on a test of this type, do you increase your expected score by guessing or marking don't know", I'll bet some of them would have said "Guess. That's obvious.")

According to my interpretation, the only reason your outcome was superior was because you made the discovery early on under your own steam. To measure whether you would be better at admitting you were wrong than your friends, we would have to give you a test where you actually had to admit you were wrong.

Anyway, guessing does increase the variance in your answer. So maybe a more complete argument where you asked your friends how many questions they expected to know and then gave them odds for getting each of "no pass", "pass", "honors" and "high honors" using guess and no-guess strategies would have been more effective.

Taking tests is one place where I've noticed focus on rationality can give you a boost. I had two classes with a very similar format - each semester had two exams and a final. Each exam had several multi-part questions that got progressively more difficult. The average score was expected to be about 50%, and not everyone was expected to finish any of the exams. Grade in the class was based on ranking - the person with the highest cumulative score got an A, e.g.

A classmate and I realized that we could use the bias other students had for wanting to focus on the easy points to our advantage. That is, the later questions were harder, but they gave much more bang for the buck. It was kind of painful to leave questions you knew you could answer easily blank (which is where overcoming the bias comes in), but it was most certainly worth it when we got the top ranks.

This... absolutely sickens me. It's bad enough when I hear my family members argue morals/politics/economics that they subscribe to for proximate lifestyle purposes - but when University students pull this, and then ignore the eminent Yvain when he councils them otherwise?

My only comforts are the harsh cold truth of schadenfreude, that such beings don't deserve an extra 5%, and that at least I only wasted three years and $20 000 at post-secondary.*

*(My degree was non-technical; Humanities students who don't want a PhD should drop out in second year, spend a year reading, and then lie on their resume.)

P.S. Excellent break down of the reasoning process, Yvain. I think you hit the nail on the head.

The students infer a social rule penalising guessing. In almost all cases exploiting a technicality that works around a social rule is penalised. Nobody likes munchkins.

I would expect to observe a tendency for people rationalise reasons to not guess even if loss aversion was contolled for.

The idea of doing significantly worse than chance by guessing on the test sounds absurd, but I recall that when I was on Chemistry Team in high school, many of the teams we competed against managed it, with teams of four getting average scores of below 20% on four-option multiple choice tests.

If the students were really guessing at random, they ought to expect to do better, but answering questions to the best of their limited knowledge may cause them to do significantly worse than chance if the questions are designed to trip up people with common misunderstandings or gaps in their knowledge of the subject.

I think you got your math wrong

If you get 20 out of 30 questions wrong, you are break even, therefore the probability of losing points by guessing is

Sum( (i 30), i = 21..30) / 2^30 ~ 2.14% > 1%

When I took my high school's AP Calculus classes these last two years, the teacher pointed out that since, on average, guessing would give the same result as leaving questions blank, you might as well guess. As far as I know, nobody disagreed with him.

(Actually, he said it's better to guess, because leaving a question blank means running the risk of accidentally putting the next question's answer in the wrong place--which, in one case, led to a student answering practically every question in one section wrongly. But that's relatively impertinent.)

It sounds like your fellow students understood the concept of a guessing penalty, but did not realise that the guessing penalty was too low in this case. One approach to convince them might have been:

Assume you get -0.0001 points for guessing an incorrect answer. Obviously, you should answer every question, because the penalty for guessing is so low. Now, assume that the guessing penalty is -20 points. Again, you obviously shouldn't guess. What would the penalty have to be where you're indifferent between guessing and not guessing? Obviously, when the penalty is -1 point. You guess two answers, one is correct and the other not, and your expected score is 0. In this case the penalty is -0.5, which is closer to -0.0001 than to -20, therefore you should always guess.

NB At my university, multiple choice exams always feature four possible answers, and you lose .33 for guessing incorrectly. Every student understands this concept perfectly. If they had to take your exam, they would've guessed every single time. It's strange to see that there are universities where the guessing penalty is not well calibrated. It seems like an elementary thing to do.

Possibly part of the loss aversion is the desire not to look foolish. I mean, if the teacher is reviewing your exam results, and he sees you answered 100 questions correctly and said "don't know" for the rest, then you look like a pretty smart and modest guy compared to the schmuck who answered 125 questions correctly and 25 questions incorrectly.

Probably in our evolutionary history, if you looked foolish it was bad news.

But anyway, I am mainly posting in this thread to state that as an attorney I can attest that loss aversion is a big issue in civil litigation. A lot of people are scared to death of going to court and losing even when the downside is pretty minimal. The most successful attorneys I know (at least on the Plaintiffs' side) lose in various ways on a regular basis.

Suggested intervention if anyone finds themselves in this situation in the future: distribute copies of Ender's Game and/or Methods. Subjectively it feels to me like identifying with Ender / MoR!Harry has made me better at noticing and more motivated to take advantage of these kinds of optimization opportunities.

Regarding penalizing guessing, if you're going to penalize it you might as well go all the way. My high school math club once hosted a competition which included a round with a ridiculous guessing penalty (free response, 1 point for a correct answer, 0 for a blank, and -3 or -5 for an incorrect answer). Exactly one person out of a hundred-ish got a positive score.

This is incredible. As others have said the most likely explanation is that people could see the system was intended to dis-incentivise guessing and that this design intent shaped the way they saw the test.

Now I want an exam on "logic and probability" which uses this system. The surprise being that the grading system indicated is in fact a lie. You fail if you put a single "don't know". Otherwise you pass with 100%. 

(A teacher of mine at school once set us a reading test. It had a big line at the top saying "read this entire test before starting". Then there were paragraphs of text interpresed with questions about it, "How many people were in the study?". Right at the end it said "Now you have finished reading, please provide a blank sheet of paper. Do not answer any of the questions." I was stung.)

This reminds me of a class I had as an undergraduate. To avoid taking another class with a lab, I took Ethnobotany to finish out my general science requirements. The tests were multiple choice with conjunctive answers. For example:

The parts of a flower include: a) Petal b) Seed c) Stamen d) Sepal

To which the correct answer is a, c, and d. It was computer scored such that the only correct answer was to bubble a, c, and d: bubbling a and c got you no points, nor did bubbling a, b, c, and d, for example.

Given this test format, I put the Conjunctive Fallacy to work for me. When I came across a question where I had no idea if one of the options was part of the answer, I never included it since including it would have made the resulting answer less likely to be true than the answer would have been by leaving it off. The result: I got several questions right that I would have otherwise missed.

This is related to Cognitive Reflection Test.

The basic mistake seems to be loss aversion, the tendency to regret losses more than one values gains.

I tend to think of loss aversion as a preference rather than a mental error, but I agree that it probably explains a lot of the "don't know" answers. What I cannot figure out is why the test designers want an individual's level of loss aversion to affect their score.

Another possible (and probably over-charitable) explanation for the lack of guessing is that students are afraid of being drawn to the wrong answer. For example, I've heard that on the SAT math portion many of the answer choices purposely contain numbers that were mentioned in the question, drawing "random" guessers because such answers look more plausible.

I would say it basically comes down to the fact that abstract rationality is slow and requires lots of processing power. For the same reasons we can usually only mentally afford to employ a certain limited set of fairly abstracted terms, and can only follow the implications of this to a limited degree. If we were all Kryptonians it would probably be pretty functionally rational to stay in 'far mode' all the time, but as the squishy, dumb bugs we are a lot of our functional capacity derives from various habitual and patterned behaviour. Far mode mostly seems to serve as a general regulator for some general patterns, perhaps in order to improve intra-plan cohesion. The whole cognitive consciousness part of this may simply be a side-effect of it being kind of overlayed over the background pattern integration that constitutes our ordinary mental processes.

College admissions and financial aid generally are gold mines of examples of non-optimizing behavior with large consequences. I fairly regularly see people losing tens of thousands of dollars or more through failure to spend a few hours doing relevant research.

FYI, the average student can expect to increase their score by about 8% by guessing, assuming the "non-guesses" were right 80% of the time. An enormous effect.

Maybe (if your goal was to get them to score higher) you could have pitched it as gambling for entertainment, i.e. record which answers you guessed on, and later compare who was luckiest in getting the biggest portion of those correct.

Most apathetic students have no qualms with guessing. It sounds like these peers of yours are either extremely diligent and motivated by fear, or unwilling to lose face by admitting that you're more clever than they.

Are you sure that none of them were pulling your leg?

I suggest that the students aren't as irrational as they appear. After all, why would the designer of the test incorporate a "don't know" option and a penalty for wrong answers, except to discourage guessing on questions that you're clueless on? And if I were a random student (instead of someone especially interested in the mathematics of decision theory), why should I take the trouble to second guess the test designer, instead of assuming that (with high probability) he is rational and competent at his job?

ETA: Also, you're supposed to maximize expected utility, not expected number of points. Increasing the variance of your score may decrease expected utility, even if it keeps the expected score the same. (I see that John Maxwell IV has made a similar point.)