Posts
Comments
[....is at the expense of rewarding knowledge of the correct answer.]
Hmm... I'm not sure that Alice has really more knowledge than Bob in your example.
[EDIT : In fact, in your example, for the quadratic scoring rule, the score of 50:50:$\epsilon:\epsilon$ is better than the score of 40:20:20:20 since $12/25 < 1/2 + 2\epsilon^2$, so that we can indeed say that Alice has more knowledge than Bob after this rule. The following example is, IMHO, more interesting. /EDIT].
Let me propose an other perspective with the following two answers for propositions A:B:C:D :
1) 50:50:0:0
2) 50:25:25:0,
where the correct answer is 100:0:0:0.
In this case, 2) has a better score than 1).
What does 1) know ? That D and C are false. He knows nothing for A and B.
What does 2) know ? That D is false. That C is not very probable. He does not know for A, like 1). But he does know moreover that B is probably not the right answer.
Suppose that 3) is someone who knows that D and C are false, and also knows that B is probably not the right answer (i.e., 3 has the knowledge of both 1 and 2). Then 3) could have given the answer 3a) 75:25:0:0, or the answer 3b) 62,5:37,5:0:0. These two answers score better than 1) and 2).
(Note that knowing that D and C are false and that B is probably not the right answer influence your knowledge about A.)
-----
For example, imagine that 2) first thinks of 50:25:25:0, but then he remembers that it can in fact not be C. We can then compute the bayesian update, and we get :
- P(A | non C) = 2/3 (vs P(A) = 1/2)
- P(B | non C) = 1/3 (vs P(B) = 1/4)
- P(C | non C) = 0 (vs P(C) = 1/4)
- P(D | non C) = 0 (vs P(D) = 0).
This is different from answer 1). In this sense, I think we can really say that 2) knows something that 1) does not know, even if 2) is not sure that C is false. Indeed, after an update of the information 'non C', the score of 2) becomes better than the score of 1). (2/3:1/3:0:0 has a better score than 1/2:1/4:1/4:0).
[... why do they score more?]
I'm not sure if these are good reasons, but it seems to me that
1) The expected answer to the quiz does not just consist in identifying A as a correct answer but also in identifying the others as incorrect answers. I mean that the expected right answer is 100:0:0:0 (and not, for example, 100:50:0:0 or whatever else).
2) Giving 25:25 for B:C is better than giving 50:0 even if answer C is 0 since 25:25 is closer to 0:0 than 50:0 (for the usual Euclidean distance). In this perspective, a better answer for the 50:50:0:0's guy would have been 50:25:0:0, which is better than 50:25:25:0.
Indeed, 1 - [(1-1/2)^2 + (1/4)^2 + 0^2 + 0^2] > 1 - [(1-1/2)^2 + (1/4)^2 + (1/4)^2 + 0^2] > 1 - [(1-1/2)^2 + (1/2)^2 + 0^2 + 0^2].
3) With this perspective, I am indeed not sure that encouraging for a student's answer the sum to be 100 is a good idea. It seems better (for the student which is answering) to focus on each proposition (i.e., A, B, C or D) separately (related to point 1 of my message). For each proposition, the answer should reflect the credence of the person in the the fact that the answer is correct/incorrect. Therefore this could also be applied for a multiple-choice quiz with zero or more than one good answer(s).
EDIT (added) :
To sum up what I think could in this case be an answer to your question, I will say that, with the "quadratic scoring rule", if the expected answer for A:B:C:D is 100:0:0:0, then the answer 1) 50:25:0:0 scores more than the answer 2) 50:50:0:0 because they are both right for C and D, they are at the same distance of the expected answer for A but 1) is closer to the expected answer for B (which is 0) than 2).
The same reasoning works for comparing 1') 50:25:25:0 with 2') 50:50:0:0, except that in this second case, it is the general distance (for the quadratic scoring rule) of 25:25 (for B:C) which is closer to 0:0 than 50:0.