Bogus Exam Questions
post by Donald Hobson (donald-hobson) · 2020-03-28T12:56:40.407Z · LW · GW · 8 commentsContents
An exam question only measures if you have enough understanding to answer it correctly. The amount of understanding needed to answer a question can be lower than it appears. None 10 comments
Take a look at this multiple choice question from the Open University BSc on psychology. Can you tell what is wrong with it. This is sampled from a set of 15 questions, some of the others aren't this bad, some are guessing the teachers password. One of the questions requires that you know that statistically significant means p<0.05 and can look at a table of p values and say which numbers are less than 0.05.
An educational psychologist believes that an individual's reading speed (in words per minute) and general comprehension abilities (scored out of 10) may be related to how well they do on a timed exam. To study the relative importance of these variables and the association between them, she runs a multiple regression and produces the following predictive model for exam score:
Exam Score =7.78 + (0.05*Reading Speed)+(5.77*Comprehension)
What exam score would you expect students to achieve if they have a reading speed of 120 w.p.m and a comprehension score of 5?
Select one:
o 4726
o 55.67
o 66.44
o 42.63
What level of understanding is necessary to fully understand the question?
From a mathematicians perspective, Multiple linear regression is an algorithm based on variational calculus (Optimizing over a space of functions), to select the function that maximizes the probability density that a model consisting of a linear function from the vector space to plus a Gaussian random variable (with arbitrary standard deviation, that term happens to cancel out) assigns to the observed data. This involves function spaces, matrix inversions, multivariate integrals ect. To understand the mathematical structure that leads up to multivariate regression, you need the idea of jacobians and vector calculus just to integrate your to find the normalizing constant on your Gaussian random variables. You need continuus probability spaces, but probability isn't defined over arbitrary sets of real numbers, so you need sigma algebras, which brings you to set theory. You need to prove that the integrals exist, that the optimum slope exists and is unique, ect.
In short you need to understand a lot of maths to deeply understand what is going on.
How much understanding do you need to get the answer right. Suppose you have basic English comprehension, but all you know about the symbols "385+*)" is that they are some mysterious code that you can type into a calculator and it will reply in the same code.
You know that the answer has to be in this mysterious code, and the calculator doesn't have buttons for letters. You know its asking for an exam score, so you type in the code for "Exam score", but replacing the word "Reading speed" with the mysterious symbols "120", and the word "Comprehension" with the symbol "5".
Your calculator spits out an answer, it is one of the options available, you put it down. (You get an exact answer, no rounding needed)
Lets suppose your grasp of English was little better than your grasp of arithmetic.
You have no idea what an "educational psychologist" is.
You have a rough grasp of capital letters, you know that the first letter in words sometimes looks big or funny, enough to suspect that "Reading Speed" and "reading speed" are the same thing. None of the mysterious numbery things in the question match any of the answers, so you try typing the big numbery thing into a calculator. The calculator doesn't have any lettery buttons, so you replace the lettery things with the numbery things that are near them in the text. If you guess the wrong numbery thing the first time, you don't get an answer on the list, so if you know what a multiple choice test is, you will know to try another.
Maybe I am being a little scathing there, I am certainly assuming that the hypothetical person answering the question knows some exam technique, and are able go along with plausible guesses. But I would expect most people who knew basic English and arithmetic to be able to answer it correctly.
The basic problem is
An exam question only measures if you have enough understanding to answer it correctly.
The amount of understanding needed to answer a question can be lower than it appears.
The question references all sorts of advanced maths, giving the appearance of serious academic learning, but instead of asking you to show an understanding of that maths, it asks you something much simpler.
This is like guessing the teachers password [LW · GW], except that the password is a procedure, not a direct answer.
8 comments
Comments sorted by top scores.
comment by jbash · 2020-03-28T14:16:48.225Z · LW(p) · GW(p)
Erm, the students are not expected to understand the math, and are not being tested on their understanding of the math. The professor doesn't understand the math either. I mean that there is epsilon chance that any given psychology professor, especially an educational psychology professor, has ever heard the phrase "sigma algebra". If they have, it's because they're math hobbyists, not because it's ever come up in their professional work.
In a psychology course, "runs a multiple regression" means "follows a specific procedure analogous to a computer program". The whole thing is a black box. The decision about when it's valid to use that procedure is made based on various rules of thumb, which are passed around mostly as folklore, and are themselves understood and followed with varying degrees of conscienciousness. The same applies to the question of what the results mean.
It's absolutely a valid criticism that people in fields like psychology tend to misapply statistical methods and misunderstand statistical results. They do need an intuitive understanding of what they're doing, good enough to know when they can apply the method and what the results actually show. And it's true that most of them probably don't have that understanding.
On the other hand, it's also true that you don't need to understand the math at a very deep level to use the techniques practically. They don't need to be able to derive the method from first principals, nor to be able to rigorously prove that everything works, nor to recognize pathological corner cases that will literally never be encountered in real applications. Those are unreasonable things to ask. Remember that their goal is to understand psychology, not to understand mathematics.
Students in all kinds of science and engineering are allowed to use instruments that they couldn't build themselves, and practitioners still more so. They're not expected to understand every possible corner-case limitation of those instruments, either. At most they're given some rules for when they can or can't use and rely on an instrument.
It's still a really lame question, though, and the fact that it's asked does show a problem. Nobody seems to be looking for even an intuitive grasp of all the stuff that's lurking in that word "expect".
comment by Pattern · 2020-03-29T05:36:29.643Z · LW(p) · GW(p)
It could be an attempt to be more reality based*. (If you know the names of processes models are produced by, you can start to learn about them. Predictive models could start out more simply, and the course could have sophisticated pre-requisites.) But perhaps models coming from somewhere is a start, not an end.
*Like calculus problem that "involve" physics.
comment by Dagon · 2020-03-29T17:04:12.536Z · LW(p) · GW(p)
The saddest part is how many students will miss it. Does Open University publish aggregate scores or details of pass/fail rates?
This probably gives the test author too much credit, but they MIGHT be trying to test whether a student can ignore the irrelevant details and just do the calculation.
Replies from: Ericfcomment by lsusr · 2020-06-04T19:03:08.876Z · LW(p) · GW(p)
I once counted the number of facts per minute in a psychology lecture and a chemistry lecture at the same university. The information density differed by an order of magnitude. This wasn't because the subject with harder individual facts had fewer of them. It was the other way around.
Replies from: Ericfcomment by shannon (shannon-1) · 2022-10-20T13:56:12.085Z · LW(p) · GW(p)
I know the answer is 42.63, but I don't know how this was calculated can someone explain, please?