Exams and Overfitting

What's going to be on the exam?

This would hopefully have good correlation with "what's important?"

Sometimes you teach things that aren't the core of the class, but you want to expose the students to. Telling students what will be on the exam is a way of focusing their study, removing random study attentional noise from the grade.

As a professor I mostly agree with you except that professors when writing exams take into account how the kind of exams they create will impact student incentives. I might tell you that X will be on the exam because I believe my students receive a high marginal benefit from studying X, and this is, hopefully, useful information if you intend to maximize what you learn.

Manuel Blum generally has one open* problem on his exams. Manuel Blum was one of the best teachers I ever had (he taught complexity theory at Berkeley -- he's at CMU now). It is possible to get 1000 out of 100 on his exams, apparently.

• In the sense of "no one bothered writing it up, but generally you should be able to do it in a timed setting if you thought about a class problem in depth as is encouraged but not required."

Consider that if a somewhat "out of the left field" problem is on the exam, it is there to separate the upper end of the curve. One could imagine some reasons a professor might want to do that.

When I take a class, my goal is to learn the material.

Do note that it is NOT "pass the exam with the top grade".

I see no contradiction between wanting to learn and wanting to munchkin the exam. I don't have much respect for university exams.

I used to teach physics to pre-med students (a nearly-identical situation to [2] in the original post). I tried to write my exams so that simply memorizing a large set of very specific algorithms for solving a problems wouldn't work, but nobody would have to be very clever in order to get a good grade.

In addition to this, I looked at the course material and asked "Is there anything thing on here that a doctor really needs to know?". I decided it was good for doctors to know how half-lives work, since this is important for things like drug dosing, as well as probably other things I don't even know about (since anything who's rate of decay is proportional to it's value will behave the same way). So, I explained to my students that a discharging capacitor was mathematically identical to the way that some drug concentrations decrease over time, and that there absolutely, positively would be a question about it on the exam. I didn't say anything else very specific about what would be on the rest of the exam. That exam had one question about a discharging capacitor, followed by a second question that was the same as the first, but reworked in terms of drugs. Most students got the first one right, but fewer got the second.

I think that part of my distaste, as an instructor, for students knowing a lot more about what is on the exam was that I wound up talking a lot more about the same things, and it got boring.

1) I get upset when professors construct their classes such that overfitters are rewarded to the extent that the opportunity cost associated with actually learning the material might cause one to fall behind. It happens in 70% of introductory courses and 35% of advanced courses. Also, 90% of multiple choice formats are guilty of this.

2) Random samples of the material inherently reward "over-fitters" because they do a shallow memorization of everything so that they can pass, rather than the true learner who does an in-depth treatment of the course topics that interest them in particular. I'm not sure how to solve this problem.

As a post-doc who has to teach a lot of intro classes I may have some perspective on this that someone who has only been on the other end may not have.

If I know in advance what topics will be covered on the exam, and if I then prepare for the exam by learning only those topics, then I am screwing up this whole process.

Yes, up to a point. For example, last semester I told my Calc I students that there would be optimization on the exam. But optimization covers so many different aspects of calculus that telling that doesn't really narrow the set of things one should study for.

It is also worth noting that sometimes we tell people what is on exams because we're trying to actually get better feedback between different collections of students. Again, I'll use an example from last semester: the main reason I told the students there would be an optimization problem (and also told them that there would be an implicit differentiation problem and told them that there would be a graphing problem) was because I had two classes of students. However, due to scheduling issues, one had their exam on Monday and the other on Wednesday. I was trying to keep both classes on the same curve so I would have trouble if I made radically different exams. But then there was a high probability that some of the Wednesday students would find out from Monday people what was on the exam- this would both wreck keeping the curves intact and would penalize the Monday students as well Wednesday students who didn't have talkative friends in the class with the Monday exam. However, by giving a very large amount of information about what would be on the exam, this ensured that information leakage from Monday wouldn't matter.

When I take a class, my goal is to learn the material.

That does not imply that when anyone takes a class their goal is to learn the material.

Having been on both sides, my main beef is with the profs who structure their exams poorly. Specifically,

On an exam you only test the skills previously trained in the course. Anything extra should be a bonus question.

It is not uncommon to see a final exam containing problems completely unlike anything practiced in the lectures or tutorials. They are technically about the material studied, but they tend to be a level or two harder, or assume familiarity with a previously unused prerequisite, or require connections between the parts of the course never made explicit before. In this case, usually you get someone very bright figuring it all out and getting a high mark, while the rest of the class is left with barely passing and frustrated.

Of course, I have little sympathy to the students constantly asking "will this be on the test?". Every problem similar to those covered in the lectures, tutorials and assignments is fair game for the test. But it must be similar.

The point of the test is to prove that you know the material. If you're taking the class to learn the material, then the only person that you need to prove it to is yourself. You can skip the test altogether. If you also consider getting the degree to be a bonus, you might as well optimize the grade a bit.

You find what is going to be on exam, you memorize it, you pass the exam, you forget it.

Why?

Because you don't care for knowledge, you just want a diploma.

Why?

Because companies don't care for knowledge that university gives (and don't really need it), they just want to see your diploma. If you don't have it, good luck finding a decent job.

Why?

Because one who finished the university at least isn't completely dumb and lazy. If you have such a method to filter job applicants, why not use it?

That is how it worked for me. Is it different in countries where higher education is not state-funded?

Here's my rough model of your situation (which may be totally wrong): you value deep learning, and feel that the sacrifices you've made socially for it should be reflected in some sort of payoff. Basically, you want a better grade, and feel indignant because that others studying to the test is a shortcut that cheapens the effort you put in.

I think you'll be a lot happier if you decouple your learning from the grades you receive. Grades are grades, and as far as they are used to land you a better job or graduate level position, they follow their own system. If you're not gaming that system like everyone else, you're not playing well. Your learning can be something else entirely, and insofar as you think deep learning is innately valuable or will give you a competitive advantage, it has its own merits. But don't conflate the two - you seem to be projecting a lot of your frustration at having two competing goals (learning and a social life that both eat up time) onto people who have a different set of priorities.

I don't mean to sound harsh here. But your days will only ever have 24 hours, and if you can dig down through the things you want to figure out some core values, you might be able to build better tradeoffs and balance your life.

Not sure if I am saying the obvious thing, but I think the teacher should have a larger database of questions, and choose the question for the test randomly. Not necessarily in a way "here are 100 questions, choose 10 of them", but if the class consists of two equally important parts, it should be "here are 50 questions for the first part, 50 questions for the second part, choose 5 and 5". Even within each part, you could choose e.g. 2 easy questions, 2 medium-hard questions, and 1 hard question. If some questions are central to the lesson, you can make them appear on the test with certainty. On the other hand, if there are many non-central, exotic questions, you can make the test always contain exactly 1 exotic question, even if you have 100 possible exotic questions in the database.

Using this system, anytime a student asks "will this be on exam?", you can answer "with certain probability, yes". More central questions will have higher probability, different tests will be relatively balanced, but you also need to learn the exotic questions if you want to have to achieve 100% score.

Try something like this. Close the book, take your dog for a walk, and when you stop 'buzzing' with all that semi-new shiny knowledge, look around you and try to think about what you see using what you learned. Write down your questions. When you come home, calculate what proportion of your questions you have already answered in class (and so expect to turn up on the test.) If all of them, then you are either lucky, not interested in the course, or your dog needs more exercise:) if there are a few you haven't thought of before... it's worked.

I think this essay is relevant.

I don't think that overfitting is a good metaphor for your problem. Overfitting involves building a model that is more complicated than an optimal model would be. What exactly is the model here, and why do you think that learning just a subset of the course's material leads to building a more complicated model?

Instead, your example looks like a case of sampling bias. Think of the material of whole course as the whole distribution, and of the exam topics as a subset of that distribution. "Training" your brain with samples just from that subset is going to produce a learning outcome that is not likely to work well for the whole distribution.

Memorizing disconnected bits of knowledge without understanding the material - that would be a case of overfitting.

My first thought is that often times there are things worth mentioning in the lecture, but that aren't part of the material that they want to test for. But I do agree with your central points.