Footage of the Michael Fox lecture

Thank you for linking to the original paper. I read it, and I no longer think this was a completely valid experiment. Or, at least, it is not as strong as it seems to be.

The questionnaire that they gave subjects read:

• Did he dwell upon the obvious?
• Did he seem interested in his subject?
• Did he use enough examples to clarify his material?
• Did he present his material in a well-organized form?
• Did he stimulate your thinking?
• Did he put his material across in an interesting way?
• Have you read any of this speaker's publications?

Few of these seem completely incompatible with illogical nonsense. I'm sure he sounded interested in his subject, and for all I know he used lots of examples and put things across in an interesting way (Something like "educating physicians is much like hunting tigers, because of the part with the stethoscopes" is interesting and provides examples, but still total nonsense).

Another section asked for written comments, and they received comments like "Enjoyed listening", "Has warm manner", "Good flow, seems enthusiastic", all of which I'm sure were true, as well as a few like "Too intellectual a presentation", "left out relevant examples", and my favorite, "He misses the last few phrases which I believe would have tied together his ideas for me." These last ones seem to me like face-saving ways of saying "I didn't actually understand the slightest bit of what he was talking about".

If you have a really nice, warm presenter, probably after a lot of old stuffy guys who have bored everyone in the conference to death, and you ask for evaluations that don't really ask the questions you're interested in but give enough waffle room to allow respondents to praise the presentation they don't understand, I'm not at all surprised that people would do that.

Why, oh why, couldn't the experimenters have included a simple "Did you or did you not understand what this man was talking about?" It almost seems suspicious, like they were worried they wouldn't get as interesting a result.

...or maybe it's just normal incompetence. I have this same problem with course evaluations in my own university: they consist entirely of closed questions on peripheral issues that force me to end up giving very positive evaluations to awful classes. For example, it might ask me to rate from 1 to 5 the answers to lots of questions like "Was the professor always available to help students?" and "Was the work load reasonable?" and other things I am forced to admit were satisfactory, but nothing like "Did the professor drone on in a monotone about his own research interests for two hours a day and never actually get to covering the course material?"

(BTW, feel free not to vote this up. I'm just relaying a link I found interesting.)

I'd prefer that links with short explanations go in the Discussion area, and that Main be reserved for more substantial pieces.

I think the authors of this experiment were afraid to draw the correct (and obvious) conclusions from it. If a phony "expert" doesn't immediately stand out among the "real" ones during collaborative expert work, there are only two plausible hypotheses. The milder one is that the particular business transacted there is just a phony pretense, and the stronger one is that the entire field of expertise is as phony as a three dollar bill.

The experimenters, however, themselves have prominent positions within the same system, so instead we get a rambling discussion that evades the obvious issues.

Yeah, I'll give a high performance evaluation if I enjoyed the course, and I can enjoy the course either because I learned something or because the instructor is a skilled performance artist. I don't think the issue is that I can't tell I'm not learning anything - I think the issue is that I'm inclined to reward things other than teaching me things even in an environment ostensibly about learning.

Is there a video of the full lecture?

I know somebody is going to say, "Teachers and philosophers - so what?" So I'll say it first, and request that anybody else wishing to harp on the point provide some data showing that teachers and philosophers are not very smart or not very critical.