You Are Not Hiring the Top 1%
post by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2007-03-02T05:34:27.000Z · LW · GW · Legacy · 12 commentsContents
12 comments
Today's statistical fallacy (slightly redacted by editor) comes from Joel on Software:
Everyone thinks they're hiring the top 1%. Martin Fowler said, "We are still working hard to hire only the very top fraction of software developers (the target is around the top 0.5 to 1%)." I hear this from almost every software company. "We hire the top 1% or less," they all say. Could they all be hiring the top 1%? Where are all the other 99%? General Motors?
When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%? Think about what happens to the other 199 that you didn't hire. They go look for another job.
The entire world could consist of 1,000,000 programmers, of whom the worst 199 keep applying for every job and never getting them, but the best 999,801 always get jobs as soon as they apply for one. So every time a job is listed the 199 losers apply, as usual, and one guy from the pool of 999,801 applies, and he gets the job, of course, because he's the best, and now, in this contrived example, every employer thinks they're getting the top 0.5% when they're actually getting the top 99.9801%.
I'm exaggerating a lot, but the point is, when you select 1 out of 200 applicants, the other 199 don't give up and go into plumbing (although I wish they would... plumbers are impossible to find). They apply again somewhere else, and contribute to some other employer's self-delusions about how selective they are.
This might explain some other phenomena I've heard of, such as the "slush heap" of inconceivably awful stories received in the mail by every fiction publisher that accepts unsolicited manuscripts. When a story is good enough to be published, it's accepted and removed from the system. Otherwise the hapless author sends it to another publisher!
Perhaps most novice writers aren't quite as dreadful (on average) as editors seem to believe? Editors will disproportionately encounter the work of novice writers who are not only awkward, but so incompetent as to be unaware of their own incompetence, and so proud that they can't take a hint.
PS: Two other areas where this might apply: Students who apply to your program/department for admission. And, grant proposals.
PPS: Robert Scarth comments, "This might also explain why some women often think that 'all men are bastards' - there are a few cads out there in continual circulation and with no intention to settle down, whereas men with honourable intentions have much fewer relationships, and settle down more quickly."
12 comments
Comments sorted by oldest first, as this post is from before comment nesting was available (around 2009-02-27).
comment by JMG3Y · 2007-03-02T14:34:54.000Z · LW(p) · GW(p)
And papers. First, it goes to that top journal, the one with the highest citation index, on a wing and a prayer. Then when the almost inevitable rejection slip comes, the paper is reformatted, the reviewers comments are sort of addressed, and goes to the next journal on the list. And so on until ultimately, unless it is really a turkey, it is provisionally accepted, subject to addressing the reviewers' comments at least in the editor's mind. Again the comments are addressed and it becomes another line on the CV.
On the other hand, some of the 20% in "Teaching vs Research" thread on Greg Mankiw's blog were rejected over and over because they were such paradigm shifts. Sort of like picking the cream of the crop students. Some who are picked are duds (false positives) and some of the rejected ones go on to do great work at other institutions (false negatives), leaving one to wonder how good the selection process really is but not being able to do any really good controlled trials to find out.
comment by rcriii · 2007-03-02T19:40:30.000Z · LW(p) · GW(p)
When you get 200 resumes, and hire the best person, does that mean you're hiring the top 0.5%? Think about what happens to the other 199 that you didn't hire.
This assumes that you (1) know what constitutes the best person, (2) that you are even capable of identifying that person, and (3) that the best person for you is the best for all.
To me the problem in hiring is not finding the best 1% (or 5 or 10%), but sorting the good from the bad. Once you have the good, then the problem of which are good for you almost sorts itself out.
comment by Eliezer Yudkowsky (Eliezer_Yudkowsky) · 2007-03-02T19:51:49.000Z · LW(p) · GW(p)
rcriii, the original "Joel on Software" post discusses that part.
comment by Robert_Scarth · 2007-03-02T20:34:02.000Z · LW(p) · GW(p)
This might also explain why some women often think that "all men are bastards" - there are a few cads out there in continual circulation and with no intention to settle down, whereas men with honourable intentions have much fewer relationships, and settle down more quickly.
comment by Doug_S.2 · 2007-03-03T04:29:31.000Z · LW(p) · GW(p)
There are an awful lot of awful novice writers out there, though; just go to www.fanfiction.net and start reading. On the other hand, there are a couple of really brilliant writers there, too.
Replies from: Matt_Simpson↑ comment by Matt_Simpson · 2011-05-03T05:10:32.289Z · LW(p) · GW(p)
Looks at date of comment. Heh.
comment by Stuart_Armstrong · 2007-03-03T13:30:56.000Z · LW(p) · GW(p)
Any company who think they are hiring the top 1% are deluded or maybe microsoft. Any company that actually try and really hire the top 1% are heading for swift bankruptcy - you can't afford to take the time and effort to get "the best" of anything in business; you have to go with something that's good and fast.
But people do give up after a while if they can't get anywhere (especially in academia). Maybe you can help overcome the bias if there is some way of measuring how often people have applied in the past?
Replies from: taryneastcomment by Alan2 · 2009-05-18T08:31:32.000Z · LW(p) · GW(p)
As a writer, I know the slush pile example is a good one. But I always try to edit and improve a rejected story before sending it on, especially if I've had constructive comments from the editor. Hopefully enough people do this to skew the bias back again, if only a little bit!