Can we hold intellectuals to similar public standards as athletes?

Prediction-tracking systems, particularly prediction markets with dedicated virtual currencies, seem like an obvious contender here. One could imagine the scope of those being expanded to more fields. So I'm going to focus on what's not captured or inadequately captured by them:

• The book Superforecasters notes that generating good questions is as important as predicting answers to questions effectively, but prediction-tracking is completely unable to assign credit to the thinkers who generate good questions.
• This is similar to the idea that we should judge people separately on their ability to generate hypotheses and their ability to accurately score hypotheses. Hypothesis generation is incredibly valuable, and we would not want to optimize too much for correctness at the expense of that. This is an idea I attribute to Scott Garrabrant.
• Another important idea from Scott is epistemic tenure [LW · GW]. This idea argues against scoring an intellectual in too fine-grained a way. If a person's score can go down too easily (for example, if recent work is heavily weighted), this could create a fear of putting bad ideas out there, which could severely dampen creativity.
• Ability to produce proofs and strong evidence seems potentially under-valued. For example, if a conjecture is believed to be highly likely, producing a proof would enable you to get a few points on a prediction-tracker (since you could move your probability to 100% before anyone else), but this would severely underrate your contribution.
• Ability to contribute to the thought process seems under-valued. Imagine a prediction market with an attached forum. Someone might contribute heavily to the discussions, in a way which greatly increases the predictive accuracy of top users, without being able to score any points for themselves.

I asked a tangentially related question [LW · GW] two years ago (to the day, as it happens). I made the comparison with an eye to performance maintenance/enhancement, rather than accountability, but it feels like they are two sides of the same coin.

I disagree with most of the other commenters about the problems of generating metrics. The simplest answer is that a lot of metric-style things are already known and used when comparing two different academics aside from citations or IQ, such as:

• Age of first degree
• Age of first publication
• Scores on standardized tests, like SAT/ACT/GRE
• Placement in national competitions/exams
• Patents with their name
• Patents based on their work

These are the kinds of things people reach for when comparing historical academics, for example. Then you could highlight other performance metrics that aren't necessarily their core area of expertise, but indicate a more general ability:

• Publications outside their core specialization
• Credentials outside their core specialization
• Correctness of grammar/spelling of the languages in which they publish
• Number of languages spoken
• Number of languages in which they published
• Success of their students

Then you can consider subjective and informal things:

• Evidence of reputation among their peers
• Popularity of talks or lectures not about one of their own publications

For example, consider that Von Neumann is considered a strong candidate for the smartest human ever to live, and it seems to me when this is being discussed the centerpiece of the argument boils down to all the other legendary minds of the day saying something to the effect of "this guy makes me feel like an idiot" in private correspondence.

But this information is all buried in biographies and curricula vitae. It isn't gathered and tracked systematically, largely because it seems like there is no body incentivized to do so. This is what I see as the crux of it; there is no intellectual institution with similar incentives to the sports leagues or ESPN.

I like the thought. Though unlike sports, intellectual work seems fundamentally open-ended, and therefore doesn't seem to allow for easy metrics. Intellectuals aren't the ones playing the game, they're the ones figuring out the rules of the game. I think that's why it often is better to focus on the ideas rather than the people.

A similar question also applies within academia. There, citation counts already serve as a metric to measure intellectual accomplishment. The goodharting of that metric probably can tell you a lot about the challenges such a system would face. What metrics do you have in mind?

In a way, this problem is just scaling up the old reputation/prestige system. You find people you respect and trust, and then you see who they respect and trust, and so on. While regularly checking for local validity [LW · GW] of course. Maybe some kind of social app inspired by liquid democracy/quadratic voting might work? You enter how much you trust and respect a few public intellectuals (who themselves entered how much they trust other intellectuals) and then it computes how much you can trust everyone else.

I think this is fundamentally not possible, because the world does not come pre-labeled with rules and win/lose conditions the way a sport or game does. Any attempt to do this would require you to take certain concepts as presumptively valid and unquestionable, but the main point of being an intellectual is to question accepted concepts and develop new ones.

Baseball and American sports in general are a bit of an exception though. Most sports are much more difficult to evaluate statistically. For example in chess the elo ranking offer good prediction of average performances but fails totally to evaluate 1) head-to-head win rate, 2) one-game results or even tournaments results, 3) relative strength on different parts of the game (for example "who is good at opening preparation ?" is typically answered by the same sort of heuristics as "who is good at maths" or "who is good at psychology ?".

Somewhere I read that a big reason IQ tests aren't all that popular is because when they were first introduced, lots of intellectuals took them and didn't score all that high.  I'm hoping prediction markets don't meet a similar fate.

Here is a quick list of things that spring to mind when I evaluate intellectuals. Any score does not necessarily need to cash out in a ranking. There are different types of intellectuals that play different purposes in the tapestry of the life of the mind. 

How specialized is this person's knowledge?
What are the areas outside of specialization that this person has above average knowledge about?
How good is this person at writing/arguing/debating in favor of their own case?
How good is this person at characterizing the case of other people?
What are this person's biggest weaknesses both personally and intellectually?
 

I enjoyed writing this post, but think it was one of my lesser posts. It's pretty ranty and doesn't bring much real factual evidence. I think people liked it because it was very straightforward, but I personally think it was a bit over-rated (compared to other posts of mine, and many posts of others). 

I think it fills a niche (quick takes have their place), and some of the discussion was good. 

An objection might be that athletes have good metrics. But this ignores selection effects. Athletes have good metrics in part because there is money to be made in finding good metrics and then arbitraging against teams who use worse metrics. See Moneyball. Universities do draft elite intellectuals, and that is partially based on how much money that intellectual will be able to bring to that university. University donors want to affiliate with elite intellectuals. 

Even someone who scores terribly on most objective metrics because of e.g. miscalibration can still be insightful if you know how and when to take their claims with a grain of salt. I think making calls on who is a good thinker is always going to require some good judgment, though not as much good judgment as it would take to form an opinion on the issues directly. My sense is there's more returns to be had from aggregating and doing AI/statistics on such judgment calls (and visualizing the results) than from trying to replace the judgment calls with objective metrics.

You'd have to narrow and categorize different domains of intellectualism - the VAST majority of athletes don't get any attention at all, and those that do comprise a very small volume of athelticism-space.  And you'd have to turn it into a contest that median humans could identify with.

I don't think either of these changes would make things better.  The top few percent of intellectuals could make a bunch of money (mostly through sponsorships, which may be unfortunate if we prefer them to be independent).  A bunch of pretty-good intellectuals would learn that they'll never make it big and be less effective than they now are.

Summary: Academia and intellectualism might be sub-optimal, but sports is horrible and should not be copied.

Other difficulties include type and durability of output.  Paid/popular sports are performances - the value is in the action and instantaneous demonstration of talent and physique.  Intellectual value (to observers) is in the output - papers, ideas, explanations, validation of models, etc.   This is both far more durable and less repeatable than sporting events.  A true meritocracy could very well lead to such dominance that we're forced to sabotage the smartest/best.

Sabermetrics basically came out of the observation that a team that used the metrics to make hiring decisions outcompetes teams that don't. It's the same as universities using citation count metrics for their tenure decisions. In both cases metrics get used to make decisisions about which individual is supposed to be hired.

The problem is that the metrics for athletics are better for their usages then the metrics the universities have available for their tenure decisions. 

Part of the standards we have for professional atheletes is that we have a lot of rules that cap their performance. Having rules that cap performance by engaging in some non-standard behavior is useful if you want to rank people by a fair metric but not useful if you care about the quality of the output. 

if I want to know how much to trust and value Jonathan Haidt I'm honestly not sure what to do

The most basic approach which most academics would use: look at the citations on his papers. Then you could see a lot of the same metrics as in baseball: how his performance has been changing over time, how good he is compared with others in the field, and so on. You could look at his h-index, or at the impact factors of the journals he's been published in. It seems odd that you didn't mention any of these, since they're such a standard part of academic evaluation.

Yes, we can hold intellectuals to similar public standards as athletes. Using GPT-4/5 we could use it to create a set of questions to check if the intellectual can answer the questions correctly avoiding every kind of bias already explained here on LW. For each bias already explained before, we can create new questions that show when a human fall on them, assigning a new score to that human. I would like each human to write down all his knowledge with the help of an automatic writing system, we could create a visual tree of all knowledge the system detected the human acquired on the past, and evaluate how well he performs in answering questions about the fields he visited/he claims to know about. What's the point of asking your credentials when I can evaluate your knowledge in real-time with GPT-n systems? 

On the tree of knowledge we could see which humans score higher in which domains and why. What are the questions they can answer that others can't. Don't ask me my credentials, ask me a hard question/ give me a hard problem to solve and let's see who solves it first or better. GPT-n could babble about the solution presented by different humans, and other group of humans that score high on these domains could also rate/evaluate the solutions by others, choosing the score they assign for each solution. 

An interesting follow-up (but I have no path to implementation, and don't even know the desirability) is "can we hold amateur opinion-holders to the same standards we hold casual sports-players?"

A whole lot of people play pickup basketball or soccer among friends, and a subset (but still large amount) play in organized non-professional leagues.  Nobody takes them very seriously.  Why do we let idiots vote?  Why do we care what they say on Facebook?

While, if successful, such an epistemic technology would be incredibly valuable, I think that the possibility of failure should give us pause. In the worst case, this effectively has the same properties as arbitrary censorship: one side "wins" and gets to decide what is legitimate, and what counts towards changing the consensus, afterwards, perhaps by manipulating the definitions of success or testability. Unlike in sports, where the thing being evaluated and the thing doing the evaluating are generally separate (the success or failure of athletes doesn't impede the abilities of statisticians, and vice versa), there is a risk that the system is both its subject and its controller.

My impression of the proposed idea is to create a "hard" intellectual accountability system for intellectuals by sampling some "falsifiable" subspace of idea space, similar to what exists for superpredictors and athletes. This certainly seems helpful in some areas, and I think is similar to the purpose of politifact.

But then there's the risk of falling into the Marxist mistake: that if something isn't quantifiable or "hard" it is not useful. The idea that "hard" production matters (farmers, factory workers, etc.) while "soft" production (merchants, market researchers, entertainers) does not, which is at the base of Marxism, has been disgraced by modern economics. But this is kind of hard to explain, and Marxism seemed "obviously correct" to the proto-rationalists of the early 20th century. 

The sphere of intellectual expertise, especially the even "softer" side of people who've taken it on themselves to digest ideas for the public, is much harder still to get right than economics. No matter how many parameters an analysis like this tries to take into account, it is likely to miss something important. While I like the idea of using something like this on the margin, to boost lower-status people who get an unusually high number of things right or flag people who are obviously full of it, I think it would be a bad idea to use something like this to replace existing systems.

This reminds me of attempts to rate the accuracy of political pundits. Maybe this was in Superforecasting? Pundits are a sort of public intellectual. I wonder if one place to start with this intellectual-sabermetrics project would be looking for predictions in the writings of other intellectuals, and evaluating them for accuracy.

Find a way to bet on the outcomes of intellectual performance, make that work public and entertaining in some way, and the bookies will figure out the rest. #slightlyGlib

I'm sure professional athletes said the same thing when public metrics began to appear. Generally new signals get push back. There will be winners and losers, and the losers fight back much harder than the winners encourage. In this case the losers would likely be the least epistemically modest of the intellectuals, a particularly nastybunch. But if signals can persist, they get accepted as part of the way things are and life moves on.

This seems to ignore the possibility of flaws in the metrics. For example, if such metrics had been created some time ago, and had included something involving p-values...

There are definitely ways of doing this poorly, but the status quo is really really bad.

In order to create a metric, you should know what the metric is for. 

Tenure? I think those metrics already exists. (For better or for worse.)

Citation stats - not a good measure by itself. (It's independent of both replication and non-replication.)

For one, Expert Political Judgement provided a fair amount of evidence for just how poorly calibrated all famous and well esteemed intellectuals seem to be.

It determined that by asking them to make predictions? If you want good predictors, then ditch the intellectuals - and get good forecasters/forecasts.

A lot of people's beliefs are influenced by their political affiliation. I'd be wary of a rating system. If one political party got ahold of it, they might use it to completely discredit their opponents, when the issue is just not very well understood in the first place and having multiple perspectives is very useful.

Climate change is the issue that comes to mind first.

The most basic approach which most academics would use: look at the citations on his papers. Then you could see a lot of the same metrics as in baseball: how his performance has been changing over time, how good he is compared with others in the field, and so on. You could look at his h-index, or at the impact factors of the journals he's been published in. It seems odd that you didn't mention any of these, since they're such a standard part of academic evaluation.