Prediction Markets Don't Reveal The Territory

This is basically the same problem as Gears vs Behavior [LW · GW], specialized to the context of prediction markets. To a large extent, we can use prediction markets to pull out insights into system gears using tricks similar to those discussed in that piece. In particular, causal models [? · GW] are easily adapted to prediction markets: just use conditional bets, which only activate when certain conditions are satisfied. Robin Hanson talks about these fairly often; they're central to a lot of his ideas about prediction-market-driven decision-making systems (see e.g. here).

If the point is to not just know what the market says, but to know how the world works, then prediction markets in themselves may not be of much help. Here are two quick examples to demonstrate illustrate the point:

They can also be made conditional.

My other comment was about the ambitious side of prediction markets. This will be about the unambitious side, how they don't have to do much to be better than the status quo.

above technical problems. We are optimistic that these can be overcome.

What problems do you mean, the paragraphs one and three before? That could be clearer. Are you really optimistic, or is this apophasis in which you deniably assert problems? Well, I'm going to talk about them anyway.

ad­di­tional limi­ta­tion to pre­dic­tion mar­kets is that peo­ple have to be in­ter­ested enough to take part in them

Robin Hanson has always said you get what you pay for. If information is valuable to you, pay for it by subsidizing the market. Betting markets aren't free, but are they cheaper or more accurate than the alternative? Start with things that bettors care about, like politics.

Pope Fran­cis’ next pro­nounce­ment

Having a market on his next pronouncement would encourage leaks. I'm not sure whether that would be good or bad. Having a market for the first papal pronouncement of 2021 that closed a year ahead probably wouldn't produce leaks. Nor would it produce a precise answer, but it would produce some kind of average that might be interesting. For comparison, the Nobels rarely leak, so the markets don't vary much from year to year. Is it useful to know that Haruki Murakami is usually at the top of the list? Some people are skeptical, though.

When in­for­ma­tion is not widely dis­tributed or dis­cov­er­able, pre­dic­tion mar­kets are not use­ful. Pre­dic­tion mar­kets for WMDs

Those are two problems and they apply both to WMDs.

As for discoverability, in 2000 you could have a market over what inspectors would find in a year. You could also have a market over what inspectors would find in a decade. You could imagine a market over what would be the consensus in 2010, but it is more speculative how that would work. In 2002 it would be straightforward to have a conditional market, conditional on invasion. I hope that simply setting up a market would have encouraged precision, such as chemical vs nuclear, stockpiles vs production, and quantity. Such distinctions seem like an easy way to improve the public debate.

As for wide distribution, so what? We want an opinion, even if it is not very certain. In fact, open sources should have been enough to beat the CIA in Iraq. Partly that is because the CIA is incompetent, but partly it is because the CIA is not on our side. I think that open source amateurs have done a pretty good job of predicting the North Korean nuclear missile program. How well did Intrade do in predicting North Korea missile tests in 2006? I don't know, but they did a lot better than the DOD at postdiction. (In fact, I was somewhat surprised that the administration accepted a lack of WMD in Iraq and did not fabricate them.)

Of course, we don't have direct access to the territory, only the map. Prediction markets can only be judged by the future map. I am extremely pessimistic about our ability to create a collective map, so I think prediction markets have only a very low bar to clear. From your user name, you sound like a scholastic apologist, whereas I am very cynical about the schools. I don't dispute that they house expertise, but they abuse that position, by, among many other things simply lying about the consensus [LW · GW] in their field. A very simple step forward would be to use surveys to assess consensus. And when fields interact, it is even worse. As I've said elsewhere:

1. If you want to make an Appeal to the Authority of the Scientific Consensus, you have to be able to figure out what that consensus is. What does it matter that nutritionists have a good consensus if you don’t know what it is, and instead believe the medical school curriculum? Similarly, your psychology textbook lied to you.

2. It is very common, both in your economics example, and in the Nurture Assumption case, that there is an explicit consensus that method X is better than method Y, but people will just keep using method Y. It seems very fair to me to describe that as an implicit consensus that method Y is good enough. Moreover, it is common that the consensus accepts the aggregate results of the inferior method, just because of the volume of publications; and thus the explicit consensus on the object level is made by methods that violate the explicit consensus of methodology. (Also, most of the replication crisis details were discussed at length by leading psychologist Paul Meehl fifty years ago. Everyone abased themselves in shame and proceeded to change nothing. Is this time different?)

3. You might say that (1) is a special case of (2). Everyone accepts that you should look to experts with good methods, but they don’t actually pay attention to nutritionists. I think that it is fair to call the exclusion of nutritionists “the scientific consensus.”

A lot of information technologies provide value simply by creating conflict. The internet makes it easy to find people who disagree with you, if you want to. Wikipedia provides a focal point for disagreeing parties to fight over, forcing both sides to acknowledge the other's existence, making it easy for ignorant amateurs to notice the breakdown of consensus. Similarly, prediction markets provide opportunity for disagreement on a more fine-grained level.

And let me close with a less hostile, more amusing example [LW · GW] of lack of consensus.

There's a lot I don't follow here. In particular, you say a bunch of things and it's not clear if you think that they are the same thing or related or unrelated. Some of that may be the excerpt nature.

What does the "territory" of the title mean? Snappy titles are good, but you should also explain the metaphor. Perhaps you mean the laws of science, rather than the concrete observations or even interventional claims about specific experiments? Robin Hanson's response is: Just Do It: make decade long bets on vaguely worded claims. He proposes lots of infrastructure to fix these problems and it doesn't seem very convincing to me, but the proposal seems built incrementally, so it is easy to start small.

What does it matter if prediction markets don't do X? If people are proposing prediction markets as an additional institution, then it matters what they do, rather than what they don't do. If they are proposed as a substitute for existing institutions, then it matters if they are as good as the existing ones. But there is a serious instance of status quo bias that people pretend that existing institutions work, whereas they often don't. Seemingly unambitious that work may well be an improvement over ambitious institutions that don't. Robin Hanson does propose substituting prizes for research grants, so there he would have to make that argument. But research funding is highly divisible, so it is easy to start small and see what happens.

Prediction markets are really fascinating. You should also touch on the idea of assassination markets, which I'd say are a subset of prediction markets where the contract price can influence the likelihood of the outcome.