How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage

Broke: Prediction markets are an aggregation of opinions, weighted towards informed opinions by smart people, and are therefore a trustworthy forecasting tool on any question.

Woke: Prediction markets are MMOs set in a fantasy world where, if someone is Wrong On The Internet, you can take their lunch money.

Hi! I’m the person that wrote that Manifold newsletter. My opinion: you’re totally right, and it was sloppy to include that as justification that Whitmer was a reasonable choice.

In the counterfactual universe where that conditional market didn’t exist, I might have written that section anyway; it started because I personally believed that a Harris-Whitmer ticket would have about those odds, and it just so happened that the conditional market agreed so I cited it. (In my head it was fine because Whitmer had some chance to be the VP in the past, so the odds couldn’t be totally garbage, but any sense of internal ranking among the contenders was pretty silly.) I’m glad I gave it some disclaimer, but I think it wasn’t a strong enough disclaimer.

As mentioned below, I think conditional markets aren’t always useless; especially those with a >25% chance of occurring and where the correlation-causation issues aren’t too severe. But I’m embarrassed Manifold was promoting this one so much, and I’ll try to fight to make the site more intellectually honest going forward.

Thanks for raising this issue! (…and Whitmer’s probability)

Isn't the fact that Manifold is not really a real-money prediction market very important here? If there was real money on the table, for example, it's less likely that the 1/1/26 market would have been "forgotten" -- the original traders would have had money on the line to discipline their attention.

Every time someone calls Manifold (or Metaculus) a "prediction market", god kills an arbitrageur [even though both platforms are still great!].

The title "How I Learned To Stop Trusting Prediction Markets and Love the Arbitrage" isn't appropriate for the content of the post. That there is a play-money prediction market in which it costs very little to make the prices on conditional questions very wrong does not provide significant reasons to trust prediction markets less. That this post got 193 karma leading me to see it 2 months later is a sign of bad-voting IMO. (There are many far better, more important posts that get far less karma.)

In conditional prediction markets with exclusive conditionals, it should be possible to let people invest $N and then have $N to trade with in each of the markets.

(I think Manifold considered adding this functionality at some point.)

I made that market. Some thoughts

1.Seems kind of inaccurate to not put in Matt's tweet particularly if you're gonna call it "objective sounding".

 

Matt himself calls says "seriously but not literally". So he agrees with you, I think. 

2.Regarding fees on conditional markets, I don't know.

3.

all but a very few markets are pure popularity contests, dominated by those who don't mind locking up their mana for a month for a guaranteed 1% loss.

I don't know that this is as big a problem as it seems. A popularity contest where it costs something to vote is a better kind than we usually see. Overall I agree that conditional markets aren't to be taken too seriously. But I think the tone of this is probably too negative to this audience. This one went fine.

4.

Out of epistemic cooperativeness as well as annoyance, I spent small amounts of mana on the markets where it was cheap to reset implausible odds closer to Harris' overall odds of victory.

Thanks. I did the same. Overall I thought the markets seemed to say pretty sensible things.

I learned this lesson looking at the conditional probabilities of candidates winning given they were nominated in 2016, where the candidates with less than about 10% chance of being the nominee had conditional probabilities with noise between 0 and 100%. And this was on the thickly traded real-money markets of Betfair! I personally engage in, and also recommend, just kinda throwing out any conditional probabilities that look like this, unless you have some reason to believe it's not just noise. 

Another place this causes problems is in the infinitely-useful-if-they-could-possibly-work decision markets, where you want to be able to evaluate counterfactual decisions, except these are counterfactuals so you don't make the decision so there's no liquidity and it can take any value.

These problems seem less acute if you instead break apart conditionals into a base question and a conjunctive market (like this one for the Democratic VP question), where you don't have the possibility of a N/A resolution blowing a hole in the pricing.

I think the solution to this particular problem is to report the liquidity of conditional markets weighted by the probability of the condition being satisfied, and for people to only care about high liquidity markets.

I read your post but didn't understand it immediately. After thinking about it, I wrote what I feel the main points are in my own words. Maybe this makes it more clear to others too. Or maybe the OP will comment that I misunderstood something.

The "Harris market" can be seen as (is functionally equivalent to) a collection of markets. Each market has the following form:

Would Harris win, if {person} were her Vice Presidential nominee in 2024?

Each market resolves as follows:

• Yes, if {person} is the nominee and Harris wins.
• No, if {person} is the nominee and Harris loses.
• N/A, if {person} not the nominee.

Each market is independent. I mean this in the sense that there is no manifold mechanic that ties them together, except that they are visually grouped under "the Harris market".

There is a problem with this kind of market. When people see a high probability of a person in the market, they mistake it as meaning that the person is a likely choice or a good choice.

This is a mistake because it does not follow from the market's incentives. Imagine someone that will obviously not be nominee, like "Batman". You know that Batman will not be nominee, so you know that this market will resolve to N/A. This makes bets on Batman meaningless. You can bet Yes or No, it doesn't matter. The market will resolve to N/A and you get your money back. Some people do this because they find it fun to see that Batman has a high Yes probability.

Batman can't be nominee because he doesn't exist, but you have the same problem for obviously terrible candidates. Imagine the person most hated by all potential Harris voters, "the Joker". (Assume Joker was a real person.) The true probability of Harris winning with Joker is tiny. Yet, in the market, there is little incentive to correctly predict this. Everyone knows Joker is a terrible choice, so Harris is not going to choose him as nominee, so the market will resolve N/A.

The author of the Substack you quote makes this mistake. They did not consider what the market incentivizes.

They also made a separate mistake that applies to all prediction markets, not just misleading conditional markets. They did not take into account how much money it takes to intentionally influence the market. It is easy (cheap) for people to push small markets (markets with little money invested) in their desired direction.

And it underscores for me that conditional prediction markets should almost never be taken seriously, and indicates that only the most liquid markets in general should ever be cited.

This conclusion seems too strong. For almost all of those options the probability of resolving N/A was very high, so the incentive to correct the price was low. That's not true of many conditional markets, like "If Republicans/Democrats win the presidency, will border crossings be above [number]", and then the probability of resolving N/A is more like 50% and the incentive to correct prices remains. Does that really justify "almost never" taking conditional markets seriously? Same for "only the most liquid markets" "should ever be cited" - the market I think you said that people had forgotten about only had about 22 traders when it was at 60%, which isn't a lot. I don't see why this means markets that aren't the most liquid but have 500 traders, or markets on polymarket that have ten thousand $ in volume (small for polymarket) should not ever be cited. Manifold markets do have problems with accuracy but it's not that bad.

So the sharps would have no reason to get involved if even one of the contenders has numbers that are off by a couple points from a sane prior.

Yes 50% chance of N/A would add a 'couple points' of noise, but that doesn't make the market useless if it shows a larger difference. In the market you took a screenshot of Shapiro appears to have been at a reasonable 46% for most of the market's existence.

Conditional prediction markets could resolve to the available options weighted by calibration on similar subjects of the holders in unconditional markets, rather than N/A. Such markets might end up looking like predicting what well-calibrated people will pick, or following on after they bet (implying not expecting significant well-calibrated disagreement). Well-calibrated people could then expect to earn a profit by betting in conditional markets if they bet closer to the consensus than the market does, partly weighted in their favor for being better calibrated relative to the whole market.

The LessWrong Review [? · GW] runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?