How well did Manifold predict GPT-4?

post by David Chee (david-chee) · 2023-03-15T23:19:06.477Z · LW · GW · 5 comments

Contents

  How well did we predict the launch date?
  Insider Trading
  What else are people predicting about GPT-4?
  Markets you can still predict on
None
5 comments

Chat GPT-4 is already here!! Who could have seen that coming… oh wait Manifold (kinda) did? 😅

I thought I’d write a short piece on how Manifold Markets was used to predict the launch of GPT-4 and its attributes. Both its successes and its failures. Disclaimer I work at Manifold.

How well did we predict the launch date?

Throughout the end of last year, people were bullish on a quick release, which began to decline as we entered the start of this year.

The first spike in February corresponds to the release of Bing’s chatbot which people speculated was Chat CPT-4. Turns out it actually was! Although Open AI did a fantastic job at concealing this with our market on it hovering at a stubborn 50-60%.

There was a lot of uncertainty on if GPT-4 would be released before March. However, on the 9th of March Microsoft Germany CTO Andreas Braun mentioned at an AI kickoff event that its release was imminent which caused the market to jump.

Although the market graphs are a beautiful representation of hundreds of traders’ predictions, did they actually give us any meaningful information? One thing that stands out about these graphs in particular is the strong bets away from the baseline towards YES throughout February. Is this just noise, or is something more going on?

Insider Trading

Being the socialite I am, I go to a whopping one (1) social gathering a month!! At 100% of these, the SF Manifold Markets party and Nathan Young’s Thursday dinner, I spoke to someone who claimed they were trading on the Chat GPT-4 markets based on privileged insider information.

One of them got burnt as allegedly there were delays from the planned launch and they had gone all-in on the GPT-4 being released by a certain date.

I love knowing people with privileged information are able to safely contribute to public forecasts which wouldn’t be possible without a site like Manifold Markets. As they were trading from anonymous accounts I have no way of knowing whether they are the ones responsible for the large YES bets, but I suspect some of them are. That said, someone with insider knowledge would be better off placing a large limit order to buy YES just above the current baseline which would exert strong pressure to hold the market at/slightly above its current probability. Placing a large market order which causes the spikes gives them less profit than they otherwise could have earned.

What else are people predicting about GPT-4?

Jacy Reese Anthis, an American social scientist of the Sentience Institute, created a market on if credible individuals with expertise in the space will claim GPT-4 is sentient. 16% seems surprisingly high to me, but the market has only just been created and needs more traders. Go now and place your bets!

One of our most popular markets, which failed in spectacular fashion, was whether it would get the Monty Fall problem correct (note - this is not the same as the Monty Call problem, click through to the market description for an explanation).

This might be the single most consistent upward-trending market I have ever seen on our site. I wonder if GPT-4 hadn’t been released yet how much further it would have continued to trend upwards before plateauing.

Part of the confidence came from Bing’s success in answering correctly when set to precise mode. Many speculated GPT-4 was going to be even more powerful than Bing, even though they turned out to be the same. I’m not exactly sure what the difference is using the “precise” setting, if anyone knows let me know!

Markets you can still predict on

Here are some more open markets for you to go trade-in. It’s free and uses play money!

Thanks for reading! Hope it was interesting to see the trends on Manifold, even if not a particularly in-depth analysis this time around.

5 comments

Comments sorted by top scores.