Super-forecasters as a service
post by frcassarino · 2021-02-12T13:35:19.062Z · LW · GW · 3 commentsThis is a link post for https://federicorcassarino.substack.com/p/super-forecasters-as-a-service
I've just started building a website yesterday that I think would be super interesting, but I'm not sure if it's legal or if Metaculus/Elicit would be fine with it.
The idea: Super-forecasters as a Service.
Say I wanted to know if I should book a flight to Costa Rica this summer, but I'm hesitant because of Covid flight restriction uncertainty. I could create the following question using Elicit: "Will commercial flight BA2490 from the UK to Costa Rica in July 24 be cancelled?"
The website would let you embed your elicit question, and pay people for predicting. People get paid based on their Points-per-question in Metaculus. You can "auction" 100 dollars on a question, and pick a base price per prediction multiplied by the Points-per-question metric.
The more money you auction, the more predictions you'll get. The higher the multiplier, the more predictions will tend to come from top-predictors.
Benefits:
- It would give all of us access to predictions from the top forecasters in Metaculus.
- It would incentivize people to become better predictors in Metaculus, so they can get paid more per prediction in my website.
- It would lend credibility to prediction markets [LW · GW], if the best predictors are making money for predicting.
Drawbacks:
- It could encourage more people to game Metaculus.
- It would not be as efficient as a straight prediction market, as people have no incentive to make an effort to make good predictions in this website, as there's no scoring. They are incentivized to make as many predictions as possible to make more money.
Questions:
- I'm I right to assume that this would be legal, as I'm paying people a fixed fee per prediction (based on their Metaculus score), rather than paying based on correctly predicting the question?
- Would Metaculus be fine with this idea? Would they be fine with me scraping https://metaculusextras.com/points_per_question?page=1 and using that data to determine the score for each person?
- Would Elicit be fine with this idea? Would they be willing to give me access to their API, or a way to embed questions directly on my Website? By the way, I tried to reach out to you folks by joining your Slack, but it seems to be invite only.
Originally posted to Substack: https://federicorcassarino.substack.com
3 comments
Comments sorted by top scores.
comment by ChristianKl · 2021-02-12T21:58:47.413Z · LW(p) · GW(p)
A prediction by a good predictor that's made in one minute isn't as valuable as one made by spending a few hours of research. "What will be my income in 2025 if I switch to X career?" is a question for which making a good forcast not only needs forcasting skills but also research or a lot of existing knowledge.
The ranked list doesn't tell you whether or not someone is a superforcasters. Metaculus could tell you but they don't offer that kind of information publically. They use predictive skill to calculate their metaculus prediction in a way that's completely separate from the ranking score.
Superforcasting as a service also exists but more in a consulting model that sells a higher quality product for a higher price.
comment by ojno · 2021-02-12T13:49:58.765Z · LW(p) · GW(p)
Isn't singling out individual market participants as 'superforecasters', and privileging their individual predictions based on past performance, missing the whole point of prediction markets?
Replies from: frcassarino↑ comment by frcassarino · 2021-02-12T13:56:16.578Z · LW(p) · GW(p)
Yes. I would still use it though. For grand-scheme-of-things stuff, Metaculus is great. For stuff that's personally relevant (What will be my income in 2025 if I switch to X career?) predictions from people with great track records is good enough for me.
Note that the title is super-forecasters as a service, not prediction-markets as a service.