On Metaculus, people have attempted to solve this problem by having moderators review each question before it goes live. The result has generally been that most vague questions require substantial re-editing before they can appear on the website. 

Also, over time, users have become quite skilled at noticing when questions are likely to resolve ambiguously. 

Even with this cultural difference between the sites, many questions still resolve ambiguously on Metaculus. It's just really, really hard to say exactly what you mean when predicting the future.

David here from Manifold.

Really cool to see you taking the time to present your thoughts in a way that is constructive whilst not holding back with your criticisms. A lot of people would have just gotten frustrated with their experience on the site and have just moved on, so thank you for this post! We definitely think about these things a lot as we design Manifold and this post could definitely have an impact on where we lean within tough design problems.

First off, let me briefly discuss why we are bullish on using a creator-based resolution mechanism.

• Creator reputation. This is definitely the most important point. Regular users quickly learn which markets are run by credible creators who know to set clear criteria. And in the rare instance an unexpected variable occurs making the resolution less clear, these creators will often defer to what the comments agree should be the correct resolution. Admittedly this is definitely not obvious for new users who are interacting with our markets for the first time and are often times trading in good faith that all the markets are reliable. We are aiming to get a reputation system set up at some point, but we haven't had the capacity to do so yet
• Creator resolution facilitates types of questions that otherwise couldn't exist.
  • Fun, personal markets that often only the creator will know the results. eg. this one from James Medlock.
  • Subjective/recommendation markets. 
• Creators resolving works >95% of the time. For the majority of markets, there are no issues. Very rarely does a creator's internal bias or ulterior motive affect the resolution. It's unfortunate that the first market you interacted with may be one of the minority and we will be more conscious about ensuring big/featured markets are reliable. I strongly believe if you were to use the site more that you would find a lot of your concerns are exacerbated by your experience with an outlier. 
• Interpretation of a result is often necessary. As others have alluded to in the comments, there is often conflict between a market that is useful and answers what you want to know vs one that is extremely well-defined in advance. It will always be the case that questions come up where someone has to make a call and it makes sense that the person creating the market is the best person to do so. We have heard some good suggestions regarding crowd-sourcing resolutions so might be open to possible types of implementations of that.

Now that that's out of the way, let's discuss some of the specific criticisms you outlined. The following is more a reflection of my own thoughts and hasn't necessarily been discussed in depth with the team.

• The market probability becomes more reflective of the creator's tendencies and bias and not the event itself.
  • I will agree that this is definitely true with some of our markets. But I actually think that is okay! One thing that sets Manifold apart from other prediction markets is that we support a wide range of markets that is only possible by giving creators full autonomy. I think it is a good thing that each individual creator is expressed in their market. Not all markets have to provide actionable, informative data even if this may not align with what is traditionally sought after with forecasting. Just to be clear though, one of our primary goals is to facilitate markets that provide actionable and informative discussion and data. But, I suspect as the site grows that it will become more obvious what the intent of each individual market is. There very well could be multiple instances of the same market which will actually have different probabilities based on who is the creator and subtle differences in resolution criteria.
• Markets on Manifold are too ambiguous.
  • I don't have too much to say on this as I agree with you that our markets are generally too ambiguous and what Jack said covers a lot of my own thoughts. We do encourage creators to try and be as clear as possible, but many people are still learning how to create questions. One of the main purposes of the site is to lower the entry-barrier to prediction markets and allow anyone to create one easily so it isn't a surprise a large number of markets aren't that great or clear. Fortunately, there is already a clear culture developing where users will comment when they see unambiguity or a possible loophole in a market. And I think you would find that our big serious markets tend to be fairly unambiguous at this point.
• Manifold is untrustworthy.
  • I'm not sure I agree with this but that may definitely be my bias haha. I do think though that users should regard every market they bet on with mild scepticism and not default to trusting the creator. Now, as a site is it optimal to have a bunch of unreliable content? Maybe not, particularly for the new user experience and I can empathise with why this may cause you to mistrust the site.  But it's unrealistic to expect a site that gives its users full autonomy and does minimal curation to have only trustworthy content. But it is something we will need to think about more. Hopefully, it will be mostly solved once we introduce a reputation system of some sort. And from my experience, the number of markets that have been "untrustworthy" can be counted on one hand so far.

Thanks again for taking the time to try out our site (even if it didn't go as you expected) and for doing this write-up. Am happy to discuss things further. Maybe what you've outlined is a bigger problem than we've realised and we need to spend a lot more effort designing the site around it, but I'm not yet convinced. It is good to know though that it is turning some users away.

I agree that ambiguity is bad, and most Manifold markets are probably too imprecise and ambiguous. My usual style is trying to be fairly precise in the forecasting questions I write, and I definitely second your recommendations!

However, I want to point out that the problem isn't just ambiguity, but really complexity. The more you try to nail down the resolution criteria, the more likely it becomes for there to be a serious mismatch between the top-line summary of the question (the question that you are actually interested in answering) and the detailed decision tree used to resolve it.

Here's a great example with a real-money, CFTC-regulated prediction market "Will student loan debt be forgiven?" on Kalshi. The rules were as unambiguous as possible up-front, specifying what would count as "broad-based debt relief" and things like that - things that people would reasonably think are important details to check. Unfortunately, it turned out that there was a huge detail in the resolution criteria that the market missed - the resolution rules required that the forgiveness be done via Executive Order or law. And in this case, the debt relief was clearly done but it wasn't done by law or executive order. So lots of traders lost lots of real money, and the prediction market clearly failed at its job. Link to the market: https://kalshi.com/events/SDEBT/markets/SDEBT-23JAN01. (Note: the page has been updated after the fact to highlight this issue.) 

This is to highlight the point that writing unambiguous resolution criteria that also match the top-line summary question is really really hard! It's not uncommon on Metaculus for there to be a fairly clear subjective answer that differs from the actual resolution because of some issue that most people agree is a mere technicality. So I think that trying to disambiguate is good, but being willing to step back from technicalities is also potentially good. There are advantages to both approaches. I agree with you that the average market on Manifold is way too underspecified and ambiguous, but if you require Metaculus levels of precision then many people wouldn't be willing to spend the effort to ask the questions at all, and that is also a loss.

Traders have the option, where a question is ambiguous, of asking the resolver how they would resolve it in some hypothetical scenario. This is true on Manifold as on Metaculus. I find this is normally more profitable for me than trying to get inside the head of the resolver.

There is a separate issue of resolver reputation, where some resolvers have a history of being biased in favor of their own positions, or just getting wrong. Definitely a weakness of current Manifold.

This post has a lot of good advice that I agree with, thanks for writing it.

FYI the creator of the market decided to resolve the market to N/A.

This means everyone's mana is fully restored as if they never interacted with the market.

Strongly agree about the existence of the problem. It's something I've put a bit of thought into.

One thing I think could help, in some cases, would be to split the market definition into

1. the question definition, and
2. the resolution method

And then specify the relationship between them. For example:

Question: How many reported covid cases will there be in the US on [DATE]?

Resolution method: Look at https://covid.cdc.gov/covid-data-tracker/ a week after [DATE] for the reported values for [DATE].

Resolution notes: "Whatever values are reported that day will be used unconditionally." or "If the values change within the following week, the updated values will be used instead." or "The resolution method is a guideline only; ultimately the intent of the question, as interpreted by the question author, overrides."

This will only solve a subset of ambiguous resolutions, but I think it would still be a big help to spell some of these things out more clearly.

Markets should probably have multiple outcomes (which in politically controversial scenarios should ideally also be evaluated by different people with different biases) and allow people to make correlated bets over the outcomes.

An off-hand attempt at market mechnanism to solve this. Have "resolution issue" predictors. Like "There will be a news paper article of lenght more than 20 cm written by a full time journalist about biological weapons AND the vague biological weapons question will resolve in the negative".

(Engineer at Manifold here.) I largely agree! Letting people make markets on anything means that many people will make poorly operationalized markets. Subjective resolution is good for bets on personal events among friends, which is an important use case, but it bad for questions with an audience bigger than that.

We need to do a better job of: 
1. resolution reliability, like by adding a reputation system or letting creators delegate resolution to more objective individuals / courts.
2. helping users turn their vague uncertainties into objective questions - crucial but less straightforward.
3. surfacing higher quality content

For the "Will Russia use chemical or biological weapons in 2022?" question, the creator provided information about an ambiguous outcome, though it seems very subjective:

If, when the question closes, there is widespread reporting that Russia did the attacks and there is not much reported doubt, then I will resolve YES. If it seems ambiguous I will either resolve as N/A or at a percentage that I think is reasonable. (Eg. resolve at 33% if I think there’s a 2/3 chance that the attacks were false flag attacks.) This of course would also go the other way if there are supposed Ukrainian attacks that are widely believed to be caused by Russia.

I really buy the argument Sinclair makes about reducing trivial inconveniences here. Let’s make a model.

Ambiguity has two main negative effects, according to me:

1. Reducing precision in the prediction, because some of the prediction is based around interpretation of the creator’s foibles rather than a “true” resolution of a better-specified question.
2. Making the forecaster’s lives worse, because they want to be forecasting world events, not guessing as to the creators behavior in unclear situations.

Let’s set 1. aside for now. 2. seems like a big deal for sure. But also big is the drag on creating prediction markets imposed by Metaculus-style process. The way to balance these two seems to me to be a question that hinges on what you want your impact to be. If you’re trying to make the world as good a place as possible, you might have quite a strong preference for there being plenty of markets that can be made with low overhead. If the experience for forecasters is bad enough, then you won’t get predictions on those questions, but my empirical belief is that Manifold is striking a better balance right now than Metaculus.

As for 1., again, we can answer it according to, what’s most useful for our goals, and again, I want to claim Manifold is doing well here.

tl;dr There are real tradeoffs here.

I like the post btw!