Towards no-math, graphical instructions for prediction markets

post by ryan_b · 2019-01-04T16:39:58.479Z · score: 29 (12 votes) · LW · GW · 9 comments

You are probably familiar with prediction markets. As of the last general update, Robin Hanson notes that while there is increasing research interest and a few general platforms are coming soon, the real goal of implementing inside organizations remains elusive.

I strongly suspect that it will remain so until companies are founded which use them from the beginning. This is because introducing people to new tools is hard. Introducing them to new tools that don’t even have a direct bearing on doing their jobs is even harder. We know that trivial inconveniences [LW · GW] are important; it was the dominant theme of the crypto autopsy [LW · GW]. If it happens in an environment like Lesswrong with crypto, I expect it will be worse in virtually all other contexts.

If prediction markets are going to be widely adopted, they need to be widely applicable, which basically means that any given schlub needs to be able to make a bet in a market without any math/economics/computing background. Happily, we have extremely robust evidence that any given schlub is perfectly capable of making a bet, courtesy of casinos.

One way to help with this problem is to make it as easy and intuitive as possible to learn. To wit: interactive visualizations. The biggest inspirations are Up and Down the Ladder of Abstraction by Brett Victor, and Metaculus. The first is about exploring system behavior without math, and the training for the second does a pretty good job of letting you make predictions based on the shape of the probability curve (though this doesn’t work as well for understanding the payoffs). Another relevant tool is the Sankey Diagram, which makes the relative sizes of flows apparent at a glance. I don't know of anything which makes such a thing interactive though - which is a shame, because I bet it would make controls comprehensible to people who don't know any differential equations or linear algebra. I am currently impressed by Seeing Theory. I ran across a few other websites which are designed to communicate math, and even one about risk, but for some reason all these savages are using Flash.

However, all of these is mostly for teaching math, and that is not actually the goal. Fundamentally the idea is information flows into the market, and only payments need to flow back out. So what we need is a way for people to specify what they think the outcome will be, with what confidence, without numbers.

Naively I am leaning in this direction:

I feel like the instructions could be collapsed into a few drag-and-drop shapes, and a few heuristic guidelines for how to manipulate them to represent a belief accurately. Training people how to do this should easily fit into a single-session course, not more than a day if we include motivation, the payoffs, and a bunch of trial runs.

9 comments

comment by RobinHanson · 2019-01-05T17:44:58.792Z · score: 10 (5 votes) · LW · GW

There have long been lots of unexplored good ideas for interface improvements. But they need to be tested in the context of real systems and users.

comment by uncomputable · 2019-01-04T17:29:23.244Z · score: 6 (4 votes) · LW · GW

I made an astonishing pile of points on SciCast from people who did not understand what the simple "user friendly" interface was doing. Turning "what i think the the distribution is" into a single trade is a fraught process; permit the user to do it repeatedly for a given question and I will actually show up and take absolutely all of their money.

comment by ryan_b · 2019-01-04T18:34:08.438Z · score: 3 (2 votes) · LW · GW

I'd be interested in hearing more about this; I'm unfamiliar with the SciCast interface.

I notice you put user friendly in quotes - do you think it was not in fact user friendly, or was it misleading in an important dimension?

For comparison, I don't think the Metaculus interface is simple enough to do the job. I agree it would be a fraught process, but my starting assumption is that developing an intuitive and correct UI is always a fraught process.

comment by uncomputable · 2019-01-04T21:41:35.458Z · score: 12 (5 votes) · LW · GW

I realized after my initial comment that it was a bit too terse to be productive. Sorry.

(pre-note: Metaculus is not a market, and I don't consider it interesting. I'm addressing and interested only in the case of an actual market where binary securities on events are being traded, with other market participants directly, or all through a market maker like SciCast.)

SciCast had two (major & relevant) modes for placing a trade, I remember them as "easy" and "normal". Normal presented you with a slider from 0% to 100%, you could move the slider back and forth, and as you moved it, you were presented with a continuously updating quote for the "cost" of making the trade. The cost could be negative (equivalent to selling a previously held long position, or closing an existing short position), or positive (going long, or covering a short, whatever). After you made a trade, the interface immediately reflected the percentage that you'd chosen as the new value of the market. While you could go terribly wrong with this interface (and a lot of people did), it was at least possible to do reasonable things.

The Very Bad interface was the "easy" one. Instead of a slider, and instead of being given any feedback about the costs of your trade, you were presented simply with five choices, probably something like <20%, 20-40%, 40-60%, 60-80% and >80%. I don't remember the exact equations that were used at this point, but it caused you to submit a trade that moved the market from wherever it was to as close as it get to the midpoint(?) of the region, without spending more than 1/2 (1/4? something) of your remaining points.

You weren't shown how many points the trade was about to cost you. And neither interface showed you how many points you already had invested in the positions you were adjusting!

About the worst thing (I remember) that happened to naive users (through no fault of their own, IMO) was to come along to a question where the current state of the market was maybe 5%, and then do an easy mode trade for "<20%". The system would spend their points to move the market from 5% to 10%.

The worst thing (which the bad interface encouraged), was users who thought that they should be trading until the market forecast matched their forecast. So if the easy mode trade didn't move the market to where they wanted (or I'd moved the market back in the seconds following their trade), they'd do another easy mode trade. Over and over, with no idea of the position they were building up on a single question.

The short version of the problem, in real-world stock market terms, is: just because you think APPL is only going to go up in the long run doesn't mean that you dump the entirety of your savings into it via market orders.

Think about kelly betting for a minute, just because the user has given you what they believe the odds to be isn't enough to compute the correct bet for them. You also need to know their bankroll. And if you're letting them trade on lots of questions simultaneously, you should be taking in a lot of probability estimates from the user, comparing them all to the state of the market, and making the trades that are profitable: ones where 1) the user believes the market is very wrong and 2) where corrections can be made inexpensively.

(For bonus points the user gives you the conditional probabilities between all those things, too, so that you can avoid betting too much on correlated trades.)

thats, uh, probably enough for now. i'll keep an eye out for any questions.

comment by RobinHanson · 2019-01-05T17:45:41.255Z · score: 8 (4 votes) · LW · GW

Your critique is plausible. I was never a fan of these supposedly simple interfaces.

comment by Charles R. Twardy (charles-r-twardy) · 2019-01-07T21:11:55.744Z · score: 7 (4 votes) · LW · GW

Interesting! We designed "safe mode" to be safe against the common newbie failure of losing all cash in a few big trades. It achieved that much at least. Each trade spent max of about *1%* of your cash, and moved it at most halfway towards your goal. A dedicated newbie could still burn cash, but it took effort.

Still, from your report, that may have happened more than I thought. If so, it may not even have achieved its main design goal. :-(

"The worst thing" was indeed atrocious. We fixed that fairly quickly, but ... ugh.

Thanks for the thoughtful critique. We're about to reuse the engine. I'm giving this some thought.

comment by uncomputable · 2019-01-09T18:39:43.814Z · score: 2 (2 votes) · LW · GW

i'm not sure if it would fix easy mode, but i feel that the points that go into a trade should be some function of the points that have already gone into that question, such that a user's total loss on a given question is capped.

comment by ryan_b · 2019-01-04T22:25:25.865Z · score: 4 (3 votes) · LW · GW

That is brutally bad. What was the training information like? Was it even possible for a naive user to become less naive without losing their ass a bunch of times?

comment by uncomputable · 2019-01-09T18:49:35.356Z · score: 3 (2 votes) · LW · GW

I remember the tutorials as focusing on easy mode, and treating it as the thing to start with. With Charles refreshing my memory about how easy mode behaved (thank you), I think there was plenty of time for a slow and cautious user to come in and perhaps question the nature of easy mode before they'd done any damage to their account. Certainly I used it a bit at the start, and still recovered nicely.

Hoping that your users will be slow and/or cautious isn't an ideal plan, though.