PredictionBook: A Short Note
post by Jayson_Virissimo · 2011-11-10T15:10:28.480Z · LW · GW · Legacy · 38 commentsContents
New Updates My Experience So Far Admonition/Guilt Trip The Future of PredictionBook None 38 comments
New Updates
PredictionBook has been updated and the speed improvements are massive. I was considering abandoning it because navigating the site was so slow (especially on my Android smartphone), but now I'm here to stay. Since there is no news feed on the website itself, it seemed appropriate to announce the update here. Also, the layout has changed slightly (for the better IMO) and the probability assignments have been made more intuitive (which many of the newcomers were having trouble with). The updates, along with the (slowly) growing user base, make now a better time then ever to join. Thanks TrikeApps and Ivan Kozik!
My Experience So Far
I was one of those that was convinced to give PredictionBook a shot based on gwern's article. It has only been a month, so I am using up my willpower preventing myself from drawing any firm conclusions about my web of belief or sanity from such a small sample size. Ultimately, my goal is to know what 5% feels like and I already believe I have made a small step towards doing so. Even if PredictionBook turns out to be a failure for improving my calibration to the extent that I can intuitively feel what my degree of belief in a proposition is in percentage terms, it would still be worth it for the simple reason that I now have an inkling about how poorly calibrated I am. Finally, on at least three occasions, I have found myself trying to cook up some kind of rationalization in order to put off doing some task that I probably should be doing, but have felt additional pressure not to give in to akrasia because "my prediction will be judged false".
Admonition/Guilt Trip
Honestly, PredictionBook is a rationalist's dream tool. How would you even know if you are actually becoming less wrong without tracking your predictions? Eliezer Yudkowsky, Robin Hanson, Lukeprog, Yvain, Alicorn, Phil Goetz, wedrifid, Anna Salamon, Wei Dai, and cousin_it where the hell are you? Do you doubt the potential of PredictionBook to improve your calibration and diminish akrasia or is rationality really about "affiliating with an ideology or signaling one’s authority" and not about having the map reflect the territory (yes, I realize this is a false dichotomy, but it makes for better rhetoric)?
The Future of PredictionBook
I would like to see a few changes in the way PredictionBook works and a few new features. Sometimes when you are entering dates into the date field it shows an incorrect number of minutes/days/years (but seems to fix itself once the prediction is submitted). This seems like a minor nitpick that should be easily fixable. Another minor change would be to make it more intuitive how the judging buttons work, since some of the newer members try pushing them instead of assigning a probability to the statement in the prediction.
It would be nice to have a measure of the user's calibration that can be compared across time, so that the user can easily determine if their calibration is getting better or worse. This could be something like a measure of how closely an OLS line of the user's predictions matches the line-of-perfect-calibration in the graph on the user's page. Also, I would like to see some of the community conventions written down in an FAQ-like document (specifically how conditional predictions should be judged), perhaps after they have been discussed.
I should add that the source code is now available on github, but it is written in Ruby (which I don't have any experience with yet). If there are any Ruby programmers on the Less Wrong Public Goods Team, this might be something worth considering for a project.
Note: Now that I called some people out, my probability estimates concerning who will make predictions on PredictionBook will have to be adjusted.
EDIT: There already exists a measure of calibration somewhat like the one I hinted at above; it's called a Brier score.
EDIT 2: lukeprog has joined the PredictionBook community.
38 comments
Comments sorted by top scores.
comment by [deleted] · 2011-11-10T22:49:51.498Z · LW(p) · GW(p)
gwern told me about PB in January 2011, and given my heavy use, I've spent about two days tweaking it and fixing bugs. If you like the changes, feel free to upvote this comment (I could use more than 0 karma). Note that TrikeApps is responsible for making the database faster, not me.
Also, if you see old predictions with bizarre estimates (e.g. 99% instead of 1%), it's because almost everyone made mistakes due the old copy displaying "99% against" instead of "1%" for 1% - the new version doesn't do this.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2011-11-11T01:49:06.120Z · LW(p) · GW(p)
I edited the article to give you some credit.
comment by [deleted] · 2011-11-10T20:55:50.260Z · LW(p) · GW(p)
Me, two months ago: "PB is mostly useless. Sure, things like elections can be judged, but I believe plenty of things that would be really hard for anyone to confirm. How would I ever turn them into predictions?"
Me, two weeks ago: *click* "That's the whole point, isn't it? Time to get active."
comment by shokwave · 2011-11-11T10:52:31.528Z · LW(p) · GW(p)
Started a beeminder about making a number of predictions a week, and made a prediction about whether I'll keep on top of the beeminder. Which counts as a prediction for this week, so I immediately added it to the beeminder.
I am ... not sure I am using these tools in the intended manner.
comment by RobertLumley · 2011-11-10T16:01:09.489Z · LW(p) · GW(p)
I've decided to sign up. I don't know why I dismissed this several times the first few times I saw it on here, it looks really interesting...
Edit: Why are you allowed to see what others predicted before you make a prediction yourself? That seems like it would wholly bias people...
Replies from: Jayson_Virissimo, gwern, moridinamael↑ comment by Jayson_Virissimo · 2011-11-10T16:10:29.513Z · LW(p) · GW(p)
Why are you allowed to see what others predicted before you make a prediction yourself? That seems like it would wholly bias people...
This is a good point. Depending on what exactly you are using it for, you might or might not want to see others' predictions before you make your own. It would be a useful feature to be able to toggle that functionality on/off in the user settings.
↑ comment by gwern · 2011-11-10T17:26:00.032Z · LW(p) · GW(p)
Edit: Why are you allowed to see what others predicted before you make a prediction yourself? That seems like it would wholly bias people...
One should be able to usefully incorporate other people's opinions, if only to adjust in the opposite direction!
As well, it'd be something of a UI/usability nightmare. If you really want to avoid contamination/additional information, make your own prediction and set it private or something.
↑ comment by moridinamael · 2011-11-10T16:24:10.845Z · LW(p) · GW(p)
I contemplated a strategy of always taking the average of the listed predictions. But that would be boring.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2013-11-12T18:16:02.356Z · LW(p) · GW(p)
A less boring strategy: make two accounts, one assigning probabilities based on your intuition alone; the other based on the average of the listed predictions. Compare their calibrations.
comment by lavalamp · 2011-11-10T17:10:47.765Z · LW(p) · GW(p)
You convinced me, I signed up and made some predictions.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2011-11-11T08:44:11.997Z · LW(p) · GW(p)
Welcome to PredictionBook!
comment by Alicorn · 2011-11-10T18:50:03.028Z · LW(p) · GW(p)
Alicorn ... where the hell are you?
I don't think in numbers, so I would expect to do really really badly for a really really long time, and I'm shy about things I realize I'm bad at. If I were going to embark on a long-term project to log lots of predictions for my own calibration and learning-what-percentages-feel-like, a text file on my own computer would do, wouldn't it? I'd know if I tampered with it.
On occasions when I'm confident about a specific prediction and don't feel shy about it, I prefer to just take some money from a nearby disagreer. I suppose if I wanted to make a longer-term bet, PredictionBook might be an okay place to stash the commitment if it's set up for that? Is it? But I feel like I'd be likely to forget about a long term bet so I'd rather avoid making them.
Replies from: JoshuaZ, shokwave, printing-spoon↑ comment by JoshuaZ · 2011-11-10T22:21:25.509Z · LW(p) · GW(p)
I'm probably more numerically inclined than you are but one thing I've found to be really helpful is that PredictionBook has given me much more of a feel for what things like "60%" or "95%" mean in the real world. How confident does that really translate into to be accurate? The effort involved is minimal and the payoff can be subtle but high.
Replies from: Alicorn↑ comment by Alicorn · 2011-11-10T22:29:57.494Z · LW(p) · GW(p)
But how does PredictionBook help with that over a little text file where I would write things like "60%: That fleebs are spruckled by 2012; that thus-and-such experiment confirms that glox is a form of spolk; that abritsens are publicly repudiated by Mr. Blafwem before such time as he resigns" or whatever things I might actually be disposed to register predictions about?
Replies from: gwern, JoshuaZ↑ comment by gwern · 2011-11-11T00:05:30.756Z · LW(p) · GW(p)
Graphing is pretty nice to have, as it fixes a rather more-than-trivial inconvenience; I'm not sure even I care enough to figure out how to munge a raw text file into a proper calibration graph (generated using... what, GraphViz? I don't even know where to start) to look at and go 'oh, I'm really underconfident in the 10/90% range, why is that?'
↑ comment by JoshuaZ · 2011-11-10T22:32:25.839Z · LW(p) · GW(p)
It doesn't help that aspect that directly at all.
Although I think having the ability to see what other people have predicted helps prevent lies to oneself especially of the form "well, that was just a black swan". When someone else predicted the other direction or simply had much reduced confidence one isn't able to say that.
↑ comment by printing-spoon · 2011-11-10T20:51:19.725Z · LW(p) · GW(p)
I don't think in numbers, so I would expect to do really really badly for a really really long time, and I'm shy about things I realize I'm bad at. If I were going to embark on a long-term project to log lots of predictions for my own calibration and learning-what-percentages-feel-like, a text file on my own computer would do, wouldn't it?
You can make your predictions private.
Replies from: Alicorncomment by matt · 2011-11-10T22:57:41.692Z · LW(p) · GW(p)
TrikeApps doesn't get all of the credit here - we pushed in the speed improvements, but the layout and copy changes were contributed by Ivan Kozik. Kudos to you, Ivan.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2011-11-11T01:22:37.398Z · LW(p) · GW(p)
I fixed the article to better share credit.
comment by lukeprog · 2011-11-12T22:17:03.484Z · LW(p) · GW(p)
My first PredictionBook prediction.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2011-11-13T07:45:35.901Z · LW(p) · GW(p)
Welcome Luke! I wagered against you joining, but I'm glad you're here.
comment by CarlShulman · 2011-11-10T23:06:24.117Z · LW(p) · GW(p)
I lost my password for the site, and there doesn't seem to be any way to get a replacement password emailed to me. Does anyone know how to arrange that?
Replies from: mattcomment by beoShaffer · 2011-11-10T21:02:04.731Z · LW(p) · GW(p)
On a related note it looks like you may be getting an influx of David Brin fans to PB. http://davidbrin.blogspot.com/2011/11/world-of-wonders-and-worries.html
Replies from: gwerncomment by falenas108 · 2011-11-10T18:49:38.633Z · LW(p) · GW(p)
There's an issue I've been having with PredictionBook with the amount of "personal" predictions, like if a person will finish a task by a certain time. Could there be a separate category for this type of prediction?
Replies from: JoshuaZ, Jayson_Virissimo↑ comment by JoshuaZ · 2011-11-10T22:23:19.733Z · LW(p) · GW(p)
In general, there's not a good way of categorizing predictions right now. A few people have tried when they do predictions to make tags by hand by just commenting in the form "[personal]" or "[politics]" or something similar. But this just assists in searching for specific types of predictions. Being able to really categorize and to do boolean searches of categories would be nice addition.
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2021-12-27T20:45:45.339Z · LW(p) · GW(p)
PredictionBook now has a basic tagging functionality. Props to CFAR and Bellroy for supporting me in getting the feature added.
↑ comment by Jayson_Virissimo · 2011-11-11T04:50:44.085Z · LW(p) · GW(p)
Upvoted, we very much need a way to tag predictions and a way to see our calibration on particular topics. I can't believe I forgot to include this in my article.
comment by JoshuaZ · 2011-11-16T14:22:52.287Z · LW(p) · GW(p)
Is this a good spot to make requests for additional functionality? If someone has the time, two things I'd really like to see is an ability to link efficiently in comments to different predictions using their numbers. The ideal syntax would be something like [3847] going to prediction number 3847.
comment by Morendil · 2011-11-11T10:30:54.729Z · LW(p) · GW(p)
This post and the comments inspired me to take a quick look at my history with PB.
- joined: 2009-10-14, the day it was announced on LW
- entered estimates on 5 "easy" predictions from others, one non-trivial, and created 3 private predictions
- entered 2 private predictions and 2 estimates the next month, november 2009
- briefly back in february 2010, 1 private prediction and one estimate
- briefly back in september 2010, for 5 private predictions of professional interest
- started using it again more regularly september 2011, with most of my activity occurring since then (25 transactions)
My participation in the Good Judgment project was the trigger for renewed interest in predictions and PB.com.
comment by CuSithBell · 2011-11-10T18:57:37.627Z · LW(p) · GW(p)
Do you doubt the potential of PredictionBook to improve your calibration and diminish akrasia or is rationality really about "affiliating with an ideology or signaling one’s authority" and not about having the map reflect the territory (yes, I realize this is a false dichotomy, but it makes for better rhetoric)?
Sounds tongue in cheek, but hopefully it doesn't actually make for better rhetoric here... right?
Replies from: Jayson_Virissimo↑ comment by Jayson_Virissimo · 2011-11-11T04:51:37.490Z · LW(p) · GW(p)
I was just being a little bit silly.