What are the best tools for recording predictions?

post by Raemon · 2020-05-24T19:15:24.033Z · score: 14 (4 votes) · LW · GW · 3 comments

This is a question post.

Contents

  Answers
    7 NunoSempere
    4 peter_hurford
    4 Thomas Kwa
    4 bgold
    1 dvasya
None
3 comments

Preferably, follow rules from "Best Textbooks in any Subject." I'm interested in people who have tried 2-3 prediction-recording tools, and can argue why one is better than the others.

I've decided I finally want to get gud at calibration. 

I'm personally just interested in tracking my own predictions, often about private things. I'd prefer something very low-friction where I can record my own predictions, mark them as true/false later on using my own judgment, and automatically see a graph of how calibrated I am.

Answers

answer by NunoSempere · 2020-05-25T08:19:19.992Z · score: 7 (2 votes) · LW(p) · GW(p)

Personally, I've used Foretold, Google Sheets, CSVs, an R script, and my own bash script (PredictResolveTally) (which writes to a csv.).

Personally, I like my own setup best (it does work at the 5 second level), but I think you'd be better off just using a CSV, and then analyzing your results every so often with the programming language of your choice. For the analysis part, this is a Python library I'm looking forward to using.

answer by peter_hurford · 2020-05-25T15:21:08.491Z · score: 4 (2 votes) · LW(p) · GW(p)

PredictionBook is really great for lightweight, private predictions and does everything you're looking for. Metaculus is great for more fully-featured predicting and I believe also supports private questions, but may be a bit of overkill for your use case. A spreadsheet also seems more than sufficient, as others have mentioned.

answer by Thomas Kwa · 2020-05-25T01:17:09.974Z · score: 4 (3 votes) · LW(p) · GW(p)

I've tried Metaculus private questions, Roam, and Google Sheets, and unfortunately find Google Sheets the least tedious. Metaculus questions are best when you revise predictions dozens of times, and Roam can't do much automatically yet.

Columns in the spreadsheet:

  • Date: date I make the prediction
  • Personal?: whether the prediction is about my own actions
  • Prediction: e.g. "I have 1000 LW karma by 2021"
  • Pro, Con: main reasons for/against, in a few words
  • %: predicted probability e.g .60
  • Outcome: 0/1 (haven't tried numerical data yet nor do I think it'll be worthwhile)
  • Hindsight: the probability I would have given in hindsight

I'm working on calibration, but also trying to identify patterns in mispredictions of myself that I can gain self-knowledge from, hence the extra information. It gets slow to load around 200 entries, but entering predictions using Google Forms could mitigate this (though I haven't tried it). The main advantage of a spreadsheet is the ability to customize graphs with relatively little effort.

comment by Raemon · 2020-05-25T02:14:05.273Z · score: 2 (1 votes) · LW(p) · GW(p)

Heh, I do often find spreadsheets to work the best, even if they're a bit janky/ugly, because I can customize them to be exactly what I want.

But it actually looks like PredictionBook may be superior to a spreadsheet (for me at least), by virtue of being pretty simple to enter data, as well as automatically composing your "correct predictions" graph, and sending you reminder emails when the prediction is due to resolve.

answer by bgold · 2020-05-24T23:00:46.050Z · score: 4 (3 votes) · LW(p) · GW(p)

I've spent a fair bit of time in the forecasting space playing w/ different tools, and I never found one that I could reliably use for personal prediction tracking.

Ultimately for me it comes down to:

1.) Friction: the predictions I'm most interested in tracking are "5-second-level" predictions - "do I think this person is right", "is the fact that I have a cough and am tired a sign that I'm getting sick" etc. - and I need to be able to jot that down quickly.

2.) "Routine": There are certain sites that are toothbrush sites, aka I use them everyday. I'm much more likely to adopt a digital habit if I can use one of those sites to fulfill the function.

So my current workflow for private predictions is to use a textexpander snippet w/ Roam.

- [[Predictions]]
- {percentage}%
- [[operationalized]]:
- [[{date}]]
- {{[[TODO]]}} [[outcome]]:


It doesn't have graphs, but I can get a pretty good sense of how calibrated I am, and if I want I could quickly export the markdown and evaluate it.

Of course I want to mention foretold.io as another good site - if you want to distributions that's definitely the way to go.

comment by ozziegooen · 2020-05-27T09:55:03.151Z · score: 5 (3 votes) · LW(p) · GW(p)

For those reading, the main thing I'm optimizing Foretold for right now, is for forecasting experiments and projects with 2-100 forecasters. The spirit of making "quick and dirty" questions for personal use conflicts a bit with that of making "well thought out and clear" questions for group use. The latter are messy to change, because it would confuse everyone involved.

Note that Foretold does support full probability distributions with the guesstimate-like syntax, which prediction book doesn't. But it's less focused on the quick individual use case in general.

If there are recommendations for simple ways to make it better for individuals; maybe other workflows, I'd be up for adding some support or integrations.

comment by Raemon · 2020-05-27T14:55:54.346Z · score: 4 (2 votes) · LW(p) · GW(p)

That makes sense. Thanks for chiming in. 

comment by Raemon · 2020-05-27T15:06:00.369Z · score: 2 (1 votes) · LW(p) · GW(p)

I think I might have phrased the OP “hey, is there a reason to use Foretold or Metaculus  over Prediction Book?”, and it sounds in both cases like they’re really optimized for a different thing. 

comment by Raemon · 2020-05-24T23:08:20.781Z · score: 2 (1 votes) · LW(p) · GW(p)

Is there an option for foretold to become Very Low Friction somehow? I agree with the "5 second level predictions" thing being a key issue.

comment by NunoSempere (Radamantis) · 2020-05-25T08:26:00.171Z · score: 3 (2 votes) · LW(p) · GW(p)

Foretold has a public API; requests can be made to it from anything that sends requests. This would require some work.

comment by bgold · 2020-05-24T23:22:13.431Z · score: 2 (2 votes) · LW(p) · GW(p)

I know Ozzie has been thinking about this, because we were chatting about how to use an Alfred workflow to post to it. Which I think would be great!

answer by dvasya · 2020-06-01T04:21:10.170Z · score: 1 (1 votes) · LW(p) · GW(p)

A while back TinyCast seemed pretty friendly: https://tinycast.cultivateforecasts.com/questions/new

3 comments

Comments sorted by top scores.

comment by Raemon · 2020-05-24T23:14:37.855Z · score: 7 (4 votes) · LW(p) · GW(p)

I did just check if PredictionBook could set all predictions to "private" instead of me having to change the setting every time, and the answer is yes, and also it looks like the UI has a few other nice-to-haves that actually make "low friction prediction" achievable.

I think I might need to create a custom stylish overlay for the page so to clear away some excess clutter, so it feels a bit less overwhelming to use. But, that's a fairly simple UI shift and one that I can create for myself. So PredictionBook might just be a good solution.

comment by lifelonglearner · 2020-05-24T19:26:12.024Z · score: 2 (1 votes) · LW(p) · GW(p)

I haven't tried many different tools, so I'm just spitballing here.

It doesn't seem to hard to just rig up a simple Python script or some Excel file that can keep track, if you want something self-hosted and/or local. (I know you have programming exp, so I expect this is not going to be too hard.)

Otherwise, I think some rationalists are on PredictionBook.

comment by Raemon · 2020-05-24T19:31:37.832Z · score: 2 (1 votes) · LW(p) · GW(p)

PredictionBook is basically my BATNA here. Seems better than homebrewing something. But I wanted to check if there were any better options that had come out in the past few years.