Raising the forecasting waterline (part 1)

I really, really want to answer "shortly" and leave it at that.

But since you ask, 45% chance I'll do it tomorrow, 25% over the week-end, 10% monday, and 20% later than monday.

Predictions involving what I'm gonna do are trickier for me, because there's a feedback loop between the act of making a prediction, and my likelihood of taking the corresponding actions once the prediction has turned them into a public commitment; it's a complicated one which sometimes triggers procrastination, sometimes increased motivation.

Yes, near the extremes it makes a difference - but we're using a Brier scoring rule, averaged over all days a forecast is open. That makes thinking in logs less important - 99% isn't much worse than 100% on errors. I'll discuss that in pt.2 under 'loss function'.

It depends on whether you're using probabilities epistemically or instrumentally. Changing the probability of A from 90% to 95% doesn't affect your expected utility any more than changing it from 50% to 55%.

This is a theme of multiple papers in the 2001 anthology Principles of Forecasting (a PDF of which is findable online), to give a specific citation.

I vaguely recall some academic work showing this to be true, or more generally if you're predicting the variable X_t over time, the previous period's value tends to be a better predictor than more complicated models.

These get called AR(1) models, for autoregressive 1.

Most complicated models that I'm familiar with include both the previous value and other factors (since there is generally more going on than a random walk).

Dunno. I have trouble with odds, for some reason, and rarely if ever think in terms of odds.

provided I think a question and its negation are equally likely to have been asked, there is a 50% chance that the answer to the question you have asked is yes.

Well, yes. But ought I believe that a yes/no question I have no idea about is as likely as its negation to have been asked? (Especially if it's being asked implicitly by a situation, rather than explicitly by a human?)

I disagree with this. The reason you shouldn't assign 50% to the proposition "I will win the lottery" is because you have some understanding of the odds behind the lottery. If a yes/no question which I have no idea about is asked, I am 50% confident that the answer is yes. The reason for this is point 2: provided I think a question and its negation are equally likely to have been asked, there is a 50% chance that the answer to the question you have asked is yes.

That's only reasonable if some agent is trying to maximize the information content of your answer. The vast majority of possible statements of a given length are false.

The reason you shouldn't assign 50% to the proposition "I will win the lottery" is because you have some understanding of the odds behind the lottery.

Yup. Similarly you don't assign 50% to the proposition "X will change", where X is a relatively long-lasting feature of the world around you - long-lasting enough to have been noticed as such in the first place and given rise to the hypothesis that it will change. (In the Le Pen prediction, the important word is "cease", not "Le Pen" or "election".)

ETA: what I'm getting at is that nobody gives a damn about the class of question "yes/no question which I have no idea about". The subthread about these questions is a red herring. When a question comes up about "world events", you have some idea of the odds for change vs status quo based on the general category of things that the question is about. For instance many GJP questions are of the form "Will Prime Minister of Country X resign or otherwise vacate that position within the next six months?". Even if you are not familiar with the politics of Country X, you have some grounds for thinking that the "No" side of the question is more likely than the "Yes" side - for having an overall status quo bias on this type of question.

Is there any established method

Yes: use a scoring rule to rate your predictions, giving you an overall evaluation of their quality. If you use, say, the Brier score, that admits decompositions into separate components, for instance "calibration" and "refinement"; if your "refinement" score was high on the lottery drawings, meaning that you'd assigned higher probabilities of winning to the people who did in fact win (as opposed to correctly calling the probabilities of winning overall), you'd be a suspect for game-rigging or psi powers. ;)

Maybe a similar rule in forecasting is "home ground advantage plus personal or professional stakes, or don't bother."

On 16 questions currently scored, I've done better than the team average at 15. Two of the questions where I outperformed the team by a large margin where the Syrian refugee question, basically a matter of extrapolating a trend and predicting status quo with respect to the conflict, and the Kismayo question, basically a matter of knowing my loss function. I had zero home ground advantage on either question.

Some of my wins resulted purely from general knowledge rather than from having any idea of the specifics of the situation: for instance, in mid-August I answered 40% to "Will Kuwait commence parliamentary elections before 1 October 2012?", reflecting only status quo bias in that a date for the election had not yet been announced. However, early in September I downgraded this to 10%, because I know that as a rule of thumb it takes at least one month to convene an election. The week before, I went to 5% (and even that was quite a generous margin), while several of my teammates made predictions, after I published mine, of 15%, 19%, 33% and even 51% (!).

This felt like entering a poker tournament where people routinely raise pre-flop with a "beer hand" (seven and two - when you play this, either you've had too many beers, or it's time you have one). Elections aren't a mysterious thing, we participate in one every so often. You need to print ballots, set up voting booths, audit voter registration records, give people time to campaign on national media, all very mundane stuff. Even dictatorships make at least a half-hearted attempt at this, and it's not like anyone in Kuwait had any particular interest in meeting an October deadline, this was strictly an internal-to-GJP deadline.

So while this question had to do, ostensibly, with something happening in Kuwait, all you needed to make a call at least as good as mine was background knowledge about extremely mundane, practical stuff that, if I had any hint that you wouldn't factor that in when making a close-to-home prediction, I wouldn't trust you with organizing so much as the PTA president election. Maybe a birthday party.

I wouldn't go so far as to claim that "skill at forecasting macro trends transfer to microeconomic moves".

But I'd take a stand on "demonstrated incompetence at the most elementary moves of forecasting, in a macro domain, is a strong indicator of likely incompetence at forecasting in any micro domain, other than the few narrow ones you might happen to be good at".

Isn't Tool 0 of forecasting 'Mind your own business'?

In a nutshell, no.

Consider some practicalities. An advantage of forecasting world events is that it permits participation by a much broader population. I could run a forecasting contest on when the city of Paris will complete a construction project on the banks of the Seine, which is "my backyard" compared to Syria. Nobody would bother.

The point is to find out something about how you think, and comparing yourself to other people will yield information that you can't get by sitting on your own, minding your own business. (On the other hand, there's nothing preventing you from doing both.)

Finally, I'm not aware that people routinely make explicit, quantified forecasts even about their own business. Rather, it seems plain that most of the time, we think "probable" the things we would like to happen, and as a result fail to plan for contingencies we don't like to think about.

To go from not forecasting at all to making forecasts in any domain is progress. It would certainly be useful to many to make forecasts about their daily lives (which I now do, a little bit). But let's imagine this were taught in schools as a life skill: I suspect you would have people practicing precisely on events that they have no control over and that allow interpersonal comparison.

Isn't Tool 0 of forecasting 'Mind your own business'?

Thanks for inspiring the following bit of staircase wit, which might make it into some further version of the post: Tool 0 of forecasting is "forecast". If you don't do it, you can't become better at it.

Gwern prefers PredictionBook - where you can, if you want, record private predictions - to GJP. For my part I prefer GJP, precisely because they ask me questions that might not occur to me otherwise, and the competitive aspect suits me. You could also do just fine by recording your own forecasts in a spreadsheet or a notepad, on whatever topics you like.

The key to accuracy is having fewer moving parts, all of which are visible and known by you.

Is accuracy what you're after? Which component of accuracy? I can get perfect calibration by throwing a thousand coin flips and predicting 50% all the time. What I seek is debiasing, making the most of whatever information is available without overweighting any part of it (including my own hunches, feelings and fears); and I'm most vulnerable to bias when there are many moving parts, many of which are hidden from me or unknown to me.

Isn't Tool 0 of forecasting 'Mind your own business'? One rule of the thumb in poker is "Jacks or better", meaning that you shouldn't even consider playing with anything less. Maybe a similar rule in forecasting is "home ground advantage plus personal or professional stakes, or don't bother."

No, tool 0 is more like 'mind your base rates' or 'don't predict what you would like, predict what you really think would happen'. I dunno where you're getting Tool 0 as 'Mind your own business' from; certainly I or Morendil didn't write it.

How well does skill at forecasting macro trends transfer to microeconomic moves?

I dunno, did you look into any research?

I'm guessing successful policymakers know a lot about their colleagues' personalities, histories, and connections, and that good policy comes from navigating those instead of forecasting macrotrends accurately.

Per the huge amount of material on Outside View vs Inside View and performance of SPRs already discussed on LW, I would guess quite the opposite.

How much would someone boost their batting average by limiting opinions to things of concern, such as for example health-related science, psychology as a science, or which people to draw closer or keep away? Entering a service-based deal with someone who turns out to be incompetent is costly, painful. Or failing to size up new "friends" fast enough for you to retreat before they waste your time. Many people perform well with evolved and practiced instinct by making sure that their forecasts never leave the home ground of their concern.

Do you know that, or are you just guessing, as you said you were before?

Or was your entire comment just an excuse to do an awful lot of rhetorical questions?

The methods that Morendil is discussing here are pretty general forecasting techniques, not limited to a particular domain. Some skills are worth developing, even if you're practicing them in domains you don't care about.

Personal example: I was a bio major in college, and I found it very difficult to care about organic chemistry, because we were mostly learning about chemicals that had no biological relevance. Consequently, I didn't learn it very well, which came back to bite me pretty hard when I took biochemistry.

What's the connection with the OP? I'm not seeing it...

Are there self consistent ways for people to believe that trickle-down economic policies should be encouraged but also to believe that small businesses are the primary drivers of growth?

Sure. Just to pick an obvious example, I might believe that trickle-down economic policies will benefit a subset of the population, and also believe that small businesses are primary growth drivers for the population as a whole, and believe that trickle-down economic policies should be encouraged because I consider benefits to the former subset more valuable than growth.