Scott Alexander 2021 Predictions: Market Prices - Resolution

OK, so I am obviously biased but I'll look to see if I think this is fair.

First of all, I didn't look at market prices for a lot of the things (where I did, I mentioned it). If I had done this more I would have done considerably better. Instead, I was saying whether I would trade on Scott's markets based on my current knowledge level. Does that count as predicting that number when comparing to the market? That's up to you to decide. 

You could of course just say 'should have done the research.' You could also say 'I'm comparing your ability to predict to what a market would do, on arbitrary questions, so tough that you only had Scott's prediction' or something. Again, not my call?

Second of all, the procedure for deciding what I meant seems to not match the way I was making predictions. In general, it would be fair to say that 'buy to X%' is actually saying 'it's at least X%' so my 'fair' must be higher than that ,and reverse for selling. 

But it's pretty bad to be doing this now, in hindsight, if we want to do Briar we need to specify those numbers at that time. 

So for e.g. Biden's approval, 80% was a dumb prediction and I should have sold it down somewhat. But Starlink I would strongly push back. Basic summary:

1. Biden approval: Giving me 80% outright is a tad unfair, but I take the L on this. Dumb.
2. Court packing: Meh.
3. Yang: My fault for looking at the prediction market, ironically. Should have been lower.
4. Newsome: Actually did make money selling this btw.
5. Tokyo: Not convinced I was right to buy this but I got away with it. Probably went too high.
6. Russia/Ukraine: I am confused why my 'maybe buy to 10' got interpreted as 15 here, whereas my 'sell down to X' in other places got interpreted as X and compared to a lower market. I think being lower than market was right here, although 15 was likely a better prediction than 10.
7. Netanyahu: Getting punished for this one feels wrong - I basically said sell while I'm above market, that's not exactly a statement that I should be higher than market - except that my explanation was that I was going to be slightly higher than market by default. I could argue 25 is fair. Can't really judge but I feel like I'd slightly buy this again at the new market fair if we ran another Everett branch?
8. Prospera: This is a 'trust Scott because I know nothing and there's no market' rather than anything else. In hindsight, people who get excited enough to write posts about X are a little too excited about X so I should have been moderately lower and sold a bit. Whoops. But note that if there had been a market I would have mostly defaulted to it.
9. GME: Yeah, I still dunno what to think of this, no further comments at this time.
10. Bitcoin: I noticed there was an easy arbitrage here, I think market was being dumb. Couldn't hedge the way this was scored, but notice that what I did was "Sell January 1 BTC 100k calls and buy spot BTC" and that trade seems like it does fine depending on the ratio, was definitely good. I think you gotta give me the market price or lower.
11. Ether: My trade here is "Sell ETH January 1 5k calls and buy ETH at 2300" and that's... a very very good trade and damn I shoulda done that. Feels weird to penalize me on that trade. But then again, if you'd told me ETH calls were trading at 11%, yeah, probably would have bought some cause that's too low, so maybe I lose anyway? Again, it's weird. 
12. Dow: Yeah, I think this is exactly right, I get market prices here. I'm not challenging the EMH on this one.
13. Unemployment: Definitely taking the L. Bigger L in terms of my fair at the time but would have come down a bit if I'd seen market. I am surprised.
14. Starship: I literally said 'no idea' and didn't trade, which is another way of saying market. 
15. Vaccination: 62% of the US population is 'fully vaccinated' which is lower than 66%, and the Metaculus market currently predicts January 22, 2022. I think 77% was clearly too high and also it's not clear it happened. 
16. Vitamin D: See my edit on 4/27, I did not predict 50%, that was me adjusting to a false understanding of Scott's prediction and therefore not selling this as far down as all that. E.g.

EDITED VERSION 4/27: I updated a lot on Scott being at 30% for this (e.g. 70% for this being recognized) in the original, and moved it to 50%. With Scott at 70% instead, we’re much closer, but I think I still want to nudge a little higher and buy this to 75%, instead of moving 30% to 50%. This is a sign of how much I’m reluctant to move a reasonable person’s odds in this type of exercise; if you’d asked me before seeing Scott’s number, I’d have said recognition is very unlikely, and put it at something like 85%-90%, and my true probability is still likely 80% or so.

I think when I say my 'true probability is 80% for not happening' you need to give me a 20% for happening.

17. Astrazeneca: Probably was actually slightly lower having only seen Scott, but seeing market would have undone that. 20 seems fine.

The big adjustment is that I took a big knock for the 50% on Q16, and that's just a misread, should be 20%. 

I'll let Simon decide what to do with the rest. I also find it super weird to be punished vs. market for when I said "this is the wrong price, do an arbitrage' in the correct direction, and made money even vs. market prices doing the trade, but hey.

Thanks for doing this!

I think the post should mention the fact that Zvi's forecasts were made after reading Scott's.

It looks to me as though your computation of log scores in the Google Sheet are wrong, and it’s not just a sign error: The correct log-score (Y log p + (1-Y) log (1-p), where Y is the outcome and p is the prediction) should be 0 for a perfect prediction (e.g. p=0 and the event didn’t happen) and should approach -infinity as the prediction becomes more and more confidently wrong. However, in the formula you used (-Y log (1-p) - (1-Y) log p), as we approach a perfect prediction, our score becomes infinitely large, whereas at the other extreme, the score is just 0. This can’t be a proper scoring rule, because the guesser would be incentivized to always predict p=0 or p=1.

Woop great post and great thread!

If anyone can figure out how to format that table, I would appreciate it, thanks!