2022 Prediction contest performance self-review.
General note: I’m trying to move my blog here from my wordpress page. I’ll probably copy my old posts over eventually so that it’s all here in the history.
This post is a review of my predictions for the 2022 ACX prediction contest.
My overall score seems okay given the level of the competition and that this was my first real try at this. My calibration seems mostly good, but maybe a bit overconfident (although the overconfidence on the lest implies I'm giving 50% odds in places where I should be giving 30% odds, so it might be a form of underconfidence - maybe it means I should be more willing to give low odds when both me and Scott are around 50%).
Key Takeaways
My biggest relative mistake was giving 50% chance odds. My reasoning here was that the trend was upwards, but 2021 had been unusually warm so there was reason to expect mean reversion and this should roughly cancel out. I think I lost here to both the people who did ignorant heuristics thinking ("things are generally getting warmer so this year probably will too") and the people who spent a bit more time looking into specific dynamics (presumably there were some projections due to climate factors for why this was actually more likely). Main conclusion here is to spend some of my prediction time looking for some kind of studies, not just trying to figure out my priors.
My other big mistake was dismissing the odds of a Russia-Ukraine conflict escalation. My reasoning here was that it came together with similar questions about Israel-Palestine (where I was familiar enough with the situation to confidently dismiss the odds of a significant escalation) and China/Taiwan (which I do still follow somewhat, and also had some insight from my pro-annexation Chinese then-girlfriend), so I lumped Ukraine in with them instead of looking more closely. I should notice when extrapolating to cases when I don't know the details and try to look up some details.
A few questions (like the Dow Jones average) were places where I could have sat down and done some straightforward math, and got bad results because I didn’t. I should remember to do that when possible.
I seem to have had an advantage in predicting how fast people wanted to get over covid. I’ll remember that I’m correct about people wanting things to be normal, mostly.
Taking Scott’s estimate (or the average where multiple odds were given) then adjusting a bit based on research worked better than just going by the average, when I bothered to do it.
The people who took the top few spots said they deliberately took risky bets to maximize win odds, not expected score. I specifically didn't do that last year (or this year either), I don't think I'm ready to plausibly win the eternal glory first prize and for now I'm just trying to do well reliably. Similarly, I try to avoid not making predictions, even in places where I probably shouldn’t if I’m just score optimizing.
Specific Predictions
Biden approval rating (as per 538) is greater than fifty percent. Prediction: 18%. Resolved False. I think this one was pretty obvious at the time - the 538 trend has been incredibly stable for the last few presidents and Biden was already well underwater at the time. I may have been underconfident (I can imagine a world where inflation doesn’t happen and Biden’s approval is a few points higher, but even there going over 50 is a stretch), but given that Scott was giving this 40% maybe I had reason to be cautious.
At least $250 million in damage from a single round of mass protests in US. Prediction 12%, resolved False. I may have been overconfident on this one - Scott gave it 10%, and I don’t remember a clear reason for going higher. On the other hand maybe not - this wasn’t too long after the Floyd/BLM riots, and while we’ve been trending away from that it’s not unimaginable that we could have seen another spike (or some different cause causing mass protests, like the truck drivers in Canada).
PredictIt thinks Joe Biden is most likely 2024 Dem nominee. Prediction 85%, resolved True. Not much to say here - I went higher than the baseline 80% and still think I was right to do so.
PredictIt thinks Donald Trump is most likely 2024 GOP nominee. Prediction 50%, resolved False. Again, I think this was around the right ballpark. Could I have gotten this one better based on what I knew at the time? I don’t know - it required both assuming that Trump would lose to the field and that a clear challenger would emerge, neither of which was obvious before the midterms.
Major flare-up (worse than past 5 years) in Russia/Ukraine conflict. Prediction 38%, resolved True. I did badly on this one, compared to both Scott’s baseline prediction and the market average. My reasons here are mentioned above - I extrapolated too hard on my prior that major conflicts are unlikely based on familiarity with Israel/Palestine (and the prior of looking at how often Russia does invasions), and didn’t look enough into the details.
Major flare-up (worse than past 10 years) in Israel/Palestine conflict. Prediction 3%, resolved False. I wouldn’t bet this low for this year, given the new government, but given the combination of an unusually reasonable Israeli government, no special messes, and a high threshold to overcome (there’s been some pretty big escalations in the last decade), this did seem pretty unlikely. The only way I see it changing was if Abbas died (about 11% for a man his age) and the succession led to a major conflict. I give that lower than 25% conditional odds, so 3% seems right here.
Major flare-up (worse than in past 50 years) in China/Taiwan conflict (prediction 5%, resolved False). I went with prediction here, which worked out.
Honduran ZEDEs legally crippled to the point where no reasonable person would invest in them further. Prediction 15%, resolved False. I’m not sure about this one - maybe I should have looked deeper and gone lower, but ZEDEs have had enough legal issues that 5% seems too low in hindsight. I did end up at around the average score here (though behind Scott).
New ZEDE approved in Honduras. Prediction 20%, resolved False. Seems reasonable, maybe could have gone lower.
Gamestop stock price still above $100. Prediction 23%, resolved False. I’m not sure how I feel about this one - Gamestop was still above $100 (adjusted for the stock split) at the start of December, and only went down as part of the big tech crash. On the other hand, a general stock market crash was always plausible (especially given how high tech valuations started the year), and it could also have decayed more purely due to losing attention. So on the fence about whether I should have gone lower or higher here.
Bitcoin above 100K. Prediction 15%, resolved False. I think I was too high here - I tend to argue against crypto hype, but I still got more caught up in it than I should have.
Ethereum above 5K. Prediction 25%, resolved False. I went too high on this one for the same reason as Bitcoin. I think the relative probabilities mostly made sense - Ethereum was a lot closer to 5K than Bitcoin was to 100K. In retrospect it also had the proof of stake shift coming up, which increased volatility and thus should have increased the odds it would have another jump. Both of these predictions were way too optimistic on crypto, but I think the ratio makes sense.
Ethereum above 0.05 BTC. Prediction 80%, resolved True. I think this was about reasonable - there’s a lot of things that could have happened that didn’t, and over the last few years Ethereum has been below 0.05BTC for significant chunks of time and hasn’t uniformly gone up. Maybe I could have anchored harder to Scott’s 90%, but I think 80% was already anchoring pretty hard.
Dow above 35K. Prediction 85%, resolved False. I should have just sat down and done the math on this one. Dow jones average returns are 8.7% and average annual volatility is about 17.5% (but this be left-skewed, which would have improved both predictions). This would have required -3.7% returns, and working out the math this comes to needing to be over -0.65 standard deviations, which has odds of 75%. Eyeballing it at 85% was overconfident, even without going into any weird econ weeds.
Dow above 37.5K. Prediction 65%, resolved False. Going by the math from question 14, this would have required 3.2% returns, which has about 62%. I was closer here with my eyeballing, although I still could have done better. Adjusting both this question and the previous with a left skew, 72% and 60% seem like reasonable priors, which would have gone better for both cases.
Inflation for the year below five percent. Prediction 93%, resolved False. In my defence that’s really high inflation, but given the level of pandemic spending that didn’t seem to be going down I should still have been less confident here (the aggregate score was around 70%! This was not unknowable!). Understanding the Ukraine situation better might also have helped, as would trying to look up some economic data.
Unemployment below five percent in December. Prediction 80%, resolved True. I think this was a more tempered and reasonable version of my inflation prediction, and largely made sense (I actually bothered to look up the trends for this one).
Google widely allows remote work, no questions asked. Prediction 30%, resolved True. This was my best relative score, which I feel good about - my reasoning about both Google being less worker-friendly than they pretend, and remote work being less effective, were both correct. I think I could reasonably have gone lower here.
Starship reaches orbit. Prediction 82%, resolved False. I beat the market on this one, and I think I acted reasonably. I looked it up enough to realize that Scott had some reason to be optimistic, but adjusted somewhat for Musk’s unreliability. I don’t think doing that more would have been a good heuristic, even though it would have worked in this specific case.
Fewer than 10K daily average official COVID cases in US in December 2022. Prediction 10%, resolved False. I looked up the trends and beat the market. Seems good.
Fewer than 50K daily average COVID cases worldwide in December 2022. Prediction 1%, resolved False. Yeah, no one thought this would happen and it didn’t. Congratulations to everyone involved (except for the appropriately-named row EW).
>66% US population fully vaccinated (by current standards) against COVID. Prediction 80%, resolved True. No comment.
India's official case count is higher than US. Prediction 12%, resolved False. A bit too high. I can imagine a new variant causing it, but 12% seems high for that specific type of new variant. Historical trends were clear enough that this should have been low without something weird and new showing up.
Medical establishment reverses course and officially says any of Vitamin D, HCQ, or ivermectin is actually effective against COVID.Prediction 5%, resolved False. I went over Scott’s 1% estimate but eh, I don’t feel too bad about it.
FDA approves a COVID indication for fluvoxamine. Prediction 60%, resolved False. I think just going with Scott was the right move here, I get my information about covid and the FDA through him anyway and would just have been adding noise.
Some new variant not currently known is greater than 25% of cases. Prediction 66%, resolved False. I just did prior math on the rate of new variants over the last couple of years, which seems to have beaten the market average.
Most people Scott sees in his local grocery store on December 31st are wearing masks. Prediction 35%, resolved False. I think I beat Scott on this one because New York was a few months ahead of the curve over California, so I had some more first hand information on how fast people wanted to get over masks.
Masks still required on US domestic flights. Prediction 40%, resolved False. Once again, “people want to be over covid already” won out.
CDC recommends that triple-vaxxed people get at least one more vax. Prediction 45%, resolved True. People might have wanted to move on from covid, but the CDC sure didn’t. my bad.
China has fewer than 100,000 COVID cases this year (official estimate). Prediction 30%, resolved False. I was generally more skeptical of Chinese containment than most people, and did a good job comparing their trajectory to neighboring countries.
No new real-money prediction market becomes bigger than Polymarket. Prediction 70%, resolved True. I just went with Scott here, since I get my prediction markets info from him anyway.
Manifold Markets is still alive and active. Prediction 35%, resolved True. Not sure why I went over Scott here, but it did work out.
New legal US real-money prediction market at least half as big as Kalshi. Prediction 7%, resolved False. Again not sure why I parted with Scott, this one didn’t work out.
New illegal but easy-to-use market satisfying the above. Prediction 22%, resolved False. Same issue.
Inflation for the year under three percent. Prediction 80, resolved False. I just went with Scott and did really badly on this one (and a lot of people didn’t). Again, should have researched inflation a bit more.
Democrats will lose their majorities in the House and Senate. Prediction 85%, resolved False. Possibly could have gone a bit lower since the Senate seems reasonably good for Democrats, but I did go lower than Scott and Matt, and Anchoring to them did make some sense.
Supreme Court will overturn Roe v. Wade. Prediction 55%, resolved True. I went below Scott and Matt, and might have been too willing to dismiss the new court’s willingness to just win this fight now that it could. More thinking about their perspective might have helped.
Emmanuel Macron will be reelected president of France. Prediction 65%, resolved True. I went with Scott, but could probably have looked up some French politics and gone a bit higher.
Jair Bolsonaro will be reelected president of Brazil. Prediction 52%, resolved False. I just went with the Scott/FP average, which I think was reasonable given my ignorance of Brazilian politics. Doing some independent research might have been good though.
Bongbong Marcos will be elected president of the Philippines. Prediction 57%, resolved True. Same thing.
China will NOT reopen its borders at any point during the first half of 2022. Prediction 88%, resolved True. I went 80% of the way towards the more confident estimate, which seems reasonable, I think.
Chinese GDP will continue to grow for the first 3/4 of the year. Prediction 92%, resolved True. Again, just went with the average.
20% of US kids between 0.5 and 5 years old will get at least one COVID vaccine by year's end. Prediction 65%, resolved False. Again went with the average, should have questioned a bit.
WHO will designate another Variant Of Concern. Prediction 79%, resolved False. Went a bit over the average (maybe due to a Zvi post? don’t remember.)
12 billion COVID shots will be given out globally by November 1st. Prediction 90%, resolved True. Don’t remember why I went over the average here, evidently it worked out. Maybe I looked harder at the vaccination rates?
At least one country will have less than 10% of people vaccinated with two shots by November 1st. Prediction 95%, resolved True. I just went with Scott’s higher estimate, since “at least one country” is a pretty low bar.
A psychedelic drug will be decriminalized/legalized in at least one more US state. Prediction 75%, resolved True. Again, just went with the average.
AI will discover a new drug promising enough for clinical trials. Prediction 83%, resolved True. Adjusting down from average failed on this one.
US government will NOT renew the ban on funding gain-of-function research. Prediction 60%, resolved True. Average again.
2022 will be warmer than 2021. Prediction 50%, resolved True. Should have both done more research (beyond just looking up a table for priors) and anchored harder (both of them going 30% over me should be more of a red flag). This one was my biggest mistake relative to the market.
Kenneth Branagh's Belfast will win Best Picture. Prediction 35%, resolved False. Adjusted a bit from Scott towards FP, maybe I should have adjusted harder off of this not being a Hollywood woke movie. This movie is excellent, strongly recommend seeing it.
Democrats lose at least two Senate seats. Prediction 85%, resolved False. Should have anchored less on this one, based on a look at the senate map I should have given it a lower score.
Democrats lose fewer than six Senate seats. Prediction 85%, resolved True. Could have gone higher on this one, this would have required losing some pretty safe seats.
Joe Biden is still president. Prediction 95%, resolved True. This one was mostly based off actuarial tables - I don’t know why other people went so low.
At least one Biden cabinet-rank official resigns. Prediction 78%, resolved True. Not sure why I went over the average here (maybe looking at priors?), but good move.
No military conflict between the PRC and Taiwan. Prediction 95%, resolved True. This question appeared twice.
Saudi Arabia and Israel establish diplomatic relations. Prediction 65%, resolved False. Should have stuck closer to the average unless I asked my brother about it.
Fewer US Covid deaths in 2022 than in 2021. Prediction 94%, resolved True. Not sure why I was this confident (a new variant could have broken this, but maybe giving 6% to a new deadlier variant was a reasonable calculation? hard to say.)
Liz Cheney loses primary. Prediction 80%, resolved True. Went with the average.
Some version of USICA passes Congress. Prediction 70%, resolved True. Average again.
Lula elected president of Brazil. Prediction 55%, resolved True. My prediction here wasn’t even consistent with my Bolsonero prediction. Bad, no.
China officially abandons Covid Zero. Prediction 70%, resolved True. Went with the average again.
Additional booster shots of mRNA vaccines authorized for seniors. Prediction 80%, resolved True. Went with the average.
The Fed ends up doing more than its currently forecast three interest rate hikes. Prediction 60%, resolved True.
Viktor Orbán loses power in Hungary. Prediction 45%, resolved False. I went well below the average here, which I think was justified.
Sinn Fein becomes the largest party in the Northern Ireland assembly. Prediction 60%, resolved True. Average again.
Democrats lose at least one governor on net. Prediction 60%, resolved False. Average again.