2024 Prediction Contest Self Review
Epistemic status: Long post, mostly written for self-review.
The 2024 ACX Prediction contest results are out. I got 79th place this year1, which is a significant improvement over last year (rank 501st), which was itself a significant improvement over 2022. Furthermore, I achieved my true dream goal and managed to score higher than Zvi2.
Overall, while I hope to keep improving, this is about as good as it gets. The average superforcaster score was around 70th percentile even in last year’s bigger contest, so the baseline here is high. And prediction contests are inherently noisy, which means to maximize your odds of winning you need to act more confident than your true probabilities imply3, which I explicitly decided not to do. So in terms of raw score, getting consistently in the top 5% is probably as far as skill can get me.
That said, there are lessons to be learned from some of the specific questions from 2024. So without further ado, let’s get to those.
Conclusions
After going through the below questions, a few trends
Most of the places where I could have made more of a difference were the ones where I already significantly deviated from consensus (both my biggest wins and my biggest losses). It would have been worth putting more time and effort into those questions specifically.
Technopessimism (on both AI and rockets) mostly won out, with the exception of the tower catching the rocket.
Overall I should adjust more towards the market when I have low credence and adjust less towards it when it has low credence. In cases where I’m not sure where the gap comes from, it’s worth putting in more effort to understand it.
Individual contest questions in detail
We go through the questions in order of best to worst performance on my part4. This is the exhaustive section that’s probably mostly worth skipping.
I list my probability and average probability at contest cutoff for each question, and then the score (which should be roughly how much better I did relative to the mean average final prediction before the cutoff).
1. Will Benjamin Netanyahu remain Prime Minister of Israel throughout 2024? Yes
My probability: 82% Average probability at contest cutoff: 57% Score: 73.083
This one was my easiest win, and shows the value of context familiarity (and also just how bad international media is when it comes to middle east politics). I could probably have been even more confident here than I was.
2. Will OpenAI publish information describing Q* (Q-Star) in 2024? No
My probability: 20% Average probability at contest cutoff: 40% Score: 67.237
I adjusted to go closer to market consensus. I’d have gotten a higher score if I hadn’t but I’m unfamiliar enough with OpenAI inner workings that I’m not sure that would’ve been worth the risk. Maybe if I’d done some more work to quantify how often they publish models.
3. Will a member of the United States Congress introduce legislation limiting the use of LLMs in 2024? Yes
My probability: 80% Average probability at contest cutoff: 55% Score: 58.577
I matched manifold and we somehow both significantly outpredicted metaculus. Maybe people just had no idea and were anchoring, and didn’t think about the question details? This was a pretty weak requirement.
4. Will Donald Trump be convicted of a felony before the 2024 presidential election? Yes
My probability: 52% Average probability at contest cutoff: 33% Score: 58.274
I mostly adjusted down to be closer to the market, otherwise I could’ve done better.
5. Will there be faithless electors in the 2024 US Presidential election? No
My probability: 32% Average probability at contest cutoff: 49% Score: 50.184
Another one where adjusting towards the market was bad for me.
6. Will an AI win a coding contest on Codeforces in 2024? No
My probability: 5% Average probability at contest cutoff: 15% Score: 49.123
I went by the heuristic that people overhype AI, and it worked out.
7. Will Mike Johnson remain Speaker for all of 2024? Yes
My probability: 79% Average probability at contest cutoff: 70% Score: 40.389
Standard reasoning here (Working off priors + adjustments) led to a solid result. No comment.
8. Will Ukraine control central Bakhmut at the end of 2024? No
My probability: 10% Average probability at contest cutoff: 20% Score: 36.210
9. Who will win the 2024 Mexican Presidential Election? Claudia Sheinbaum
My probability: 85.5% Average probability at contest cutoff: 80% Score: 34.686
Once again adjusted towards the market to my detriment.
10. Will there be 100 or more military conflict deaths between Ethiopia and Eritrea in 2024?
My probability: 18% Average probability at contest cutoff: 12% Score: 30.115
Another harmful adjustment, but makes sense to be low credence here since I don’t know their political situation well.
11. Will the New Glenn launch vehicle reach an altitude of 100 kilometers in 2024? No
My probability: 40% Average probability at contest cutoff: 44% Score: 28.696
Technopessimism remains undefeated. Another harmful adjustment, but one that I think makes sense.
12. Will the WHO declare a global health emergency (PHEIC) in 2024? Yes
My probability: 25% Average probability at contest cutoff: 30% Score: 28.620
I went down from 30% to 25% due to covid adjustment, which I think was overdoing it (especially since there’s an argument that covid gave them fun attention and they might want more now, so it could’ve gone either way). Not doing that would’ve helped a bit.
13. Will SpaceX attempt to catch a Starship booster with the tower in 2024? Yes
My probability: 32% Average probability at contest cutoff: 32% Score: 28.066
Technopessimism loses this time (especially since they actually succeeded in catching it). Maybe it doesn’t always win.
14. Will Ali Khamenei cease to be supreme leader of Iran in 2024? No
My probability: 10% Average probability at contest cutoff: 15% Score: 26.501
Slightly beat the odds on this one by correctly saying he wouldn’t lose power if he didn’t die.
15. Which party will win the 2024 US presidential election? Republicans
My probability: 49.8% Average probability at contest cutoff: 46.7% Score: 25.842
I started out at 60% Republicans and adjusted down to 50/50 based on the market. Given that Trump actually beat his polls, I’m going to say betting on democrats to beat theirs (as the market was assuming) was a bad heuristic and I should’ve stuck to the polls.
16. In 2024 will there be any change in the composition of the US Supreme Court? No
My probability: 12% Average probability at contest cutoff: 15% Score: 24.569
I went with actuarial tables, which seems right.
17. Will the Fed Funds Rate on December 31, 2024 be below 4%? No
My probability: 40% Average probability at contest cutoff: 40% Score: 22.700
18. Will there be a bilateral cease-fire or peace agreement in the Russo-Ukraine conflict in 2024? No
My probability: 18% Average probability at contest cutoff: 15% Score: 19.822
Interesting that I had this much higher (but with low credence) and adjusted a lot based on Zvi/Manifold. Adjustment wins this time.
19. Will there be a US government shutdown before January 1, 2025? No
My probability: 31% Average probability at contest cutoff: 20% Score: 17.815
20. Will the FDA or EMA withdraw approval of semaglutide for the treatment of obesity or diabetes in 2024? No
My probability: 2% Average probability at contest cutoff: 5% Score: 16.275
21. Will a debate be held between Joe Biden and Donald Trump before the 2024 US presidential election? Yes
My probability: 52% Average probability at contest cutoff: 66% Score: 15.852
Another one where adjusting to the market helped. I should probably have had a higher prior on this, going all the way down to 15% was way too low.
22. Will there be 10 or more armed forces conflict deaths between India and Pakistan in 2024? No
My probability: 21% Average probability at contest cutoff: 20% Score: 15.759
23. Will there be a serious radiation incident at any nuclear plant in Ukraine before 2025? No
My probability: 4% Average probability at contest cutoff: 8% Score: 15.252
24. Will X declare bankruptcy in 2024? No
My probability: 7% Average probability at contest cutoff: 10% Score: 14.692
25. Will there be 10 or more armed forces conflict deaths between China and Taiwan in 2024? No
My probability: 10% Average probability at contest cutoff: 6% Score: 10.864
I assumed a uniform probability distribution over the next 10 years, but doing a bit more reading about China’s military buildup could have shown me it’s a lot more likely later on, since they’re still building up naval capabilities. adjusting for that would’ve gotten me lower on this.
26. Will the S&P 500 index go up over 2024? Yes
My probability: 73%. Average probability at contest cutoff: 75% Score: 9.147
27. Will SpaceX's Starship reach orbit in 2024? No
My probability: 75% Average probability at contest cutoff: 70% Score: 6.546
Technopessimism wins this one
28. Will a nuclear weapon detonation kill at least 10 people in 2024? No
My probability: 1% Average probability at contest cutoff: 2% Score: 4.677
29. Will the Shanghai (SSE) Composite Index go up over 2024? Yes
My probability: 49% Average probability at contest cutoff: 51% Score: 2.401
30. Will Ilya Sutskever still lead OpenAI’s Superalignment team at the end of 2024? No
My probability: 59% Average probability at contest cutoff: 54% Score: -5.734
My decision to refuse to follow OpenAI internal politics drama was very wise.
31. Will cannabis be removed from Schedule I of the Controlled Substance Act before 2025? No
My probability: 55% Average probability at contest cutoff: 34% Score: -21.976
I’m still confused about this one. Did it require an act of congress or not? If not, why wasn’t it included in Biden’s spree of executive orders? Either way, Nothing Ever Happens implies I should have been below 50%, probably significantly below, despite what Zvi said.
32. Will the US unemployment rate be above 4% in November 2024? Yes
My probability: 28% Average probability at contest cutoff: 45% Score: -32.902
I could probably have built some kind of time series model on this one instead of going with my gut, which led me to go too low.
33. Will Bitcoin go up over 2024? Yes
My probability: 40% Average probability at contest cutoff: 66% Score: -33.550
I probably should’ve stayed closer to pure gaussian motion, which would have led me to ~50% odds on this one.
34. Will annual US core CPI inflation be above 3% in December 2024? Yes
My probability: 27% Average probability at contest cutoff: 40% Score: -36.597
35. Will US refugee admissions exceed 100,000 in fiscal year 2024? Yes
My probability: 12% Average probability at contest cutoff: 20% Score: -73.235
Oh boy, this was a weird one
To quote one guy from the Metaculus comments
This resolved positively by a sliver, with the resolution source (the Refugee Processing Center) reporting 100,034 admissions in the fiscal year. Based on data from 2001 onward, the monthly admission numbers seem to be strongly seasonal, often with relatively high admissions reported in the last one or two months of each fiscal year. This kind of seasonality is common for rolling admissions systems with fixed quotas or targets in consecutive non-overlapping periods.
ETA: Looking at Metaculus's track record page, this seems to have broken the all-time record for the worst-scoring binary Metaculus prediction (0.2%) at closing. The previous record was 0.4%.
I still think my guess was reasonable here. Even aside from the trend, allowing this to happen was the single biggest political own goal in recent history and probably cost Democrats the election. It should not have been predictable that they’d mess up the refugee situation this badly.
36. Will the 2024 light duty electric vehicle sales share exceed 11% in the US through November 2024? No
My probability: 88% Average probability at contest cutoff: 63% Score: -95.791
My single biggest mistake, but it was a big one. There were enough warnings (skepticism, initially low manifold odds, rumors about the rental car market causing issues) that I should have been more careful about looking into this one (or even if I wasn’t doing more actual research due to time constraints, at least be less overconfident about this).
Review of personal predictions
Will I be living in the US in a year? The main consideration I overlooked was the possibility that I’d end up wanting to be in a third country that’s neither the US or Israel. I don’t know that it would have changed my probability estimate much, but I should have considered it more as a factor.
Which countries will I visit in 2024? No comment about the prediction, but I ended up visiting quite a lot of countries (ten if you include Israel. I also eventually did China, but only in February 2025, which was outside the limit).
Who will be the author of my favourite new book in 2024? Not much to say on this one. 2024 was a disappointing year for new books.
It’s a little unclear what percentile this puts me in. If I unroll the full list I see 1339 listed contestants, which would put me at the 94th percentile, but I see a similar number for the 2023 contest despite it having had (officially) 3297 contestants.
I’m guessing the 1339 number is correct, and this year just had fewer contestants because it required actively registering to metaculus, which the unlisted contestants last year (including me) just didn’t do. This does imply a tougher market this year, on the assumption that the people willing to go through the effort of signing up for metaculus and predicting every question were more serious about the actual predictions too. So overall I think comparing absolute rank with last year’s makes more sense, although I should probably adjust it down somewhat since at least a few of the missing contestants would probably have passed me.
One more note here - last year I complained about missing a misleadingly-stated question, without which I would have jumped to rank 195/95th percentile. That basically matches my score this year without any adjustments, which I’m reasonably happy about.
Zvi got place 95. I don’t think Scott competed this year, his name doesn’t seem to be on the list.
I still can’t find a published form of this, but the gap is a closed problem - stating your true probabilities maximizes your expected value, but being overconfident increases variance at the cost of EV, and as the number of contestants with approximately correct models grows this improves your odds of getting the single highest score. There’s a whole theory of how to hit the win probability frontier, which requires more overconfidence as the number of contestants grows, but the details are beyond the scope of this post.
Once again I am stymied by substack’s inability to let me just put things in a table without going through some dysfunctional mess like datawrapper.