It was our first time trying to build somewhat explicit models and be as quantitative as possible (I’d previously entered one ACX contest with a method most accurately described as squinting at a random number generator). Suffice to say we did way better than we were expecting.

The Method

We wanted to minimise correlated failure modes, so we spent the initial research and modelling time working independently and avoiding sharing any significant information except for clarifications about resolution criteria and the like. We wanted to get through this in a weekend, so allocated 7 minutes per question for individual research and 3 minutes for discussion. In practice this almost always ran over slightly, occasionally by a factor of 3.

We recorded our individual predictions in a spreadsheet, then updated them after our discussions. These updated predictions are what we both submitted to Metaculus. We predicted on every question.

My research methodology almost always consisted of attempting to calculate a meaningful base rate, then looking for idiosyncratic effects that could update me away from this. The 2024 contest seemed to have many questions that lent themselves to this kind of quantitative analysis quite well. For example:

• Question: Will the WHO declare a global health emergency (PHEIC) in 2024?

  Strategy: Pick a representative timeline and sample. In this case it was clearly going up over time, so I weighted more recent years more heavily.

• Question: In 2024 will there be any change in the composition of the US Supreme Court?

  Strategy: Look up everyone’s age, and US death rates by age. Subtract a bit from each due to wealth then multiply out. Add epsilon for chance of drastic constitutional changes.

• Question: Will {Bitcoin | SSE | S&P500} go up over 2024.

  Strategy: Pick a representative long timeline and sample years to get a base probability. Update upwards a bit because this whole AI thing might be a big deal.

These tended to be the sorts of questions where I think we were best calibrated, although it can also be quite hard to gain significant leaderboard points on these, as frequently our predictions would come in pretty much on top of the community prediction.

Most of our questions ended up being close to the community prediction, and I estimate that around 50% of my score came from questions where I was a few percentage points off consensus in the direction of resolution. These are probably quite boring to discuss, so I’ll focus on ones where I disagreed with Metaculus.

What went well?

My second highest scoring question was

When we first saw this we both thought we must have accidentally input 1-P instead of P. In reality though, I think this is largely attributable to resolution criteria lawyering. The question sounds like it’s asking if the November datapoint is going to be higher than 11%, when in reality it requires the average across all of Jan-Nov to be higher than 11%. Given LEV sales were at 8.5% at the end of 2023, this would mean hitting around 13.5% by November assuming a linear growth rate, for a nearly 60% YoY increase. The rate ended up being 9.7%, and not even any of the individual months breached 11%, although some did get quite close. Interestingly enough, this is also the question on which I most affected my co-predictor J’s prediction (if you look at the Question Breakdown graph in his post), and second by effect on Brier score.

Another question I feel quite good about is:

This was actively being talked about by the Biden administration at the time, which had a track record of being somewhat on the ball regarding AI. A large component of my prediction came from this executive order Biden issued, which seemed uncharacteristically detailed and binding. For example, it frequently referred to specific timelines, many of which were 90 or 180 days long.

Within 1 year of the date of this order and consistent with applicable law, the Secretary of Defense, in consultation with the Secretary of Commerce, the Secretary of Energy, the Secretary of Homeland Security, the Director of National Intelligence, and the Assistant to the President for National Security Affairs, shall issue regulations that prescribe heightened safeguards to protect computing hardware acquired, developed, stored, or used on any sites on which frontier AI infrastructure is located and that are managed by the Department of Defense, as needed to implement or build upon the objectives of, or the requirements established pursuant to, subsection 4(g)(iv).

None of it explicitly referenced limitations on AI, but given how popular conversations about copyright were at the time I had high individual probabilities assigned to both some kind of restrictions on training data, and separately on export restrictions on China. I thought the frequent referral to the Department of Defense made the idea of restrictions on foreign countries more likely. Given how soon this resolved, I feel good about my probability.

One more example (my third highest scoring question) was

Since 1950, Israel has had 7 premature prime ministerial resignations (I discount deaths and the one guy who got depressed), for a base rate of 10% per year. I decided to very slightly increase Netanyahu’s chances, since as a rule of thumb support for country leaders increases during times of conflict. My main takeaway is that one should be wary of going too far from base rate, especially since Israel has had many conflicts, so the base rate ought to still be somewhat representative.

What went not so well?

Over 3/4 of my negative points came from two questions, both of which were not amenable to base-rate analysis.

My worst question was

I remember finding it hard to be quantitative about this question, although again I tried to take the number of predicted launches and assigning them an individual probability. I think the downside to this approach is that it can magnify errors in your individual probabilities. For example, with 4 launches, if each has a 60% chance of failure, we get 13% overall, whereas if each has 70% we instead get 24%. This model also completely sucks, because launch failures are likely to be very correlated, so assigning them an individual percentage is pretty meaningless. I think my key takeaway is that putting 90% for a question I don’t especially trust my modelling on is dumb - ie I should probably account for Knightian uncertainty.

My second worst question was

With this one, I remember thinking that it would be a massive publicity nightmare if they didn’t release more information about it. This was patently false, I’ve not heard anyone mention Q* at all recently. I also assumed that this must be the next big thing, based on it leaking, and that therefore the next model release would at least in passing describe Q*.

My main takeaway here is that it’s really hard to model company politics and publicity effects. Also that publicity effects from not doing something are pretty much non existent. How often do you see articles titled “X still hasn’t explained Y”? How long would you wait before publishing this? Who’s the target audience? What else do you say in your article that isn’t in the title?

The Debacle

Given it was our first time participating, we were hoping to come in the top 50%, but not much better than that, so we didn’t think too much of entering a third submission containing our average predictions. This way we could get an interesting comparison of how we did as a team, vs how we did individually, and could get some nice warm fuzzies when our joint account came in the top 30%. We thought about telling Metaculus at the time, but that seemed entirely ridiculous (“Just in case we as complete prediction market first timers do come in the top 5, please be aware…”) so we decided to just leave it.

We checked the leaderboards a few times as the questions started resolving, and were pleasantly surprised to see ourselves slowly notching our ways up to the top 50. Then the leaderboard switched to spot scoring, and suddenly our accounts were 3rd, 4th and 6th.

At this point we reached out to Metaculus, and we agreed that our shared account l1z1x should be deactivated, and that we should forgo our cash prizes. We are very grateful to Metaculus for being so pleasant to deal with about this, for listening to our explanations with an open mind and goodwill, and for allowing us to keep our spots on the leaderboard. For their account of this please see the winners announcement.

On the brighter side, the joint account came 3rd, so it seems wisdom of crowds works despite the saying “two's company, three's a crowd”!

Takeaways

• The style of collaboration was great, I am fully committed to independent research time + discussion later.

  • A lot of the time we came up with similar models, and our updates were relatively small. In this case it might have been worth us collaborating during research, as some effort was duplicated between us.

  • But sometimes we came up with very different models. When these converged on similar probabilities, this was a strong sign we were onto something. When these disagreed we were usually both able to provide strong arguments that the other had missed, leading to us converging on a more accurate number. There were even times when we both updated in the same direction post discussion - we’d found different data that supported the same prediction!

  • I think it’s really important that the research is independent. Trying to work collaboratively all the way through is likely to lead to correlated models and errors.

• Trust the base rate!

  • I got a lot of points by trusting the base rate when popular opinion differed from it by quite a lot. I think there’s a human tendency to over-update on recent events. When you see lots of recent news articles complaining about a given prime minister, it’s easy to assume this means a high likelihood they step down. But all PMs have articles complaining about them all the time! This feels somewhat related to Scott’s post Against Learning From Dramatic Events.

  • I didn’t think of doing this for the 2025 contest, but in 2026 I’m planning to write down my calculated base rate explicitly, and compare that against my final prediction. Subscribe now for more posts in only 644 days time!

    Subscribe now

• Don’t submit joint account predictions on Metaculus

• Mike Johnson 4 eva

  • Our highest probability for any question ended up being “Will Mike Johnson remain Speaker for all of 2024?” at 95%. We thought about this carefully, and I maintain this was a well-calibrated estimate, but it was also a source of endless hilarity whenever Mike Johnson was in the news for anything, and made me way more invested in his speakership than I had any right to be. I also got a very healthy 58 points for this, so I can feel good in hindsight.

  • More generally, predicting these things made me way more invested in the news than I otherwise would have been. I think this alone is a good reason to participate.

We all know life insurance premiums get more expensive as you get older, because your chance of death in any given year increases with age (citation needed). Here’s a table where we can see this happening.1

At first glance these numbers don’t seem too crazy, but let’s figure out the total cost of the policy assuming you want coverage until you’re 80.

And now the arb becomes clear. Let’s suppose you’re a healthy non-smoking Indian male aged 55. All your friends are getting life insurance until they’re 80, but you’ve read Error Margin so you laugh in their face, approach your friendly local neighbourhood reinsurance company and sell them a life insurance policy on yourself for a juicy 28k, travel back to the year 1995, head to the cinema to watch Toy Story, then find your 25yo self and force them at gunpoint to take out life insurance until they’re 80. Congrats my healthy non-smoking Indian male friend, you just locked in a nearly 6k risk free profit for an instantaneous 25% return. Do this 31 times and you’ve earned yourself 1000x your capital and probably some questions from the insurance company.

OK but I don’t live in India?

No problem, are you perhaps a 50yo healthy male British non-smoker? I'll spare you the maths but you can do the arb too. 30 years of £100k insurance at age 30 costs you £2084. 10 years at age 50 costs you £2282 for a 9.5% return if you time travel.2

It's hard to find good publicly available data on this but I initially stumbled upon it when looking for life insurance for myself, which makes me think it's not just cherry picked.

Also, Canadian insurer Equitable published this graph that makes a similar point visually. It’s not quite the same exact thing because their premiums aren’t fixed for the whole term when you renew, but if I’m reading this correctly then the total cost of a 10 year term when you’re 60 is the area under the navy line from year 20 to 30, which is higher than the area under the whole cyan line of your (initially higher) premiums if you take the 30 year term at age 40.3

Weirdly enough, I couldn’t recreate this with the US data from Nerd Wallet. Is there a reason for this? Almost certainly. Do I know what it is? Nope. Do I think it’s because actual time travellers arbed the US market? No comment.

This might be to do with regulatory differences between countries in how much risk the insurance companies are able to take with their investments, but this would require the US to be unusually strict in its regulations, which seems pretty unlikely. I’d be interested to hear any theories to explain this.

But what about the efficient markets hypothesis?

OK well for starters this isn't really a market. Even if it was, it's very difficult to short — you need a time machine and the ability to sell life insurance on yourself, so there’s no mechanism to correct this.

But this may actually make more sense than it seems to at first. I’ve read4 that insurance companies make only small profits on premiums minus payouts after accounting for operating expenses, and the way they top this up is by investing the premiums until they need the cash. So the insurance company genuinely does have more opportunity to make money if you start young and take out a long contract. The returns of 25% and 9.5% I quote above only annualise out to 0.75% and 0.43% respectively given the 30y and 20y timescales involved.5

But… I don’t have a time machine?

I never said arbitrage was easy!

I guess the takeaway for those of us stuck in the 2020s is that as a consumer holding off on life insurance when you’re young may actually cost you more in total over the long run. This is obviously not financial advice, and I’m sure rules and regulations will make this differ a lot by country, but when getting life insurance it may be worth considering how old you’re going to be when it renews, and whether you can optimise the length of your term to minimise total cost to whatever age you want cover for.

B.b.b..but… where’s the rest of your blog?

Work in progress! I wanted to get at least one post out ahead of the ACX announcement of the prediction contest winners. I expect to post maybe 5 - 10 more things from my backlog over the next few weeks, then you’ll be at the mercy of whenever I have blog-worthy thoughts. Please subscribe to my mailing list for forecasting, scientific self-experimentation, video games, and lots of analysis of random interesting bits of data I come across in the wild. Also a post mortem of the 2024 ACX Prediction Contest, though if you want to skip the wait please check out this awesome post from my co-conspirator.

Subscribe now