AI forecasters now equal or beat top human performance at traditional forecasting questions (eg ForecastBench, the Metaculus Cup).
asserted
forecasters → equal → questions
These are usually specialized AIs with forecasting harnesses, created by forecasting startups (eg FutureSearch, Mantic, Preseen), but foundation models like ChatGPT and Claude are only about six months behind the specialized models, so by the end of the year they’ll match top humans too.
In the past, the options for getting top-human-level forecasts were slim and inconvenient.
asserted
options → specialize → forecasts
You could hire a team of superforecasters to work on the problem, but this would cost thousands of dollars and take weeks.
uncertain
this → hire → weeks
Or you could check prediction markets (or Metaculus), but they only have a small sample of general-interest questions and are constantly trying to screw you over by gaming resolution criteria.
uncertain
they → check → criteria
The rise of AI forecasters solves these problems more cleanly and comprehensively than almost anyone thought possible just a year ago.
asserted
anyone → solve → problems
The “science of forecasting”, which only recently felt like trying to pick the least-bad of various bad options, now faces tasks more like “what kind of magnifying glass should we use to inspect our giant novelty lottery check as we confirm that it’s real?” and “how do we spend our winnings?”
asserted
we → feel → winnings
Specifically: AI forecasters now equal or beat top human performance at traditional forecasting questions.
asserted
forecasters → equal → questions
We need to learn how this success decays as we move to different types of non-traditional forecasting questions.
asserted
we → need → questions
I recently got to talk to some people at the Forecasting Research Institute, one of the leaders in this field.
asserted
I → get → field
Here’s my impression of the remaining challenges:
asserted
impression → ’ → challenges
1: Can AI do conditional forecasts?
asserted
AI → do → forecasts
A traditional forecasting question asks something like: will the Democrats win the next election?
asserted
Democrats → ask → election
A conditional forecasting question is more like: if the Democrats win the next election, how will that affect the chances of a stock market crash?
asserted
that → win → crash
Conditional forecasting questions are more exciting than traditional ones because they directly speak to policy outcomes.
asserted
they → speak → outcomes
If we defund the police, how will that affect crime rates (compared to the counterfactual where we hold police funding constant)?
asserted
funding → defund → counterfactual
If we mandate COVID vaccinations, how will that affect COVID mortality and myocarditis mortality (compared to the counterfactual where vaccines are voluntary / banned)?
asserted
vaccines → mandate → counterfactual
Prediction markets have been slow to experiment with these, partly because of annoying logistical issues (you have to refund people’s money if the condition doesn’t come true) and partly because they entice fewer customers than new forms of degenerate sports gambling.
asserted
they → experiment → gambling
AI forecasters don’t have this problem; you can ask an AI forecaster any of those questions above right now, and the big companies (FutureSearch, Preseen, Mantic) have it as an explicit feature.
asserted
companies → have → feature
What remains is to prove that AI forecasters’ great performance on traditional questions carries over to conditional questions.
asserted
performance → remain → questions
About a year ago, FRI tested this and found that it didn’t - the AIs of the time did significantly worse on conditional questions.
asserted
AIs → test → questions
It seemed like some kind of weird misunderstanding where the AI didn’t entirely understand the concept.
asserted
AI → seem → concept
Will this flip like a switch?
asserted
this → flip → switch
At some point, the AI is smart enough to fully grasp conditionals, and it becomes exactly as good at conditional forecasting as the regular type?
asserted
it → grasp → type
The experts I talked to thought no - as it gets smarter, its conditional forecasting ability will improve gradually.
asserted
ability → talk → ?
The experts weren’t willing to hazard a guess on whether it was better or worse than the average smart human, although they did think it was still worse than top superforecasters.
asserted
it → hazard → superforecasters
They’re working on a benchmarking experiment that should give us a better measure of this in a few months.
asserted
that → work → months
2: Can AI do truly causal forecasts?
asserted
AI → do → forecasts
When we ask “if the Democrats win the next election, will the economy go down?”, what we really want to know is a causal question - does voting for Democrats hurt the economy?
asserted
voting → ask → economy
But that’s not what we asked.
asserted
we → ’ → what
Suppose we’re wondering this in 2025, so we ask “If the Democrats win in 2026, is the economy more likely to be bad in 2027?”
asserted
economy → suppose → 2027
But maybe the Democrats will win in 2026 because there’s a already a bad economy (eg the Republican president implements tariffs and declares war on Iran, raising prices and making him historically unpopular), and then that bad economy continues into 2027.
asserted
economy → win → 2027
So the answer to the question “If the Democrats win in 2026, is the economy more likely to be bad in 2027?” is yes, but we still don’t know whether voting for Democrats in 2026 is good or bad for the economy.
asserted
voting → win → economy
Prediction markets can’t answer this question, because they can only be used to predict observable events that let us give payouts to successful gamblers.
asserted
us → answer → gamblers
Might AI superforecasters succeed where the markets fail?
uncertain
markets → succeed → ?
Could you just prompt them with “please answer the unobservable causal question, instead of the easily-resolved conditional one”?
uncertain
you → prompt → one
The experts I talked to weren’t excited about that particular prompt, but they did point out that a causal forecast is basically a combination of conditional forecasts.
asserted
forecast → talk → forecasts
That is, if your original conditional forecast implicitly considered the four different combinations of two variables - that is:
p(Democrats win & economy good)
p(Democrats win & economy bad)
p(Republicans win & economy good)
p(Republicans win & economy bad)
…then things like “the previous economy was good” or “we’re at war with Iran” are just more variables you can add to the stew.
asserted
you → consider → stew
It would be crazy to ask humans to forecast 2^10 branches so you could combine ten variables, but for AIs that could just mean a bit higher API spend.
uncertain
that → ask → spend
If you capture enough relevant variables in your “conditional” forecast, and find that the economy is still worse when the only one you’ve changed is whether or not the Democrats win, that’s a causal forecast.
asserted
that → capture → forecast
At least one company already tries something like this, though there are no studies of how well it works.
asserted
it → try → this
…and 112 more, not listed.