Photo via Silver Bulletin

How to judge an election model

Read the original at Silver Bulletin ↗
Cite

Silver, N. (2026, November 3). How to judge an election model. Silver Bulletin. https://www.natesilver.net/p/how-to-judge-an-election-model

Dated from the outlet's own feed. Check the original for its byline.

Silver Bulletin · collected 2026-11-03 · by Nate Silver commentary

Quick Summary

The article discusses how to evaluate probabilistic election models by explaining the importance of calibration over multiple true out-of-sample predictions rather than relying on backtested results. The author emphasizes that for a model to be reliable, its forecasts must accurately reflect actual outcomes across many instances; for example, if an event with a 30% chance is predicted and happens roughly 30% of the time, the model is well-calibrated. The piece highlights the distinction between fitting past data and making accurate predictions for future events, advocating for transparency about why a model works rather than showcasing historical successes.
Written by AI on 2026-11-03, from this article's own text only (the story summary below draws on all the coverage).

AI analysis

Story summary

Nate Silver's latest newsletter from the Silver Bulletin discusses the upcoming final FLIPR forecast ahead of an election. The newsletter emphasizes that the model has a track record spanning 18 years and includes detailed predictions for both elections and sporting events. Silver acknowledges the unusually high frequency of recent posts, typically publishing only about three items per week. He also addresses common questions about how to judge probabilistic forecasts, noting it is more complex than simply expecting 30 percent chances to occur 30 percent of the time.

Written for “Election Model Evaluation” on 2026-11-21, grounded in this article and the 0 other(s) covering the same event.

Compared with similar articles

This article reads unscored and hedges 11% of its claims. Each row says how that neighbour differs.
Times of India
Hedges 3 of 29 claims Outlet record 59/100
“The articles discuss different aspects of the election day context and potential outcomes, rather than reporting on a single specific incident.”
New York Post
Centre Hedges 2 of 9 claims Outlet record 66/100
“The articles discuss different aspects related to elections but do not describe the same specific incident.”
CBS News
Outlet record 66/100
“The articles discuss different aspects of an election, with Article A focusing on judging election models and providing coverage details, while Article B focuses on exit polls from a midterm voting day.”
NPR
Hedges 9 of 87 claims Outlet record 60/100
“The articles discuss different aspects of election coverage with distinct focuses and timing.”
TIME
Hedges 0 of 37 claims Outlet record 60/100
“The articles discuss different aspects of election forecasts and do not describe the same specific incident or occurrence.”
Silver Bulletin
Leans left Hedges 7 of 50 claims
“The articles discuss similar topics related to election predictions but do not describe the same specific incident.”

Story

Election Model Evaluation
Politics · 1 article(s) covering the same event.
How this article was measured signals, leaning charts, publisher record, byline, topics and claims

Signals How these are calculated →

Claims extracted
88
sentences that state something as happening or true
Uncertain
11%
10 of 88 hedged ("reportedly", "may", "according to")
Leaning
not political
takes no side on a contested political question
Publisher signals
not measured
these signals count newsroom corrections; Commentary has none to count
Outlets on this story
1
Politics
Same framing elsewhere
1
articles that frame the event the way this one does
Analyzed 2026-11-03 · how these are computed

How this is being covered How these are calculated →

Article leaning vs. publisher reliability
Source leaning vs. consistency

Publisher

Silver Bulletin · 72 article(s) · 0 correction(s) detected
No corrections detected for this publisher. That may mean careful reporting, or simply that nothing has been checked.

Who wrote this

Nate Silver
49 article(s) here · 1 carrying a prediction
And I don’t think it should be, although maybe I’d feel differently if I were stranded on the tarmac at LaGuardia at the time.
2026-11-19 · assertive framing · The White House needs a ground stop
We would consider a full-time and/or more experienced hire for an exceptional candidate who can persuade us of the value they’d bring to the site.
2026-11-17 · assertive framing · Silver Bulletin is hiring an editor and a developer
As you may know, I’m fond of finding different ways to classify Democrats.
2026-11-15 · assertive framing · Fighters vs. uniters and the big blue future
The timing might lead one to downplay the magnitude of the Nvidia story.
2026-11-13 · assertive framing · The NVIDIA story is a BFD
After considering how the extraordinary midterm landslide might affect Democrats’ 2028 plans, we turned to Republicans this time.
2026-11-12 · assertive framing · JD Vance and the VP curse
Since it’s a mostly progressive audience, we figure we’ll spend most of our time talking about how Democrats might respond in the wake of their extraordinary night.
We also may be doing some hiring.
2026-11-04 · assertive framing · After an extraordinary wave, what comes next?
Since we’ve recently published some very detailed analysis of basically every race that might matter, I’m not going to go through them state-by-state.
2026-11-03 · assertive framing · It's Judgment Day for Trump (and for pollsters)
One more newsletter for you before we publish our final FLIPR forecast shortly after midnight.
2026-11-03 · assertive framing · How to judge an election model
Our election-night page will go live at silverbulletin.com before coverage begins.
2026-11-02 · assertive framing · Election Night Live with Silver Bulletin and guests
Also by Nate Silver
The White House needs a ground stop
2026-11-19 · Silver Bulletin
Fighters vs. uniters and the big blue future
2026-11-15 · Silver Bulletin
The NVIDIA story is a BFD
2026-11-13 · Silver Bulletin
Nothing else under this byline is closely related to this article, so these are simply their most recent.
All 49 articles by Nate Silver →

Topics

ELWAY FLIPR FiveThirtyEight NYT the Super Bowl XVI

Subjects

FLIPR ORG · 2× FiveThirtyEight ORG · 2× Clare PERSON · 1× Disney ORG · 1× ELWAY ORG · 1× Galen PERSON · 1× Haley Stevens PERSON · 1× Internet Archive ORG · 1× NYT ORG · 1× Silver Bulletin ORG · 1×

Narrative

In the case of the Silver Bulletin/FiveThirtyEight lineage of models, however, we have something fairly unique: an 18-year-track record of making true out-of-sample predictions in advance of elections and sporting events, which were published ahead of time, and indeed have received a ton of public scrutiny.
framing: assertive · carried by 1 article(s) · first seen 2026-11-03
One more newsletter for you before we publish our final FLIPR forecast shortly after midnight.
2026-11-03 · Silver Bulletin
How to judge an election model · assertive framing

Claims (88 extracted, 10 hedged)

One more newsletter for you before we publish our final FLIPR forecast shortly after midnight. asserted
we → publish → midnight
If you’re new here, we do not usually post this much: it’s typically ~3 items a week, not three a day. asserted
it → ’re → much
But we hope you’ll join us for our final flurry of coverage, including our Election Night livestream and live blog tomorrow. asserted
you → hope → livestream
And we have a bonus episode of Still Counting out: asserted
we → have → episode
Reviews, comments, likes and subscribes are always greatly appreciated. asserted
Reviews → appreciate → ?
We covered a lot of ground on the show. asserted
We → cover → show
But I wrote about those NYT polls and Haley Stevens yesterday, so I’m going to use this as an excuse to cover another topic that Galen, Clare and I were only able to scratch the surface of. asserted
Galen → write → surface
I thought it would be nice to write about this ahead of time so it doesn’t feel like spin if a lot of FLIPR’s forecasts don’t come true. asserted
lot → think → forecasts
I’ll sometimes get the question, “How do you know if a probabilistic forecast is ever really wrong?” which is an entirely reasonable thing to ask. asserted
which → get → question
In fact, let me give you a four-part answer. asserted
me → let → answer
It’s certainly not quite as simple as saying “30 percent chances are supposed to happen 30 percent of the time. asserted
chances → ’ → time
That’s true insofar as it goes, but it’s incomplete. asserted
it → ’ → ?
And if you’re being Bayesian about it, you can’t always use this as a get-of-of-jail-free card. asserted
you → use → card
A 0.03 percent chance happening is much more suspect for your model than a 30 percent hitting. asserted
chance → happen → hitting
You want your probabilistic forecasts to be well-calibrated over a sufficiently large sample of true out-of-sample predictions. asserted
forecasts → want → predictions
In other words, if you’ve forecasted 100 events that you said had a 30 percent probability of happening, and 27 of them or 34 of them actually did “hit” asserted
27 → forecast → them
, your model is doing well. asserted
model → do → ?
Backtesting doesn’t count. asserted
Backtesting → count → ?
Any halfway-competently-designed model should fit the past data well; that’s the easy part. asserted
that → design → data
But the model may be misfit or overfit, keying off of essentially random factoids in past data that have no true predictive power and aren’t likely to repeat themselves. uncertain
that → key → themselves
You may have noticed that in documenting our models, we tend to not make claims like “ELWAY would have gotten the exact score right in the Super Bowl XVI” or “FLIPR would have nailed the 1994 midterm.” uncertain
FLIPR → notice → midterm
Who cares. asserted
Who → care → ?
We already know what happened. asserted
what → know → ?
I generally think it’s a negative sign when forecasters brag about their backtested results instead of explaining why they think their models make structural sense. asserted
models → think → sense
In the case of the Silver Bulletin/FiveThirtyEight lineage of models, however, we have something fairly unique: an 18-year-track record of making true out-of-sample predictions in advance of elections and sporting events, which were published ahead of time, and indeed have received a ton of public scrutiny. asserted
which → have → scrutiny
At FiveThirtyEight, we actually had an interactive that diligently tracked our forecasts (although the Internet Archive version of the interactive is a little hinky and Disney nuked the original version). asserted
Disney → have → version
Our sports forecasts were extremely well-calibrated: over a large sample size, the 30 percent chances really did hit about 30 percent of the time, and so on. asserted
chances → calibrate → time
Our politics forecasts were pretty well-calibrated too, although actually a bit underconfident: there were actually too few upsets relative to what our models expected. asserted
models → calibrate → what
It’s hard to know whether this is a statistically significant differential1, but it makes me a bit less worried when FLIPR is more confident about a race than the consensus. asserted
FLIPR → ’ → consensus
A second way is a very Nate way that some of you won’t care for. asserted
some → care → you
Would the model have made you money if you’d been betting on it — relative to other models, the conventional wisdom as far as it can be sussed out, or betting markets? asserted
it → make → models
If you trusted our model in 2016, you should actually have made a large bet on Trump, because you could get roughly 5-1 or 6-1 odds when our model thought he’d win around 1 time in 3 instead. uncertain
he → trust → 3
We could get into a more detailed discussion about things like Brier scores — you not only want your model to be well-calibrated, you also want it to discriminate clearly between relatively more and relatively less likely outcomes. uncertain
it → get → outcomes
You’ll be calibrated perfectly if you always forecast every race as 50/50 but the model doesn’t tell you anything. asserted
model → calibrate → anything
But I actually think some of the most important ways to evaluate a model are qualitative. asserted
some → think → model
The third thing I always look at is whether a model responds proportionally to new evidence. asserted
model → look → evidence
If either the magnitude or certainly the direction of changes in our model doesn’t pass a common-sense test in accordance with the new evidence, that’s often a sign it’s poorly constructed. asserted
it → pass → evidence
For instance, when that set of very strong NYT/Siena polls for Democrats came out last night, their win probability in the Senate improved by about 2 percentage points in FLIPR. asserted
probability → come → FLIPR
That seems fairly rational. asserted
That → seem → ?
And the NYT/Siena polls have generally been on the higher side of the averages for Dems, so FLIPR was expecting some relatively good numbers. asserted
FLIPR → expect → numbers
…and 48 more, not listed.
Give feedback · no sign-in needed

No sign-in needed for now. Your note is sent without a name, so it cannot be answered.

History Support