In September, CNN reported that the United States came close to military confrontation with China.
asserted
States → report → China
According to the outlet, a military analyst using an AI chatbot to prepare an intelligence assessment of a Chinese vessel incorrectly concluded that the ship was transporting nuclear materials.
uncertain
ship → accord → materials
American aircraft were reportedly in the air to intercept the ship by the time the mistake was caught.
uncertain
mistake → intercept → time
An extensive discussion about how an AI model could make such a dangerous error followed.
uncertain
model → make → error
But as that deliberation dies down, it remains concerning how quickly an AI-assisted assessment may have moved through the chain of command and acquired the authority of vetted intelligence.
uncertain
assessment → die → intelligence
While we debate the impact of a future superintelligence on humanity, it is increasingly clear that AI will very soon enter the Situation Room, if it hasn’t already.
asserted
it → debate → Room
As anyone who has been involved in a national-security crisis can tell you, there is often too much information, some of it contradictory, and not a lot of time or clear methods to sort through it all.
asserted
some → involve → it
AI can process volumes of intelligence inputs more than any human team, detect patterns and anomalies buried in the data, and make sense of complex information in remarkably little time.
asserted
AI → process → time
But what will it do to the president’s judgment?
asserted
it → do → judgment
Based on my experience studying crisis decision-making and assessing intelligence, I believe there is a real risk that AI will amplify leaders’ own biases, handing them back their own assumptions with the authority of objective analysis and the ease of a clear-cut outcome.
asserted
AI → base → outcome
We tend to talk about AI as if it were wholly independent, a more rational intelligence system that can compensate for our weaknesses.
asserted
that → tend → weaknesses
But in reality, humans have built these AI models and trained them on data produced by humans.
asserted
humans → build → humans
We then ask these AI models questions shaped by our own beliefs.
asserted
We → ask → beliefs
The flaws in human decision-making are well-known.
asserted
flaws → know → making
Presidents and their advisers have always had to guard against confirmation bias, our tendency to interpret evidence in light of our pre-existing beliefs and to under-weigh or dismiss facts that point to the contrary.
asserted
that → have → contrary
We know humans have selective attention, an instinct to hone in on facts that are more vivid and to overlook others.
asserted
that → know → others
Presidential decisions represent countless underlying judgments.
asserted
decisions → represent → judgments
Reasonable people can differ on each of the estimates, and those differences are part of what presidential judgment must resolve.
asserted
judgment → differ → what
Flattening them up into a probability assessment obscures more than it reveals because it does not show which indicators were seen as credible and which were ignored.
asserted
which → flatten → more
President Barack Obama had gathered his advisors to assess intelligence that indicated Osama Bin Laden was hiding in a compound in Abbottabad, Pakistan.
asserted
Laden → gather → Abbottabad
If the intelligence was correct, he would send in American troops.
asserted
he → send → troops
But his trusted, experienced advisors, all looking at substantially the same data, offered individual estimates ranging from roughly 30% to 95%.
asserted
all → trust → %
Now imagine a powerful AI system in that room and synthesized those assessments to give a 63.7% probability that bin Laden was there.
asserted
Laden → imagine → probability
We would certainly gain precision.
asserted
We → gain → precision
But what would be lost?
asserted
what → lose → ?
The number would not tell the president why there was such a marked difference in the assessments.
asserted
number → tell → assessments
It would not inform him of the competing assumptions on which those assessments were built, or how the implications of action versus inaction were weighed.
asserted
implications → inform → inaction
But worst of all, it would not alert the president to the fact that there was disagreement at all.
asserted
it → alert → fact
Such a “clean” number can make a question look resolved.
asserted
question → make → ?
Now factor in that the CIA, DIA, State Department, military services, and other agencies will likely rely on the same foundation models built from overlapping data and sources.
asserted
CIA → factor → data
You could theoretically end up with 10 assessments pointing in the same direction but all derived from the same fundamental materials.
uncertain
all → end → materials
The AI version of that risk is synthetic consensus: institutional agreement that looks independent but is partly generated by common analytical machinery and reproduced at scale.
asserted
that → look → scale
But if the problems are well-known from human decision-making, so too are the strategies to counteract them.
asserted
strategies → know → them
We can design our systems to act as benevolent antagonists and train our decision-makers to engage with them as such.
asserted
makers → design → them
On the technology side, we need to build a system that talks back.
asserted
that → need → system
AI used in the national security context should not be optimized just for speed of response.
asserted
AI → use → response
It should be built to help identify which evidence is credible; to generate the strongest competing explanations; to distinguish facts from inference and prediction; and to make uncertainty visible rather than manufacture precision.
asserted
uncertainty → build → precision
A system designed to be agreeable is dangerous.
asserted
system → design → ?
The points of friction and disagreement are often what’s most revealing and valuable for leaders making tough decisions.
asserted
points → ’ → decisions
On the human side, we need to ask better questions.
asserted
we → need → questions
…and 11 more, not listed.
The Aporia
