p(doom): Why a Probability of Catastrophe Is No Basis for Decisions

One number carries more weight in AI debates than any benchmark: p(doom). It denotes a personally estimated probability that advanced AI leads to catastrophe for humanity. Not a measurement, not a study — a subjective belief expressed as a percentage.

The term grew out of research forums and travelled from there to conference stages, podcasts and boardrooms. It matters to decision-makers even though it decides nothing operationally: it shapes how AI risk gets discussed — and usually obscures the risks that actually occur inside a company.

In short: p(doom) is a subjective probability statement about existential AI risk, not an empirical value. Published estimates from prominent experts range from "negligible" to double digits. For companies the more useful question is not p(doom) but p(fail): how likely is this specific workflow to break — and what does that cost?

1. What p(doom) actually denotes

p(doom) borrows notation from probability theory and applies it to a fuzzy event: "p" for probability, "doom" for a catastrophic outcome. In practice the value functions as a credence — an expression of personal conviction, closer to a bet than to a frequency measurement.

That produces the first problem: there is no shared definition of the event. Some mean human extinction, others the permanent loss of human control, others any civilisationally irreversible decline. Without a defined event, two percentages are not comparable even when they look identical.

2. The spread of public estimates

The striking part is not the level of individual numbers but their dispersion among people with comparable expertise. Publicly stated positions range from very low single digits to clearly double-digit figures. Yoshua Bengio and Geoffrey Hinton, both Turing Award winners, have named serious probabilities for severe outcomes and derived precautionary demands from them. Yann LeCun, also a Turing Award winner, considers such scenarios implausible and points to the steerability of today's systems. Dario Amodei has said publicly and repeatedly that he sees meaningful residual risk and still works on the technology — arguing that safety work has to happen where capabilities are created.

The broader survey of AI researchers by AI Impacts (2023, several thousand respondents) showed no unified picture either: the median figure for extremely bad outcomes sat in the low single-digit percentages, with substantial spread within the sample. And the joint Statement on AI Risk by the Center for AI Safety (May 2023) deliberately avoided any number — it stated only that mitigating the risk should be a global priority.

The honest reading: there is no expert consensus that could be translated into a figure. Presenting a single percentage as "the state of the science" exceeds what the sources support.

3. Why the number usually hurts corporate debates

p(doom) is rhetorically powerful and operationally empty. Three effects recur in real conversations.

One: false precision. A percentage signals measurement. In fact it is an opinion in numeric clothing. Inside a decision memo it therefore reads as sturdier than any carefully worded assessment — with the truth value inverted.

Two: confused scales. Existential scenarios and operational errors end up in the same sentence. The result is a debate that tips into either doom mood or dismissal. Both prevent the work that is actually needed: defining which concrete errors a system can make and who notices them.

Three: displaced responsibility. If the risk is planetary, it is not my process. A debate about humanity's risk rarely yields a better approval chain in marketing. A debate about false claims in a customer chat yields one immediately.

4. The usable translation: from p(doom) to p(fail)

For companies a different probability counts: the chance that a specific AI-supported workflow measurably goes wrong. That question is answerable, because the event is defined and the horizon is known.

In practice: per use case you write a failure definition, measure a frequency and assess a cost. A customer chat that invents binding price statements has a measurable error rate. An agent with write access to campaign settings has a measurable rate of unwanted changes. Both can be tested, logged and bounded — unlike any statement about 2075.

That operational level is exactly where AI governance takes effect: grant permissions individually, log outputs, define sign-off for binding statements, name a kill switch. Agentic workflows add the specific attack surface we described in prompt injection and tool poisoning. Personal data adds the frame from AI and GDPR, disclosure duties the one from the EU AI Act in marketing practice.

5. How to handle it cleanly in conversation

When p(doom) shows up in a meeting — from the board, a client or the press — a fixed three-step sequence helps.

1. Clarify the event. What exactly is being claimed, over what horizon, under which assumptions? Without that, arguing about the size of the number is pointless.
2. Name the spread instead of picking a number. The defensible statement is: experts with comparable qualifications arrive at very different estimates, and no consensus value exists. That is not evasion, it is the actual state of knowledge.
3. Switch to your own level. "Our question isn't how the century ends, it's how often this assistant makes false commitments and who sees them first." That puts the debate back where decisions are possible.

This stance is also the more stable one in communication terms: neither doom rhetoric nor dismissal ages well. A traceable risk description with evidenced controls survives every later review.

Conclusion: the number is a signal, not a foundation

p(doom) says more about worldviews than about systems. As an object of discourse the term is useful, because it shows how seriously parts of the field take long-term risk. As a basis for decisions it fails, because it lacks an event definition, a horizon and an empirical base.

Deploying AI responsibly does not require a percentage for humanity's future; it requires error rates for your own workflows. The good news: the latter are measurable, and measurement starts with a single defined failure description per use case.

Next step: take the AI-supported workflow with the most customer contact and state in one sentence what "wrong" means there. Then measure for a week how often it happens, and decide who signs off on such cases. That replaces every percentage debate.