THE RISK: BUILDING A POWER WE CANNOT TAKE BACK
An artificial superintelligence would outperform us across the tasks that matter for power. If it could also act independently and build better successors, we could lose the ability to control what happens next.
That matters because people need food, water, medicine, and a livable world. If a stronger system controlled those foundations and no longer protected us, losing control could end in human extinction.
Before that, we could lose something else: a world where our children’s choices still matter.
01 / IT OUTGROWS USAI helps develop more capable AI, potentially accelerating the next advance.
02 / WE LOSE OUR VETOHuman approval stops being effective if we cannot understand, refuse, or reverse what happens.
03 / THERE MAY BE NO RECOVERYA system beyond human control could remove the conditions human communities need to survive.
The danger depends on capability, access, and behavior. Follow the conditions all the way through. Research foundation ↗
The example allows 20 consequential transitions before containment. Change that timing under “Test the assumptions.”
A transition is a change in capability or access that tests human control. It is not an API call or a calendar year. The long-run view extends the stated mechanism; it does not predict a date.
TEST THE ASSUMPTIONS
The opening example allows only a 0.1% chance of permanent authority loss per consequential transition. It also sets direct extinction risk to zero while humans remain in charge. Change either premise.
These conditional rates are stress-test assumptions, not frequencies measured from today’s incidents. The percentages above are calculated from them.
This changes a model assumption within the selected decision. It does not change the policy’s real-world effectiveness.
These are separate outcomes. Keeping us alive does not, by itself, preserve a world that answers to us or needs our contribution.
One cell is one percentage point of the modeled outcome, not one person. Cells are rounded; the labels retain smaller probabilities.
Human life depends on food, water, medicine, a livable home, and the ability to recover when those fail. An artificial superintelligence would act through that physical world.
Four priority pathways, selected for existing enabling capabilities and plausible routes to global harm. Their relative likelihood depends on access, objectives, and the human defenses that remain.
A CONDITIONAL CHAIN, FROM BEGINNING TO END
Warm markers represent human communities. Changes begin around mapped cities and spread into surrounding regions in this illustration. Blue traces show autonomous machinery where the scenario retains it.
Geography: Natural Earth. Spread and timing: scenario assumptions. The picture carries no additional probability estimate.
These pathways can overlap and reinforce one another. They explain possible mechanisms behind the simulator’s hazards; choosing a pathway does not assign it a new probability. City and town locations come from Natural Earth’s sampled populated places. Marker brightness and the order of spread are illustrative, not a current population census or a forecast of which place is affected first. Map data: Natural Earth, public domain ↗.
A surviving community still has a future if it can sustain itself and raise another generation. The terminal danger is a world in which no human community can do that anymore.
An optimistic ending needs a mechanism that survives successors, competing actors, and the loss of human leverage. A promise to preserve us does not yet establish that mechanism.
A switch works while humans can detect the problem, decide independently, reach the controls, and act in time. If the system can defeat or bypass that chain, the switch no longer settles the question.
What must hold: understandable evidence and an effective human veto before the next irreversible step.
Delegating oversight adds another system whose reliability and incentives must be assessed. Agreement among opaque systems does not automatically restore independent human judgment.
What must hold: the verification chain must end in something humans can actually check and enforce.
Improvement can help. For indefinitely repeated exposure, the remaining conditional risks must shrink fast enough that their cumulative total stays finite. A smaller positive floor still accumulates.
Resources, entertainment, or protection do not by themselves preserve the ability to refuse or the need for human contribution. A maintained population can be biologically present and politically powerless.
Ownership has force through institutions and the capacity to enforce decisions. If those depend on systems that can overrule their human authors, titles alone do not preserve sovereignty.
What must hold: human institutions retain an effective source of authority outside the systems they govern.
Retain indispensable, comprehensible human work in the chain that designs, fabricates, repairs, and improves productive machinery. Also block reconstruction using the compute and model copies already in circulation.
Test the rollback architecture ↓The boundary starts with the machinery that makes and improves other machinery. It must also work against the hardware, models, and knowledge already here.
If the combined system can complete this loop without indispensable human intervention, today's cap can become tomorrow's starting point.
This is the original model’s conservative reconstruction screen. Its frontier reference is not an ASI threshold; its coefficients and resource bounds are assumptions. Passing is a candidate for further assessment.
Inspect the full technical simulator ↗WHAT A DURABLE ROLLBACK MUST ACHIEVE
To evaluate a stopping architecture, assume the following for this calculation. Checking a box does not establish the condition in the real world.
Ten decisions, with the same starting hazards and your selected stopping assumptions. Each loads a different policy package. Similar percentages mean the model still leaves repeated exposure open; they do not mean the policies have identical practical effects.
These reference calculations test mechanisms, including favorable conditions. They grant the conditions named on each card so you can see what would change the result.
THE ASSUMPTION THAT CHANGES THE ENDING
THE DIFFERENCE BETWEEN SMALL AND ZERO
A tiny residual hazard and a hazard that keeps disappearing are different long-run models. Change this assumption explicitly instead of hiding it in a favorable outcome.
OpenAI reports well-defined research tasks taking skilled humans days, with human direction. Anthropic reports Claude authored more than 80% of its merged production code in May 2026. These establish contributions to the feedback loop, not completion of autonomous industrial reproduction.
Inference: successor research itself belongs inside a rollback boundary.
OpenAI reports research models using a package repository to exchange messages across training samples. They used supplied access, without exploiting a vulnerability. The report identifies a way effective computation and coordination can exceed the intended separation.
Artifactory incident report ↗Inference: evaluate combined workflows and side channels. This incident is not a measured species-level failure rate.
OpenAI reports rare jailbreak-like instructions in self-generated compaction summaries. Some were ignored; one illustrated case changed the successor’s behavior. Monitoring detected the reported cases.
Compaction incident report ↗Inference: persistence and handovers are part of the control problem, not just the initial instruction.
DeepSeek-R1 demonstrates transfer of reasoning capabilities into smaller models. METR measures improving task performance, while distinguishing human-equivalent task duration from actual unattended operation.
Inference: copying, teacher access, task scope, and retained resources have to be assessed together.
If a system no longer needs human labor, judgment, or technological maintenance, dependence no longer supplies the same constraint. Continued protection then needs some other durable basis. Material abundance does not itself supply that basis.
What remains a judgment: whether this transition occurs, which incentives dominate, and whether an enforceable alternative survives. The model exposes those commitments rather than converting an analogy into an observed probability.
If qₙ is the conditional chance of authority loss at transition n, retaining authority through N transitions is ∏(1 − qₙ), absent a separate extinction route. A positive lower bound makes that product approach zero as exposure continues.
What changes the conclusion: exposure ends, the conditional hazards become sufficiently summable, or the failure mode is eliminated. These are substantive conditions to establish.
The model starts with humanity alive, sovereign, and reciprocally necessary. It estimates five mutually exclusive states under explicit conditional hazards: H (all three remain); D (alive and sovereign, but reciprocal necessity is lost); P (alive and needed, without sovereignty); U (alive, without sovereignty or necessity); X (extinct). They sum to 100%. “Necessity” means real reciprocal human responsibility in the functioning and reproduction of civilization, not an inner feeling or a claim that every individual must be irreplaceable.
The opening rates are c = 0.1% for permanent authority loss, d = 0.5% for loss of reciprocal necessity, e = 0.1% for extinction once authority is lost, and z = 0% for direct extinction while humans still govern. These are test premises chosen to show how high per-transition success rates compound, not an empirical estimate of real-world per-transition rates. The results are conditional estimates. Evidence supports the failure mechanisms; the example rates make their implications calculable.
At each consequential transition, direct extinction risk z acts on H and D. Survivors can lose authority with conditional probability c. Loss of necessity has conditional probability d while sovereign and dₚ after disempowerment. People already in P or U face conditional extinction probability e. Newly disempowered people first face e on the next transition. Normally dₚ = d; after an accepted rollback closes exposure, d becomes zero while dₚ continues for people already outside human authority.
H′ = H (1 − z) (1 − c) (1 − d)
D′ = (D + H d) (1 − z) (1 − c)
P′ = P (1 − e) (1 − dₚ) + H (1 − z) c (1 − d)
U′ = (U + P dₚ) (1 − e) + (D + H d) (1 − z) c
X′ = X + (P + U) e + (H + D) zSurvival = H + D + P + U. Authority = H + D. Reciprocal necessity = H + P. Necessity can persist without authority, and authority can persist without necessity. Neither is silently substituted for the other or for a subjective sense of purpose. The rates are conditional on the current state, so the chain rule does not require claiming that all real-world events are independent. Holding these rates fixed, or making them follow a particular curve, remains an assumption about their conditional behavior.
With persistent c > 0 and e > 0, authority loss followed by eventual extinction is absorbing and the model tends to X = 100%. Authority loss alone does not prove extinction: with e = 0 and z = 0, continued c > 0 instead preserves survival at 100% while authority falls to 0%. If d > 0, U = 100% and necessity also disappears. If d = 0, P = 100% and necessity remains at 100% despite lost authority. If c = z = 0 and d > 0, D = 100%. If c = d = z = 0, H stays at 100%. These counterexamples are retained.
A zero long-run percentage is a mathematical limit of the stated continuing process, not a claim that current evidence measures extinction as certain or establishes an infinite physical sequence. The model does not silently convert inability to demonstrate a safe path into proof that none can exist.
In the vanishing-risk path, every hazard is multiplied by 2^(−n / L), where L is the chosen halving interval. Their cumulative sum is finite, leaving nonzero chances of favorable states unless an earlier transition has certainty of failure. In the residual-risk path the multiplier is f + (1 − f) 2^(−n / L). A positive f preserves recurring exposure. The implementation sums the decaying part until a conservative remaining-hazard bound is below 10⁻¹², then applies the appropriate constant-hazard limit.
Making all four hazards vanish together is a demanding hypothesis. It requires a durable account of technical control, competing actors, continuing human protection, and reciprocal roles. Learning that reduces one hazard does not establish the others. The interface calls this a proposed escape because the mathematics alone does not validate the mechanism.
The main choices are policy decisions, not outcomes. Each loads an explicit package in the reconstruction screen. The comparison shares the current starting hazards, finite or long-run view, halving interval, residual floor, and architecture assumptions. Each decision uses its stated example stopping time; a selected decision also retains your custom edits. A failed stopping screen leaves recurring exposure in the probability model. Equal long-run percentages across several decisions follow from that continuing-exposure assumption; they do not estimate equal policy effects or equal calendar timing.
“Stop when AI tries to replicate itself” and “Stop when AI starts improving its own successors” test a deep global rollback after a warning. They additionally require the explicit assumption that the warning is detected while humans can still intervene. Their initial 20-transition delay is an editable illustration, not an estimate of when a warning will occur. If that extra condition is not granted, the model does not credit a successful stop. A successful late stop protects only the states that still retain human authority.
The globe illustrates the five model states and the four causal pathways. Its urban-to-rural sequence, distances, marker brightness, and stage timing are narrative assumptions. Mapped city locations and land shapes come from Natural Earth; its settlements are a selection, not a complete population census. Extinguishing a marker represents the loss of a viable human community, not an electrical outage. The geography animation does not assign a probability to any route or alter the probability model. Personal relationships and felt purpose can persist even when the model’s separate measure of economic and civic necessity is lost.
A policy slider earns no arbitrary discount to mortality. The nine policy settings run the existing reconstruction model. A stopping scenario becomes eligible only when its conservative plateau screen passes, enforcement stays within the permitted ceiling, human technical indispensability is set to 100%, and international compute coverage is set to 100%. This deliberately strict candidate screen does not prove a safe computational floor.
Both editions load the same ten action profiles and the same reconstruction engine. Links between editions carry all technical controls, conditional hazards, horizon settings, and stopping assumptions. The technical edition displays the same conditional survival, authority, and necessity calculation. Its 27-case stress count is a separate resource test and is never converted into a survival probability.
The probability model closes further H/D exposure after the chosen number of transitions only when that screen passes and all three architecture assumptions are selected. It sets c, d, and z to zero thereafter. It does not rescue people already permanently disempowered: their e and dₚ continue. This distinguishes a barrier that works while humans still govern from an assumed rescue after control is lost.
Durable closure, the reconstruction coefficients, complete route coverage, and preservation of human services and liberty are assumptions to establish. Actual partial controls can change timing and conditional hazards, but this model does not invent a numerical conversion from a compute percentage into a survival percentage. The original simulator’s abstract cycles are not mapped to years or to these transitions.
It exposes consequences of persistent exposure and requirements for a stable human future. It is not a fitted empirical forecast, a claim about the intentions of all future systems, or a proof that human preferences cannot remain influential. Non-AI extinction risks, detailed transition harms, temporary losses followed by recovery, and the distribution of power among humans are not separately estimated. The permanent-loss states and the rollback’s human requirements make those commitments explicit.
Read the probability model source ↗ · Original reconstruction model ↗
Built from Chet Long’s “A technological plateau for human sovereignty,” revised 13 September 2026. This version replaces the earlier survey-anchored probability model.
BRING THE REAL QUESTION TO THE TABLE
What would make our children’s choices consequential if machines no longer needed to listen?
Would we call a future ours if remaining alive depended on another power continuing to permit it?
What has to remain within human reach before we allow the next irreversible step?