超越p（厄运）：人类机构如何塑造现实世界的人工智能结果（英）2026

Beyond p(doom) How human agencyis shaping real‑worldAI outcomes Hélène Galy, Jen Daffron,Lucy Stanbrough TA B L E O F C O N T E N TS Executive summary03 The realities of AI and p(doom)05 Why p(doom) is discussed and what it misses06 AI risk is not one thing07 Where the lack of AI governance todaybecomes tomorrow’s losses08 How governance gaps turn into real‑world losses09 Case studies10 Key questions to guide practical AI governance11 What management should focus on12 Human agency matters13 Board‑level requirements for a resilient AI future16 The practical takeaway: Ensuring progressdoes not outpace governance17 Contacts18 Executive summary AI adoption is accelerating acrossall sectors, moving rapidly fromexperimentation to embedded use incore operations and decision making.While public debate often gravitatestoward extreme or speculativescenarios, such as existential risk,these abstractions are of no value toboards and overshadow the real risksposed by AI. What risk managers anddirectors need instead are practical,actionable strategies to govern AIresponsibly — managing near termrisks while capturing the tangibleopportunities already emerging. Definition:p(doom) /’pi: du:m/ •The estimated probability of catastrophicAI outcomes•A common shorthand in technicaland policy debates•Signals concern but isn’t a practical risk tool Why this matters now Key AI failure modes Challenge questions for risk and decision makersto unlock action AI is altering risk architecture faster than governanceframeworks were designed to handle. Currentdeployments introduce new liabilities — includingmisinformation, bias, privacy breaches, correlatedtechnology risk and governance failures — which canescalate quickly when embedded in critical workflows. The extreme eventualities described in p(doom)scenarios are modes of failure that companiesface today and often fall under one of the followingsix themes: •Where are we over reliant on specific models,vendors or cloud infrastructure?•Which AI‑related risk combinations couldbreach our risk appetite?•Which strategic assumptions fail underplausible AI‑driven scenarios (operational,financial or market‑level)?•Are AI insights leading to clearchanges in controls, workflow designor deployment decisions?•How does today’s investment in AIshape future resilience and optionality? 1.Loss of control/misaligned AI.Goals divergingfrom human intent due to mis‑specification,instrumental convergence or misalignment A single AI error can shift instantly from a technicalflaw to a regulatory, conduct or customer fairnessissue depending on where it lands. Leadership mustensure the organization has the visibility, capabilityand controls to detect and intervene early. 2.Rapid uncontrolled growth.Fasteracceleration of capabilities than oversight,occasionally framed as “hard takeoff” orrecursive self‑improvement 3.Harmful human use.Cyber attacks,disinformation, biological misuse orauthoritarian surveillance 4.AI-enabled social collapse.Economic shocks,automated systemic failures or large‑scaleinterdependent infrastructure disruptions Closing the governance gap Most AI losses will not come from dramatic failures.Instead, they will arise from common operationalweaknesses: over‑trusting model outputs, scalinga workflow too quickly or eroding human oversight.Risk managers and boards have a critical role inensuring the organization maintains agency overhow AI is deployed, monitored and controlled. 5.Accidental failures (technical/systemic).Unexpected behavior in high‑stakes systems,control instabilities or cascading errors 6.Multi-agent failures.Emergentconflicts or coordination failuresacross interacting systems The lessons laid out in this report are clear: AI doesnot fail catastrophically by default. It fails when humanagency is absent, weak or outpaced by adoption. A governable AI environment is one where everyidentified failure mode leads to a clear action,owned by a specific leader, embedded in a disciplinedgovernance framework. This is the foundation forusing AI safely while capturing its strategic upside. Conversely, when organizations invest in technicalcontrols, cultivate knowledge and capability,understand the pervasiveness of AI across theiroperations and operate within robust regulatoryguardrails, AI remains a transformative asset ratherthan a systemic liability. For risk managers and boards,the focus is not how likely a catastrophic scenariomay be but how everyday failures accumulate intostrategic, financial and regulatory risks. We hope this research will help you stay ahead ofthe curve and anticipate where opportunities andvulnerabilities lie. For further information on AI,please contact our specialists. The realities of AI and p(doom) The challenge is that, while p(doom) can be auseful conversation starter, it is not a practical riskmanagement tool. It collapses a wide range ofscenarios into a single figure, often without a sharedtime frame, explicit assumptio