Probing the improbable: methodological challenges for risks with low probabilities and high stakes
Open accessAbstract
Some risks have extremely high stakes. For example, a worldwide pandemic or asteroid impact could potentially kill more than a billion people. Comfortingly, scientific calcultions often put very low probabilities on the occurrence of such catastrophes. In this paper, we argue that there are important new methodological problems which arise when assessing global catastrophic risks and we focus on a problem regarding probability estimation. When an expert provides a calculation of the probability of an outcome, they are really providing the probability of the outcome occurring, given that their argument is watertight. However, their argument may fail for a number of reasons, such as a flaw in the underlying theory, a flaw in the modelling of the problem or a mistake in the calculations. If the probability estimate given by an argument is dwarfed by the chance that the argument itself is flawed, then the estimate is suspect. We develop this idea formally, explaining how it differs from the related distinction between model and parameter uncertainty. Using the risk estimates from the Large Hadron Collider as a test case, we show how serious the problem can be when it comes to catastrophic risks and how best to address it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
The Logic of Scientific Discovery
Bhaskar, R. (2013) · Routledge eBooks · doi:10.4324/9780203090732-9 · Cited by 7,765 · see citing works
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,628 · see citing works
Witten, E. (1984) · Physical Review D · doi:10.1103/physrevd.30.272 · Cited by 3,452 · see citing works
Cartwright, N. D. (1999) · Cambridge University Press eBooks · doi:10.1017/cbo9781139167093 · Cited by 2,437 · see citing works
Revue Internationale de Philosophie
Meyer, M. (2005) · Cited by 1,359 · see citing works
Black Holes at the Large Hadron Collider
Dimopoulos, S., & Landsberg, G. (2001) · Physical Review Letters · doi:10.1103/physrevlett.87.161602 · Cited by 979 · see citing works
Experience and prediction: An analysis of the foundations and the structure of knowledge.
Reichenbach, H. (1952) · University of Chicago Press eBooks · doi:10.1037/11656-000 · Cited by 903 · see citing works
Busby, J. S., & Barton, S. C. (1996) · Manufacturing Engineer · doi:10.1049/me:19960307 · Cited by 842 · see citing works
Suppes, P. (1957) · CERN Document Server (European Organization for Nuclear Research) · Cited by 766 · see citing works
Junk, W. (1916) · Springer eBooks · doi:10.1007/978-94-017-6613-5_16 · Cited by 586 · see citing works
Giere, R. N. (1999) · Medical Entomology and Zoology · Cited by 473 · see citing works
What we know about spreadsheet errors
Panko, R. R. (1998) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.1998040102 · Cited by 464 · see citing works
Catastrophe: Risk and Response
Posner, R. A. (2004) · Cited by 388 · see citing works
Problems and Changes in the Empiricist Criterion of Meaning
Hempel, C. G. (1950) · Revue internationale de philosophie · Cited by 290 · see citing works
Lord Kelvin and the Age of the Earth
Burchfield, J. D. (1975) · doi:10.1007/978-1-349-02565-7 · Cited by 273 · see citing works
The Galactic Habitable Zone and the Age Distribution of Complex Life in the Milky Way
Lineweaver, C., Fenner, Y., & Gibson, B. (2003) · Science · doi:10.1126/science.1092322 · Cited by 234 · see citing works
A Scientist's Nightmare: Software Problem Leads to Five Retractions
Miller, G. (2006) · Science · doi:10.1126/science.314.5807.1856 · Cited by 205 · see citing works
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181 · see citing works
Effect of Computer Order Entry on Prevention of Serious Medication Errors in Hospitalized Children
Walsh, K. E., Landrigan, C. P., Adams, W. G., Vinci, R. J., Chessare, J. B., Cooper, M. R. C., Hebert, P. M., Schainker, E. G., McLaughlin, T., & Bauchner, H. (2008) · PEDIATRICS · doi:10.1542/peds.2007-0220 · Cited by 170 · see citing works
Decision Making Under Great Uncertainty
Hansson, O. (1996) · Philosophy of the Social Sciences · doi:10.1177/004839319602600304 · Cited by 169 · see citing works
Retractions in the research literature: misconduct or mistakes?
Nath, S. B., Marcus, S. C., & Druss, B. G. (2006) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00504.x · Cited by 161 · see citing works
Garay, L. J., Anglin, J. R., Cirac, J. I., et al. (2000) · Cited by 131 · see citing works
Review of speculative “disaster scenarios” at RHIC
Jaffe, R. L., Busza, W., Wilczek, F. A., & Sandweiss, J. H. (2000) · Reviews of Modern Physics · doi:10.1103/revmodphys.72.1125 · Cited by 120 · see citing works
Drug administration errors and their determinants in pediatric in-patients
Prot, S., Fontan, J. E., Alberti, C., et al. (2005) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzi066 · Cited by 120 · see citing works
How many scientific papers should be retracted?
Cokol, M., Iossifov, I., Rodriguez‐Esteban, R., & Rzhetsky, A. (2007) · The EMBO Reports · doi:10.1038/sj.embor.7400970 · Cited by 113 · see citing works
Astrophysical implications of hypothetical stable TeV-scale black holes
Giddings, S. B., & Mangano, M. L. (2008) · Physical review. D. Particles, fields, gravitation, and cosmology/Physical review. D. Particles and fields · doi:10.1103/physrevd.78.035009 · Cited by 99 · see citing works
Turner, M. S., & Wilczek, F. A. (1982) · Nature · doi:10.1038/298633a0 · Cited by 89 · see citing works
Cognitive biases potentially affecting judgement of global risks
Yudkowsky, E. (2008) · Oxford University Press eBooks · doi:10.1093/oso/9780198570509.003.0009 · Cited by 80 · see citing works
Modelling nature : Between physics and the physical world
Morrison, M. (1998) · Cited by 75 · see citing works
Detection of nitrogen at {100} platelets in diamond
Berger, S. D., & Pennycook, S. J. (1982) · Nature · doi:10.1038/298635a0 · Cited by 74 · see citing works
The chromosome number in humans: a brief history
Gartler, S. M. (2006) · Nature Reviews Genetics · doi:10.1038/nrg1917 · Cited by 59 · see citing works
Prescription errors in psychiatry – a multi-centre study
Stubbs, J. H., Haw, C., & Taylor, D. M. (2006) · Journal of Psychopharmacology · doi:10.1177/0269881106059808 · Cited by 55 · see citing works
Will relativistic heavy-ion colliders destroy our planet?
Arnon Dar, A., Rújula, A. D., & Heinz, U. (1999) · Physics Letters B · doi:10.1016/s0370-2693(99)01307-6 · Cited by 55 · see citing works
Is a doomsday catastrophe likely?
Tegmark, M., & Bostrom, N. (2005) · Nature · doi:10.1038/438754a · Cited by 54 · see citing works
A selected history of expectation bias in physics
Jeng, M. (2006) · American Journal of Physics · doi:10.1119/1.2186333 · Cited by 53 · see citing works
Probabilistic climate forecasts and inductive problems
Frame, D. J., Faull, N., Joshi, M., & Allen, M. R. (2007) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2007.2069 · Cited by 44 · see citing works
Anthropogenic climate change: Scientific uncertainties and moral dilemmas
Hillerbrand, R., & Ghil, M. (2008) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2008.02.015 · Cited by 41 · see citing works
Self-citationA selected history of expectation bias in physics
Jeng, M. (2005) · CERN Document Server (European Organization for Nuclear Research) · Cited by 40 · see citing works
A Critical Look at Risk Assessments for Global Catastrophes
Kent, A. (2004) · Risk Analysis · doi:10.1111/j.0272-4332.2004.00419.x · Cited by 39 · see citing works
Hut, P., & Rees, M. J. (1983) · Nature · doi:10.1038/302508a0 · Cited by 37 · see citing works
Might a laboratory experiment destroy planet earth?
Calogero, F. (2000) · Interdisciplinary Science Reviews · doi:10.1179/030801800679224 · Cited by 6 · see citing works
Consequences of coarse-grained Vlasov equations
Morawetz, K., & Walke, R. (2003) · Physica A Statistical Mechanics and its Applications · doi:10.1016/s0378-4371(03)00507-7 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Existential Risk Prevention as Global Priority
Bostrom, N. (2013) · Global Policy · doi:10.1111/1758-5899.12002 · Cited by 588 · see citing works
Liu, H.-Y., Lauta, K. C., & Maas, M. M. (2018) · Futures · doi:10.1016/j.futures.2018.04.009 · Cited by 59 · see citing works
A Fate Worse Than Warming? Stratospheric Aerosol Injection and Global Catastrophic Risk
Tang, A., & Kemp, L. (2021) · Frontiers in Climate · doi:10.3389/fclim.2021.720312 · Cited by 58 · see citing works
“Understanding risks which are plausible, high-impact, but low or unknown-probability is critical for robust decision making under uncertainty (Ord et al., 2010; Kunreuther et al., 2013; Wagner and Weitzman, 2015).”
Background
- Authors
- Ord, T., Hillerbrand, R., & Sandberg, A.
- Affiliations
- University of Oxford (GB)
- RWTH Aachen University (DE)
- Corresponding author
- Ord, Toby
- Venue
- Journal of Risk Research
- Publisher
- Informa UK Limited
- Published
- 2010-03
- Volume
- 13
- Issue
- 2
- Pages
- 191-205
- ISSN
- 1366-9877 (ISSN-L)
- 1466-4461
- OpenAlex
- W2123253865
- Cited by
- 72
- References
- 42
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 6 · 2023: 3 · 2022: 1 · 2021: 6 · 2020: 9 · 2019: 7 · 2018: 7 · 2017: 8 · 2016: 4 · 2015: 3 · 2014: 3 · 2013: 2 · 2012: 6
- FWCI
- 10.37
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Journal of Risk Research (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- RWTH Publications (RWTH Aachen) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Sociology and Political Science › Risk Perception and Management
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Risk and Safety Analysis
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Keywords
- global catastrophic risk, probability estimation, model uncertainty, parameter uncertainty, expert opinion, Large Hadron Collider, risk assessment methodology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
