The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Use (and abuse) of expert elicitation in support of decision making for public policy
Open accessAbstract
The elicitation of scientific and technical judgments from experts, in the form of subjective probability distributions, can be a valuable addition to other forms of evidence in support of public policy decision making. This paper explores when it is sensible to perform such elicitation and how that can best be done. A number of key issues are discussed, including topics on which there are, and are not, experts who have knowledge that provides a basis for making informed predictive judgments; the inadequacy of only using qualitative uncertainty language; the role of cognitive heuristics and of overconfidence; the choice of experts; the development, refinement, and iterative testing of elicitation protocols that are designed to help experts to consider systematically all relevant knowledge when they make their judgments; the treatment of uncertainty about model functional form; diversity of expert opinion; and when it does or does not make sense to combine judgments from different experts. Although it may be tempting to view expert elicitation as a low-cost, low-effort alternative to conducting serious research and analysis, it is neither. Rather, expert elicitation should build on and use the best available research and analysis and be undertaken only when, given those, the state of knowledge will remain insufficient to support timely informed assessment and decision making.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810
Climate change 2007: the physical science basis
Solomon, S., Qin, D., Manning, M., Marquis, M. C., Averyt, K. B., Tignor, M., Miller, H. L., & Chen, Z. (2007) · Cited by 24,159
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Climate Change 2007: The Physical Science Basis.
Schiavon, S., & Zecchin, R. (2007) · eScholarship (California Digital Library) · Cited by 16,994
Climate Change 2001: The Scientific Basis
Cooper, R. N., Houghton, J. T., McCarthy, J. J., & Metz, B. (2002) · Foreign Affairs · doi:10.2307/20033020 · Cited by 12,889
Barker, T. J., Bashmakov, I., Bernstein, L., Bogner, J. R., Bosch, P. C., Dave, R., Davidson, O., Fisher, B. S., Grubb, M. J., Gupta, S. K., et al. (2007) · Socio-Environmental Systems Modeling · Cited by 9,160
Climate Change 2007: Impacts, Adaptation and Vulnerability
Parry, M. L., Canziani, O. F., Palutikof, J. P., van der Linden, P., & Hanson, C. (2007) · Centre for Environmental Data Analysis Digital Repository (Centre for Environmental Data Analysis) · doi:10.5281/zenodo.7356334 · Cited by 9,136
Judgment under Uncertainty: Heuristics and Biases
(1982) · doi:10.1017/cbo9780511809477 · Cited by 8,240
Climate Change 2007: Mitigation of Climate Change
Change, I. P. O. C. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511546013 · Cited by 7,729
(2013) · Cited by 6,752
Griggs, D. J., & Noguer, M. (2002) · Weather · doi:10.1256/004316502320517344 · Cited by 5,754
Judgment Under Uncertainty: Heuristics and Biases.
Shafer, G. R., Kahnerman, D., Slovic, P., & Tversky, A. (1984) · Journal of the American Statistical Association · doi:10.2307/2288362 · Cited by 5,657
Probability theory the logic of science
Jaynes, E. T., & Bretthorst, G. L. (2003) · Cited by 4,575
Probability Theory: The Logic of Science
Jaynes, E. T. (2003) · doi:10.1017/cbo9780511790423 · Cited by 4,045
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis
Morgan, M. G., & Henrion, M. (1990) · Cambridge University Press eBooks · doi:10.1017/cbo9780511840609 · Cited by 1,711
Self-citationUncertainty and Quality in Science for Policy
Funtowicz, S. O., & Ravetz, J. R. (1990) · doi:10.1007/978-94-009-0621-1 · Cited by 1,575
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Experts in uncertainty: opinion and subjective probability in science
(1992) · Choice Reviews Online · doi:10.5860/choice.29-5666 · Cited by 1,129
Experts In Uncertainty: Opinion and Subjective Probability in Science
Cooke, R. M. (1991) · doi:10.1093/oso/9780195064650.001.0001 · Cited by 1,107
Randolph, P. H. (1970) · Technometrics · doi:10.1080/00401706.1970.10488715 · Cited by 986
Combining Probability Distributions From Experts in Risk Analysis
Clemen, R. T., & Winkler, R. L. (1999) · Risk Analysis · doi:10.1111/j.1539-6924.1999.tb00399.x · Cited by 816
Statistical Methods for Eliciting Probability Distributions
Garthwaite, P. H., Kadane, J. B., & O'Hagan, A. (2005) · Journal of the American Statistical Association · doi:10.1198/016214505000000105 · Cited by 801
Exceptional Paper—Probability Encoding in Decision Analysis
Spetzler, C. S., & Staël Von Holstein, C.-A. S. (1975) · Management Science · doi:10.1287/mnsc.22.3.340 · Cited by 662
Theory of Probability, A Critical Introductory Treatment.
Kingman, J. F. C., de Finetti, B., Machi, A., & Smith, A. (1975) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2345252 · Cited by 601
Clark, W. C., Tomich, T. P., van Noordwijk, M., Guston, D., Catacutan, D., Dickson, N. M., & McNie, E. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0900231108 · Cited by 568
A framework for dealing with uncertainty due to model structure error
Refsgaard, J. C., van der Sluijs, J. P., Brown, J., & van der Keur, P. (2006) · Advances in Water Resources · doi:10.1016/j.advwatres.2005.11.013 · Cited by 535
Woudenberg, F. (1991) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(91)90002-w · Cited by 520
Measuring the vague meanings of probability terms.
Wallsten, T. S., Budescu, D. V., Rapoport, A., Zwick, R., & Forsyth, B. (1986) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.115.4.348 · Cited by 517
Nuclear Regulatory Commission, W., Budnitz, R. J., Nuclear Regulatory Commission, W., Lab., C. (. S. L. L. N., USDOE, W., USDOE, W., Apostolakis, G., & Boore, D. M. (1997) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/479072 · Cited by 501
Physicians' use of probabilistic information in a real clinical setting.
Christensen-Szalanski, J. J. J., & Bushyhead, J. B. (1981) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.7.4.928 · Cited by 443
Van Der Sluijs, J. P., Craye, M., Funtowicz, S., Kloprogge, P., Ravetz, J., & Risbey, J. (2005) · Risk Analysis · doi:10.1111/j.1539-6924.2005.00604.x · Cited by 426
Advances in Decision Analysis: From Foundations to Applications
(2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511611308 · Cited by 398
Subjective Judgments by Climate Experts
MORGAN, M. G., & KEITH, D. W. (1995) · Environmental Science & Technology · doi:10.1021/es00010a753 · Cited by 357
Self-citationA route to more tractable expert advice
Aspinall, W. (2010) · Nature · doi:10.1038/463294a · Cited by 332
TU Delft expert judgment data base
Cooke, R. M., & Goossens, L. L. H. J. (2008) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2007.03.005 · Cited by 317
Assessing uncertainty in physical constants
Henrion, M., & Fischhoff, B. (1986) · American Journal of Physics · doi:10.1119/1.14447 · Cited by 316
The use of expert elicitation in environmental health impact assessment: a seven step procedure
Knol, A. B., Slottje, P., van der Sluijs, J. P., & Lebret, E. (2010) · Environmental Health · doi:10.1186/1476-069x-9-19 · Cited by 277
Oppenheimer, M., O'Neill, B. C., Webster, M., & Agrawala, S. (2007) · Science · doi:10.1126/science.1144831 · Cited by 265
(2001) · Choice Reviews Online · doi:10.5860/choice.38-6233 · Cited by 245
Learning from integrated assessment of climate change
Morgan, M. G., & Dowlatabadi, H. (1996) · Climatic Change · doi:10.1007/bf00139297 · Cited by 201
Self-citationExpert elicitation on ultrafine particles: likelihood of health effects and causal pathways
Knol, A. B., de Hartog, J. J., Boogaard, H., Slottje, P., van der Sluijs, J. P., Lebret, E., Cassee, F. R., Wardekker, J. A., Ayres, J. G., Borm, P. J., et al. (2009) · Particle and Fibre Toxicology · doi:10.1186/1743-8977-6-19 · Cited by 192
Wardekker, J. A., van der Sluijs, J. P., Janssen, P. H. M., Kloprogge, P., & Petersen, A. C. (2008) · Environmental Science & Policy · doi:10.1016/j.envsci.2008.05.005 · Cited by 189
A web-based tool for eliciting probability distributions from experts
Morris, D. E., Oakley, J. E., & Crowe, J. A. (2014) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2013.10.010 · Cited by 177
Elicitation of Expert Judgments of Climate Change Impacts on Forest Ecosystems
Morgan, M. G., Pitelka, L. F., & Shevliakova, E. (2001) · Climatic Change · doi:10.1023/a:1010651300697 · Cited by 148
Self-citationMorgan, G., Dowlatabadi, H., Henrion, M., Keith, D. W., Lempert, R. J., McBrid, S., Small, M. J., & Wilbanks, T. J. (2009) · U.S. Climate Change Science Program eBooks · Cited by 131
Technical Uncertainty in Quantitative Policy Analysis — A Sulfur Air Pollution Example
Morgan, M. G., Morris, S. C., Henrion, M., Amaral, D. A. L., & Rish, W. R. (1984) · Risk Analysis · doi:10.1111/j.1539-6924.1984.tb00139.x · Cited by 130
Self-citationExpert Assessments of Future Photovoltaic Technologies
Curtright, A. E., Morgan, M. G., & Keith, D. W. (2008) · Environmental Science & Technology · doi:10.1021/es8014088 · Cited by 129
Self-citationZickfeld, K., Levermann, A., Morgan, M. G., Kuhlbrodt, T., Rahmstorf, S., & Keith, D. W. (2007) · Climatic Change · doi:10.1007/s10584-007-9246-3 · Cited by 129
Self-citationEvaluation of potential cost reductions from improved amine-based CO2 capture systems
Rao, A. B., Rubin, E. S., Keith, D. W., & Morgan, M. G. (2005) · Energy Policy · doi:10.1016/j.enpol.2005.08.004 · Cited by 124
Self-citationRoman, H. A., Walker, K. D., Walsh, T. L., Conner, L., Richmond, H. M., Hubbell, B., & Kinney, P. L. (2008) · Environmental Science & Technology · doi:10.1021/es0713882 · Cited by 122
Cooke, R. M., Wilson, A., Tuomisto, J. T., Morales‐Nápoles, O., Tainio, M., & Evans, J. S. (2007) · Environmental Science & Technology · doi:10.1021/es0714078 · Cited by 117
Eliciting Probabilities from Experts
Hora, S. C. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511611308.009 · Cited by 112
‘Combining Probability Distributions from Experts in Risk Analysis’
Kaplan, S. (2000) · Risk Analysis · doi:10.1111/0272-4332.202015 · Cited by 110
Expert assessments of the cost of light water small modular reactors
Abdulla, A., Azevedo, I. L., & Morgan, M. G. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1300195110 · Cited by 109
Self-citationComment on Cooke's classical method
Clemen, R. T. (2008) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2008.02.003 · Cited by 98
Zickfeld, K., Morgan, M. G., Frame, D. J., & Keith, D. W. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0908906107 · Cited by 97
Self-citationUse of Probabilistic Expert Judgment in Uncertainty Analysis of Carcinogenic Potency
Evans, J. S., Gray, G. M., Sielken, R. L., et al. (1994) · Regulatory Toxicology and Pharmacology · doi:10.1006/rtph.1994.1034 · Cited by 95
Elicitation of Expert Judgments of Aerosol Forcing
Morgan, M. G., Adams, P. J., & Keith, D. W. (2006) · Climatic Change · doi:10.1007/s10584-005-9025-y · Cited by 88
Self-citationvon Krauss, M. P. K., Casman, E. A., & Small, M. J. (2004) · Risk Analysis · doi:10.1111/j.0272-4332.2004.00546.x · Cited by 77
Expert Judgments about RD&D and the Future of Nuclear Energy
Anadón, L. D., Bosetti, V., Bunn, M., Catenacci, M., & Lee, A. (2012) · Environmental Science & Technology · doi:10.1021/es300612c · Cited by 75
Probabilistic Coherence Weighting for Optimizing Expert Forecasts
Karvetski, C. W., Olson, K. C., Mandel, D. R., & Twardy, C. R. (2013) · Decision Analysis · doi:10.1287/deca.2013.0279 · Cited by 47
Commentary: Uncertainty Analysis in Risk Assessment
Morgan, M. G. (1998) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.1998.11009680 · Cited by 43
Self-citationNOAA Data Buoy Office Programs
Hamilton, G. D. (1980) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1980)061<1546:sipfit>2.0.co;2 · Cited by 33
Uncertainty Assessment of Voc Emissions from Paint in the Netherlands Using the Nusap System
van der Sluijs, J. P., Risbey, J. S., & Ravetz, J. R. (2005) · Environmental Monitoring and Assessment · doi:10.1007/s10661-005-3697-7 · Cited by 32
A cautionary note on global recalibration
Kadane, J. B., & Fischhoff, B. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500004460 · Cited by 12
Cited by
The role of artificial intelligence in achieving the Sustainable Development Goals
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S., Felländer, A., Langhans, S. D., Tegmark, M., & Fuso Nerini, F. (2020) · Nature Communications · doi:10.1038/s41467-019-14108-y · Cited by 3,294
Future cost and performance of water electrolysis: An expert elicitation study
Schmidt, O., Gambhir, A., Staffell, I., Hawkes, A., Nelson, J., & Few, S. (2017) · International Journal of Hydrogen Energy · doi:10.1016/j.ijhydene.2017.10.045 · Cited by 2,067
Mapping synergies and trade-offs between energy and the Sustainable Development Goals
Fuso Nerini, F., Tomei, J., To, L. S., Bisaga, I., Parikh, P., Black, M., Borrion, A., Spataru, C., Castán Broto, V., Anandarajah, G., et al. (2017) · Nature Energy · doi:10.1038/s41560-017-0036-5 · Cited by 1,090
Climate as a risk factor for armed conflict
Mach, K. J., Kraan, C. M., Adger, W. N., Buhaug, H., Burke, M., Fearon, J. D., Field, C. B., Hendrix, C. S., Maystadt, J.-F., O’Loughlin, J., et al. (2019) · Nature · doi:10.1038/s41586-019-1300-6 · Cited by 619
Iterative near-term ecological forecasting: Needs, opportunities, and challenges
Dietze, M. C., Fox, A., Beck-Johnson, L. M., Betancourt, J. L., Hooten, M. B., Jarnevich, C. S., Keitt, T. H., Kenney, M. A., Laney, C. M., Larsen, L. G., et al. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1710231115 · Cited by 604
Metadata
- Authors
- Morgan, M. G.
- Venue
- Proceedings of the National Academy of Sciences
- Published
- 2014-05-12
- Volume
- 111
- Issue
- 20
- Pages
- 7176-7184
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- OpenAlex
- W2111275873
- Cited by
- 702
- References
- 68
- FWCI
- 55.04
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.