The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
A practical guide to structured expert elicitation using the IDEA protocol
Open accessAbstract
Expert judgement informs a variety of important applications in conservation and natural resource management, including threatened species management, environmental impact assessment and structured decision‐making. However, expert judgements can be prone to contextual biases. Structured elicitation protocols mitigate these biases, and improve the accuracy and transparency of the resulting judgements. Despite this, the elicitation of expert judgement within conservation and natural resource management remains largely informal. We suggest this may be attributed to financial and practical constraints, which are not addressed by many existing structured elicitation protocols. In this paper, we advocate that structured elicitation protocols must be adopted when expert judgements are used to inform science. In order to motivate a wider adoption of structured elicitation protocols, we outline the IDEA protocol. The protocol improves the accuracy of expert judgements and includes several key steps which may be familiar to many conservation researchers, such as the four‐step elicitation, and a modified Delphi procedure (“Investigate,” “Discuss,” “Estimate” and “Aggregate”). It can also incorporate remote elicitation, making structured expert judgement accessible on a modest budget. The IDEA protocol has recently been outlined in the scientific literature; however, a detailed description has been missing. This paper fills that important gap by clearly outlining each of the steps required to prepare for and undertake an elicitation. While this paper focuses on the need for the IDEA protocol within conservation and natural resource management, the protocol (and the advice contained in this paper) is applicable to a broad range of scientific domains, as evidenced by its application to biosecurity, engineering and political forecasting. By clearly outlining the IDEA protocol, we hope that structured protocols will be more widely understood and adopted, resulting in improved judgements and increased transparency when expert judgement is required.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Confirmation Bias: A Ubiquitous Phenomenon in Many Guises
Nickerson, R. S. (1998) · Review of General Psychology · doi:10.1037/1089-2680.2.2.175 · Cited by 7,267
On the psychology of prediction.
Kahneman, D., & Tversky, A. (1973) · Psychological Review · doi:10.1037/h0034747 · Cited by 6,202 · cited 1×
“This is often due to a range of contextual biases and heuristics such as anchoring, availability, and representativeness (Kahneman & Tversky, 1973), groupthink (Janis, 1971), overconfidence (Soll & Klayman, 2004) and difficulties associated with communicating knowledge in numbers and probabilities…”
Background(2004) · Choice Reviews Online · doi:10.5860/choice.42-1645 · Cited by 2,925
Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis
Morgan, M. G., & Henrion, M. (1990) · Cambridge University Press eBooks · doi:10.1017/cbo9780511840609 · Cited by 1,711
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
A literature review of the anchoring effect
Furnham, A., & Boo, H. C. (2011) · The Journal of Socio-Economics · doi:10.1016/j.socec.2010.10.008 · Cited by 1,179
Experts In Uncertainty: Opinion and Subjective Probability in Science
Cooke, R. M. (1991) · doi:10.1093/oso/9780195064650.001.0001 · Cited by 1,107
How social influence can undermine the wisdom of crowd effect
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008636108 · Cited by 1,017
Superforecasting: the art and science of prediction
(2016) · Choice Reviews Online · doi:10.5860/choice.196400 · Cited by 919
Rational choice in an uncertain world the psychology of judgment and decision making
Hastie, R., & Dawes, R. M. (2010) · Cited by 904
Page, S. (2008) · Princeton University Press eBooks · doi:10.1515/9781400830282 · Cited by 881
Eliciting Expert Knowledge in Conservation Science
MARTIN, T. G., BURGMAN, M. A., FIDLER, F., KUHNERT, P. M., LOW‐CHOY, S., MCBRIDE, M., & MENGERSEN, K. (2012) · Conservation Biology · doi:10.1111/j.1523-1739.2011.01806.x · Cited by 829
Self-citationStatistical Methods for Eliciting Probability Distributions
Garthwaite, P. H., Kadane, J. B., & O'Hagan, A. (2005) · Journal of the American Statistical Association · doi:10.1198/016214505000000105 · Cited by 801
Simple tools for understanding risks: from innumeracy to insight
Gigerenzer, G., & Edwards, A. (2003) · BMJ · doi:10.1136/bmj.327.7417.741 · Cited by 763
Probabilistic Risk Analysis: Foundations and Methods
Bedford, T., & Cooke, R. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813597 · Cited by 754
Mastrandrea, M. D., Field, C. B., Stocker, T. F., Edenhofer, O., Ebi, K. L., Frame, D. J., Held, H., Kriegler, E., Mach, K. J., Matschoss, P., et al. (2010) · MPG.PuRe (Max Planck Society) · Cited by 705
Use (and abuse) of expert elicitation in support of decision making for public policy
Morgan, M. G. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319946111 · Cited by 702
Measuring the vague meanings of probability terms.
Wallsten, T. S., Budescu, D. V., Rapoport, A., Zwick, R., & Forsyth, B. (1986) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.115.4.348 · Cited by 517
Overconfidence in Interval Estimates.
Soll, J. B., & Klayman, J. (2004) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.30.2.299 · Cited by 493
Runge, M. C., Converse, S. J., & Lyons, J. E. (2011) · Biological Conservation · doi:10.1016/j.biocon.2010.12.020 · Cited by 450
A guide to eliciting and using expert knowledge in Bayesian ecological models
Kuhnert, P. M., Martin, T. G., & Griffiths, S. P. (2010) · Ecology Letters · doi:10.1111/j.1461-0248.2010.01477.x · Cited by 441
The Role of Expert Opinion in Environmental Modelling
Krueger, T., Page, T., Hubacek, K., Smith, L., & Hiscock, K. (2012) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.01.011 · Cited by 413
Acting fast helps avoid extinction
Martin, T. G., Nally, S., Burbidge, A. A., Arnall, S., Garnett, S. T., Hayward, M. W., Lumsden, L. F., Menkhorst, P., McDonald‐Madden, E., & Possingham, H. P. (2012) · Conservation Letters · doi:10.1111/j.1755-263x.2012.00239.x · Cited by 374
Reducing Overconfidence in the Interval Judgments of Experts
Speirs-Bridge, A., Fidler, F., McBride, M., Flander, L., Cumming, G., & Burgman, M. (2010) · Risk Analysis · doi:10.1111/j.1539-6924.2009.01337.x · Cited by 351
Self-citationA route to more tractable expert advice
Aspinall, W. (2010) · Nature · doi:10.1038/463294a · Cited by 332
A note on aggregating opinions
Hogarth, R. M. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90037-5 · Cited by 321
An expert judgement assessment of future sea level rise from the ice sheets
Bamber, J. L., & Aspinall, W. P. (2013) · Nature Climate Change · doi:10.1038/nclimate1778 · Cited by 316
Burgman, M. A., McBride, M., Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F., Rumpff, L., & Twardy, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022998 · Cited by 314
Self-citationPsychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312 · cited 7×
“Importantly, it should account for each step of the elicitation including the recruitment of experts, the framing of questions, the elicitation and aggregation of their judgements, using procedures that have been tested and clearly demonstrated to improve judgements (e.g. Cooke, 1991; Mellers et al., 2014).”
BackgroundMethodInfluentialPerformance-based assessment of expertise: How to decide if someone is an expert or not
Shanteau, J., Weiss, D. J., Thomas, R. P., & Pounds, J. C. (2002) · European Journal of Operational Research · doi:10.1016/s0377-2217(01)00113-8 · Cited by 309
Eliciting probabilities from experts in complex technical problems
Keeney, R. L., & von Winterfeldt, D. (1991) · IEEE Transactions on Engineering Management · doi:10.1109/17.83752 · Cited by 309
Choy, S. L., O'Leary, R., & Mengersen, K. (2009) · Ecology · doi:10.1890/07-1886.1 · Cited by 283
The use of expert elicitation in environmental health impact assessment: a seven step procedure
Knol, A. B., Slottje, P., van der Sluijs, J. P., & Lebret, E. (2010) · Environmental Health · doi:10.1186/1476-069x-9-19 · Cited by 277
Trasande, L., Cortes, J. E., Landrigan, P. J., Abercrombie, M. I., Bopp, R. F., & Cifuentes, E. (2010) · Environmental Health · doi:10.1186/1476-069x-9-1 · Cited by 257
Armstrong, J. S. (2001) · International Series in Operations Research & Management Science · doi:10.1007/978-0-306-47630-3_19 · Cited by 237
Trusting Judgements: How to Get the Best out of Experts
Burgman, M. A. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9781316282472 · Cited by 232
Self-citationIs It Better to Average Probabilities or Quantiles?
Lichtendahl, K. C., Grushka‐Cockayne, Y., & Winkler, R. L. (2013) · Management Science · doi:10.1287/mnsc.1120.1667 · Cited by 208
McBride, M. F., Garnett, S. T., Szabo, J. K., Burbidge, A. H., Butchart, S. H. M., Christidis, L., Dutson, G., Ford, H. A., Loyn, R. H., Watson, D. M., et al. (2012) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00221.x · Cited by 186
Self-citationQuantifying scientific uncertainty from expert judgement elicitation
Aspinall, W. P., & Cooke, R. M. (2013) · Cambridge University Press eBooks · doi:10.1017/cbo9781139047562.005 · Cited by 151
Procedures Guide for Structural Expert Judgement in Accident Consequence Modelling
Cooke, R. M., & Goossens, L. H. J. (2000) · Radiation Protection Dosimetry · doi:10.1093/oxfordjournals.rpd.a033152 · Cited by 151
I nvestigate D iscuss E stimate A ggregate for structured expert judgement
Hanea, A. M., McBride, M. F., Burgman, M. A., Wintle, B. C., Fidler, F. M., Flander, L. B., Twardy, C. R., Manning, B., & Mascaro, S. (2016) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2016.02.008 · Cited by 151
Self-citationProbability Judgments for Continuous Quantities: Linear Combinations and Calibration
Hora, S. C. (2004) · Management Science · doi:10.1287/mnsc.1040.0205 · Cited by 146
Use of expert knowledge to elicit population trends for the koala (Phascolarctos cinereus)
Adams‐Hosking, C., McBride, M. F., BAXTER, G. S., Burgman, M. A., de Villiers, D., Kavanagh, R. P., Lawler, I. R., Lunney, D., Melzer, A., Menkhorst, P., et al. (2016) · Diversity and Distributions · doi:10.1111/ddi.12400 · Cited by 142
Self-citationCross validation for the classical model of structured expert judgment
Colson, A. R., & Cooke, R. M. (2017) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2017.02.003 · Cited by 126
French, S. (2011) · Revista de la Real Academia de Ciencias Exactas Físicas y Naturales Serie A Matemáticas · doi:10.1007/s13398-011-0018-6 · Cited by 109
Classical meets modern in the IDEA protocol for structured expert judgement
Hanea, A. M., McBride, M. F., Burgman, M. A., & Wintle, B. C. (2016) · Journal of Risk Research · doi:10.1080/13669877.2016.1215346 · Cited by 98
Self-citationShip-borne Nonindigenous Species Diminish Great Lakes Ecosystem Services
Rothlisberger, J. D., Finnoff, D., Cooke, R. M., & Lodge, D. M. (2012) · Ecosystems · doi:10.1007/s10021-012-9522-6 · Cited by 96
Benefits of integrating complementarity into priority threat management
Chadès, I., Nicol, S., van Leeuwen, S. J., Walters, B., Firn, J. L., Reeson, A. F., Martin, T. G., & Carwardine, J. (2014) · Conservation Biology · doi:10.1111/cobi.12413 · Cited by 89
Words Of Estimative Probability
Kent, S. (1964) · PhilPapers (PhilPapers Foundation) · Cited by 86
Use of structured expert judgment to forecast invasions by bighead and silver carp in Lake Erie
Wittmann, M. E., Cooke, R. M., Rothlisberger, J. D., Rutherford, E. S., Zhang, H., Mason, D. M., & Lodge, D. M. (2014) · Conservation Biology · doi:10.1111/cobi.12369 · Cited by 80
McBride, M. F., Fidler, F. M., & Burgman, M. A. (2012) · Diversity and Distributions · doi:10.1111/j.1472-4642.2012.00884.x · Cited by 78
Self-citationThe Consistency of Extinction Risk Classification Protocols
Regan, T. J., Burgman, M. A., McCarthy, M. A., et al. (2005) · Conservation Biology · doi:10.1111/j.1523-1739.2005.00235.x · Cited by 72
Self-citationTiming of Protection of Critical Habitat Matters
Martin, T. G., Camaclang, A. E., Possingham, H. P., Maguire, L. A., & Chadès, I. (2016) · Conservation Letters · doi:10.1111/conl.12266 · Cited by 69
Ban, S. S., Pressey, R. L., & Graham, N. A. J. (2014) · Global Environmental Change · doi:10.1016/j.gloenvcha.2014.04.018 · Cited by 68
Improving visual estimation through active feedback
Wintle, B. C., Fidler, F. M., Vesk, P. A., & Moore, J. L. (2012) · Methods in Ecology and Evolution · doi:10.1111/j.2041-210x.2012.00254.x · Cited by 54
Self-citationFirn, J. L., Martin, T. G., Chadès, I., Walters, B., Hayes, J. F., Nicol, S., & Carwardine, J. (2015) · Journal of Applied Ecology · doi:10.1111/1365-2664.12500 · Cited by 49
Expert frailties in conservation risk assessment and listing decisions
Burgman, M. A. (2004) · doi:10.7882/fs.2004.054 · Cited by 43
Self-citationA Practical Approach to Address Uncertainty in Stakeholder Deliberations
Gregory, R., & Keeney, R. L. (2017) · Risk Analysis · doi:10.1111/risa.12638 · Cited by 41
Cited by
Expert Knowledge Elicitation: Subjective but Scientific
O’Hagan, A. (2019) · The American Statistician · doi:10.1080/00031305.2018.1518265 · Cited by 368
Regenerative tourism: a conceptual framework leveraging theory and practice
Bellato, L., Frantzeskaki, N., & Nygaard, C. A. (2023) · Tourism Geographies · doi:10.1080/14616688.2022.2044376 · Cited by 320
How many bird and mammal extinctions has recent conservation action prevented?
Bolam, F. C., Mair, L., Angelico, M., Brooks, T. M., Burgman, M., Hermes, C., Hoffmann, M., Martin, R. W., McGowan, P. J. K., Rodrigues, A. S. L., et al. (2021) · Conservation Letters · doi:10.1111/conl.12762 · Cited by 286
“Delphi process We used a Delphi expert elicitation method following Hemming et al. (2018) and Mukherjee et al. (2015).”
MethodSelf-citationQuantifying Earth system interactions for sustainable food production via expert elicitation
Chrysafi, A., Virkki, V., Jalava, M., Sandström, V., Piipponen, J., Porkka, M., Lade, S. J., La Mere, K., Wang-Erlandsson, L., Scherer, L., et al. (2022) · Nature Sustainability · doi:10.1038/s41893-022-00940-6 · Cited by 41
Progress and opportunities in advancing near-term forecasting of freshwater quality
Lofton, M. E., Howard, D. W., Thomas, R. Q., & Carey, C. C. (2022) · doi:10.1002/essoar.10512426.1 · Cited by 38
Metadata
- Authors
- Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C.
- Venue
- Methods in Ecology and Evolution
- Publisher
- Wiley
- Published
- 2017-09-05
- Volume
- 9
- Issue
- 1
- Pages
- 169-180
- ISSN
- 2041-210X (ISSN-L)
- OpenAlex
- W2741638098
- Cited by
- 528
- References
- 59
- FWCI
- 59.40
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.