The DOI matches its Crossref record; 101 of the 101 references it cites were found in a registry or an index.
Statistical Methods for Eliciting Probability Distributions
Open accessAbstract
Elicitation is a key task for subjectivist Bayesians. Although skeptics hold that elicitation cannot (or perhaps should not) be done, in practice it brings statisticians closer to their clients and subject-matter expert colleagues. This article reviews the state of the art, reflecting the experience of statisticians informed by the fruits of a long line of psychological research into how people represent uncertain information cognitively and how they respond to questions about that information. In a discussion of the elicitation process, the first issue to address is what it means for an elicitation to be successful; that is, what criteria should be used. Our answer is that a successful elicitation faithfully represents the opinion of the person being elicited. It is not necessarily “true” in some objectivistic sense, and cannot be judged in that way. We see that elicitation as simply part of the process of statistical modeling. Indeed, in a hierarchical model at which point the likelihood ends and the prior begins is ambiguous. Thus the same kinds of judgment that inform statistical modeling in general also inform elicitation of prior distributions. The psychological literature suggests that people are prone to certain heuristics and biases in how they respond to situations involving uncertainty. As a result, some of the ways of asking questions about uncertain quantities are preferable to others, and appear to be more reliable. However, data are lacking on exactly how well the various methods work, because it is unclear, other than by asking using an elicitation method, just what the person believes. Consequently, one is reduced to indirect means of assessing elicitation methods. The tool chest of methods is growing. Historically, the first methods involved choosing hyperparameters using conjugate prior families, at a time when these were the only families for which posterior distributions could be computed. Modern computational methods, such as Markov chain Monte Carlo, have freed elicitation from this constraint. As a result, now both parametric and nonparametric methods are available for low-dimensional problems. High-dimensional problems are probably best thought of as lacking another hierarchical level, which has the effect of reducing the as-yet-unelicited parameter space. Special considerations apply to the elicitation of group opinions. Informal methods, such as Delphi, encourage the participants to discuss the issue in the hope of reaching consensus. Formal methods, such as weighted averages or logarithmic opinion pools, each have mathematical characteristics that are uncomfortable. Finally, there is the question of what a group opinion even means, because it is not necessarily the opinion of any participant.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Availability: A heuristic for judging frequency and probability
Tversky, A., & Kahneman, D. (1973) · Cognitive Psychology · doi:10.1016/0010-0285(73)90033-9 · Cited by 9,984 · cited 2×
“Hence, most people judge that “r” is more likely to be the first letter of a word, rather than the third letter, although the reverse is true (Tversky and Kahneman, 1973).”
BackgroundSocial Choice and Individual Values.
Shackle, G. L. S., & Arrow, K. J. (1951) · Economica · doi:10.2307/2549613 · Cited by 7,792
On the psychology of prediction.
Kahneman, D., & Tversky, A. (1973) · Psychological Review · doi:10.1037/h0034747 · Cited by 6,202 · cited 2×
“X being a librarian (Kahneman and Tversky 1973).”
BackgroundThe growth of logical thinking: From childhood to adolescence.
Inhelder, B., & Piaget, J. (1958) · Basic Books · doi:10.1037/10034-000 · Cited by 4,596
Degroot, M. H. (1974) · Journal of the American Statistical Association · doi:10.1080/01621459.1974.10480137 · Cited by 4,004
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246 · cited 2×
“Tversky and Kahneman (1971) refered to this false logic as the “law of small numbers,” which asserts that the law of large numbers applies to small numbers as well.”
BackgroundKendall's Advanced Theory of Statistics.
AA, Stuart, A., & Ord, K. (1995) · Journal of the American Statistical Association · doi:10.2307/2291187 · Cited by 2,619
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Decision Analysis: Introductory Lectures on Choices Under Uncertainty.
Klevorick, A. K., & Raiffa, H. (1969) · The Journal of Finance · doi:10.2307/2325714 · Cited by 1,928
Cited by
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Bayesian estimation supersedes the t-test
Kruschke, J. K. (2013) · Journal of Experimental Psychology: General · doi:10.1037/a0029146 · Cited by 1,530
Bayes factor design analysis: Planning for compelling evidence
Schönbrodt, F. D., & Wagenmakers, E.-J. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1230-y · Cited by 729
Use (and abuse) of expert elicitation in support of decision making for public policy
Morgan, M. G. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319946111 · Cited by 702
Good practices for quantitative bias analysis
Lash, T. L., Fox, M. P., MacLehose, R. F., Maldonado, G., McCandless, L. C., & Greenland, S. (2014) · International Journal of Epidemiology · doi:10.1093/ije/dyu149 · Cited by 646
A practical guide to structured expert elicitation using the IDEA protocol
Hemming, V., Burgman, M. A., Hanea, A. M., McBride, M. F., & Wintle, B. C. (2018) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12857 · Cited by 528
A guide to eliciting and using expert knowledge in Bayesian ecological models
Kuhnert, P. M., Martin, T. G., & Griffiths, S. P. (2010) · Ecology Letters · doi:10.1111/j.1461-0248.2010.01477.x · Cited by 441
The Role of Expert Opinion in Environmental Modelling
Krueger, T., Page, T., Hubacek, K., Smith, L., & Hiscock, K. (2012) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2012.01.011 · Cited by 413
Imprecise probability assessment of tipping points in the climate system
Kriegler, E., Hall, J. W., Held, H., Dawson, R., & Schellnhuber, H. J. (2009) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0809117106 · Cited by 393
Modeling uncertainty in multi-criteria decision analysis
Durbach, I. N., & Stewart, T. J. (2012) · European Journal of Operational Research · doi:10.1016/j.ejor.2012.04.038 · Cited by 378
Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment
Authority, E. F. S. (2014) · EFSA Journal · doi:10.2903/j.efsa.2014.3734 · Cited by 330
Burgman, M. A., McBride, M., Ashton, R., Speirs-Bridge, A., Flander, L., Wintle, B., Fidler, F., Rumpff, L., & Twardy, C. (2011) · PLoS ONE · doi:10.1371/journal.pone.0022998 · Cited by 314
Assurance in clinical trial design
O'Hagan, A., Stevens, J. W., & Campbell, M. J. (2005) · Pharmaceutical Statistics · doi:10.1002/pst.175 · Cited by 294
Self-citationClaxton, K. P., & Sculpher, M. J. (2006) · PharmacoEconomics · doi:10.2165/00019053-200624110-00003 · Cited by 279
Using Bayesian Deep Learning to Capture Uncertainty for Residential Net Load Forecasting
Sun, M., Zhang, T., Wang, Y., Strbac, G., & Kang, C. (2020) · IEEE Transactions on Power Systems · doi:10.1109/tpwrs.2019.2924294 · Cited by 262
Determining the Effective Sample Size of a Parametric Prior
Morita, S., Thall, P. F., & Müller, P. (2008) · Biometrics · doi:10.1111/j.1541-0420.2007.00888.x · Cited by 257
Probabilistic inference for future climate using an ensemble of climate model evaluations.
Rougier, J. (2007) · Climatic Change · doi:10.1007/s10584-006-9156-9 · Cited by 218
“Therefore even if its detailed prescriptions seem to climate scientists to be inappropriate (although there is no evidence from current practice that they should be), the general approach should serve as a template for all model-based probabilistic inference for climate.”
BackgroundDetermining informative priors for cognitive models
Lee, M. D., & Vanpaemel, W. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-017-1238-3 · Cited by 120
Reliable support: measuring calibration of likelihood ratios
Ramos, D., & Gonzalez-Rodriguez, J. (2013) · Forensic Science International · doi:10.1016/j.forsciint.2013.04.014 · Cited by 118
Bayesian estimation in random effects meta‐analysis using a non‐informative prior
Bodnar, O., Link, A., Arendacká, B., Possolo, A., & Elster, C. (2017) · Statistics in Medicine · doi:10.1002/sim.7156 · Cited by 73
“previous, similar meta-analyses or from experts, and formal elicitation procedures are available [42,43].”
ResultCan LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., & Hooi, B. (2023) · arXiv · doi:10.48550/arxiv.2306.13063 · Cited by 618
Quantifying Earth system interactions for sustainable food production via expert elicitation
Chrysafi, A., Virkki, V., Jalava, M., Sandström, V., Piipponen, J., Porkka, M., Lade, S. J., La Mere, K., Wang-Erlandsson, L., Scherer, L., et al. (2022) · Nature Sustainability · doi:10.1038/s41893-022-00940-6 · Cited by 41
Metadata
- Authors
- Garthwaite, P. H., Kadane, J. B., & O'Hagan, A.
- Venue
- Journal of the American Statistical Association
- Publisher
- Informa UK Limited
- Published
- 2005-06
- Volume
- 100
- Issue
- 470
- Pages
- 680-701
- ISSN
- 0162-1459 (ISSN-L)
- 1537-274X
- OpenAlex
- W2079025608
- Cited by
- 801
- References
- 101
- FWCI
- 25.28
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.