The effects of averaging subjective probability estimates between and within judges.
Abstract
The average probability estimate of J > 1 judges is generally better than its components. Two studies test 3 predictions regarding averaging that follow from theorems based on a cognitive model of the judges and idealizations of the judgment situation. Prediction 1 is that the average of conditionally pairwise independent estimates will be highly diagnostic, and Prediction 2 is that the average of dependent estimates (differing only by independent error terms) may be well calibrated. Prediction 3 contrasts between- and within-subject averaging. Results demonstrate the predictions' robustness by showing the extent to which they hold as the information conditions depart from the ideal and as J increases. Practical consequences are that (a) substantial improvement can be obtained with as few as 2-6 judges and (b) the decision maker can estimate the nature of the expected improvement by considering the information conditions.
Cites (7)
- Calibration of probabilities: the state of the art to 1980Lichtenstein, Sarah, Fischhoff, Baruch, Phillips, Lawrence D. · 1982 · Judgment under Uncertainty
- Probabilistic mental models: a Brunswikian theory of confidenceGigerenzer, Gerd, Hoffrage, Ulrich, Kleinbölting, Heinz · 1991 · Psychological Review
- Reasons for confidence.Koriat, Asher, Lichtenstein, Sarah, Fischhoff, Baruch · 1980 · Journal of Experimental Psychology: Human Learning and Memory
- Varieties of Confirmation BiasKlayman, Joshua · 1995 · Psychology of Learning and Motivation
- Simultaneous over- and underconfidence: the role of error in judgment processesErev, Ido, Wallsten, Thomas S., Budescu, David V. · 1994 · Psychological Review
- Reliability of Subjective Probability Forecasts of Precipitation and TemperatureMurphy, Allan H., Winkler, Robert L. · 1977 · Applied Statistics
- Confidence in judgmentHarvey, Nigel · 1997 · Trends in Cognitive Sciences
Cited by (14)
- Group Performance and Decision MakingKerr, Norbert L., Tindale, R. Scott · 2004 · Annual Review of Psychology
- Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.Pleskac, Timothy J., Busemeyer, Jerome R. · 2010 · Psychological Review
- Strategies for revising judgment: How (and how well) people use others’ opinions.Soll, Jack B., Larrick, Richard P. · 2009 · Journal of Experimental Psychology: Learning, Memory, and Cognition
- Intuitions about combining opinions: misappreciation of the averaging principleLarrick, Richard P., Soll, Jack B. · 2006 · Management Science
- Measuring the Crowd Within: Probabilistic Representations Within IndividualsVul, Edward, Pashler, Harold · 2008 · Psychological Science
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The effects of averaging subjective probability estimates between and within judges. https://app.cochat.ai/papers/10.1037//1076-898x.6.2.130
- The paper
- Ariely, D., Tung Au, W., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H., Wallsten, T. S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied. https://doi.org/10.1037//1076-898x.6.2.130
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.