The effects of averaging subjective probability estimates between and within judges.

Ariely, Dan, Tung Au, Wing, Bender, Randall H., Budescu, David V., Dietz, Christiane B., Gu, Hongbin, Wallsten, Thomas S., Zauberman, Gal·2000·Journal of Experimental Psychology: Applied

Abstract

The average probability estimate of J > 1 judges is generally better than its components. Two studies test 3 predictions regarding averaging that follow from theorems based on a cognitive model of the judges and idealizations of the judgment situation. Prediction 1 is that the average of conditionally pairwise independent estimates will be highly diagnostic, and Prediction 2 is that the average of dependent estimates (differing only by independent error terms) may be well calibrated. Prediction 3 contrasts between- and within-subject averaging. Results demonstrate the predictions' robustness by showing the extent to which they hold as the information conditions depart from the ideal and as J increases. Practical consequences are that (a) substantial improvement can be obtained with as few as 2-6 judges and (b) the decision maker can estimate the nature of the expected improvement by considering the information conditions.

Cites (7)

  1. Calibration of probabilities: the state of the art to 1980
    Lichtenstein, Sarah, Fischhoff, Baruch, Phillips, Lawrence D. · 1982 · Judgment under Uncertainty
  2. Probabilistic mental models: a Brunswikian theory of confidence
    Gigerenzer, Gerd, Hoffrage, Ulrich, Kleinbölting, Heinz · 1991 · Psychological Review
  3. Reasons for confidence.
    Koriat, Asher, Lichtenstein, Sarah, Fischhoff, Baruch · 1980 · Journal of Experimental Psychology: Human Learning and Memory
  4. Varieties of Confirmation Bias
    Klayman, Joshua · 1995 · Psychology of Learning and Motivation
  5. Simultaneous over- and underconfidence: the role of error in judgment processes
    Erev, Ido, Wallsten, Thomas S., Budescu, David V. · 1994 · Psychological Review
  6. Reliability of Subjective Probability Forecasts of Precipitation and Temperature
    Murphy, Allan H., Winkler, Robert L. · 1977 · Applied Statistics
  7. Confidence in judgment
    Harvey, Nigel · 1997 · Trends in Cognitive Sciences

Show all 7 references

Cited by (14)

  1. Group Performance and Decision Making
    Kerr, Norbert L., Tindale, R. Scott · 2004 · Annual Review of Psychology
  2. Two-stage dynamic signal detection: A theory of choice, decision time, and confidence.
    Pleskac, Timothy J., Busemeyer, Jerome R. · 2010 · Psychological Review
  3. Strategies for revising judgment: How (and how well) people use others’ opinions.
    Soll, Jack B., Larrick, Richard P. · 2009 · Journal of Experimental Psychology: Learning, Memory, and Cognition
  4. Intuitions about combining opinions: misappreciation of the averaging principle
    Larrick, Richard P., Soll, Jack B. · 2006 · Management Science
  5. Measuring the Crowd Within: Probabilistic Representations Within Individuals
    Vul, Edward, Pashler, Harold · 2008 · Psychological Science

See all 144

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The effects of averaging subjective probability estimates between and within judges. https://app.cochat.ai/papers/10.1037//1076-898x.6.2.130
The paper
Ariely, D., Tung Au, W., Bender, R. H., Budescu, D. V., Dietz, C. B., Gu, H., Wallsten, T. S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied. https://doi.org/10.1037//1076-898x.6.2.130

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.