The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Identifying Expertise to Extract the Wisdom of Crowds
Abstract
Statistical aggregation is often used to combine multiple opinions within a group. Such aggregates outperform individuals, including experts, in various prediction and estimation tasks. This result is attributed to the “wisdom of crowds.” We seek to improve the quality of such aggregates by eliminating poorly performing individuals from the crowd. We propose a new measure of contribution to assess the judges' performance relative to the group and use positive contributors to build a weighting model for aggregating forecasts. In Study 1, we analyze 1,233 judges forecasting almost 200 current events to illustrate the superiority of our model over unweighted models and models weighted by measures of absolute performance. In Study 2, we replicate our findings by using economic forecasts from the European Central Bank and show how the method can be used to identify smaller crowds of the top positive contributors. We show that the model derives its power from identifying experts who consistently outperform the crowd. Data, as supplemental material, are available at http://dx.doi.org/10.1287/mnsc.2014.1909 . This paper was accepted by James Smith, decision analysis.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Applied Linear Statistical Models
Ziegel, E. R., Neter, J., Wasserman, W., & Kutner, M. (1992) · Technometrics · doi:10.2307/1269588 · Cited by 9,907
Applied Linear Statistical Models
(1997) · Technometrics · doi:10.1080/00401706.1997.10485141 · Cited by 7,185
Heuristics and Biases: The Psychology of Intuitive Judgment
Bottom, W. P., Gilovich, T., Griffin, D., & Kahneman, D. (2004) · The Academy of Management Review · doi:10.2307/20159081 · Cited by 3,536
(2004) · Choice Reviews Online · doi:10.5860/choice.42-1645 · Cited by 2,925
Constructive Consumer Choice Processes
Bettman, J. R., Luce, M. F., & Payne, J. W. (1998) · Journal of Consumer Research · doi:10.1086/209535 · Cited by 2,831
Heuristics and Biases: The Psychology of Intuitive Judgment
(2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511808098 · Cited by 2,192
Combining forecasts: A review and annotated bibliography
Clemen, R. T. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90012-5 · Cited by 2,164
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,966
Combining forecasts: A review and annotated bibliography
(1990) · Long Range Planning · doi:10.1016/0024-6301(90)90316-v · Cited by 1,465
Applied Linear Statistical Models
John, R. C. S. (1986) · Journal of Quality Technology · doi:10.1080/00224065.1986.11978989 · Cited by 1,307
Experts In Uncertainty: Opinion and Subjective Probability in Science
Cooke, R. M. (1991) · doi:10.1093/oso/9780195064650.001.0001 · Cited by 1,107
Principles of Forecasting: A Handbook for Researchers and Practitioners
(2001) · International Series in Operations Research & Management Science · doi:10.1007/978-0-306-47630-3 · Cited by 1,041
How social influence can undermine the wisdom of crowd effect
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008636108 · Cited by 1,017
Combining Probability Distributions: A Critique and an Annotated Bibliography
Genest, C., & Zidek, J. V. (1986) · Statistical Science · doi:10.1214/ss/1177013825 · Cited by 1,001
Infotopia How Many Minds Produce Knowledge
Sunstein, C. R. (2006) · doi:10.1093/oso/9780195189285.001.0001 · Cited by 913
Probabilistic Risk Analysis: Foundations and Methods
Bedford, T., & Cooke, R. (2001) · Cited by 825
Combining Probability Distributions From Experts in Risk Analysis
Clemen, R. T., & Winkler, R. L. (1999) · Risk Analysis · doi:10.1111/j.1539-6924.1999.tb00399.x · Cited by 816
Probabilistic Risk Analysis: Foundations and Methods
Bedford, T., & Cooke, R. (2001) · Cambridge University Press eBooks · doi:10.1017/cbo9780511813597 · Cited by 754
Averages of Forecasts: Some Empirical Results
Makridakis, S., & Winkler, R. L. (1983) · Management Science · doi:10.1287/mnsc.29.9.987 · Cited by 621
The Robust Beauty of Majority Rules in Group Decisions.
Hastie, R., & Kameda, T. (2005) · Psychological Review · doi:10.1037/0033-295x.112.2.494 · Cited by 462
Strategies for revising judgment: How (and how well) people use others’ opinions.
Soll, J. B., & Larrick, R. P. (2009) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0015145 · Cited by 416
Modis, T. (1993) · Technological Forecasting and Social Change · doi:10.1016/0040-1625(93)90030-b · Cited by 403
Information Sampling and Adaptive Cognition
(2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511614576 · Cited by 396
State of the Art—Encoding Subjective Probabilities: A Psychological and Psychometric Review
Wallsten, T. S., & Budescu, D. V. (1983) · Management Science · doi:10.1287/mnsc.29.2.151 · Cited by 350
Self-citationClemen, R. T., & Winkler, R. L. (1986) · Journal of Business & Economic Statistics · doi:10.1080/07350015.1986.10509492 · Cited by 340
A route to more tractable expert advice
Aspinall, W. (2010) · Nature · doi:10.1038/463294a · Cited by 332
A note on aggregating opinions
Hogarth, R. M. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90037-5 · Cited by 321
TU Delft expert judgment data base
Cooke, R. M., & Goossens, L. L. H. J. (2008) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2007.03.005 · Cited by 317
The Wisdom of Many in One Mind: Improving Individual Judgments With Dialectical Bootstrapping
Herzog, S. M., & Hertwig, R. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02271.x · Cited by 304
Mannes, A. E., Soll, J. B., & Larrick, R. P. (2014) · Journal of Personality and Social Psychology · doi:10.1037/a0036677 · Cited by 255
Comparing the Calibration and Coherence of Numerical and Verbal Probability Judgments
Wallsten, T. S., Budescu, D. V., & Zwick, R. (1993) · Management Science · doi:10.1287/mnsc.39.2.176 · Cited by 240
Self-citationDavis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014) · Decision · doi:10.1037/dec0000004 · Cited by 178
Self-citationThe Scientist Speculates: An Anthology of Partly-baked Ideas
Page, T. (1964) · American Journal of Physics · doi:10.1119/1.1970298 · Cited by 151
Intuitive Biases in Choice versus Estimation: Implications for the Wisdom of Crowds
Simmons, J. P., Nelson, L. D., Galak, J., & Frederick, S. (2010) · Journal of Consumer Research · doi:10.1086/658070 · Cited by 146
Some Comparisons Among Quadratic, Spherical, and Logarithmic Scoring Rules
Bickel, J. E. (2007) · Decision Analysis · doi:10.1287/deca.1070.0089 · Cited by 142
Evaluating and Combining Subjective Probability Estimates
Wallsten, T. S., Budescu, D. V., Erev, I., & Diederich, A. (1997) · Journal of Behavioral Decision Making · doi:10.1002/(sici)1099-0771(199709)10:3<243::aid-bdm268>3.0.co;2-m · Cited by 141
Self-citationPredicting the Next Big Thing: Success as a Signal of Poor Judgment
Denrell, J. C., & Fang, C. (2010) · Management Science · doi:10.1287/mnsc.1100.1220 · Cited by 79
Forecast aggregation via recalibration
Turner, B. M., Mark Steyvers, B., Merkle, E. C., Budescu, D. V., & Wallsten, T. S. (2013) · Machine Learning · doi:10.1007/s10994-013-5401-4 · Cited by 76
Self-citationThe Wisdom of Competitive Crowds
Lichtendahl, K. C., Grushka‐Cockayne, Y., & Pfeifer, P. E. (2013) · Operations Research · doi:10.1287/opre.2013.1213 · Cited by 65
Median Aggregation of Distribution Functions
Hora, S., Fransen, B., Hawkins, N., & Susel, I. (2013) · Decision Analytics · doi:10.1287/deca.2013.0282 · Cited by 62
Aggregating Large Sets of Probabilistic Forecasts by Weighted Coherent Adjustment
Wang, G., Kulkarni, S. R., Poor, H. V., & Osherson, D. N. (2011) · Decision Analysis · doi:10.1287/deca.1110.0206 · Cited by 52
The wisdom of ignorant crowds: Predicting sport outcomes by mere recognition
Herzog, S. M., & Hertwig, R. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500002096 · Cited by 49
Why Are Experts Correlated? Decomposing Correlations Between Judges
Broomell, S. B., & Budescu, D. V. (2009) · Psychometrika · doi:10.1007/s11336-009-9118-z · Cited by 45
Self-citationThe wisdom of the crowd playing The Price Is Right
Lee, M. D., Zhang, S., & Shi, J. (2010) · Memory & Cognition · doi:10.3758/s13421-010-0059-7 · Cited by 45
Confidence in Aggregation of Opinions from Multiple Sources
Budescu, D. V. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511614576.014 · Cited by 45
Self-citationUnderstanding pooled subjective probability estimates
Wallsten, T. S., & Diederich, A. (2001) · Mathematical Social Sciences · doi:10.1016/s0165-4896(00)00053-6 · Cited by 42
A Constrained Linear Estimator for Multiple Regression
Davis‐Stober, C. P., Dana, J. D., & Budescu, D. V. (2010) · Psychometrika · doi:10.1007/s11336-010-9162-8 · Cited by 29
Self-citationCompetitive Dynamics in Forecasting: The Interaction of Skill and Uncertainty
Evgeniou, T., Fang, L. H., Hogarth, R. M., & Karelaia, N. (2012) · Journal of Behavioral Decision Making · doi:10.1002/bdm.1766 · Cited by 13
Clemen, R. T. (2008) · Springer optimization and its applications · doi:10.1007/978-0-387-77131-1_1 · Cited by 11
Cited by
Wisdom or Madness? Comparing Crowds with Expert Evaluation in Funding the Arts
Mollick, E., & Nanda, R. (2016) · Management Science · doi:10.1287/mnsc.2015.2207 · Cited by 515
A solution to the single-question crowd wisdom problem
Prelec, D., Seung, H. S., & McCoy, J. (2017) · Nature · doi:10.1038/nature21054 · Cited by 314
Will Humans-in-the-Loop Become Borgs? Merits and Pitfalls of Working with AI
Fügener, A., Grahl, J., Gupta, A., & Ketter, W. (2021) · MIS Quarterly · doi:10.25300/misq/2021/16553 · Cited by 303
Forecast combinations: An over 50-year review
Wang, X., Hyndman, R. J., Li, F., & Kang, Y. (2023) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.11.005 · Cited by 280
The gateway belief model: a large-scale replication
van der Linden, S., Leiserowitz, A., & Maibach, E. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.01.009 · Cited by 262
Metadata
- Authors
- Budescu, D. V., & Chen, E.
- Venue
- Management Science
- Published
- 2015-02
- Volume
- 61
- Issue
- 2
- Pages
- 267-280
- ISSN
- 0025-1909 (ISSN-L)
- 1526-5501
- OpenAlex
- W2007423920
- Cited by
- 360
- References
- 49
- FWCI
- 20.13
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.