The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
The wisdom of select crowds.
Abstract
Social psychologists have long recognized the power of statisticized groups. When individual judgments about some fact (e.g., the unemployment rate for next quarter) are averaged together, the average opinion is typically more accurate than most of the individual estimates, a pattern often referred to as the wisdom of crowds. The accuracy of averaging also often exceeds that of the individual perceived as most knowledgeable in the group. However, neither averaging nor relying on a single judge is a robust strategy; each performs well in some settings and poorly in others. As an alternative, we introduce the select-crowd strategy, which ranks judges based on a cue to ability (e.g., the accuracy of several recent judgments) and averages the opinions of the top judges, such as the top 5. Through both simulation and an analysis of 90 archival data sets, we show that select crowds of 5 knowledgeable judges yield very accurate judgments across a wide range of possible settings-the strategy is both accurate and robust. Following this, we examine how people prefer to use information from a crowd. Previous research suggests that people are distrustful of crowds and of mechanical processes such as averaging. We show in 3 experiments that, as expected, people are drawn to experts and dislike crowd averages-but, critically, they view the select-crowd strategy favorably and are willing to use it. The select-crowd strategy is thus accurate, robust, and appealing as a mechanism for helping individuals tap collective wisdom.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
A study of normative and informational social influences upon individual judgment.
Deutsch, M., & Gerard, H. B. (1955) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0046408 · Cited by 4,819
(2004) · Choice Reviews Online · doi:10.5860/choice.42-1645 · Cited by 2,925
The trouble with overconfidence.
Moore, D. A., & Healy, P. J. (2008) · Psychological Review · doi:10.1037/0033-295x.115.2.502 · Cited by 2,358
The difference: how the power of diversity creates better groups, firms, schools, and societies
(2007) · Choice Reviews Online · doi:10.5860/choice.45-1534 · Cited by 2,280
The person and the situation: perspectives of social psychology
(1992) · Choice Reviews Online · doi:10.5860/choice.29-3604 · Cited by 2,208
Hackman, J. R., & Lorsch, J. R. (1987) · Cited by 2,151
Group process and productivity
Steiner, I. D. (1972) · Medical Entomology and Zoology · Cited by 2,106
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,966
Adaptive strategy selection in decision making.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.14.3.534 · Cited by 1,955
Combining forecasts: A review and annotated bibliography
(1990) · Long Range Planning · doi:10.1016/0024-6301(90)90316-v · Cited by 1,465
Group Performance and Decision Making
Kerr, N. L., & Tindale, R. S. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142009 · Cited by 1,331
Why People Fail to Recognize Their Own Incompetence
Dunning, D., Johnson, K., Ehrlinger, J., & Kruger, J. (2003) · Current Directions in Psychological Science · doi:10.1111/1467-8721.01235 · Cited by 1,224
Principles of Forecasting: A Handbook for Researchers and Practitioners
(2001) · International Series in Operations Research & Management Science · doi:10.1007/978-0-306-47630-3 · Cited by 1,041
Bonaccio, S., & Dalal, R. S. (2006) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.07.001 · Cited by 1,011
Group versus individual performance: Are N?+?1 heads better than one?
Hill, G. W. (1982) · Psychological Bulletin · doi:10.1037//0033-2909.91.3.517 · Cited by 918
Advice taking in decision making: egocentric discounting and reputation formation
Yaniv, I., & Kleinberger, E. (2000) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2909 · Cited by 701
Unit weighting schemes for decision making
Einhorn, H. J., & Hogarth, R. M. (1975) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(75)90044-6 · Cited by 673
Overconfidence in Interval Estimates.
Soll, J. B., & Klayman, J. (2004) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.30.2.299 · Cited by 493
Self-citationBias in judgment: Comparing individuals and groups.
Kerr, N. L., MacCoun, R. J., & Kramer, G. P. (1996) · Psychological Review · doi:10.1037//0033-295x.103.4.687 · Cited by 491
The Robust Beauty of Majority Rules in Group Decisions.
Hastie, R., & Kameda, T. (2005) · Psychological Review · doi:10.1037/0033-295x.112.2.494 · Cited by 462
A status-enhancement account of overconfidence.
Anderson, C., Brion, S., Moore, D. A., & Kennedy, J. A. (2012) · Journal of Personality and Social Psychology · doi:10.1037/a0029395 · Cited by 453
Why we still use our heads instead of formulas: Toward an integrative approach.
Kleinmuntz, B. (1990) · Psychological Bulletin · doi:10.1037//0033-2909.107.3.296 · Cited by 437
Strategies for revising judgment: How (and how well) people use others’ opinions.
Soll, J. B., & Larrick, R. P. (2009) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/a0015145 · Cited by 416
Self-citationIntuitions about combining opinions: misappreciation of the averaging principle
Larrick, R. P., & Soll, J. B. (2006) · Management Science · doi:10.1287/mnsc.1050.0459 · Cited by 410
Self-citationLittlepage, G., Robison, W., & Reddington, K. (1997) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1997.2677 · Cited by 367
Identifying Expertise to Extract the Wisdom of Crowds
Budescu, D. V., & Chen, E. (2015) · Management Science · doi:10.1287/mnsc.2014.1909 · Cited by 360
Larrick, R. P., Burson, K. A., & Soll, J. B. (2007) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2006.10.002 · Cited by 335
Self-citationA note on aggregating opinions
Hogarth, R. M. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90037-5 · Cited by 321
Made to Stick: Why Some Ideas Survive and Others Die
Heath, C., & Heath, D. (2007) · Cited by 316
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312 · cited 1×
“Environments with high dispersion in expertise present an op portunity to identify and capitalize on the abilities of the more capable judges (Mellers et al., 2014).”
BackgroundLorge, I., Fox, D., Davitz, J., & Brenner, M. (1958) · Psychological Bulletin · doi:10.1037/h0042344 · Cited by 302
Einhorn, H. J., Hogarth, R. M., & Klempner, E. (1977) · Psychological Bulletin · doi:10.1037//0033-2909.84.1.158 · Cited by 283
Freund, P. A., & Kasten, N. (2012) · Psychological Bulletin · doi:10.1037/a0026556 · Cited by 277
The Benefit of Additional Opinions
Yaniv, I. (2004) · Current Directions in Psychological Science · doi:10.1111/j.0963-7214.2004.00278.x · Cited by 270
When are two heads better than one and why?
Koriat, A. (2012) · Science · doi:10.1126/science.1216549 · Cited by 256
Bayesian inference: Combining base rates with opinions of sources who vary in credibility.
Birnbaum, M. H., & Mellers, B. A. (1983) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.45.4.792 · Cited by 253
Expert Political Judgment: How Good Is It? How Can We Know?
Tetlock, P. E. (2009) · Princeton University Press eBooks · doi:10.1515/9781400830312 · Cited by 232
An input-process-output analysis of influence and performance in problem-solving groups.
Littlepage, G. E., Schmidt, G. W., Whisler, E. W., & Frost, A. G. (1995) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.69.5.877 · Cited by 226
Proper analysis of the accuracy of group judgments.
Gigone, D., & Hastie, R. (1997) · Psychological Bulletin · doi:10.1037//0033-2909.121.1.149 · Cited by 215
What we really know about our abilities and our knowledge
Ackerman, P. L., Beier, M. E., & Bowen, K. R. (2002) · Personality and Individual Differences · doi:10.1016/s0191-8869(01)00174-x · Cited by 192
Organizational Decision Making: An Information Aggregation View
Csaszar, F. A., & Eggers, J. P. (2013) · Management Science · doi:10.1287/mnsc.1120.1698 · Cited by 189
Member variation, recognition of expertise, and group performance.
Libby, R., Trotman, K. T., & Zimmer, I. (1987) · Journal of Applied Psychology · doi:10.1037//0021-9010.72.1.81 · Cited by 187
Overconfidence in interval estimates: What does expertise buy you?
McKenzie, C. R. M., Liersch, M. J., & Yaniv, I. (2008) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2008.02.007 · Cited by 179
Davis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014) · Decision · doi:10.1037/dec0000004 · Cited by 178
The effects of member expertise on group decision-making and performance
Bonner, B. L., Baumann, M. R., & Dalal, R. S. (2002) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(02)00010-9 · Cited by 174
Individual versus group problem solving: An empirical test of a best-member strategy
Yetton, P. W., & Bottger, P. C. (1982) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(82)90248-3 · Cited by 170
Are We Wise About the Wisdom of Crowds? The Use of Group Judgments in Belief Revision
Mannes, A. E. (2009) · Management Science · doi:10.1287/mnsc.1090.1031 · Cited by 165
Self-citationExpertise and air time as bases of actual and perceived influence in problem-solving groups.
Bottger, P. C. (1984) · Journal of Applied Psychology · doi:10.1037//0021-9010.69.2.214 · Cited by 148
Evaluating and Combining Subjective Probability Estimates
Wallsten, T. S., Budescu, D. V., Erev, I., & Diederich, A. (1997) · Journal of Behavioral Decision Making · doi:10.1002/(sici)1099-0771(199709)10:3<243::aid-bdm268>3.0.co;2-m · Cited by 141
Miner, F. C. (1984) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(84)90014-x · Cited by 134
Sah, S., Moore, D. A., & MacCoun, R. J. (2013) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2013.02.001 · Cited by 132
Group-based forecasting?: A social psychological analysis
Kerr, N. L., & Tindale, R. S. (2010) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2010.02.001 · Cited by 130
Improving Group Judgment Accuracy: Information Sharing and Determining the Best Member
Henry, R. A. (1995) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1995.1042 · Cited by 130
Lamberson, P. J., & Page, S. E. (2011) · Management Science · doi:10.1287/mnsc.1110.1441 · Cited by 129
The effects of asymmetry among advisors on the aggregation of their opinions
Budescu, D. V., Rantilla, A. K., Yu, H.-T., & Karelitz, T. M. (2003) · Organizational Behavior and Human Decision Processes · doi:10.1016/s0749-5978(02)00516-2 · Cited by 117
Is the judgment of the group better than that of the average member of the group?
Stroop, J. (1932) · Journal of Experimental Psychology · doi:10.1037/h0070482 · Cited by 106
Group Decision Making and Quantitative Judgments: A Consensus Model
Davis, J. H. (2014) · doi:10.4324/9781315806372-10 · Cited by 101
Combining forecasts: What information do judges need to outperform the simple average?
Fischer, I., & Harvey, N. (1999) · International Journal of Forecasting · doi:10.1016/s0169-2070(98)00073-9 · Cited by 100
Judgmental aggregation strategies depend on whether the self is involved
Soll, J. B., & Mannes, A. E. (2010) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2010.05.003 · Cited by 96
Self-citationTapping into the wisdom of the crowd—with confidence
Hertwig, R. (2012) · Science · doi:10.1126/science.1221403 · Cited by 89
Hinsz, V. B. (1999) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1999.2853 · Cited by 80
Moore, D. A., & Klein, W. M. P. (2008) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2008.02.005 · Cited by 76
Effects of judges' forecasting on their later combination of forecasts for the same outcomes
Harvey, N., & Harries, C. (2003) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2003.09.012 · Cited by 73
Expertise in Group Problem Solving: Recognition, Social Combination, and Performance.
Bonner, B. L. (2004) · Group Dynamics Theory Research and Practice · doi:10.1037/1089-2699.8.4.277 · Cited by 45
Galton, F. (1907) · Nature · doi:10.1038/075509f0 · Cited by 44
Cited by
Inoculating the Public against Misinformation about Climate Change
van der Linden, S., Leiserowitz, A., Rosenthal, S., & Maibach, E. (2017) · Global Challenges · doi:10.1002/gch2.201600008 · Cited by 1,058
Forecasting: theory and practice
Petropoulos, F., Apiletti, D., Assimakopoulos, V., Babai, M. Z., Barrow, D. K., Ben Taieb, S., Bergmeir, C., Bessa, R. J., Bijak, J., Boylan, J. E., et al. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2021.11.001 · Cited by 921
Use of Delphi in health sciences research: A narrative review
Shang, Z. (2023) · Medicine · doi:10.1097/md.0000000000032829 · Cited by 657
Identifying Expertise to Extract the Wisdom of Crowds
Budescu, D. V., & Chen, E. (2015) · Management Science · doi:10.1287/mnsc.2014.1909 · Cited by 360
Cognitive bias in clinical medicine
O'Sullivan, E., & Schofield, S. (2018) · Journal of the Royal College of Physicians of Edinburgh · doi:10.4997/jrcpe.2018.306 · Cited by 334
Metadata
- Authors
- Mannes, A. E., Soll, J. B., & Larrick, R. P.
- Venue
- Journal of Personality and Social Psychology
- Published
- 2014
- Volume
- 107
- Issue
- 2
- Pages
- 276-299
- ISSN
- 0022-3514 (ISSN-L)
- 1939-1315
- OpenAlex
- W2011399107
- Cited by
- 255
- References
- 66
- FWCI
- 19.30
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.