A solution to the single-question crowd wisdom problem
The DOI matches its Crossref record; 26 of the 26 references it cites were found in a registry or an index.
Abstract
Once considered provocative, the notion that the wisdom of the crowd is superior to any individual has become itself a piece of crowd wisdom, leading to speculation that online voting may soon put credentialed experts out of business. Recent applications include political and economic forecasting, evaluating nuclear safety, public policy, the quality of chemical probes, and possible responses to a restless volcano. Algorithms for extracting wisdom from the crowd are typically based on a democratic voting procedure. They are simple to apply and preserve the independence of personal judgment. However, democratic methods have serious limitations. They are biased for shallow, lowest common denominator information, at the expense of novel or specialized knowledge that is not widely shared. Adjustments based on measuring confidence do not solve this problem reliably. Here we propose the following alternative to a democratic vote: select the answer that is more popular than people predict. We show that this principle yields the best answer under reasonable assumptions about voter behaviour, while the standard 'most popular' or 'most confident' principles fail under exactly those same assumptions. Like traditional voting, the principle accepts unique problems, such as panel decisions about scientific or artistic merit, and legal or historical disputes. The potential application domain is thus broader than that covered by machine learning and psychometric methods, which require data across multiple questions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The latest research indicates that exploiting the knowledge from the crowd will facilitate the task (Prelec et al., 2017).”
From Deep learning for sensor-based activity recognition: A survey (2019) · cites this paper as Prelec et al., 2017
Citations
Cites
A decision-theoretic generalization of on-line learning and an application to boosting
Freund, Y., & Schapire, R. E. (1997) · Journal of Computer and System Sciences · doi:10.1006/jcss.1997.1504 · Cited by 20,466
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239
Self-citationModels of ecological rationality: The recognition heuristic.
Goldstein, D. G., & Gigerenzer, G. (2002) · Psychological Review · doi:10.1037/0033-295x.109.1.75 · Cited by 1,352
Experts In Uncertainty: Opinion and Subjective Probability in Science
Cooke, R. M. (1991) · doi:10.1093/oso/9780195064650.001.0001 · Cited by 1,107
GALTON, F. (1907) · Nature · doi:10.1038/075450a0 · Cited by 1,080
How social influence can undermine the wisdom of crowd effect
Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008636108 · Cited by 1,017
Infotopia How Many Minds Produce Knowledge
Sunstein, C. R. (2006) · doi:10.1093/oso/9780195189285.001.0001 · Cited by 913
Use (and abuse) of expert elicitation in support of decision making for public policy
Morgan, M. G. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1319946111 · Cited by 702
A Bayesian Truth Serum for Subjective Data
Prelec, D. (2004) · Science · doi:10.1126/science.1102081 · Cited by 530 · cited 1×
“To discourage such behaviour, one can impose truth-telling incentives with the Bayesian truth serum, which also elicits respondents’ vote predictions 24,25 .”
BackgroundSelf-citation(2008) · doi:10.1126/science.1157679 · Cited by 406
Cited by
Items 201–300 of 312
Gruen, A., Mattingly, K., Morwitch, E., Bossaerts, F., Clifford, M., Nash, C., Ioannidis, J. P. A., & Ponsonby, A. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104783 · Cited by 4
Where’s Waldo, Ohio? Using Cognitive Models to Improve the Aggregation of Spatial Knowledge
Montgomery, L. E., Baldini, C. M., Vandekerckhove, J., & Lee, M. D. (2024) · Computational Brain & Behavior · doi:10.1007/s42113-024-00200-0 · Cited by 2
Kleňha, J. (2025) · foresight · doi:10.1108/fs-12-2023-0252 · Cited by 2
Follow the money, not the majority: A mechanism predicting unresolvable events
Baillon, A., Tereick, B., & Wang, T. V. (2025) · Journal of Risk and Uncertainty · doi:10.1007/s11166-025-09463-9 · Cited by 1
Eliciting subjective real-valued beliefs
Leo, G., & Stelnicki, S. (2025) · Experimental Economics · doi:10.1017/eec.2025.10023 · Cited by 1
Chen, Y. (2019) · Computer communications and networks · doi:10.1007/978-3-030-25590-9_1 · Cited by 1
FinTech Econometrics: Privacy Preservation and the Wisdom of the Crowd
Kou, S. (2021) · Springer series in supply chain management · doi:10.1007/978-3-030-81945-3_10 · Cited by 1
Führungstätigkeiten und Führungsinstrumente
Unger, F., Sann, U., & Martin, C. (2022) · Basiswissen Sozialwirtschaft und Sozialmanagement · doi:10.1007/978-3-658-36119-8_5 · Cited by 1
Robust Forecast AggregationPreprint
Areili, I., Babichenko, Y., & Rann Smorodinsky, B. (2017) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1710.02838 · Cited by 1
Machine Truth SerumPreprint
Luo, T., & Liu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.13004 · Cited by 1
Zeng, S., & Shen, J. (2020) · arXiv (Cornell University) · Cited by 2
Moustakas, A. (2021) · Research Square · doi:10.21203/rs.3.rs-583289/v1 · Cited by 2
Automation of Data Acquisition Strategies in Model Calibration for System Models: Sensor PlacementPreprint
Li, T., & Dahleh, M. A. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3619653 · Cited by 1
Harvesting the wisdom of crowds for election predictions using the Bayesian Truth SerumPreprint
Olsson, H., de Bruin, W. B., Galešić, M., & Prelec, D. (2019) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/rdjhs · Cited by 2
Gruen, A., Mattingly, K., Morwitch, E., Bossaerts, F., Clifford, M., Nash, C., Ioannidis, J. P. A., & Ponsonby, A. (2023) · medRxiv · doi:10.1101/2023.01.19.23284578 · Cited by 1
Group problem solving: Harnessing the Wisdom of the CrowdsPreprint
Hills, T. T. (2023) · doi:10.31234/osf.io/eyrzg · Cited by 1
Follow the fast: A simple algorithm for extracting wisdom from crowdsPreprint
de Calseyde, P. P. F. M. V., Efendić, E., van Dolder, D., van den Assem, M. J., Evans, A. M., Staal, J., Zwaan, L., Sherbino, J., & Norman, G. R. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4959736 · Cited by 1
How to find the right answer when the 'wisdom of the crowd' fails
Ross, E. (2017) · Nature · doi:10.1038/nature.2017.21370
Can Investors Profit from Information Diversity?Preprint
Mostipan, I. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3076118
Ballpark Crowdsourcing: The Wisdom of Rough Group ComparisonsPreprint
Hope, T., & Shahaf, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.04828
Annotation Scaffolds for Object Modeling and ManipulationPreprint
Frank-Bolton, P., & Simha, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.06679
Essays on Information and Beliefs in Dynamic Choices
Descamps, A. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.121077
Surrogate Scoring Rules and a Uniform Dominant Truth SerumPreprint
Liu, Y., & Chen, Y. (2018) · arXiv (Cornell University)
Zhou, F. (2018) · Queensland University of Technology · doi:10.5204/thesis.eprints.121497
Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds
Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., & Sigman, M. (2017) · arXiv · doi:10.48550/arxiv.1703.00045
К вопросу о стимульно-согласованных исследованиях с использованием апостериорных вероятностей
Cvitanić, J., Prelec, D., Radas, S., & ikic, H. C. (2020) · Теория вероятностей и ее применения · doi:10.4213/tvp5351
Kim, M., & Billfeld, N. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.3119495
Moustakas, A., & Firdaus, M. (2021) · Research Square · doi:10.21203/rs.3.rs-692921/v1
Surprisingly Popular Algorithm-based Adaptive Euclidean Distance Topology Learning PSOPreprint
Wu, X., Han, J., Cui, Q., Chen, L., Liang, Y., Huang, H. P., Lee, H. P., Zhou, Y., & Wu, C. (2021) · arXiv (Cornell University)
Improving the reliability of citizen science data
Mugford, J. (2021) · University of Canterbury Research Repository (University of Canterbury) · doi:10.26021/12154
The limitations of scientific consensus
Prelec, D. (2021) · doi:10.5644/pi2021.200.11
Ograničenja naučnog konsenzusa
Prelec, D. (2021) · doi:10.5644/pi2021.200.01
Cabrerizo, F. J., González-Quesada, J. C., Pérez, I. J., & Herrera‐Viedma, E. (2022) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2022.262
General identifiability of possible world models for crowd wisdomPreprint
Prelec, D., & McCoy, J. (2022) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/gaep3 · Cited by 3
Fujisaki, I., Honda, H., & Ueda, K. (2022) · Research Square · doi:10.21203/rs.3.rs-1497903/v1
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0024
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0034
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0028
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0029
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0023
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0006
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0017
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0035
Klintman, M. (2021) · Manchester University Press eBooks · doi:10.7765/9781526158703.00014
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0014
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0010
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0022
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0026
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0020
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0016
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0008
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0033
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0032
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0009
Vox Peritorum: Capitalizing on Confidence and Projection to Characterize ExpertisePreprint
Grüning, D. J., & Krueger, J. I. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/6vndh
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0013
Klintman, M. (2021) · Manchester University Press eBooks · doi:10.7765/9781526158703.00015
Atkinson, M. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3316169
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0019
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0036
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0030
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0007
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0038
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0027
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0039
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0004
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0012
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.003.0040
Shirasuna, M., & Honda, H. (2022) · doi:10.31234/osf.io/fpw6m
Kitle Kaynak Lojistik Kavramının Teorik Çerçevede İncelenmesi
DÜNDAR, A. O. (2022) · İstanbul Gelişim Üniversitesi Sosyal Bilimler Dergisi · doi:10.17336/igusbd.856279
Incentives for self-extremized expert judgments to alleviate the shared-information problem.
Peker, C. (2022) · Decision · doi:10.1037/dec0000198
A response time selection algorithm for more accurate crowdsPreprint
Efendić, E., Kurz, J., Goukens, C., & de Calseyde, P. P. F. M. V. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/7vykm
Fujisaki, I., Yu, L., Tsukamura, Y., Yang, K., & Ueda, K. (2023) · doi:10.31234/osf.io/57byp
Schools’ air quality monitoring for health and education: methods and protocols of the SAMHE projectPreprint
Chatzidiakou, L., Archer, R. J., Beale, V., Bland, S., Carter, H., Castro-Faccetti, C., Edwards, H., Finneran, J., Hama, S. M. L., Jones, R. L., et al. (2023) · Research Square · doi:10.21203/rs.3.rs-3324349/v1
Atkinson, M. (2023) · Contributions to phenomenology · doi:10.1007/978-3-031-40776-5_5
Coordinating several mental strategies favors integration: Evidence from human judgmentPreprint
Hoffmȧnn, J., Albrecht, R., & von Helversen, B. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9yv57
Assessment of bias in a crowdsourcing accessibility system using serious games
Barguil, J. M. M. (2023) · Digital Library of Theses and Dissertations (Universidade de São Paulo) · doi:10.11606/t.45.2023.tde-02012024-175653
Understanding resistance to knowledge and change
Becker, P. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95640-6.00009-9
Raising Social Stakes Raises ConfidencePreprint
Esmaily, J., Abharzad, E., Knogler, S., Deroy, O. D., & Bahrami, B. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4844364
k-Clustering with Comparison and Distance Oracles
Galhotra, S., Raychaudhury, R., & Sintos, S. (2024) · Proceedings of the ACM on Management of Data · doi:10.1145/3695830
Optional disclosure and observational learning
Peng, D., Rao, Y., Sun, X., & Xiao, E. (2024) · Journal of Economic Behavior & Organization · doi:10.1016/j.jebo.2024.106817 · Cited by 2
Belief Decomposition: A Mechanism for Collective InferencePreprint
McCoy, J. P., & Prelec, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5170127
Surprisingly popular-based multivariate conceptual knowledge acquisition method
Xin, X., Yuan, S., Li, T., Xue, Z., & Wang, C. (2025) · International Journal of Approximate Reasoning · doi:10.1016/j.ijar.2025.109419
Dm 2: Decision-making market for DApps
Fan, J., Wu, Q., Wang, Q., Wang, K., & Dai, X. (2025) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2025.110364
Cognitive simplicity drives collective route improvements in homing pigeons
Banerjee, S. C., Francisco, F. A., & Kao, A. B. (2025) · eLife · doi:10.7554/elife.108054
Crowd Assessment of the Military Utility of Future Technologies
Dansarie, M., Andersson, K., & Silfverskiöld, S. (2025) · Scandinavian Journal of Military Studies · doi:10.31374/sjms.339 · Cited by 1
On the Truthfulness of Surprisingly Likely Responses of Large Language Models
Goel, N. (2025) · ACM Collective Intelligence Conference (CI) · doi:10.1145/3715928.3737471 · Cited by 2
Measuring Democracy Using the Wisdom of the CrowdsPreprint
Canen, N., & Chakraborty, A. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5380850
Variational Galton Machine: From Point Predictions to a Probabilistic AggregatePreprint
Jia, Y., Keppo, J., & Satopää, V. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5392541 · Cited by 1
An ensemble method utilising multiple thinking styles that boosts the wisdom-of-inner-crowd effect
Fujisaki, I., Yu, L., Tsukamura, Y., Yang, K., & Ueda, K. (2025) · Scientific Reports · doi:10.1038/s41598-025-14740-3 · Cited by 1
Swarmter: A Crowd Wisdom based Collective Decision Making System for Decentralized Applications
Fan, J., Wu, Q., Wang, Q., Xiong, S., & Dai, X. (2025) · doi:10.1109/gbc60041.2025.11134455
Crowd Dynamics in Online Lending: Unveiling Indicators of Collective Insight
Desai, H. N. (2025) · IEEE Access · doi:10.1109/access.2025.3605316
A tale of two birds: cognitive simplicity drives collective route improvements in homing pigeonsPreprint
Banerjee, S. C., Francisco, F. A., & Kao, A. B. (2025) · eLife · doi:10.7554/elife.108054.1 · Cited by 1
Messengers: breaking echo chambers in collective opinion dynamics with homophily
Raoufi, M., Hamann, H., & Romańczuk, P. (2025) · npj Complexity · doi:10.1038/s44260-025-00052-0 · Cited by 3
A Social Choice Analysis of Optimism’s Retroactive Project Funding
Briman, E., Talmon, N., Kreitenweis, A., & Idrees, M. (2025) · Springer proceedings in business and economics · doi:10.1007/978-3-032-03273-7_5 · Cited by 1
Additional file 1 of Does the “surprisingly popular” method yield accurate crowdsourced predictions?
Rutchick, A. M., Ross, B. J., Calvillo, D. P., & Mesick, C. C. (2020) · Figshare · doi:10.6084/m9.figshare.13225934
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.002.0001
Hanson, R. (2016) · Oxford University Press eBooks · doi:10.1093/oso/9780198754626.002.0002
Aggregation of Antagonistic Contingent Preferences: When Is It Possible?
Deng, X., Tao, B., & Wang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-08560-3_1 · Cited by 3
Metadata
- Authors
- Prelec, D., Seung, H. S., & McCoy, J.
- Affiliations
- Massachusetts Institute of Technology (US)
- Princeton University (US)
- Corresponding author
- Prelec, Dražen
- Venue
- Nature
- Published
- 2017-01
- Volume
- 541
- Issue
- 7638
- Pages
- 532-535
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- OpenAlex
- W2580299352
- Cited by
- 314
- References
- 26
- Citations by year
- 2026: 13 · 2025: 28 · 2024: 23 · 2023: 33 · 2022: 29 · 2021: 40 · 2020: 31 · 2019: 28 · 2018: 35 · 2017: 19
- FWCI
- 37.04
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Nature (publishedVersion)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Funders
- U.S. Department of the Interior
- Interior Business Center
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Physical Sciences › Computer Science › Information Systems › Expert finding and Q&A systems
- Keywords
- wisdom of crowds, online voting, democratic voting, expert opinion, prediction markets, confidence calibration, prediction accuracy, group decision making
- MeSH terms
- Algorithms, Art, Bayes Theorem, Choice Behavior, Commerce, Democracy, Expert Testimony, Geography, Humans, Judgment, Public Opinion, Logistic Models, Knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.