A Bayesian Truth Serum for Subjective Data
Abstract
Subjective judgments, an essential information source for science and policy, are problematic because there are no public criteria for assessing judgmental truthfulness. I present a scoring method for eliciting truthful subjective data in situations where objective truth is unknowable. The method assigns high scores not to the most common answers but to the answers that are more common than collectively predicted, with predictions drawn from the same population. This simple adjustment in the scoring criterion removes all bias in favor of consensus: Truthful answers maximize expected score even for respondents who believe that their answer represents a minority view.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
An extensive line of work in mechanism design focuses on incentivizing high quality reporting in peer grading and other crowdsourcing environments.
“In response to this issue, an extensive line of work in the mechanism design literature focuses on incentivizing high quality reporting in peer grading and other crowdsourcing environments (Prelec 2004;Jurca andFaltings 2009, 2005;Faltings, Li, and Jurca 2012;Witkowski and Parkes 2012;Witkowski et al. 2013;Radanovic andFaltings 2013, 2014;Riley 2014;Kamble et al. 2015;Kong, Ligett, and Schoenebeck”
From Better Peer Grading through Bayesian Inference (2023) · cites this paper as Prelec 2004
Cites
Scoring Rules and the Evaluation of Probability Assessors
Winkler, R. L. (1969) · Journal of the American Statistical Association · doi:10.1080/01621459.1969.10501037 · Cited by 271 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling
John, L. K., Loewenstein, G., & Prelec, D. (2012) · Psychological Science · doi:10.1177/0956797611430953 · Cited by 2,239 · see citing works
Self-citationA solution to the single-question crowd wisdom problem
Prelec, D., Seung, H. S., & McCoy, J. (2017) · Nature · doi:10.1038/nature21054 · Cited by 314 · see citing works
Scaling up fact-checking using the wisdom of crowds
Allen, J., Arechar, A. A., Pennycook, G., & Rand, D. G. (2021) · Science Advances · doi:10.1126/sciadv.abf4393 · Cited by 234 · see citing works
Street-Level Algorithms: A Theory at the Gaps Between Policy and Decisions
Alkhatib, A., & Bernstein, M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300760 · Cited by 188 · see citing works
Optimal incentives for collective intelligence
Mann, R. P., & Helbing, D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618722114 · Cited by 112 · see citing works
Mitra, T., Hutto, C. J., & Gilbert, E. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702553 · Cited by 102 · see citing works
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
Better Peer Grading through Bayesian Inference
Zarkoob, H., D'Eon, G., Podina, L., & Leyton-Brown, K. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i5.25757 · Cited by 6 · see citing works
Prevalence, Incidence, and Rates of Contract Cheating
Curtis, G. J., & Clare, J. (2024) · Springer International Handbooks of Education · doi:10.1007/978-3-031-54144-5_110 · Cited by 6 · see citing works
- Authors
- Prelec, D.
- Affiliations
- Massachusetts Institute of Technology, Sloan School of Management, E56-320, 38 Memorial Drive, Cambridge, MA 02139, USA.
- Venue
- Science
- Published
- 2004-10-15
- Volume
- 306
- Issue
- 5695
- Pages
- 462-466
- ISSN
- 0036-8075 (ISSN-L)
- 1095-9203
- OpenAlex
- W2063782051
- Cited by
- 530
- References
- 1
- Citations by year
- 2026: 17 · 2025: 28 · 2024: 16 · 2023: 28 · 2022: 23 · 2021: 48 · 2020: 38 · 2019: 33 · 2018: 37 · 2017: 39 · 2016: 71 · 2015: 20 · 2014: 33 · 2013: 24 · 2012: 20
- FWCI
- 14.97
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- Bayesian Truth Serum, subjective judgment, honesty, scoring rules, minority perspectives
- MeSH terms
- Bayes Theorem, Games, Experimental, Humans, Judgment, Mathematics, Motivation, Truth Disclosure, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
