The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Abstract
Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created—and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 2×
“Ericsson, Krampe, and Romer (1993) have argued that expert performance is the end result of prolonged, deliberate practice, and even among elite per-formers, performance skill correlates with the amount of deliberate practice.”
BackgroundMindset: The New Psychology of Success
(2006) · Choice Reviews Online · doi:10.5860/choice.44-2397 · Cited by 7,804
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911
Cacioppo, J. T., & Petty, R. E. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.42.1.116 · Cited by 5,359
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Behavioral Study of obedience.
Milgram, S. (1963) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0040525 · Cited by 4,918 · cited 2×
“However, the Zimbardo prison “experiment” (Zimbardo, 1973) and the original Milgram obedience study (Milgram, 1963) were also heavy-handed, multidimensional interventions.”
Method“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Wason, P. C. (1968) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640746808400161 · Cited by 1,476
Research works printed but not linked: 40
- Atanasov P., Rescober P., Stone E., Servan-Schreiber E., Tetlock P., Ungar L., Mellers B. (2014). Distilling the wisdom of crowds: Prediction markets versus prediction polls. Manuscript under review. · DOI
- 10.1177/0734282909343763 · DOI
- Baron J., 2000, Thinking and deciding
- 10.1006/obhd.1997.2690 · DOI · Paper page
- 10.1016/S0272-7757(98)00044-2 · DOI · Paper page
- 10.1037/0022-3514.45.4.792 · doi:10.1037/0022-3514.45.4.792 · Paper page
- 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 · DOI · Paper page
- 10.1207/s15327752jpa4803_13 · DOI · Paper page
- 10.1111/j.1745-3984.1978.tb00065.x · DOI · Paper page
- 10.1016/0749-5978(91)90010-Q · DOI · Paper page
- 10.1016/0883-9026(88)90020-1 · DOI · Paper page
- Desvouges W. H., 1993, Contingent valuation, A critical assessment, 91
- 10.1016/B978-0-444-53707-2.00003-7 · DOI · Paper page
- 10.1111/0162-895X.00058 · DOI · Paper page
- 10.1017/CBO9780511808098 · DOI · Paper page
- Haran U., 2013, Journal of Decision Making, 8, 188
- 10.1086/296525 · doi:10.1086/296525
- Heisenberg W., 1999, Physics and philosophy: The revolution in modern science
- 10.1037/1076-8971.2.3-4.447 · doi:10.1037/1076-8971.2.3-4.447 · Paper page
- 10.1080/00222895.1977.10735113 · DOI
- Jervis R., 2010, Why intelligence fails
- Kahneman D., 2011, Thinking, fast and slow
- 10.1016/0095-0696(92)90019-S · DOI · Paper page
- Kahneman D., 1985, Judgment under uncertainty: Heuristics and biases, 153
- 10.1037/a0017905 · DOI · Paper page
- 10.1146/annurev.so.14.080188.001535 · DOI · Paper page
- 10.1177/0272989X0102100105 · DOI · Paper page
- Mellers B. A., Stone E., Atanasov P., Rohrbaugh N., Metz E. S., Ungar L. Tetlock P. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript under review. · DOI · Paper page
- 10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2 · DOI · Paper page
- 10.1016/0749-5978(88)90005-2 · DOI
- Rosenthal R., 1966, Experimenter effects in behavioral research
- 10.1016/j.ijforecast.2013.09.009 · DOI
- 10.1214/14-AOAS739 · DOI
- 10.1006/jeem.1994.1006 · DOI · Paper page
- Shipley W. C., 2009, Shipley–2 manual
- 10.1016/0001-6918(81)90005-6 · DOI · Paper page
- 10.1111/1529-1006.001 · DOI · Paper page
- 10.1111/1467-9280.00418 · DOI
- 10.1037/0096-3445.124.4.424 · DOI · Paper page
- 10.1016/0010-0277(72)90014-5 · DOI · Paper page
Cited by
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. (2018) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 392
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
Self-citationChang, W., Chen, E., Mellers, B., & Tetlock, P. (2016) · Judgment and Decision Making · doi:10.1017/s1930297500004599 · Cited by 89
“Other future research includes training the best forecasters, so-called superforecasters (Mellers et al., 2015) with more advanced cognitive debiasing modules to see if training can push the upper limit of forecasting performance even further.”
BackgroundSelf-citationConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
Self-citationCommunicating uncertainty using words and numbers
Dhami, M. K., & Mandel, D. R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.002 · Cited by 69
Metadata
- Authors
- Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-05
- Volume
- 10
- Issue
- 3
- Pages
- 267-281
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2165949980
- Cited by
- 236
- References
- 17
- FWCI
- 26.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions. https://app.cochat.ai/papers/10.1177/1745691615577794
- The paper
- Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P. (2015). Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions. Perspectives on Psychological Science. https://doi.org/10.1177/1745691615577794
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.