The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Abstract
Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created—and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 2×
“Ericsson, Krampe, and Romer (1993) have argued that expert performance is the end result of prolonged, deliberate practice, and even among elite per-formers, performance skill correlates with the amount of deliberate practice.”
BackgroundMindset: The New Psychology of Success
(2006) · Choice Reviews Online · doi:10.5860/choice.44-2397 · Cited by 7,804
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911
Cacioppo, J. T., & Petty, R. E. (1982) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.42.1.116 · Cited by 5,359
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Behavioral Study of obedience.
Milgram, S. (1963) · The Journal of Abnormal and Social Psychology · doi:10.1037/h0040525 · Cited by 4,918 · cited 2×
“However, the Zimbardo prison “experiment” (Zimbardo, 1973) and the original Milgram obedience study (Milgram, 1963) were also heavy-handed, multidimensional interventions.”
MethodThe Efficient Assessment of Need for Cognition
Cacioppo, J. T., Petty, R. E., & Feng Kao, C. (1984) · Journal of Personality Assessment · doi:10.1207/s15327752jpa4803_13 · Cited by 2,913
“Clinical versus Actuarial Judgment.”
Dawes, R. M., Faust, D., & Meehl, P. E. (1989) · Science · doi:10.1126/science.2648573 · Cited by 2,298
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Wason, P. C. (1968) · Quarterly Journal of Experimental Psychology · doi:10.1080/14640746808400161 · Cited by 1,476
Why g matters: The complexity of everyday life
Gottfredson, L. S. (1997) · Intelligence · doi:10.1016/s0160-2896(97)90014-3 · Cited by 1,408
Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006) · Psychological Science · doi:10.1111/j.1467-9280.2006.01720.x · Cited by 1,180
Evidential impact of base rates
Tversky, A., & Kahneman, D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.011 · Cited by 591
Baron, J., Scott, S., Fincher, K., & Emlen Metz, S. (2015) · Journal of Applied Research in Memory and Cognition · doi:10.1016/j.jarmac.2014.09.003 · Cited by 346
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312
Self-citationArthur, W., Tubre, T. C., Paul, D. S., & Sanchez-Ku, M. L. (1999) · Journal of Psychoeducational Assessment · doi:10.1177/073428299901700405 · Cited by 90
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 2×
“To ensure a level playing field, we scored all research groups on the official accuracy metric—the well-known Brier scoring rule, described later (Brier, 1950).”
Method
Research works printed but not linked: 39
- Atanasov P., Rescober P., Stone E., Servan-Schreiber E., Tetlock P., Ungar L., Mellers B. (2014). Distilling the wisdom of crowds: Prediction markets versus prediction polls. Manuscript under review. · DOI
- 10.1177/0734282909343763 · DOI
- Baron J., 2000, Thinking and deciding
- 10.1006/obhd.1997.2690 · DOI · Paper page
- 10.1016/S0272-7757(98)00044-2 · DOI · Paper page
- 10.1037/0022-3514.45.4.792 · doi:10.1037/0022-3514.45.4.792 · Paper page
- 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2 · DOI · Paper page
- 10.1111/j.1745-3984.1978.tb00065.x · DOI · Paper page
- 10.1016/0749-5978(91)90010-Q · DOI · Paper page
- 10.1016/0883-9026(88)90020-1 · DOI · Paper page
- Desvouges W. H., 1993, Contingent valuation, A critical assessment, 91
- 10.1016/B978-0-444-53707-2.00003-7 · DOI · Paper page
- 10.1111/0162-895X.00058 · DOI · Paper page
- 10.1017/CBO9780511808098 · DOI · Paper page
- Haran U., 2013, Journal of Decision Making, 8, 188
- 10.1086/296525 · doi:10.1086/296525
- Heisenberg W., 1999, Physics and philosophy: The revolution in modern science
- 10.1037/1076-8971.2.3-4.447 · doi:10.1037/1076-8971.2.3-4.447 · Paper page
- 10.1080/00222895.1977.10735113 · DOI
- Jervis R., 2010, Why intelligence fails
- Kahneman D., 2011, Thinking, fast and slow
- 10.1016/0095-0696(92)90019-S · DOI · Paper page
- Kahneman D., 1985, Judgment under uncertainty: Heuristics and biases, 153
- 10.1037/a0017905 · DOI · Paper page
- 10.1146/annurev.so.14.080188.001535 · DOI · Paper page
- 10.1177/0272989X0102100105 · DOI · Paper page
- Mellers B. A., Stone E., Atanasov P., Rohrbaugh N., Metz E. S., Ungar L. Tetlock P. (2014). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. Manuscript under review. · DOI · Paper page
- 10.1175/1520-0493(1987)115<1330:AGFFFV>2.0.CO;2 · DOI · Paper page
- 10.1016/0749-5978(88)90005-2 · DOI
- Rosenthal R., 1966, Experimenter effects in behavioral research
- 10.1016/j.ijforecast.2013.09.009 · DOI
- 10.1214/14-AOAS739 · DOI
- 10.1006/jeem.1994.1006 · DOI · Paper page
- Shipley W. C., 2009, Shipley–2 manual
- 10.1016/0001-6918(81)90005-6 · DOI · Paper page
- 10.1111/1529-1006.001 · DOI · Paper page
- 10.1111/1467-9280.00418 · DOI
- 10.1037/0096-3445.124.4.424 · DOI · Paper page
- 10.1016/0010-0277(72)90014-5 · DOI · Paper page
Cited by
What Motivates Effort? Evidence and Expert Forecasts
DellaVigna, S., & Pope, D. (2018) · The Review of Economic Studies · doi:10.1093/restud/rdx033 · Cited by 392
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
“Finally, assignment of the best forecasters into so-called “superteams”, allowing elite performers to interact online with each other, yields additional improvements to forecasting accuracy (Mellers et al., 2015b; Tetlock & Gardner, 2015).”
BackgroundSelf-citationChang, W., Chen, E., Mellers, B., & Tetlock, P. (2016) · Judgment and Decision Making · doi:10.1017/s1930297500004599 · Cited by 89
“Other future research includes training the best forecasters, so-called superforecasters (Mellers et al., 2015) with more advanced cognitive debiasing modules to see if training can push the upper limit of forecasting performance even further.”
BackgroundSelf-citationConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
“In another paper, Mellers et al. (2015b) document the performance of the most accurate performers, known as superforecasters, and identify reasons for their success.”
BackgroundInfluentialSelf-citationCommunicating uncertainty using words and numbers
Dhami, M. K., & Mandel, D. R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.002 · Cited by 69
“numerate people, such as superforecasters [86], show better discrimination among the numeric meanings of terms”
Metadata
- Authors
- Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., Ungar, L., & Tetlock, P.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-05
- Volume
- 10
- Issue
- 3
- Pages
- 267-281
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2165949980
- Cited by
- 236
- References
- 18
- FWCI
- 26.19
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.