Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Open accessThe DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.
Overview
CoChat Summary
## TL;DR
This paper reports a 2-year geopolitical forecasting tournament and tests three psychological interventions—training, teaming, and tracking—to improve probabilistic forecasts. All three improved accuracy, and the top 2% of Year 1 forecasters placed into elite teams (“superforecasters”) performed best in Year 2.
## Key findings
- Training improved accuracy: In Year 1, training improved Brier scores, with probability training outperforming scenario training, and scenario training outperforming no training: F(2, 1586) = 14.29, p < .001; t(1053) = 2.23, p = .026; t(1056) = 3.25, p < .001. In Year 2, probability training again beat no training: F(1, 882) = 19.12, p < .001.
- Teaming improved accuracy: Group influence was significant in both years, F(2, 1586) = 60.68, p < .001 and F(1, 882) = 62.76, p < .001. In Year 1, teams were more accurate than crowd-belief forecasters, who were more accurate than independent forecasters: t(1036) = 5.52, p < .001 and t(1120) = 5.92, p < .001.
- Tracking the top performers worked best: The best performers from Year 1, the top 2%, were placed together in elite Year 2 teams and “outperformed all other groups by a wide margin.” The paper reports no evidence of Year 2 regression to the mean.
- Both calibration and resolution improved: Calibration improved with training in Years 1 and 2, F(2, 1586) = 3.16, p = .04 and F(1, 938) = 3.78, p < .05; resolution varied with group influence and training in both years, and tracking enhanced resolution, t(293) = 4.12, p < .001.
## Methods
- The study embedded experiments in a 2-year geopolitical forecasting tournament (Year 1: September 2011–April 2012; Year 2: June 2012–April 2013) on goodjudgmentproject.com, where participants made probability estimates for 199 geopolitical outcomes.
- Year 1 used a 3 (training: no/scenario/probability) × 4 (group influence: independent/crowd-belief/team/prediction-market) factorial design; Year 2 used a 2 (training: no/probability) × 3 (group influence: independent/team/prediction-market) design plus an elite tracking condition. Results for prediction markets are noted as not discussed in the paper.
- The main outcome was Brier score; analyses also examined calibration and resolution, using statistical tests reported as F-tests, t-tests, and regressions. The paper states that findings were robust across Brier, spherical, and logarithmic scoring rules.
## Limitations
- The paper notes that results for prediction-market forecasting are not discussed here. This limits what can be concluded from the full design as presented in this article.
- The paper itself says “No one knows how much further improvement is possible” and that there are likely “pockets of inherent uncertainty” making perfect or near-perfect Brier scores impossible. This is a stated substantive limitation of forecasting accuracy.
- Evident from the text: the Year 2 no-training condition included some participants who had previously received scenario training, which the authors say “tipped the scale against finding a statistically significant effect of training.”
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 39 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and in the middle period of Year 2, F (1, 882) = 6.10, p = .014.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both years, training and teaming remained significant predictors even when timing and updating were included as predictors.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to showing a main effect of group influence, Year 2 data showed an interaction between training and group influence, F (1, 882) = 5.78, p = .018.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Year 2, calibration also benefited from group influence, F (1, 938) = 7.07, p = .008, and tracking.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no evidence of Year 2 regression to the mean;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 44 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 6
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, as reported elsewhere, training improved the calibration and resolution of forecasters by reducing overconfidence (Mellers et al., 2014; Moore et al., 2016).”
From Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments (2016) · cites this paper as Mellers et al., 2014
Experimental — accuracy not yet measured
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom, each partial η² from its F, and each ratio, mean difference or B printed with a p checked against its own confidence interval.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 882) = 5.78, p = .018”
Reported p = .018 · recomputed p = .016
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 938) = 3.78, p < .05”
Reported p < .05 · recomputed p = .052
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (2, 1463) = 6.20, p < .001”
Reported p < .001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1056) = 3.25, p < .001”
Reported p < .001 · recomputed p = .001
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1588) = 2.70, p < .001”
Reported p < .001 · recomputed p = .007
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(1056) = 3.25reported p < .001 · recomputed p = .001
“t (1056) = 3.25, p < .001”
- Doesn’t add upF(1, 882) = 5.78reported p = .018 · recomputed p = .016
“F (1, 882) = 5.78, p = .018”
- Doesn’t add upF(2, 1463) = 6.2reported p < .001 · recomputed p = .002
“F (2, 1463) = 6.20, p < .001”
- Doesn’t add upt(1588) = 2.7reported p < .001 · recomputed p = .007
“t (1588) = 2.70, p < .001”
- Doesn’t add upF(1, 938) = 3.78reported p < .05 · recomputed p = .052
“F (1, 938) = 3.78, p < .05”
37 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Groups: Interaction and Performance.
Polley, R. B., & McGrath, J. E. (1984) · Administrative Science Quarterly · doi:10.2307/2393041 · Cited by 3,290
Norm theory: Comparing reality to its alternatives.
Kahneman, D., & Miller, D. T. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.136 · Cited by 3,024
The difference: how the power of diversity creates better groups, firms, schools, and societies
(2007) · Choice Reviews Online · doi:10.5860/choice.45-1534 · Cited by 2,280
Many Hands Make Light the Work: The Causes and Consequences of Social Loafing
Latané, B., Williams, K., & Harkins, S. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.6.822 · Cited by 2,216
Stasser, G., & Titus, W. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.48.6.1467 · Cited by 1,850 · cited 1×
“Teams have a demonstrated failure to search for unique information, unshared information, or hypothesis-disconfirming information (Kerr, MacCoun, & Kramer, 1996; Stasser & Titus, 1985; Sunstein, 2006).”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Group Performance and Decision Making
Kerr, N. L., & Tindale, R. S. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142009 · Cited by 1,331
Research works printed but not linked: 2
- Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations . Washington, DC: National Academies Press.
- Janis, I. (1982). Group-think . Boston, MA: Houghton-Mifflin.
Cited by
Items 201–300 of 321
Bayesian Ensembles of Binary-Event ForecastsPreprint
Lichtendahl, K. C., Grushka‐Cockayne, Y., Jose, V. R. R., & Winkler, R. L. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940740 · Cited by 4
Rationality, pragmatics, and sources
Collins, P. J. (2017) · BIROn (Birkbeck, University of London) · doi:10.18743/pub.00040284 · Cited by 4
Regularized Aggregation of One-Off Probability PredictionsPreprint
Satopaa, V. A. (2022) · Operational Research · doi:10.2139/ssrn.3769945 · Cited by 17
Baumeister, R. F., & Lim, K. (2021) · Springer eBooks · doi:10.1007/978-3-319-98390-5_138-1 · Cited by 3
Expectations and Adaptation to Environmental Threats
Ahmad, H. F., Gibson, M., Nadeem, F., Nasim, S., & Rezaee, A. (2022) · The Economic Journal · doi:10.1093/ej/ueaf071 · Cited by 6
12. Transforming Practice: Checklists for Delivering Change
Amano, T., Bako, L. A., Best, M., Boenisch, N., Boersch‐Supan, P. H., Browne, D., Buckley, Y. M., Burgman, M. A., Cadotte, M. W., Canessa, S., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.12 · Cited by 2
8 Making Decisions for Policy and Practice
Cook, C. N., Burgman, M. A., Rumpff, L., & Thornton, A. (2022) · Open Book Publishers · doi:10.11647/obp.0321.08 · Cited by 3
4. Presenting Conclusions from Assessed Evidence
Sutherland, W. J., Amano, T., Boenisch, N., Cheng, S. H., Christie, A. P., Godfray, H. C. J., Grainger, M., McLean, A. R., Morgan, W. H., Ockendon, N., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.04 · Cited by 2
Karger, E., Rosenberg, J., Jacobs, Z., Hickman, M. G., & Tetlock, P. E. (2025) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2024.11.008 · Cited by 12
Self-citationA Behaviorally Informed Survey-Powered Market Agent
Inchauspe, J., Atanasov, P. D., Mellers, B. A., Tetlock, P. E., & Ungar, L. (2014) · The Journal of Prediction Markets · doi:10.5750/jpm.v8i2.867 · Cited by 1
Self-citationHypermind vs. Big Data: Collective Intelligence Still Dominates Electoral Forecasting
Servan-Schreiber, E., Lumenogic Llc, & Atanasov, P. D. (2015) · Cited by 2
Self-citationCollegiality and efficiency in bureaucracy
Bergman, M., & Fredén, A. (2022) · Public Policy and Administration · doi:10.1177/09520767221093878 · Cited by 2
“Moreover, recent work by Tetlock and co-authors (e.g. Mellers et al., 2014; Horowitz et al., 2019) suggests that well-structured teams regularly outperform individuals, also when the latter are drawn from the same pool as the team members.”
BackgroundForecasting in Organizations: Reinterpreting Collective Judgment Through Mindful Organizing
Montes, E. R. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_11 · Cited by 3
Forecasting long-run causal effects
Bernard, D. R., Himmelstein, M., Karger, E., & Schoenegger, P. (2026) · Journal of Development Economics · doi:10.1016/j.jdeveco.2026.103820 · Cited by 2
The Social Psychology of the Wisdom of Crowds (with a New Section on Recent Advances)
Larrick, R. P., Mannes, A. E., & Soll, J. B. (2024) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-44424-1_7 · Cited by 4
Partial Information Framework: Aggregating Estimates from Diverse Information SourcesPreprint
Satopää, V., Jensen, S. T., Pemantle, R., & Ungar, L. (2015) · arXiv (Cornell University) · Cited by 1
Self-citationMcAndrew, T., Wattanachit, N., Gibson, G., & Reich, N. G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.11409 · Cited by 2
“One of the most successful expert opinion forecasting systems enrolled approximately 2000 participants and was called the Good Judgment Project (GJP; Mellers et al., 2014; Satopää et al., 2014; Ungar, Mellers, Satopaa, Tetlock, & Baron, 2012).”
Silverman, J., & Barnea, U. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4134386 · Cited by 2
Blinding to circumvent human biases: Deliberate ignorance in humans, institutions, and machinesPreprint
Hertwig, R., Herzog, S. M., & Kozyreva, A. (2022) · doi:10.31234/osf.io/9vdb8 · Cited by 2
Automating Hybrid Collective Intelligence in Open-Ended Medical DiagnosticsPreprint
Kurvers, R. H. J. M., Nuzzolese, A. G., Russo, A., et al. (2023) · doi:10.31234/osf.io/7sc4k · Cited by 2
Promoting erroneous divergent opinions increases the wisdom of crowdsPreprint
Barrera-Lemarchand, F., Balenzuela, P., Bahrami, B., Deroy, O. D., & Navajas, J. (2023) · doi:10.31234/osf.io/wvfm3 · Cited by 1
9. Creating Evidence-Based Policy and Practice
Downey, H., Boenisch, N., Browne, D., Buckley, Y. M., Cranston, G. R., Dicks, L. V., Hughes, J., Latrémouille, C., Morgan, W. H., Muir, M. J., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.09 · Cited by 2
Shirasuna, M., & Honda, H. (2023) · PLoS ONE · doi:10.1371/journal.pone.0280984 · Cited by 1
“This effect, known as the wisdom of crowds effect [1–8], has been observed in various fields including general knowledge tasks [9, 10], forecasts [11, 12], and matter of taste [13].”
BackgroundHobday, A. J., Walters, V. M., Stephenson, R. L., Baylis, S. M., Bessey, C., Boschetti, F., Bulman, C., Contardo, S., Dambacher, J. M., Day, J., et al. (2024) · Reviews in Fish Biology and Fisheries · doi:10.1007/s11160-024-09899-3 · Cited by 2
“While feedback can be slow at long time scales (e.g., end of century), shorter term foresighting (e.g., years) can provide more rapid feedback on our ability to think about the future (Mellers et al. 2014).”
How Wisdom-of-Crowds Research Can Help Improve Deliberative Consensus Methods
Jorm, A. F. (2024) · Springer eBooks · doi:10.1007/978-981-97-9222-1_9 · Cited by 1
The meta-aggregation approaches: Correctly choosing how to choose correctly in groups
Shinitzky, H., Parpara, O., Ezrets, M., Klein, R., & D, Y. S. M. (2025) · Collective Intelligence · doi:10.1177/26339137251315410 · Cited by 3
“…tasks and applications include solving temporal ordering problems in a distributed fashion Steyvers et al. (2009), political and economic forecasting Budescu and Chen (2014); Mellers et al. (2014), and evaluating information retrieval systems Alonso et al. (2008); Lease and Yilmaz (2012).”
Ensemble forecasts of COVID-19 activity to support Australia’s pandemic response: 2020–22
Moss, R. G., Tobin, R. J., O’Hara-Wild, M., Adekunle, A. I., Liu, D., South, T., Morris, D. J., RYAN, G. E., Hao, T., Babu, A. J., et al. (2026) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1014199 · Cited by 3
“When new model features were introduced, retrospective forecasts were validated against previously-reported data before the new model iteration was considered for inclusion in the ensemble.”
Influencing Confidence: Testing Ways to Increase or Decrease Confidence in Knowledge
Stone, E. R., Parker, A. M., Somerville, A. H., Nixon, B., Kemmerly, R., & Bongard, M. (2025) · Journal of Behavioral Decision Making · doi:10.1002/bdm.70048 · Cited by 2
Self-citationCrowdsourced versus large language models forecasting: evidence for the accuracy–correlation effect
Jeddi, Y., Segovia‐Martín, J., & Servan-Schreiber, E. (2026) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2024.0456 · Cited by 3
Condorcet and Beyond: An Empirical Comparison of Voting Rules
Douven, I., Kriegeskorte, N., Stinson, P., & Ying, J. (2026) · Cognitive Science · doi:10.1111/cogs.70242 · Cited by 3
“This is consistent with the pattern reported in the superforecaster literature (Mellers et al., 2014), in which highly accurate forecasters tend to converge with one another precisely because they are converging on the truth.”
InfluentialDoes Group Size Affect Problem Solving Performance in Crowds Working on a Hidden Profile Task?
Huang, X., & Tausczik, Y. (2018) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3170427.3188669 · Cited by 1
“Crowdsourcing applications use crowds to make judgements and make predictions about the future (e.g. [4 ]).”
Background6. Identifying Stakeholders and Collaborating with Communities
Best, M., Irvine, R. L., Bako, L. A., Citegetse, G., Field, A., Roe, D., Rose, D. C., & Spencer, J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.06 · Cited by 1
Gruen, A., Mattingly, K., Morwitch, E., Bossaerts, F., Clifford, M., Nash, C., Ioannidis, J. P. A., & Ponsonby, A. (2023) · EBioMedicine · doi:10.1016/j.ebiom.2023.104783 · Cited by 4
Wilkins, D., & Meindl, M. (2023) · The British Journal of Social Work · doi:10.1093/bjsw/bcad234 · Cited by 1
“In signal detection terms, you . . . shift your response threshold for ‘crying wolf’’ (Tetlock and Mellers, 2014, p. 11575).”
Silverman, J., & Barnea, U. (2022) · Management Science · doi:10.1287/mnsc.2022.01175 · Cited by 3
Operational Collective Intelligence of Humans and Machines
Gurney, N., Morstatter, F., Pynadath, D. V., Russell, A., & Satyukov, G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60125-5_20 · Cited by 2
Motahhar, V. K., & Gruca, T. S. (2025) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2024.12.001 · Cited by 1
Individual foresight: Concept, operationalization, and correlates
Schuler, B. A., Murmann, J. P., Beisemann, M., & Satopää, V. (2025) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2025.01.003 · Cited by 1
Finkel, A. M., & Johnson, B. B. (2025) · Human and Ecological Risk Assessment An International Journal · doi:10.1080/10807039.2025.2506582 · Cited by 1
“There are examples of training that improve intuitive probabilities (see training review in Mellers et al. 2014) and other mathematical tasks, but the time involved is much greater, and they included non-typical samples.”
The Research Optimist's Defense
Benjamin, D. M., & Kimmelman, J. (2016) · Perspectives in biology and medicine · doi:10.1353/pbm.2016.0043 · Cited by 1
동료평가 정확도 향상 방안의 비교: 평가 기준에 대한 학생들 간 토론 대 전문가 평가 사례 제시
Park, J. A., & Park, J. (2019) · 인지과학 · doi:10.19066/cogsci.2019.30.4.001 · Cited by 1
Lenience breeds strictness: The generosity-erosion effect in hiring decisions
Vives, M., Navia, T. F., Teixido-Figueras, J. J., & Serra‐Burriel, M. (2021) · Science Advances · doi:10.1126/sciadv.abe2045 · Cited by 4
Recent Developments in Cointegration
Jusélius, K. (2018) · MDPI eBooks · doi:10.3390/books978-3-03842-956-2 · Cited by 1
3. Assessing Collated and Synthesised Evidence
Sutherland, W. J., Cheng, S. H., Christie, A. P., Cooke, S. J., Koricheva, J., Randall, N., Smith, R. K., & Stewart, R. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.03 · Cited by 1
Using the 'Wisdom of (Expert) Crowds' to Forecast Mass AtrocitiesPreprint
Ulfelder, J. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2418980 · Cited by 1
Who Accurately Predicted the End of the Government Shutdown?Preprint
Martin, C. C., & Eriksson, K. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2609920 · Cited by 1
Bayesian Ensembles of Binary-Event Forecasts: When Is It Appropriate to Extremize or Anti-Extremize?Preprint
Lichtendahl, K. C., Grushka‐Cockayne, Y., Jose, V. R. R., & Winkler, R. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.02391 · Cited by 4
The Impact of Overconfidence and Ambiguity Attitude on Market EntryPreprint
Gutierrez, C., Åstebro, T. B., & Obłój, T. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3370305 · Cited by 2
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text DataPreprint
Jin, W., Khanna, R., Kim, S., Lee, D., Morstatter, F., Galstyan, A., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00792 · Cited by 24
“Unlike the accuracy metric, the Brier score penalizes over-and under-confident fore-casts (Mellers et al., 2014) — thus the best model under each metric can be different.”
MethodTetlock, P. E., Lu, L., & Mellers, B. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4013831 · Cited by 1
Self-citationPre-commitment and Updating BeliefsPreprint
Ebersole, C. R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/5vsq3 · Cited by 2
Group Decision-MakingPreprint
Tindale, S., & Winget, J. R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/kq2ft · Cited by 4
Talent Spotting in Crowd PredictionPreprint
Atanasov, P. D., & Himmelstein, M. (2022) · doi:10.31234/osf.io/rm49a · Cited by 1
Self-citationEffect of Calibration Training on the Calibration of Intelligence Analysts’ JudgmentsPreprint
Kelly, M. O., & Mandel, D. R. (2023) · doi:10.31234/osf.io/p82nx · Cited by 1
Pettine, W. W., Winters, D. E., Murray, J. D., Anticevic, A., & Jacob, S. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/n32gy · Cited by 2
Studies of Methods and Interventions for Judgment Accuracy: Methodological GuidelinesPreprint
Wilcox, J. E. (2024) · doi:10.31234/osf.io/m8ven · Cited by 1
Satopää, V., Pemantle, R., & Ungar, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.06943
Self-citationEfficient Probability Sequences
Regnier, E. (2014) · Calhoun: The Naval Postgraduate School Institutional Archive (Naval Postgraduate School)
Beszámoló a spudm 25 konferenciáról
Szántó, R., & Gáspár, J. (2015) · Vezetéstudomány / Budapest Management Review · doi:10.14267/veztud.2015.12.06
The Perceived and Actual Influence of Social Diversity on "Crowd" Judgment Accuracy
Chen, S. (2017) · Deep Blue (University of Michigan)
Dorman, R. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3732518
“Scott Armstrong, “Predicting Job Performance: The Moneyball Factor,” Foresight: The International Journal of Applied Forecasting, 31–34, 2012, https://faculty.wharton.upenn.edu/wpcontent/uploads/2012/05/Moneyball-Foresight_1.pdf. 298 Claudia Goldin and Cecilia Rouse, “Orchestrating Impartiality: The Impact of ‘Blind’…”
BackgroundInfluentialAggregated knowledge from a small number of debates outperforms the wisdom of large crowds
Navajas, J., Niella, T., Garbulsky, G., Bahrami, B., & Sigman, M. (2017) · arXiv · doi:10.48550/arxiv.1703.00045
Measuring Forecasting Skill from TextPreprint
Zong, S., Ritter, A., & Hovy, E. (2020) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2006.07425 · Cited by 19
Quantitative and Qualitative Methods for Predicting Geopolitical Events
Parente, F. J., & Finley, J. A. (2020) · Identity in a changing world · doi:10.1007/978-3-030-38270-4_26
Written Justifications are Key to Aggregate Crowdsourced Forecasts
Kotamraju, S., & Blanco, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.355
Asymptotics for pull on the complete graph
Παναγιώτου, Κ., & Reisser, S. (2023) · Stochastic Processes and their Applications · doi:10.1016/j.spa.2023.01.017
Written Justifications are Key to Aggregate Crowdsourced ForecastsPreprint
Kotamraju, S., & Blanco, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07017
The effect of iterativity on adversarial opinion forming
Παναγιώτου, Κ., & Reisser, S. (2023) · Information Processing Letters · doi:10.1016/j.ipl.2023.106453
Neuropsychological Level of Trust
(2021) · Cambridge University Press eBooks · doi:10.1017/9781108770880.007
When alternative hypotheses shape your beliefs: Context effects in probability judgmentsPreprint
Cai, X., & Pleskac, T. J. (2021) · doi:10.31234/osf.io/vaze5
How to See the Future : Forecasting and Global Policy
Moore, B. M., Ruhl, C. R., & Rosen, J. R. (2021) · doi:10.15868/socialsector.39696
Rittich, J. C. (2022) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-9110
Forecasts: Consumption, Production, and Behavioral Responses
Rezaee, A. (2019) · AEA Randomized Controlled Trials · doi:10.1257/rct.4343
Forecasts: Consumption, Production, and Behavioral Responses
Rezaee, A. (2019) · AEA Randomized Controlled Trials · doi:10.1257/rct.4343-1.0
Mercier, H. (2022) · Oxford University Press eBooks · doi:10.1093/oso/9780197636916.003.0014
Getting more Wisdom out of the Crowd: The Case of Competence-Weighted AggregatesPreprint
Goedde‐Menke, M., Diecidue, E., Jacobs, A. H., & Langer, T. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4109734
Shirasuna, M., & Honda, H. (2022) · doi:10.31234/osf.io/fpw6m
A Social Scientist’s Guide to Multiple InferencePreprint
Bowen, D. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4182132
13. Supplementary Material from Online Resources
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.13
Preface: A Vision of Transformed Conservation Practice
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.14
What Correlates with Accuracy: The Empirical Epistemology of Optimal Cognition
Wilcox, J. E. (2022) · SpringerBriefs in psychology · doi:10.1007/978-3-031-19205-0_7
How Can We Get More Accurate: Recommendations About Human Judgment
Wilcox, J. E. (2022) · SpringerBriefs in psychology · doi:10.1007/978-3-031-19205-0_8
Baumeister, R. F., & Lim, K. (2022) · Springer eBooks · doi:10.1007/978-3-030-90913-0_138 · Cited by 3
Forecaster evaluation as model comparisonPreprint
Merkle, E. C., & Flores, R. D. (2023) · doi:10.31219/osf.io/n5czv
Coordinating several mental strategies favors integration: Evidence from human judgmentPreprint
Hoffmȧnn, J., Albrecht, R., & von Helversen, B. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/9yv57
A Crowd-Sourced Approach to Identifying Hidden Critical Suppliers Nexus Supplier Index RevisitedPreprint
Shao, B. B. M., Fiechter, J. L., Minnery, B. S., Choi, T. Y., Handfield, R., Widmer, C. L., Juvina, I., & Steyvers, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4648198
Hauenstein, C. E., Thomas, R. P., Illingworth, D. A., & Dougherty, M. R. P. (2024) · doi:10.31234/osf.io/z9pxk
(2021) · Odile Jacob eBooks · doi:10.3917/oj.merci.2021.01.0411
Wilson, P., Roe, G. A. B., & Kiely, J. (2024) · Research Square · doi:10.21203/rs.3.rs-3861662/v1
RetractedCollective Wisdom and the Fermi Method: Improving the Accuracy of Deliberative Groups
Barrera-Lemarchand, F., Charreau, L. V. L., Ruiz, J., Cáceres, N., Carrillo, F., & Navajas, J. (2024) · SCT Proceedings in Interdisciplinary Insights and Innovations. · doi:10.56294/piii2024257 · Cited by 1
Beard, S., Rowe, T., & Fox, J. (2024) · Open Book Publishers · doi:10.11647/obp.0360.06
The Forecasting Proficiency Test: A General Use Assessment of Forecasting AbilityPreprint
Himmelstein, M., Zhu, S. M., Petrov, N., Karger, E., Helmer, J., Livnat, S., Bennett, A., Hedley, P., & Tetlock, P. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/a7kdx · Cited by 2
Self-citationHauenstein, C. E., Thomas, R. P., Illingworth, D. A., & Dougherty, M. R. P. (2024) · Psychological Science · doi:10.1177/09567976241266481 · Cited by 3
“Mellers et al. (2014) implemented a variant of the Brier score—the mean daily average Brier score (a sum of the Brier scores averaged over days and questions over each participant).”
InfluentialImproving Measurement and Theorizing About Implicit Bias
Krosnick, J. A., Stark, T., & Scott, A. L. (2024) · Cambridge University Press eBooks · doi:10.1017/9781108885492.023
Wilson, P. J., Roe, G. A. B., & Kiely, J. (2024) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000004989
“Research has shown that effective teamwork contributes to better decision outcomes in various contexts (26,29).”
The Illusion of Control and Willingness to Pay for LuckPreprint
Vranka, M. A., Houdek, P., & Machek, O. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5120325
Crowd Assessment of the Military Utility of Future Technologies
Dansarie, M., Andersson, K., & Silfverskiöld, S. (2025) · Scandinavian Journal of Military Studies · doi:10.31374/sjms.339 · Cited by 1
Improving group forecast accuracy via mindful organizing
Rosemberg Montes, E. (2024) · doi:10.63537/erm176340
(2025) · Dunod eBooks · doi:10.3917/dunod.compe.2025.01.0488
Conditions for a Valid Consensus
Jorm, A. F. (2025) · Springer eBooks · doi:10.1007/978-981-96-8357-4_3
Metadata
- Authors
- Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E.
- Affiliations
- University of Pennsylvania (US)
- Rice University (US)
- Department of Psychology, University of Pennsylvania
- University of California, Berkeley (US)
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-03-21
- Volume
- 25
- Issue
- 5
- Pages
- 1106-1115
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2131472525
- Cited by
- 312
- References
- 42
- Citations by year
- 2026: 20 · 2025: 22 · 2024: 28 · 2023: 29 · 2022: 47 · 2021: 31 · 2020: 14 · 2019: 19 · 2018: 25 · 2017: 28 · 2016: 19 · 2015: 19 · 2014: 11
- FWCI
- 32.31
- Type
- journal article
- Language
- en
- Open access
- green
- www.escholarship.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- U.S. Department of the Interior
- IBM Center for the Business of Government
- Intelligence Advanced Research Projects Activity
- Interior Business Center
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Sports Analytics and Performance
- Keywords
- geopolitical forecasting, forecasting tournaments, forecast accuracy, teaming, tracking, calibration, resolution, cognitive biases, reference classes, behavioral interventions, forecast aggregation, heuristics
- MeSH terms
- Adult, Algorithms, Female, Forecasting, Humans, Interpersonal Relations, Judgment, Male, Probability, Psychological Techniques, Social Behavior, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.