Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Open accessThe DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.
Abstract
Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.
Overview
CoChat Summary
## TL;DR
This paper reports a 2-year geopolitical forecasting tournament and tests three psychological interventions—training, teaming, and tracking—to improve probabilistic forecasts. All three improved accuracy, and the top 2% of Year 1 forecasters placed into elite teams (“superforecasters”) performed best in Year 2.
## Key findings
- Training improved accuracy: In Year 1, training improved Brier scores, with probability training outperforming scenario training, and scenario training outperforming no training: F(2, 1586) = 14.29, p < .001; t(1053) = 2.23, p = .026; t(1056) = 3.25, p < .001. In Year 2, probability training again beat no training: F(1, 882) = 19.12, p < .001.
- Teaming improved accuracy: Group influence was significant in both years, F(2, 1586) = 60.68, p < .001 and F(1, 882) = 62.76, p < .001. In Year 1, teams were more accurate than crowd-belief forecasters, who were more accurate than independent forecasters: t(1036) = 5.52, p < .001 and t(1120) = 5.92, p < .001.
- Tracking the top performers worked best: The best performers from Year 1, the top 2%, were placed together in elite Year 2 teams and “outperformed all other groups by a wide margin.” The paper reports no evidence of Year 2 regression to the mean.
- Both calibration and resolution improved: Calibration improved with training in Years 1 and 2, F(2, 1586) = 3.16, p = .04 and F(1, 938) = 3.78, p < .05; resolution varied with group influence and training in both years, and tracking enhanced resolution, t(293) = 4.12, p < .001.
## Methods
- The study embedded experiments in a 2-year geopolitical forecasting tournament (Year 1: September 2011–April 2012; Year 2: June 2012–April 2013) on goodjudgmentproject.com, where participants made probability estimates for 199 geopolitical outcomes.
- Year 1 used a 3 (training: no/scenario/probability) × 4 (group influence: independent/crowd-belief/team/prediction-market) factorial design; Year 2 used a 2 (training: no/probability) × 3 (group influence: independent/team/prediction-market) design plus an elite tracking condition. Results for prediction markets are noted as not discussed in the paper.
- The main outcome was Brier score; analyses also examined calibration and resolution, using statistical tests reported as F-tests, t-tests, and regressions. The paper states that findings were robust across Brier, spherical, and logarithmic scoring rules.
## Limitations
- The paper notes that results for prediction-market forecasting are not discussed here. This limits what can be concluded from the full design as presented in this article.
- The paper itself says “No one knows how much further improvement is possible” and that there are likely “pockets of inherent uncertainty” making perfect or near-perfect Brier scores impossible. This is a stated substantive limitation of forecasting accuracy.
- Evident from the text: the Year 2 no-training condition included some participants who had previously received scenario training, which the authors say “tipped the scale against finding a statistically significant effect of training.”
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
0 Valid · 0 Not valid · 5 Not checked · 39 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and in the middle period of Year 2, F (1, 882) = 6.10, p = .014.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For both years, training and teaming remained significant predictors even when timing and updating were included as predictors.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition to showing a main effect of group influence, Year 2 data showed an interaction between training and group influence, F (1, 882) = 5.78, p = .018.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Year 2, calibration also benefited from group influence, F (1, 938) = 7.07, p = .008, and tracking.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There was no evidence of Year 2 regression to the mean;”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
19 more claims
Create a free account to see all 44 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 16
- Probably not a substantial claim. 6
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Additionally, as reported elsewhere, training improved the calibration and resolution of forecasters by reducing overconfidence (Mellers et al., 2014; Moore et al., 2016).”
From Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments (2016) · cites this paper as Mellers et al., 2014
Experimental — accuracy not yet measured
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom, each partial η² from its F, and each ratio, mean difference or B printed with a p checked against its own confidence interval.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 882) = 5.78, p = .018”
Reported p = .018 · recomputed p = .016
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (1, 938) = 3.78, p < .05”
Reported p < .05 · recomputed p = .052
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“F (2, 1463) = 6.20, p < .001”
Reported p < .001 · recomputed p = .002
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1056) = 3.25, p < .001”
Reported p < .001 · recomputed p = .001
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t (1588) = 2.70, p < .001”
Reported p < .001 · recomputed p = .007
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(1056) = 3.25reported p < .001 · recomputed p = .001
“t (1056) = 3.25, p < .001”
- Doesn’t add upF(1, 882) = 5.78reported p = .018 · recomputed p = .016
“F (1, 882) = 5.78, p = .018”
- Doesn’t add upF(2, 1463) = 6.2reported p < .001 · recomputed p = .002
“F (2, 1463) = 6.20, p < .001”
- Doesn’t add upt(1588) = 2.7reported p < .001 · recomputed p = .007
“t (1588) = 2.70, p < .001”
- Doesn’t add upF(1, 938) = 3.78reported p < .05 · recomputed p = .052
“F (1, 938) = 3.78, p < .05”
37 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Groups: Interaction and Performance.
Polley, R. B., & McGrath, J. E. (1984) · Administrative Science Quarterly · doi:10.2307/2393041 · Cited by 3,290
Norm theory: Comparing reality to its alternatives.
Kahneman, D., & Miller, D. T. (1986) · Psychological Review · doi:10.1037/0033-295x.93.2.136 · Cited by 3,024
The difference: how the power of diversity creates better groups, firms, schools, and societies
(2007) · Choice Reviews Online · doi:10.5860/choice.45-1534 · Cited by 2,280
Many Hands Make Light the Work: The Causes and Consequences of Social Loafing
Latané, B., Williams, K., & Harkins, S. (1979) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.37.6.822 · Cited by 2,216
Stasser, G., & Titus, W. (1985) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.48.6.1467 · Cited by 1,850 · cited 1×
“Teams have a demonstrated failure to search for unique information, unshared information, or hypothesis-disconfirming information (Kerr, MacCoun, & Kramer, 1996; Stasser & Titus, 1985; Sunstein, 2006).”
BackgroundA Theory of Fads, Fashion, Custom, and Cultural Change as Informational Cascades
Bikhchandani, S., Hirshleifer, D., & Welch, I. (1992) · Journal of Political Economy · doi:10.1086/261849 · Cited by 1,805
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Group Performance and Decision Making
Kerr, N. L., & Tindale, R. S. (2004) · Annual Review of Psychology · doi:10.1146/annurev.psych.55.090902.142009 · Cited by 1,331
Research works printed but not linked: 2
- Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations . Washington, DC: National Academies Press.
- Janis, I. (1982). Group-think . Boston, MA: Houghton-Mifflin.
Cited by
Items 101–200 of 321
Measuring Forecasting Skill from Text
Zong, S., Ritter, A., & Hovy, E. H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.473 · Cited by 30
Scenario generation and scenario quality using the cone of plausibility
Dhami, M. K., Wicke, L., & Önkal, D. (2022) · Futures · doi:10.1016/j.futures.2022.102995 · Cited by 28
How the wisdom of crowds, and of the crowd within, are affected by expertise
Fiechter, J. L., & Kornell, N. (2021) · Cognitive Research Principles and Implications · doi:10.1186/s41235-021-00273-6 · Cited by 25
Forecasting the Accuracy of Forecasters from Properties of Forecasting RationalesPreprint
Karvetski, C. W., Meinel, C., Maxwell, D. T., Lu, L., Mellers, B. A., & Tetlock, P. E. (2021) · SSRN Electronic Journal · doi:10.1016/j.ijforecast.2021.09.003 · Cited by 32
Self-citationThe wisdom of many in few: Finding individuals who are as wise as the crowd.
Himmelstein, M., Budescu, D. V., & Ho, E. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001340 · Cited by 28
Evidence for benefits of argumentation in a Mayan indigenous population
Castelain, T., Girotto, V., Jamet, F., & Mercier, H. (2016) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2016.02.002 · Cited by 23
“lems (Laughlin, 2011; Moshman & Geil, 1998; Trouche, Sander, & Mercier, 2014), work related tasks (Mercier, 2011c), forecasting (Mellers et al., 2014; Rowe & Wright, 1999), and school tasks (Mercier, 2011b; Slavin, 1995; Smith et al.”
An IRT forecasting model: linking proper scoring rules to item response theory
Bo, Y. E., Budescu, D. V., Lewis, C., Tetlock, P. E., & Mellers, B. A. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500005647 · Cited by 21
“Observed vs. Expected Response Frequencies September 2011 and July 2014 (Mellers et al. 2014; Mellers, Stone, Murray, Minster, Rohrbaugh, Bishop, Chen, Baker, Hou, Horowitz, Ungar and Tetlock, 2015; Tetlock, Mellers, Rohrbaugh and Chen, 2014).”
Self-citationUsing prediction polling to harness collective intelligence for disease forecasting
Sell, T. K., Warmbrod, K. L., Watson, C., Trotochaud, M., Martin, E. K., Ravi, S., Balick, M., & Servan-Schreiber, E. (2021) · BMC Public Health · doi:10.1186/s12889-021-12083-y · Cited by 21
“Hypermind ’ s Pres-cience platform was developed though experience with several Intelligence Advanced Research Projects Activity (IARPA) research programs on crowd-based forecasting for the US intelligence community [ 16 – 18].”
BackgroundOn the round number bias and wisdom of crowds in different response formats for numerical estimation
Honda, H., Kagawa, R., & Shirasuna, M. (2022) · Scientific Reports · doi:10.1038/s41598-022-11900-7 · Cited by 30
Fisk, S. R. (2018) · Social Psychology Quarterly · doi:10.1177/0190272518796512 · Cited by 24
“For example, individuals who receive training to make forecasts about a highly uncertain future are better able to overcome the cognitive biases that lead most people to make highly inaccurate judgments (Mellers et al. 2014).”
Probability Forecasts Made at Multiple Lead Times
Regnier, E. (2017) · Management Science · doi:10.1287/mnsc.2016.2720 · Cited by 18
Crowds Can Effectively Identify Misinformation at ScalePreprint
Martel, C., Allen, J. N. L., Pennycook, G., et al. (2022) · doi:10.31234/osf.io/2tjk7 · Cited by 18
Hybrid forecasting of geopolitical events†
Benjamin, D. M., Morstatter, F., Abbas, A. E., Abeliuk, A., Atanasov, P. D., Bennett, S. T., Beger, A., Birari, S., Budescu, D. V., Catasta, M., et al. (2023) · AI Magazine · doi:10.1002/aaai.12085 · Cited by 41
“While there are established methods showing how crowdsourced forecasting succeeds, there are not established methods for a hybrid forecasting system (Mellers et al. 2014).”
Self-citationRegularized Aggregation of One-Off Probability Predictions
Satopää, V. (2022) · Operations Research · doi:10.1287/opre.2021.2224 · Cited by 17
Schneider, G., Hadar, M., & Bosler, N. (2017) · Journal of Peace Research · doi:10.1177/0022343316683437 · Cited by 17
Building the tower without climbing it: Progress in engineering systems
Broniatowski, D. A. (2018) · Systems Engineering · doi:10.1002/sys.21426 · Cited by 17
Partial information framework: Model-based aggregation of estimates from diverse information sources
Satopää, V., Jensen, S. T., Pemantle, R., & Ungar, L. (2017) · Electronic Journal of Statistics · doi:10.1214/17-ejs1346 · Cited by 19
Self-citationHow the “wisdom of the inner crowd” can boost accuracy of confidence judgments.
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2020) · Decision · doi:10.1037/dec0000119 · Cited by 23
“In numerous areas of real-world decision making, such as intelligence service (Betts, 1978; Mandel & Barnes, 2014; Mellers et al., 2014), eyewitness reports (Wixted, Mickes, Dunn, Clark, & Wells, 2016), the stock market, and medical diagnostics (Berner & Graber, 2008), people cannot help but rely on confidence judgmen…”
BackgroundOutcome feedback reduces over-forecasting of inflation and overconfidence in forecasts
Niu, X., & Harvey, N. (2022) · Judgment and Decision Making · doi:10.1017/s1930297500009050 · Cited by 16
Too Much Trust in Group Decisions: Uncovering Hidden Profiles by Groups and Markets
Maciejovsky, B., & Budescu, D. V. (2020) · Organization Science · doi:10.1287/orsc.2020.1363 · Cited by 11
The Psychological Aspects of Corporate Foresight
Нестик, Т. А. (2018) · Foresight-Russia · doi:10.17323/2500-2597.2018.2.78.90 · Cited by 19
“How can the aforementioned motivational and cognitive biases as well as the impact of the group dynamics on the results of corporate Foresight sessions be reduced? Studies show that training experts on the use of special techniques designed to diminish such negative effects increases the accuracy of the forecasts [Mel…”
BackgroundExtremizing and Antiextremizing in Bayesian Ensembles of Binary-Event Forecasts
Lichtendahl, K. C., Grushka‐Cockayne, Y., Jose, V. R. R., & Winkler, R. L. (2022) · Operations Research · doi:10.1287/opre.2021.2176 · Cited by 18
“Schemes to extremize the average of subjective forecasts from experts have been proposed (Erev et al. 1994, Ariely et al. 2000, Shlomi and Wallsten 2010, Mellers et al. 2014, Baron et al. 2014).”
IDEA for Uncertainty Quantification
Hanea, A. M., Burgman, M. A., & Hemming, V. (2017) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-319-65052-4_5 · Cited by 26
Bias, Information, Noise: The BIN Model of ForecastingPreprint
Satopää, V. A., Salikhov, M., Tetlock, P., & Mellers, B. (2020) · Management Sciences · doi:10.2139/ssrn.3540864 · Cited by 48
“The remainder of the article applies the model to data from a multiyear, geopolitical forecasting tour-nament to explore the mechanisms via which three experimental interventions — training, teaming, and tracking of talent — improved forecasts (Mellers et al. 2014).”
Self-citationTracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study
Mandel, D. R., & Irwin, D. (2021) · Futures & Foresight Science · doi:10.1002/ffo2.98 · Cited by 22
How feasible is the rapid development of artificial superintelligence?
Sotala, K. (2017) · Physica Scripta · doi:10.1088/1402-4896/aa90e8 · Cited by 19
Metacognition biases information seeking in assessing ambiguous news
Guigon, V., Villeval, M. C., & Dreher, J. (2024) · Communications Psychology · doi:10.1038/s44271-024-00170-w · Cited by 20
“It includes as well training with speci fi c search heuristics 11,82 – 85 and probabilitycalibration exercisesto help people improve their assessment of their own knowledge and their need for further information-seeking 83,84,86 .”
Positive and negative forms of replicability in gene network analysis
Verleyen, W., Ballouz, S., & Gillis, J. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btv734 · Cited by 11
Incentive-Compatible Forecasting Competitions
Witkowski, J., Freeman, R., Vaughan, J. W., Pennock, D. M., & Krause, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11471 · Cited by 11
Tetlock, P., & Mellers, B. (2014) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1412524111 · Cited by 10
Self-citationBeyond collective intelligence: Collective adaptationPreprint
Galešić, M., Barkoczi, D., Berdahl, A. M., et al. (2022) · doi:10.31235/osf.io/5f2ad · Cited by 12
How to detect high-performing individuals and groups: decision similarity predicts accuracyPreprint
Kurvers, R. H. J. M., Herzog, S. M., Hertwig, R., Krause, J., Moussaïd, M., Argenziano, G., Zalaudek, I., Carney, P., & Wolf, M. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ryvnw · Cited by 13
Corbett, C. J., Narayanan, S., Aloysius, J. A., Berenguer, G., Bish, E. K., Bjarnadóttir, M. V., Gao, G. G., Glover, W. J., Johnson, M. P., Kalkancı, B., et al. (2025) · Production and Operations Management · doi:10.1177/10591478251318107 · Cited by 22
“In turn, such improved information sharing reduces overconfidence in team judgments and forecasts (Keck and Tang, 2018), improves general forecasting accuracy (Lamberson and Page, 2012; Mellers et al., 2014), and enhances innovation (Chan et al., 2023; Girotra et al., 2010).”
How Effective Altruism Can Help Psychologists Maximize Their Impact
Gainsburg, I., Pauer, S., Abboub, N., Aloyo, E., Mourrat, J.-C., & Cristià, A. (2022) · Perspectives on Psychological Science · doi:10.1177/17456916221079596 · Cited by 15
“Notable existing work on this topic includes research on forecasting (Mellers et al., 2014; Tetlock & Gardner, 2015), which has shown that individual differences (e.”
BackgroundInfluentialDecision Making in Groups and Organizations
Tindale, R. S., & Kluwe, K. (2015) · Wiley eBooks · doi:10.1002/9781118468333.ch30 · Cited by 14
Rebholz, T. R., Biella, M., & Hütter, M. (2024) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2369 · Cited by 13
McAndrew, T., Cambeiro, J., & Besiroglu, T. (2022) · Vaccine · doi:10.1016/j.vaccine.2022.02.054 · Cited by 13
Recalibrating probabilistic forecasts to improve their accuracy
Han, Y., & Budescu, D. V. (2022) · Judgment and Decision Making · doi:10.1017/s1930297500009049 · Cited by 18
“Mellers et al. (2014) applied Karmarkar’s transformation to data generated by more than 2,000 forecasters in a geopolitical forecasting tournament (Aggregative Contingent Estimation ACE; https://www.iarpa.gov/research-programs/ace).”
BackgroundMethodInfluentialMellers, B. A., McCoy, J. P., Lu, L., & Tetlock, P. E. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231185339 · Cited by 16
“…that several strategies improved human forecasts, and these strategies allowed us to win the tournament each year for 4 years (Chang et al., 2016; Mellers et al., 2014; Mellers, Stone, Atanasov, et al., 2015; Satopää, Baron, et al., 2014; Satopää, Jensen, et al., 2014; Satopää et al., 2017).”
BackgroundSelf-citationWhen do people rely on algorithms
Logg, J. M. (2016) · eScholarship (California Digital Library) · Cited by 10
Keeping Rumors in Proportion: Managing Uncertainty in Rumor Systems
Krafft, P. M., & Spiro, E. S. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300876 · Cited by 9
“, [31]), or perhaps a prediction market [33], make a quantitative judgment about what is a rational probability P(R | E) that the rumor is true given public evidence; and to assess the gap between that probability and the proportion of people who express belief in the rumor either through a survey or through observati…”
BackgroundWallace, K., Bollfrass, A. K., Cooper, O., Epstein, D. S., Forrest, S. A., Lukosch, H. K., Mason, R., Meesters, K., Reid, S. A., Waring, S., et al. (2025) · Safety Science · doi:10.1016/j.ssci.2025.106898 · Cited by 14
Blinding to Circumvent Human Biases: Deliberate Ignorance in Humans, Institutions, and Machines
Hertwig, R., Herzog, S. M., & Kozyreva, A. (2023) · Perspectives on Psychological Science · doi:10.1177/17456916231188052 · Cited by 23
“This wisdom-of-the-crowd effect has been used in diverse fields, including medical diagnostics (e.g., cancer diagnosis; Kurvers et al., 2016) and economic and political forecasting (Mellers et al., 2014).”
MethodThe Epistemic Contract: Fostering an Appropriate Level of Public Trust in Experts
MacCoun, R. J. (2015) · Nebraska Symposium on Motivation/The Nebraska symposium on motivation · doi:10.1007/978-3-319-16151-8_9 · Cited by 11
A Network Model of Goals Boosts Convergent Creativity Performance
Kung, F. Y. H., & Scholer, A. A. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01910 · Cited by 8
Funk, P., Davis, A. L., Vaishnav, P., Dewitt, B., & Fuchs, E. R. H. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2020.119984 · Cited by 12
Kuroda, K., Takahashi, M., & Kameda, T. (2023) · Scientific Reports · doi:10.1038/s41598-023-42080-7 · Cited by 9
“Importantly, this contrasts with previous findings that confident members are rather willing to contribute their judgments to a group when collective performance is emphasized 17–19 .”
BackgroundResultInfluentialExperts and Laymen Grossly Underestimate the Benefits of Argumentation for ReasoningPreprint
Mercier, H., Trouche, E., Yama, H., et al. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2514166 · Cited by 9
Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
Collins, R. N., Mandel, D. R., & Budescu, D. V. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_7 · Cited by 27
Turschwell, M. P., Brown, C. J., Lacharité, M., et al. (2023) · Environmental Science & Policy · doi:10.1016/j.envsci.2023.06.008 · Cited by 8
Calibration of scientific reasoning ability
Otten, C. D., & Fischhoff, B. (2022) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2306 · Cited by 9
Talent Spotting in Crowd Prediction
Atanasov, P. D., & Himmelstein, M. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_6 · Cited by 26
Self-citationUsing scenarios to forecast outcomes of a refugee crisis
Wicke, L., Dhami, M. K., Önkal, D., & Belton, I. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.05.017 · Cited by 12
Characteristics of a Process for Subjective Probability Elicitation
Quigley, J. M., & Walls, L. (2021) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-030-46474-5_13 · Cited by 8
Compromising improves forecasting
Ferreiro, D. N., Deroy, O. D., & Bahrami, B. (2023) · Royal Society Open Science · doi:10.1098/rsos.221216 · Cited by 5
Harada, T. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.1063525 · Cited by 10
Hybrid Collective Intelligence for Decision Support in Complex Open-Ended Domains
Trianni, V., Nuzzolese, A. G., Porciello, J., Kurvers, R. H. J. M., Herzog, S. M., Barabucci, G., Berditchevskaia, A., & Fung, F. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230079 · Cited by 7
“Out of the initial 1572 cases available, we obtained 1333 cases in which the available ground truth could be exactly matched (Jaccard similarity of 1) [53].”
BackgroundBayesian Revision vs. Information Distortion
Russo, J. E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.01550 · Cited by 6
Automated calibration training for forecasters
Stone, E. R., Luu, J., Costello, C., & Somerville, A. H. (2023) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2334 · Cited by 6
Self-citationA Prediction Tournament Paradox
Aldous, D. (2019) · The American Statistician · doi:10.1080/00031305.2019.1604430 · Cited by 6
“Study of results of prediction tournaments in recent years has led to the popular book by Tetlock and Gardner (2015) and substantial academic literature—for instance Mellers et al. (2014) has 147 Google Scholar citations.”
ResultThe Impact of Overconfidence and Ambiguity Attitude on Market EntryPreprint
Gutierrez, C., Åstebro, T. B., & Obłój, T. (2018) · SSRN Electronic Journal · doi:10.2139/ssrn.2924381 · Cited by 6
Combining Information from Multiple Forecasters: Inefficiency of Central TendencyPreprint
Satopää, V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06006 · Cited by 10
“Furthermore, the calibration of human forecasters can be improved through team collaboration, training, tracking (Mellers et al., 2014), timely and accurate feedback (Murphy and Daan, 1984), representative sampling of target events (Gigerenzer et al.”
BackgroundHuman Forest vs. Random Forest in Time-Sensitive COVID-19 Clinical Trial PredictionPreprint
Atanasov, P. D., Joseph, R., Feijoo, F., Marshall, M., Conway, A., & Siddiqui, S. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3981732 · Cited by 11
Self-citationTowards Augmented Financial IntelligencePreprint
Mihov, A.-H., Firoozye, N. B., & Treleaven, P. C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4148057 · Cited by 9
How the “Wisdom of the Inner Crowd” Can Boost Accuracy of Confidence JudgmentsPreprint
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ngweh · Cited by 6
Tetlock, P. E., Lu, L., & Mellers, B. A. (2022) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2022.02.008 · Cited by 10
Self-citation1. Introduction: The Evidence Crisis and the Evidence Revolution
Sutherland, W. J. (2022) · Open Book Publishers · doi:10.11647/obp.0321.01 · Cited by 6
7. Framing the Problem and Identifying Potential Solutions
Esmail, N., Green, R. E., Petrovan, S. O., Salafsky, N., Taylor, N. G., & Wilson, J. D. (2022) · Open Book Publishers · doi:10.11647/obp.0321.07 · Cited by 6
Robust recalibration of aggregate probability forecasts using meta-beliefs
Peker, C., & Wilkening, T. S. (2024) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2024.09.005 · Cited by 6
Dissecting Discrimination: Identifying Its Various Faces and Their Sources
Villiger, D. (2021) · Entscheidungs- und Organisationstheorie · doi:10.1007/978-3-658-34569-3 · Cited by 6
How Bayesian Rationality Fails and Critical Rationality Works
Albert, M. (2017) · Homo Oeconomicus · doi:10.1007/s41412-017-0062-3 · Cited by 5
“Given a stochastic environment and the fact that probabilistic forecasts were required, the result that training in probabilistic reasoning improved forecasts (Mellers et al. 2014) is not surprising.”
BackgroundCombining and Extremizing Real-Valued ForecastsPreprint
Satopää, V., & Ungar, L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06405 · Cited by 11
“(2014), and Mellers et al. (2014), have also discussed extremization of probability forecasts.”
BackgroundMethodSelf-citationHow Effective Altruism Can Help Psychologists Maximize Their ImpactPreprint
Gainsburg, I., Pauer, S., Abboub, N., Aloyo, E., Mourrat, J.-C., & Cristià, A. (2021) · doi:10.31234/osf.io/8dw59 · Cited by 5
11. Creating a Culture of Evidence Use
Frick, W. F., Cooke, S. J., Dickson, I., Jackson, H. A., Keller, K. E., la Luz, A. D., Parks, D., McPherson, T., Simkins, A. T., Tanshi, I., et al. (2022) · Open Book Publishers · doi:10.11647/obp.0321.11 · Cited by 4
5. Improving the Reliability of Judgements
Wintle, B. C., Mukherjee, N., Hemming, V., Canessa, S., & McBride, M. F. (2022) · Open Book Publishers · doi:10.11647/obp.0321.05 · Cited by 6
Crowd prediction systems: Markets, polls, and elite forecasters
Atanasov, P. D., Witkowski, J., Mellers, B. A., & Tetlock, P. E. (2024) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2023.12.009 · Cited by 4
Self-citationThe effect of calibration training on the calibration of intelligence analysts' judgments
Kelly, M. O., & Mandel, D. R. (2024) · Applied Cognitive Psychology · doi:10.1002/acp.4236 · Cited by 6
2. Gathering and Assessing Pieces of Evidence
White, T., Amano, T., Boersch‐Supan, P. H., Christie, A. P., Freckleton, R. P., Quinzin, M. C., Rezaie, A. M., Sutherland, W. J., & Yamashita, H. (2022) · Open Book Publishers · doi:10.11647/obp.0321.02 · Cited by 6
Predicting the future with humans and AI
Mellers, B. A., Lu, L., & McCoy, J. P. (2022) · Consumer Psychology Review · doi:10.1002/arcp.1089 · Cited by 7
Self-citationThe protective scale of the Armidilo‐S: The importance of forensic and clinical outcomes
Lindsay, W. R., Steptoe, L. R., Haut, F., et al. (2018) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.12456 · Cited by 4
Friedman, J. A., Lerner, J. S., & Zeckhauser, R. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2782598 · Cited by 2
“Moreover, research programs such as the Good Judgment Project have shown that foreign policy analysts can make numeric probability assessments effectively (Mellers et al. 2014, Tetlock and Gardner 2015).”
BackgroundHuman judgment forecasts of human monkeypox transmission and burden in non-endemic countriesPreprint
McAndrew, T., Majumder, M. S., Lover, A. A., et al. (2022) · doi:10.31219/osf.io/xdt4k · Cited by 4
Artificial Intelligence and Actor-Specific DecisionsPreprint
Felin, T., Sako, M., & Hullman, J. R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5279401 · Cited by 5
Wilson, P. J., Roe, G. A. B., & Kiely, J. (2025) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2025.1664191 · Cited by 3
“While our findings align with team decision-making research in other fields (22), the sport-specific challenge of balancing support staff perspectives with the ultimate authority of the coaching staff creates unique decision-making dynamics that are not typically present in more…”
InfluentialPredictable artificial intelligence
Zhou, L., Casares, P. A. M., Martínez‐Plumed, F., Burden, J., Burnell, R., Cheke, L. G., Ferri, C. S., Marcoci, A., Mehrbakhsh, B., Moros-Daval, Y., et al. (2026) · Artificial Intelligence · doi:10.1016/j.artint.2026.104491 · Cited by 41
Three and a half stages of crowd wisdom
Douven, I., Kriegeskorte, N., & Stinson, P. (2026) · Collective Intelligence · doi:10.1177/26339137261435121 · Cited by 3
10. How Conservation Practice Can Generate Evidence
Ockendon, N., Cadotte, M. W., Eklund, J., Ferraro, P. J., Fisher, M., Hancock, M. H., & Worthington, T. A. (2022) · Open Book Publishers · doi:10.11647/obp.0321.10 · Cited by 4
Using the Wisdom of the Crowd to Improve the Condition Assessment of Residential Real Estate
Koch, D., Seiler, M. J., & Thaler, S. (2024) · Journal of Real Estate Research · doi:10.1080/08965803.2024.2352223 · Cited by 4
Short-Term Expectation Formation Versus Long-Term Equilibrium Conditions: The Danish Housing Market
Hetland, A., & Hetland, S. (2017) · Econometrics · doi:10.3390/econometrics5030040 · Cited by 3
“, cf. Mellers et al. (2014), which suggests that, in practice, expectation formation for macroeconomic outcomes goes beyond the problem of identifying an appropriate statistical model.”
When alternative hypotheses shape your beliefs: Context effects in probability judgmentsPreprint
Cai, X., & Pleskac, T. J. (2021) · PsyArXiv (OSF Preprints) · doi:10.1016/j.cognition.2022.105306 · Cited by 4
Spada, P., & Paulson, L. (2023) · Routledge eBooks · doi:10.4324/9781003215929-6 · Cited by 7
Teaching epistemic integrity to promote reliable scientific communication
Allard, A., & Clavien, C. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1308304 · Cited by 10
“Psychological research has consistently shown that statistical training can improve judgments about everyday situations, and can even improve accuracy in predicting world events (Fong et al., 1986; Nisbett et al., 1987; Mellers et al., 2014; Chang et al., 2016).”
BackgroundPromoting Erroneous Divergent Opinions Increases the Wisdom of Crowds
Barrera-Lemarchand, F., Balenzuela, P., Bahrami, B., Deroy, O. D., & Navajas, J. (2024) · Psychological Science · doi:10.1177/09567976241252138 · Cited by 16
The Bias of Crowds: Rethinking Implicit Bias in Social Context
Lundberg, K. B., & Payne, B. K. (2024) · Cambridge University Press eBooks · doi:10.1017/9781108885492.026 · Cited by 3
Terrorism-Centric Behavior Recognition and Adversarial Threat Forecasting
Regens, J. L., Mould, N. A., Jensen, C. J., & Edger, D. N. (2016) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850607.2015.1083317 · Cited by 3
Examining learning coherence in group decision-making: triads vs. tetrads
Harada, T. (2021) · Scientific Reports · doi:10.1038/s41598-021-00089-w · Cited by 2
The wisdom of collective grading and the effects of epistemic and semantic diversity
Lyon, A., & Morreau, M. (2017) · Theory and Decision · doi:10.1007/s11238-017-9643-7 · Cited by 3
“Collective wisdom has been documented in many different tasks, from Francis Galton’s classic example of guessing the weight of an ox at a county fair (Galton 1907) to forecasting geopolitical events (Mellers et al. 2014) and medical diagnosis (Wolf et al. 2015).”
MethodWilkins, D., Foster, C., Sanders, M., et al. (2020) · ORCA Online Research @Cardiff (Cardiff University) · Cited by 3
Bellucci, G., & Dreher, J. (2021) · Cambridge University Press eBooks · doi:10.1017/9781108770880.011 · Cited by 4
Overprecision in the Survey of Professional Forecasters
Campbell, S., & Moore, D. A. (2024) · Collabra Psychology · doi:10.1525/collabra.92953 · Cited by 13
Self-citation
Metadata
- Authors
- Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E.
- Affiliations
- University of Pennsylvania (US)
- Rice University (US)
- Department of Psychology, University of Pennsylvania
- University of California, Berkeley (US)
- ORCID
- Venue
- Psychological Science
- Publisher
- SAGE Publications
- Published
- 2014-03-21
- Volume
- 25
- Issue
- 5
- Pages
- 1106-1115
- ISSN
- 0956-7976 (ISSN-L)
- 1467-9280
- OpenAlex
- W2131472525
- Cited by
- 312
- References
- 42
- Citations by year
- 2026: 20 · 2025: 22 · 2024: 28 · 2023: 29 · 2022: 47 · 2021: 31 · 2020: 14 · 2019: 19 · 2018: 25 · 2017: 28 · 2016: 19 · 2015: 19 · 2014: 11
- FWCI
- 32.31
- Type
- journal article
- Language
- en
- Open access
- green
- www.escholarship.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- U.S. Department of the Interior
- IBM Center for the Business of Government
- Intelligence Advanced Research Projects Activity
- Interior Business Center
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Sports Analytics and Performance
- Keywords
- geopolitical forecasting, forecasting tournaments, forecast accuracy, teaming, tracking, calibration, resolution, cognitive biases, reference classes, behavioral interventions, forecast aggregation, heuristics
- MeSH terms
- Adult, Algorithms, Female, Forecasting, Humans, Interpersonal Relations, Judgment, Male, Probability, Psychological Techniques, Social Behavior, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.