CoChat

Psychological Strategies for Winning a Geopolitical Forecasting Tournament

Open access

The DOI matches its Crossref record; 43 of the 44 references it cites were found in a registry or an index; none of the 9 sources its claims cite is checked for support yet.

Abstract

Five university-based research groups competed to recruit forecasters, elicit their predictions, and aggregate those predictions to assign the most accurate probabilities to events in a 2-year geopolitical forecasting tournament. Our group tested and found support for three psychological drivers of accuracy: training, teaming, and tracking. Probability training corrected cognitive biases, encouraged forecasters to use reference classes, and provided forecasters with heuristics, such as averaging when multiple estimates were available. Teaming allowed forecasters to share information and discuss the rationales behind their beliefs. Tracking placed the highest performers (top 2% from Year 1) in elite teams that worked together. Results showed that probability training, team collaboration, and tracking improved both calibration and resolution. Forecasting is often viewed as a statistical problem, but forecasts can be improved with behavioral interventions. Training, teaming, and tracking are psychological interventions that dramatically increased the accuracy of forecasts. Statistical algorithms (reported elsewhere) improved the accuracy of the aggregation. Putting both statistics and psychology to work produced the best forecasts 2 years in a row.

Overview

CoChat Summary

## TL;DR

This paper reports a 2-year geopolitical forecasting tournament and tests three psychological interventions—training, teaming, and tracking—to improve probabilistic forecasts. All three improved accuracy, and the top 2% of Year 1 forecasters placed into elite teams (“superforecasters”) performed best in Year 2.

## Key findings

  • Training improved accuracy: In Year 1, training improved Brier scores, with probability training outperforming scenario training, and scenario training outperforming no training: F(2, 1586) = 14.29, p < .001; t(1053) = 2.23, p = .026; t(1056) = 3.25, p < .001. In Year 2, probability training again beat no training: F(1, 882) = 19.12, p < .001.
  • Teaming improved accuracy: Group influence was significant in both years, F(2, 1586) = 60.68, p < .001 and F(1, 882) = 62.76, p < .001. In Year 1, teams were more accurate than crowd-belief forecasters, who were more accurate than independent forecasters: t(1036) = 5.52, p < .001 and t(1120) = 5.92, p < .001.
  • Tracking the top performers worked best: The best performers from Year 1, the top 2%, were placed together in elite Year 2 teams and “outperformed all other groups by a wide margin.” The paper reports no evidence of Year 2 regression to the mean.
  • Both calibration and resolution improved: Calibration improved with training in Years 1 and 2, F(2, 1586) = 3.16, p = .04 and F(1, 938) = 3.78, p < .05; resolution varied with group influence and training in both years, and tracking enhanced resolution, t(293) = 4.12, p < .001.

## Methods

  • The study embedded experiments in a 2-year geopolitical forecasting tournament (Year 1: September 2011–April 2012; Year 2: June 2012–April 2013) on goodjudgmentproject.com, where participants made probability estimates for 199 geopolitical outcomes.
  • Year 1 used a 3 (training: no/scenario/probability) × 4 (group influence: independent/crowd-belief/team/prediction-market) factorial design; Year 2 used a 2 (training: no/probability) × 3 (group influence: independent/team/prediction-market) design plus an elite tracking condition. Results for prediction markets are noted as not discussed in the paper.
  • The main outcome was Brier score; analyses also examined calibration and resolution, using statistical tests reported as F-tests, t-tests, and regressions. The paper states that findings were robust across Brier, spherical, and logarithmic scoring rules.

## Limitations

  • The paper notes that results for prediction-market forecasting are not discussed here. This limits what can be concluded from the full design as presented in this article.
  • The paper itself says “No one knows how much further improvement is possible” and that there are likely “pockets of inherent uncertainty” making perfect or near-perfect Brier scores impossible. This is a stated substantive limitation of forecasting accuracy.
  • Evident from the text: the Year 2 no-training condition included some participants who had previously received scenario training, which the authors say “tipped the scale against finding a statistically significant effect of training.”

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/0956797614524255

Cites

43 of 44 references verified0 citation errors · 1 not checkable

Statistics

42 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Fig. 1 Effects of training, teaming, and tracking on average Brier scores in Year 1 (Y1) and Year (Y2). The bars at the left show results for the no-training ('None'), probability-training ('Prob'), and scenario-training ('Scen') conditions; the bars at the right show results for independent forecasters ('Inds'), crowd-belief forecasters ('CBs'), team forecasters ('Teams'), and superforecasters ('SFs'). Error bars represent ±2 SE s.
Fig. 1 Effects of training, teaming, and tracking on average Brier scores in Year 1 (Y1) and Year (Y2). The bars at the left show results for the no-training ('None'), probability-training ('Prob'), and scenario-training ('Scen') conditions; the bars at the right show results for independent forecasters ('Inds'), crowd-belief forecasters ('CBs'), team forecasters ('Teams'), and superforecasters ('SFs'). Error bars represent ±2 SE s.From the PDF
Fig. 2 Calibration curves for the three training conditions in Year 1. Forecasts were sorted into bins on the basis of the predicted likelihood of the event (confidence). The proportion of correct forecasts is plotted for each bin. Perfect calibration is represented by the identity line; points below the identity line represent overconfidence. The total numbers of forecasts were 29,997, 27,477, and 25,844 for the no-training, scenario-training, and probability-training conditions, respectively.
Fig. 2 Calibration curves for the three training conditions in Year 1. Forecasts were sorted into bins on the basis of the predicted likelihood of the event (confidence). The proportion of correct forecasts is plotted for each bin. Perfect calibration is represented by the identity line; points below the identity line represent overconfidence. The total numbers of forecasts were 29,997, 27,477, and 25,844 for the no-training, scenario-training, and probability-training conditions, respectively.From the PDF
Fig. 3 Calibration curves for the two training conditions and the superforecaster condition in Year 2. Forecasts were sorted into bins on the basis of the predicted likelihood of the event (confidence). The proportion of correct forecasts is plotted for each bin. Perfect calibration is represented by the identity line; points below the identity line represent overconfidence. The total numbers of forecasts were 22,888, 23,758, and 4,364 for the no-training, probability-training, and superforecaster conditions, respectively.
Fig. 3 Calibration curves for the two training conditions and the superforecaster condition in Year 2. Forecasts were sorted into bins on the basis of the predicted likelihood of the event (confidence). The proportion of correct forecasts is plotted for each bin. Perfect calibration is represented by the identity line; points below the identity line represent overconfidence. The total numbers of forecasts were 22,888, 23,758, and 4,364 for the no-training, probability-training, and superforecaster conditions, respectively.From the PDF
Fig. 4 Average resolution score as a function of group influence ('Inds' = independent forecasters; 'CBs' = crowd-belief forecasters) in Year 1 (Y1) and as a function of group influence and tracking ('SFs' = superforecasters) in Year 2 (Y2). Error bars represent ±2 SE s.
Fig. 4 Average resolution score as a function of group influence ('Inds' = independent forecasters; 'CBs' = crowd-belief forecasters) in Year 1 (Y1) and as a function of group influence and tracking ('SFs' = superforecasters) in Year 2 (Y2). Error bars represent ±2 SE s.From the PDF

Verification

Claims the paper makes

0 Valid · 0 Not valid · 5 Not checked · 39 Own results and interpretations

19 more claims

Create a free account to see all 44 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 16
  • Probably not a substantial claim. 6
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom, each partial η² from its F, and each ratio, mean difference or B printed with a p checked against its own confidence interval.

5 of 5

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(1056) = 3.25reported p < .001 · recomputed p = .001

    “t (1056) = 3.25, p < .001”

  • Doesn’t add upF(1, 882) = 5.78reported p = .018 · recomputed p = .016

    “F (1, 882) = 5.78, p = .018”

  • Doesn’t add upF(2, 1463) = 6.2reported p < .001 · recomputed p = .002

    “F (2, 1463) = 6.20, p < .001”

  • Doesn’t add upt(1588) = 2.7reported p < .001 · recomputed p = .007

    “t (1588) = 2.70, p < .001”

  • Doesn’t add upF(1, 938) = 3.78reported p < .05 · recomputed p = .052

    “F (1, 938) = 3.78, p < .05”

5 of 42 · the rest aren’t loaded

37 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 44 references

Research works printed but not linked: 2

  1. Fischhoff, B., & Chauvin, C. (Eds.). (2011). Intelligence analysis: Behavioral and social scientific foundations . Washington, DC: National Academies Press.
  2. Janis, I. (1982). Group-think . Boston, MA: Houghton-Mifflin.

Cited by

Items 301–321 of 321

Metadata

Authors
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E.
Affiliations
  • University of Pennsylvania (US)
  • Rice University (US)
  • Department of Psychology, University of Pennsylvania
  • University of California, Berkeley (US)
Venue
Psychological Science
Published
2014-03-21
Volume
25
Issue
5
Pages
1106-1115
ISSN
  • 0956-7976 (ISSN-L)
  • 1467-9280
OpenAlex
W2131472525
Cited by
312
References
42
Citations by year
2026: 20 · 2025: 22 · 2024: 28 · 2023: 29 · 2022: 47 · 2021: 31 · 2020: 14 · 2019: 19 · 2018: 25 · 2017: 28 · 2016: 19 · 2015: 19 · 2014: 11
FWCI
32.31
Type
journal article
Language
en
Open access
Licence
journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations
  • Psychological Science (publishedVersion)
  • PubMed (publishedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
Funders
  • U.S. Department of the Interior
  • IBM Center for the Business of Government
  • Intelligence Advanced Research Projects Activity
  • Interior Business Center
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Sports Analytics and Performance
Keywords
geopolitical forecasting, forecasting tournaments, forecast accuracy, teaming, tracking, calibration, resolution, cognitive biases, reference classes, behavioral interventions, forecast aggregation, heuristics
MeSH terms
Adult, Algorithms, Female, Forecasting, Humans, Interpersonal Relations, Judgment, Male, Probability, Psychological Techniques, Social Behavior, Bias

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.