CoChat

Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments

Open access
Chang, Welton, Chen, Eva, Mellers, Barbara, Tetlock, Philip · 2016 · Judgment and Decision Making

The DOI matches its Crossref record; 106 of the 106 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.

Abstract

The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.

Overview

CoChat Summary

## TL;DR

Across four years of geopolitical forecasting tournaments, a brief CHAMPS KNOW training module consistently improved forecast accuracy, with Brier score improvements of 6% to 11% over controls. Cognitive ability and targeted practice also predicted accuracy, while cognitive motivation measures and interaction effects were generally not supported.

## Key findings

  • Training improved accuracy in every tournament year: year 1 probability training improved mean Brier scores by 10% and scenario training by 11% relative to control; year 2 training improved scores by 12%; year 3 by 6%; and year 4 by 7%.
  • The abstract says the training lasted less than one hour and improved accuracy (Brier scores) by 6% to 11% over the control condition.
  • Targeted practice mediated the training–accuracy relationship in 3 of 4 years; for year 2 the indirect effect was not significant, while for year 1 the bootstrapped indirect effect was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.01), and for year 3 it was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.05).
  • Across all years, the predictors in the authors’ framework explained 10% to 20% of the variance in forecasting accuracy; cognitive ability was significant in all four years, and training was significant in all four years.

## Methods

  • Four geopolitical forecasting tournaments were run between September 2011–April 2012, June 2012–April 2013, August 2013–May 2014, and August 2014–May 2015; subjects submitted probability estimates on a custom internet platform.
  • Subjects were randomly assigned to training or control conditions after completing a survey with individual-differences measures including Raven’s Progressive Matrices, Cognitive Reflection Test, Berlin Numeracy Test, grit, and actively open-minded thinking; each year they answered between 100 and 150 national-security-relevant questions.
  • Accuracy was measured with the traditional Brier score, and analyses used mean standardized Brier scores; the paper reports ANOVAs, t-tests, mediation analyses with bootstrapping (10,000 samples), and regressions to test training, practice, experience, and individual-differences effects.

## Limitations

  • The authors state that the training was designed to win a forecasting tournament and was “solely outcome focused” rather than designed to isolate the incremental effectiveness of each principle.
  • The paper states that it did not directly measure cognitive exertion, so it could not directly test deliberate practice as conceived by Ericsson; this is a stated limitation.
  • Evident from the text: the forecasters chose which questions to answer, and the paper standardizes Brier scores within questions to control for differing question difficulty, which means comparisons rely on the study’s scoring adjustments rather than identical task exposure.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1017/s1930297500004599

Cites

106 of 106 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Years 1-4 training results.
Figure 1 Years 1-4 training results.From the PDF
Figure 2 Year 1 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)
Figure 2 Year 1 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)From the PDF
Figure 3 Year 2 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)
Figure 3 Year 2 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)From the PDF
Figure 4 Year 3 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)
Figure 4 Year 3 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)From the PDF
Figure 5 Year 4 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)
Figure 5 Year 4 training mediation model: Regression coefficients for the relationship between training and accuracy as mediated by forecasts per question. The regression coefficient between training and accuracy, controlling for forecasts per question, is in parentheses. ( p < .001, ** p < 0.01, * p < 0.05.)From the PDF
Figure 6 Years 1-4 practice over time within training conditions. The points in the graphs depict the average condition's Brier score on each day of the tournament. The regression lines were fitted with a span of two days.
Figure 6 Years 1-4 practice over time within training conditions. The points in the graphs depict the average condition's Brier score on each day of the tournament. The regression lines were fitted with a span of two days.From the PDF

Verification

Claims the paper makes

2 Valid · 0 Not valid · 15 Not checked · 64 Own results and interpretations

35 more claims

Create a free account to see all 81 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 19
  • It may state two separate claims as one. 27
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • ConsistentF(2, 381) = 12.1reported p < .001 · recomputed p < .001

    “F (2, 381) = 12.1, p < .001”

  • Consistentt(395) = 5.95reported p < .001 · recomputed p < .001

    “t (395) = 5.95, p < .001”

  • Consistentt(207) = 2.32reported p = .021 · recomputed p = .021

    “t (207) = 2.32, p = .021”

  • Consistentt(225) = 2.2reported p = .028 · recomputed p = .029

    “t (225) = 2.20, p = .028”

4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 106 references

Research works printed but not linked: 2

  1. Gigerenzer, 1999, Simple Heuristics That Make Us Smart.
  2. Surowiecki, 2005, The wisdom of crowds.

Cited by

See all 92

Metadata

Authors
Chang, W., Chen, E., Mellers, B., & Tetlock, P.
Affiliations
University of Pennsylvania (US)
Corresponding author
Chang, Welton
Venue
Judgment and Decision Making
Published
2016-09
Volume
11
Issue
5
Pages
509-526
ISSN
1930-2975 (ISSN-L)
OpenAlex
W4313425124
Cited by
89
References
103
Citations by year
2026: 7 · 2025: 8 · 2024: 17 · 2023: 10 · 2022: 12 · 2021: 16 · 2020: 7 · 2019: 5 · 2018: 5 · 2017: 2
FWCI
4.32
Type
journal article
Language
en
Open access
Locations
  • Judgment and Decision Making (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Intelligence Advanced Research Projects Activity (D11PC20061)
Topics
  • Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
  • Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
Keywords
geopolitical forecasting, judgment accuracy, probability judgment, heuristics and biases, forecasting tournaments, intelligence community, cognitive ability, practice effects, Brier score

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. https://app.cochat.ai/papers/10.1017/s1930297500004599
The paper
Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgment and Decision Making. https://doi.org/10.1017/s1930297500004599

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.