Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments
Open accessThe DOI matches its Crossref record; 106 of the 106 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.
Overview
CoChat Summary
## TL;DR
Across four years of geopolitical forecasting tournaments, a brief CHAMPS KNOW training module consistently improved forecast accuracy, with Brier score improvements of 6% to 11% over controls. Cognitive ability and targeted practice also predicted accuracy, while cognitive motivation measures and interaction effects were generally not supported.
## Key findings
- Training improved accuracy in every tournament year: year 1 probability training improved mean Brier scores by 10% and scenario training by 11% relative to control; year 2 training improved scores by 12%; year 3 by 6%; and year 4 by 7%.
- The abstract says the training lasted less than one hour and improved accuracy (Brier scores) by 6% to 11% over the control condition.
- Targeted practice mediated the training–accuracy relationship in 3 of 4 years; for year 2 the indirect effect was not significant, while for year 1 the bootstrapped indirect effect was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.01), and for year 3 it was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.05).
- Across all years, the predictors in the authors’ framework explained 10% to 20% of the variance in forecasting accuracy; cognitive ability was significant in all four years, and training was significant in all four years.
## Methods
- Four geopolitical forecasting tournaments were run between September 2011–April 2012, June 2012–April 2013, August 2013–May 2014, and August 2014–May 2015; subjects submitted probability estimates on a custom internet platform.
- Subjects were randomly assigned to training or control conditions after completing a survey with individual-differences measures including Raven’s Progressive Matrices, Cognitive Reflection Test, Berlin Numeracy Test, grit, and actively open-minded thinking; each year they answered between 100 and 150 national-security-relevant questions.
- Accuracy was measured with the traditional Brier score, and analyses used mean standardized Brier scores; the paper reports ANOVAs, t-tests, mediation analyses with bootstrapping (10,000 samples), and regressions to test training, practice, experience, and individual-differences effects.
## Limitations
- The authors state that the training was designed to win a forecasting tournament and was “solely outcome focused” rather than designed to isolate the incremental effectiveness of each principle.
- The paper states that it did not directly measure cognitive exertion, so it could not directly test deliberate practice as conceived by Ericsson; this is a stated limitation.
- Evident from the text: the forecasters chose which questions to answer, and the paper standardizes Brier scores within questions to control for differing question difficulty, which means comparisons rely on the study’s scoring adjustments rather than identical task exposure.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 0 Not valid · 15 Not checked · 64 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cognitive ability and practice also made largely independent contributions to predictive accuracy.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis of which reasoning principles were associated with greater accuracy revealed that self-reported use of comparison classes was correlated with lower Brier scores.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all years, these predictors collectively explained 10-20% of the variance in forecasting accuracy (Table 16).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This interaction effect was not significant in later years.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The effects of training were consistently significant, but a subject merely having prior experience in a year of the tournament did not yield significant improvements in Brier score (results from year 4 shown in Table 9).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
35 more claims
Create a free account to see all 81 claims and every source verdict.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 27
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For example, the GJP found that just one hour of randomly assigned training had a substantial, persistent impact on improving forecasters' Brier scores (Chang et al. 2016).”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Chang et al. 2016
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(2, 381) = 12.1reported p < .001 · recomputed p < .001
“F (2, 381) = 12.1, p < .001”
- Consistentt(395) = 5.95reported p < .001 · recomputed p < .001
“t (395) = 5.95, p < .001”
- Consistentt(207) = 2.32reported p = .021 · recomputed p = .021
“t (207) = 2.32, p = .021”
- Consistentt(225) = 2.2reported p = .028 · recomputed p = .029
“t (225) = 2.20, p = .028”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Effects of training
“Finally, in year 4, training was also significant, (trained n = 131, M Std Brier Score = -0.01, SD Std = 0.26, control n = 102, M StdBrierScore = -0.08, SD Std = 0.24), t (225) = 2.20, p = .028, a 7% score improvement.”
§ 3.2 Impact of reasoning principles on accuracy
“| Year 4 | No training | 131 | 0.38 | 0.01 | 0.10 | -0.01 | 0.26 | 0.02 | | | | Training | 102 | 0.35 | 0.01 | 0.10 | -0.08 | 0.24 | 0.02 | 7% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Effects of training
“In year 3, training was associated with better performance (trained n = 97, M Std Brier Score = -0.08, SD Std = 0.27, control n = 116, M Std Brier Score = 0.00, SD Std = 0.28), t (207) = 2.32, p = .021, with trained individuals again achieving greater accuracy than controls, a 6% score improvement.”
§ 3.2 Impact of reasoning principles on accuracy
“| Year 3 | No training | 116 | 0.34 | 0.01 | 0.09 | 0.00 | 0.28 | 0.03 | | | | Training | 97 | 0.32 | 0.01 | 0.09 | -0.09 | 0.27 | 0.03 | 6% |”
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · cited 1×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
Advances In Experimental Social Psychology
(2022) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,259
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1×
“Recognition is only the first step and activating deliberate overrides requires self-awareness of error potential, a meta-cognitive skill (Croskerry, 2003; Flavell, 1979; Graber, 2003; Nelson & Narens, 1994).”
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 1×
“Sustained improvement likely requires practice that reinforces learning and facilitates deeper thought (Ericsson, 2006, 2008; Ericsson et al., 1993).”
BackgroundInternational Norm Dynamics and Political Change
Finnemore, M., & Sikkink, K. (1998) · International Organization · doi:10.1162/002081898550789 · Cited by 8,292 · cited 1×
“The additional KNOW module encouraged forecasters to under-https (Finnemore & Sikkink, 1998; Keohane, 2005), seek out other political perspectives and be aware of potential wildcard scenarios (Taleb, 2010).”
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911
Kahneman, D., & Tversky, A. (1984) · American Psychologist · doi:10.1037/0003-066x.39.4.341 · Cited by 6,544
On the psychology of prediction.
Kahneman, D., & Tversky, A. (1973) · Psychological Review · doi:10.1037/h0034747 · Cited by 6,202 · cited 3×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Development and validation of the short grit scale (grit-s)
Duckworth, A. L., & Quinn, P. D. (2009) · Journal of Personality Assessment · doi:10.1080/00223890802634290 · Cited by 3,376
Research works printed but not linked: 2
- Gigerenzer, 1999, Simple Heuristics That Make Us Smart.
- Surowiecki, 2005, The wisdom of crowds.
Cited by
Five things you should know about cost overrun
Flyvbjerg, B., Ansar, A., Budzier, A., Buhl, S., Cantarelli, C., Garbuio, M., Glenting, C., Holm, M. S., Lovallo, D., Lunn, D., et al. (2018) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2018.07.013 · Cited by 238
Expert biases in technology foresight. Why they are a problem and how to mitigate them
Bonaccorsi, A., Apreda, R., & Fantoni, G. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2019.119855 · Cited by 160
Regression to the tail: Why the Olympics blow up
Flyvbjerg, B., Budzier, A., & Lunn, D. (2020) · Environment and Planning A Economy and Space · doi:10.1177/0308518x20958724 · Cited by 163
“2008, Batselier and Vanhoucke 2016 and 2017, Chang et al. 2016, Horne 2007). Further de-biasing should be carried out, following Kahneman et al. (2011) and Flyvbjerg (2013). Based on the dataset presented above,”
The Cost-Benefit Fallacy: Why Cost-Benefit Analysis Is Broken and How to Fix It
Flyvbjerg, B., & Bester, D. W. (2021) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2021.9 · Cited by 98
Restructuring structured analytic techniques in intelligence
Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1400230 · Cited by 97
Self-citation
Metadata
- Authors
- Chang, W., Chen, E., Mellers, B., & Tetlock, P.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Chang, Welton
- Venue
- Judgment and Decision Making
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-09
- Volume
- 11
- Issue
- 5
- Pages
- 509-526
- ISSN
- 1930-2975 (ISSN-L)
- OpenAlex
- W4313425124
- Cited by
- 89
- References
- 103
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 17 · 2023: 10 · 2022: 12 · 2021: 16 · 2020: 7 · 2019: 5 · 2018: 5 · 2017: 2
- FWCI
- 4.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Judgment and Decision Making (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Intelligence Advanced Research Projects Activity (D11PC20061)
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- geopolitical forecasting, judgment accuracy, probability judgment, heuristics and biases, forecasting tournaments, intelligence community, cognitive ability, practice effects, Brier score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. https://app.cochat.ai/papers/10.1017/s1930297500004599
- The paper
- Chang, W., Chen, E., Mellers, B., & Tetlock, P. (2016). Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments. Judgment and Decision Making. https://doi.org/10.1017/s1930297500004599
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.