Developing expert political judgment: The impact of training and practice on judgmental accuracy in geopolitical forecasting tournaments
Open accessThe DOI matches its Crossref record; 106 of the 106 references it cites were found in a registry or an index; 2 of the 29 sources its claims cite were checked for support.
Abstract
The heuristics-and-biases research program highlights reasons for expecting people to be poor intuitive forecasters. This article tests the power of a cognitive-debiasing training module (“CHAMPS KNOW”) to improve probability judgments in a four-year series of geopolitical forecasting tournaments sponsored by the U.S. intelligence community. Although the training lasted less than one hour, it consistently improved accuracy (Brier scores) by 6 to 11% over the control condition. Cognitive ability and practice also made largely independent contributions to predictive accuracy. Given the brevity of the training tutorials and the heterogeneity of the problems posed, the observed effects are likely to be lower-bound estimates of what could be achieved by more intensive interventions. Future work should isolate which prongs of the multipronged CHAMPS KNOW training were most effective in improving judgment on which categories of problems.
Overview
CoChat Summary
## TL;DR
Across four years of geopolitical forecasting tournaments, a brief CHAMPS KNOW training module consistently improved forecast accuracy, with Brier score improvements of 6% to 11% over controls. Cognitive ability and targeted practice also predicted accuracy, while cognitive motivation measures and interaction effects were generally not supported.
## Key findings
- Training improved accuracy in every tournament year: year 1 probability training improved mean Brier scores by 10% and scenario training by 11% relative to control; year 2 training improved scores by 12%; year 3 by 6%; and year 4 by 7%.
- The abstract says the training lasted less than one hour and improved accuracy (Brier scores) by 6% to 11% over the control condition.
- Targeted practice mediated the training–accuracy relationship in 3 of 4 years; for year 2 the indirect effect was not significant, while for year 1 the bootstrapped indirect effect was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.01), and for year 3 it was -0.02 with a 95% CI of -0.04 to -0.01 (p = 0.05).
- Across all years, the predictors in the authors’ framework explained 10% to 20% of the variance in forecasting accuracy; cognitive ability was significant in all four years, and training was significant in all four years.
## Methods
- Four geopolitical forecasting tournaments were run between September 2011–April 2012, June 2012–April 2013, August 2013–May 2014, and August 2014–May 2015; subjects submitted probability estimates on a custom internet platform.
- Subjects were randomly assigned to training or control conditions after completing a survey with individual-differences measures including Raven’s Progressive Matrices, Cognitive Reflection Test, Berlin Numeracy Test, grit, and actively open-minded thinking; each year they answered between 100 and 150 national-security-relevant questions.
- Accuracy was measured with the traditional Brier score, and analyses used mean standardized Brier scores; the paper reports ANOVAs, t-tests, mediation analyses with bootstrapping (10,000 samples), and regressions to test training, practice, experience, and individual-differences effects.
## Limitations
- The authors state that the training was designed to win a forecasting tournament and was “solely outcome focused” rather than designed to isolate the incremental effectiveness of each principle.
- The paper states that it did not directly measure cognitive exertion, so it could not directly test deliberate practice as conceived by Ericsson; this is a stated limitation.
- Evident from the text: the forecasters chose which questions to answer, and the paper standardizes Brier scores within questions to control for differing question difficulty, which means comparisons rely on the study’s scoring adjustments rather than identical task exposure.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
2 Valid · 0 Not valid · 15 Not checked · 64 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Cognitive ability and practice also made largely independent contributions to predictive accuracy.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis of which reasoning principles were associated with greater accuracy revealed that self-reported use of comparison classes was correlated with lower Brier scores.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across all years, these predictors collectively explained 10-20% of the variance in forecasting accuracy (Table 16).”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This interaction effect was not significant in later years.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The effects of training were consistently significant, but a subject merely having prior experience in a year of the tournament did not yield significant improvements in Brier score (results from year 4 shown in Table 9).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
35 more claims
Create a free account to see all 81 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 27
- The quote only partly states this claim. 2
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“For example, the GJP found that just one hour of randomly assigned training had a substantial, persistent impact on improving forecasters' Brier scores (Chang et al. 2016).”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Chang et al. 2016
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- ConsistentF(2, 381) = 12.1reported p < .001 · recomputed p < .001
“F (2, 381) = 12.1, p < .001”
- Consistentt(395) = 5.95reported p < .001 · recomputed p < .001
“t (395) = 5.95, p < .001”
- Consistentt(207) = 2.32reported p = .021 · recomputed p = .021
“t (207) = 2.32, p = .021”
- Consistentt(225) = 2.2reported p = .028 · recomputed p = .029
“t (225) = 2.20, p = .028”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Effects of training
“Finally, in year 4, training was also significant, (trained n = 131, M Std Brier Score = -0.01, SD Std = 0.26, control n = 102, M StdBrierScore = -0.08, SD Std = 0.24), t (225) = 2.20, p = .028, a 7% score improvement.”
§ 3.2 Impact of reasoning principles on accuracy
“| Year 4 | No training | 131 | 0.38 | 0.01 | 0.10 | -0.01 | 0.26 | 0.02 | | | | Training | 102 | 0.35 | 0.01 | 0.10 | -0.08 | 0.24 | 0.02 | 7% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.1 Effects of training
“In year 3, training was associated with better performance (trained n = 97, M Std Brier Score = -0.08, SD Std = 0.27, control n = 116, M Std Brier Score = 0.00, SD Std = 0.28), t (207) = 2.32, p = .021, with trained individuals again achieving greater accuracy than controls, a 6% score improvement.”
§ 3.2 Impact of reasoning principles on accuracy
“| Year 3 | No training | 116 | 0.34 | 0.01 | 0.09 | 0.00 | 0.28 | 0.03 | | | | Training | 97 | 0.32 | 0.01 | 0.09 | -0.09 | 0.27 | 0.03 | 6% |”
Citations
Cites
Tversky, A., & Kahneman, D. (1974) · Science · doi:10.1126/science.185.4157.1124 · Cited by 28,810 · cited 1×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
Advances In Experimental Social Psychology
(2022) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,259
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282 · cited 1×
“Recognition is only the first step and activating deliberate overrides requires self-awareness of error potential, a meta-cognitive skill (Croskerry, 2003; Flavell, 1979; Graber, 2003; Nelson & Narens, 1994).”
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 1×
“Sustained improvement likely requires practice that reinforces learning and facilitates deeper thought (Ericsson, 2006, 2008; Ericsson et al., 1993).”
BackgroundInternational Norm Dynamics and Political Change
Finnemore, M., & Sikkink, K. (1998) · International Organization · doi:10.1162/002081898550789 · Cited by 8,292 · cited 1×
“The additional KNOW module encouraged forecasters to under-https (Finnemore & Sikkink, 1998; Keohane, 2005), seek out other political perspectives and be aware of potential wildcard scenarios (Taleb, 2010).”
Grit: perseverance and passion for long-term goals
Duckworth, A. L., Peterson, C., Matthews, M. D., & Kelly, D. R. (2007) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.92.6.1087 · Cited by 6,911
Kahneman, D., & Tversky, A. (1984) · American Psychologist · doi:10.1037/0003-066x.39.4.341 · Cited by 6,544
On the psychology of prediction.
Kahneman, D., & Tversky, A. (1973) · Psychological Review · doi:10.1037/h0034747 · Cited by 6,202 · cited 3×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Development and validation of the short grit scale (grit-s)
Duckworth, A. L., & Quinn, P. D. (2009) · Journal of Personality Assessment · doi:10.1080/00223890802634290 · Cited by 3,376
Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.197 · Cited by 2,517
EXPERT AND EXCEPTIONAL PERFORMANCE: Evidence of Maximal Adaptation to Task Constraints
Ericsson, K. A., & Lehmann, A. C. (1996) · Annual Review of Psychology · doi:10.1146/annurev.psych.47.1.273 · Cited by 2,139 · cited 1×
“…on individual differences and job performance suggests, for instance, a potential interaction effect between cognitive ability and practice (i.e., repeated execution of the thinking principles relayed didac-tically via training) (Deary, 2000; Ericsson & Lehmann, 1996; Schmidt & Hunter, 2000).”
How to improve Bayesian reasoning without instruction: Frequency formats.
Gigerenzer, G., & Hoffrage, U. (1995) · Psychological Review · doi:10.1037//0033-295x.102.4.684 · Cited by 2,127
Sears, D. O. (1986) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.51.3.515 · Cited by 2,119 · cited 2×
“However, past work suffers from at least six sets of limitations: 1) overreliance on student subjects who are often neither intrinsically nor extrinsically motivated to master the task (Anderson, 1982; Petty & Cacioppo, 1984; Sears, 1986); 2) oneshot experimental tasks that limit both subjects’ opportunities to learn…”
Deliberate practice and acquisition of expert performance: a general overview
Anders Ericsson, K. (2008) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2008.00227.x · Cited by 1,975
The science of training: a decade of progress
Salas, E., & Cannon-Bowers, J. A. (2001) · Annual Review of Psychology · doi:10.1146/annurev.psych.52.1.471 · Cited by 1,703 · cited 1×
“…but lose the internal validity advantages that accrue from random assignment (Fong, Krantz & Nisbett, 1986; Lehman, Lempert & Nisbett, 1988; Lehman & Nis-bett, 1990); and 6) limited study of the moderating effects of individual differences beyond cognitive ability (Salas & Cannon-Bowers, 2001).”
Ericsson, K. A. (2006) · The Cambridge Handbook of Expertise and Expert Performance · doi:10.1017/cbo9780511816796.038 · Cited by 1,629 · cited 2×
“Ericsson (2006) stresses the power of deliberate practice: concentrated, effortful and gradual improvement on sequential tasks that feed into over-all skill.”
Judged frequency of lethal events.
Lichtenstein, S. (1978) · Journal of Experimental Psychology: Human Learning & Memory · doi:10.1037//0278-7393.4.6.551 · Cited by 1,520
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980) · Journal of Experimental Psychology: Human Learning and Memory · doi:10.1037/0278-7393.6.2.107 · Cited by 1,477
Man, the State and War--A Theoretical Analysis.
Russell, W. H., & Waltz, K. N. (1959) · Military Affairs · doi:10.2307/1984611 · Cited by 1,314
Expert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260
Self-citationThe base-rate fallacy in probability judgments
Bar-Hillel, M. (1980) · Acta Psychologica · doi:10.1016/0001-6918(80)90046-3 · Cited by 1,183 · cited 1×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
How to Make Cognitive Illusions Disappear: Beyond “Heuristics and Biases”
Gigerenzer, G. (1991) · European Review of Social Psychology · doi:10.1080/14792779143000033 · Cited by 1,159
Intuitive prediction: Biases and corrective procedures
Kahneman, D., & Tversky, A. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.031 · Cited by 1,139
Ross, L., Lepper, M. R., & Hubbard, M. (1975) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.32.5.880 · Cited by 1,052
Metacognition: Knowing about Knowing
Allen, B. (1995) · Information Processing & Management · doi:10.1016/0306-4573(95)80050-4 · Cited by 878
Individual differences in rational thought.
Stanovich, K. E., & West, R. F. (1998) · Journal of Experimental Psychology General · doi:10.1037//0096-3445.127.2.161 · Cited by 867 · cited 2×
“Higher ability has also been found to correlate with less sensitivity to framing and conjunction effects (Stanovich & West, 1998a).”
The use of statistical heuristics in everyday inductive reasoning.
Nisbett, R. E., Krantz, D. H., Jepson, C., & Kunda, Z. (1983) · Psychological Review · doi:10.1037/0033-295x.90.4.339 · Cited by 811 · cited 1×
“Nisbett, Krantz, Jepson & Kunda, 1983).”
Costs and benefits of judgment errors: Implications for debiasing.
Arkes, H. R. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.3.486 · Cited by 764 · cited 5×
“The purely didactic approach is the least costly method, but the experimental evidence suggests that it is also the least effective (Arkes, 1991; Graber, 2003).”
InfluentialJudgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1973) · doi:10.21236/ad0767426 · Cited by 756
Why Investigate Metacognition?
Nelson, T. O., & Narens, L. (1994) · Metacognition · doi:10.7551/mitpress/4561.003.0003 · Cited by 750
Measuring Risk Literacy: The Berlin Numeracy Test
Cokely, E. T., Galesic, M., Schulz, E., Ghazal, S., & Garcia-Retamero, R. (2012) · Judgment and Decision Making · doi:10.1017/s1930297500001819 · Cited by 745
On the study of statistical intuitions
Kahneman, D., & Tversky, A. (1982) · Cognition · doi:10.1016/0010-0277(82)90022-1 · Cited by 714
Psychology as a means of promoting human welfare.
Miller, G. A. (1969) · American Psychologist · doi:10.1037/h0028988 · Cited by 670 · cited 2×
“…that can improve judgments (Arkes, 1991; Croskerry, Singhal & Mamede, 2013a, 2013b; Fischhoff, 1982; Lilien-feld, Ammirati & Landfield, 2009; Miller, 1969), but it re-The mains true that significantly less attention has been paid to “debiasing” than to biases (Arkes, 1991; Graber et al.,…”
European review of social psychology
Edelmann, R. J. (1991) · Behaviour Research and Therapy · doi:10.1016/0005-7967(91)90064-a · Cited by 667
The Black Swan: The Impact of the Highly Improbable
Millkey, A. M. (2009) · Psychiatric Services · doi:10.1176/appi.ps.60.11.1564 · Cited by 657
Source factors and the elaboration likelihood model of persuasion
Petty, R. E., & Cacioppo, J. T. (1984) · ACR North American Advances · Cited by 630
Cognitive debiasing 1: origins of bias and theory of debiasing
Croskerry, P., Singhal, G., & Mamede, S. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001712 · Cited by 604 · cited 2×
“These results have sparked interest in interventions that can improve judgments (Arkes, 1991; Croskerry, Singhal & Mamede, 2013a, 2013b; Fischhoff, 1982; Lilien-feld, Ammirati & Landfield, 2009; Miller, 1969), but it re-The mains true that significantly less attention has been paid to “debiasing”…”
The effects of statistical training on thinking about everyday problems
Fong, G. T., Krantz, D. H., & Nisbett, R. E. (1986) · Cognitive Psychology · doi:10.1016/0010-0285(86)90001-0 · Cited by 597 · cited 2×
“Longer-term educational programs, such as undergraduate-level statistical training have also improved probabilistic reasoning (Fong et al., 1986; Lehman et al., 1988; Lehman & Nisbett, 1990; 1 Returning to the Iraq WMD example, nowhere in the intelligence community’s assessments of the probability…”
Evidential impact of base rates
Tversky, A., & Kahneman, D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.011 · Cited by 591
John, & Raven, J. (2003) · Handbook of Nonverbal Assessment · doi:10.1007/978-1-4615-0153-4_11 · Cited by 528 · cited 1×
“…factors), grit (abbreviated scale in years 1-2, full grit-s scale in year 3) and actively open-minded thinking (comprising motivation-related factors) (Cokely, Galesic, Schulz, Ghazal & Garcia-Retamero, 2012; Duckworth & Quinn, 2009; Frederick, 2005; Haran, Ritov & Mellers, 2013; Raven, 2003).”
Fischhoff, B. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.032 · Cited by 506
Associative processes in intuitive judgment
Morewedge, C. K., & Kahneman, D. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.07.004 · Cited by 492 · cited 1×
“…make errors in prediction tasks by using effort-saving heuristics that are either insensitive to factors that normative theories say we should take into account or sensitive to factors that we should ignore (Kahneman & Tversky, 1977, 1982; Morewedge & Kahneman, 2010; Tversky & Kahne-man, 1974).”
On the psychology of experimental surprises.
Slovic, P., & Fischhoff, B. (1977) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.3.4.544 · Cited by 485 · cited 1×
“Research in judgment and choice has found numerous flaws in people’s intuitive understanding of probability (Bar-Hillel, 1980; Kahneman & Tversky, 1973, 1984; Lichten-stein, Slovic, Fischhoff, Layman & Combs, 1978; Slovic & Fischhoff, 1977; Tversky & Kahneman, 1974).”
The Flexible Correction Model: The Role of Naive Theories of Bias in Bias Correction
Wegener, D. T., & Petty, R. E. (1997) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60017-9 · Cited by 471
Cognitive interventions to reduce diagnostic error: a narrative review
Graber, M. L., Kissam, S., Payne, V. L., Meyer, A. N. D., Sorensen, A., Lenfestey, N., Tant, E., Henriksen, K., LaBresh, K., & Singh, H. (2012) · BMJ Quality & Safety · doi:10.1136/bmjqs-2011-000149 · Cited by 449 · cited 3×
“Moreover, few organizations have embraced the debiasing methods that have been developed (Croskerry, 2003; Graber et al., 2012; Lilienfeld et al., 2009).”
Cognitive forcing strategies in clinical decisionmaking
Croskerry, P. (2003) · Annals of Emergency Medicine · doi:10.1067/mem.2003.22 · Cited by 440 · cited 2×
“Recognition is only the first step and activating deliberate overrides requires self-awareness of error potential, a meta-cognitive skill (Croskerry, 2003; Flavell, 1979; Graber, 2003; Nelson & Narens, 1994).”
Flyvbjerg, B. (2008) · European Planning Studies · doi:10.1080/09654310701747936 · Cited by 438 · cited 1×
“Finally, teaching people reference-class forecasting reduces base-rate neglect more than calling attention to the bias itself (Case, Fantino & Goodie, 1999; Fischhoff & Bar-Hillel, 1984; Flyvbjerg, 2008; Kahneman & Tversky, 1977; Lovallo, Clarke & Camerer, 2012).”
BackgroundCognitive debiasing 2: impediments to and strategies for change
Croskerry, P., Singhal, G., & Mamede, S. (2013) · BMJ Quality & Safety · doi:10.1136/bmjqs-2012-001713 · Cited by 419 · cited 1×
“Accurate probability judgments are important in domains such as law, finance, medicine and politics (Croskerry et al., 2013b; Jolls & Sunstein, 2005).”
Base-rate respect: From ecological rationality to dual processes
Barbey, A. K., & Sloman, S. A. (2007) · Behavioral and Brain Sciences · doi:10.1017/s0140525x07001653 · Cited by 409 · cited 1×
“Problem format also impacts the extent to which subjects consider the prior probabilities of events (i.e., base-rate neglect) (Barbey & Sloman, 2007; Macchi, 2000) and the degree to which they correctly process information (Radecki & Medow, 2007; Roy & Lerch, 1996).”
BackgroundEliminating the hindsight bias.
Arkes, H. R., Faust, D., Guilmette, T. J., & Hart, K. (1988) · Journal of Applied Psychology · doi:10.1037//0021-9010.73.2.305 · Cited by 396
Lichtenstein, S., & Fischhoff, B. (1980) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(80)90052-5 · Cited by 375 · cited 3×
“However, past work suffers from at least six sets of limitations: 1) over-reliance on student subjects who are often neither intrinsically nor extrinsically motivated to master the task (Anderson, 1982; Petty & Cacioppo, 1984; Sears, 1986); 2) one-shot experimental tasks that limit both subjects’ opportunities to lear…”
After Hegemony: Cooperation and Discord in the World Political Economy
KEOHANE, R. O. (2005) · Princeton University Press eBooks · doi:10.2307/j.ctt7sq9s · Cited by 368
Teaching Bayesian reasoning in less than two hours.
Sedlmeier, P., & Gigerenzer, G. (2001) · Journal of Experimental Psychology: General · doi:10.1037//0096-3445.130.3.380 · Cited by 366 · cited 1×
“Even after brief amounts of feedback, subjects may demonstrate improved Bayesian reasoning (Lichtenstein & Fischhoff, 1980; Sedlmeier & Gigerenzer, 2001).”
BackgroundDebiasing the Mind Through Meditation: Mindfulness and the Sunk-Cost Bias
Hafenbrack, A. C., Kinias, Z., & Barsade, S. G. (2014) · Psychological Science · doi:10.1177/0956797613503853 · Cited by 354
Lilienfeld, S. O., Ammirati, R., & Landfield, K. (2009) · Perspectives on Psychological Science · doi:10.1111/j.1745-6924.2009.01144.x · Cited by 353 · cited 5×
“A review of debiasing methods for confirmation bias has found mixed effects at best (Lilienfeld et al., 2009; Weinstein & Klein, 1995).”
InfluentialEricsson, K. A. (2007) · International journal of sport psychology · Cited by 348
Lehman, D. R., Lempert, R. O., & Nisbett, R. E. (1988) · American Psychologist · doi:10.1037/0003-066x.43.6.431 · Cited by 341 · cited 2×
“Longer-term educational programs, such as undergraduate-level statistical training have also improved probabilistic reasoning (Fong et al., 1986; Lehman et al., 1988; Lehman & Nisbett, 1990; 1 Returning to the Iraq WMD example, nowhere in the intelligence community’s assessments of the probability…”
Impediments to accurate clinical judgment and possible ways to minimize their impact.
Arkes, H. R. (1981) · Journal of Consulting and Clinical Psychology · doi:10.1037//0022-006x.49.3.323 · Cited by 340 · cited 2×
“…Kahneman, 1981), averaging and using crowd wisdom principles (Surowiecki, 2005), finding and utilizing predictive mathematical and statistical models (Arkes, 1981; Kahneman & Tversky, 1982), cautiously us-ing time-series and historical data, and being self-aware of the typical cognitive biases…”
Measuring the Crowd Within: Probabilistic Representations Within Individuals
Vul, E., & Pashler, H. (2008) · Psychological Science · doi:10.1111/j.1467-9280.2008.02136.x · Cited by 333 · cited 1×
“Another override strategy is to ask people the same question twice, which pushes them to evaluate the accuracy of their first-order judgments (Herzog & Hertwig, 2009; Van Der Henst, 2009; Vul & Pashler, 2008).”
The role of actively open-minded thinking in information acquisition, accuracy, and calibration
Haran, U., Ritov, I., & Mellers, B. A. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500005921 · Cited by 328 · cited 1×
“…factors), grit (abbreviated scale in years 1-2, full grit-s scale in year 3) and actively open-minded thinking (comprising motivation-related factors) (Cokely, Galesic, Schulz, Ghazal & Garcia-Retamero, 2012; Duckworth & Quinn, 2009; Frederick, 2005; Haran, Ritov & Mellers, 2013; Raven, 2003).”
Self-citationThe Blackwell handbook of principles of organizational behavior
Locke, E. A. (2000) · Cited by 318
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312 · cited 5×
“Additionally, as reported elsewhere, training improved the calibration and resolution of forecasters by reducing overconfidence (Mellers et al., 2014; Moore et al., 2016).”
InfluentialSelf-citationThe Wisdom of Many in One Mind: Improving Individual Judgments With Dialectical Bootstrapping
Herzog, S. M., & Hertwig, R. (2009) · Psychological Science · doi:10.1111/j.1467-9280.2009.02271.x · Cited by 304 · cited 2×
“Another override strategy is to ask people the same question twice, which pushes them to evaluate the accuracy of their first-order judgments (Herzog & Hertwig, 2009; Van Der Henst, 2009; Vul & Pashler, 2008).”
Looking Down on Human Intelligence
Deary, I. (2000) · doi:10.1093/acprof:oso/9780198524175.001.0001 · Cited by 296 · cited 1×
“…on individual differences and job performance suggests, for instance, a potential interaction effect between cognitive ability and practice (i.e., repeated execution of the thinking principles relayed didac-tically via training) (Deary, 2000; Ericsson & Lehmann, 1996; Schmidt & Hunter, 2000).”
The Logic of political survival
(2004) · Choice Reviews Online · doi:10.5860/choice.42-0583 · Cited by 293
Teaching the Use of Cost-Benefit Reasoning in Everyday Life
Larrick, R. P., Morgan, J. N., & Nisbett, R. E. (1990) · Psychological Science · doi:10.1111/j.1467-9280.1990.tb00243.x · Cited by 274
Attaining Excellence Through Deliberate Practice: Insights from the Study of Expert Performance
Ericsson, K. A. (2002) · Teaching and Learning · doi:10.1002/9780470690048.ch1 · Cited by 260 · cited 2×
“…for a skill also need to engage in deliberate practice, he does not deny that those with natu-ral talent can derive greater benefits from a virtuous cycle of motivation-related gains in which initial cognitive advantages are reinforced by lower barriers to deliberate practice (Ericsson, 2002, 2007).”
Jolls, C., & Sunstein, C. R. (2006) · The Journal of Legal Studies · doi:10.1086/500096 · Cited by 250 · cited 1×
“Accurate probability judgments are important in domains such as law, finance, medicine and politics (Croskerry et al., 2013b; Jolls & Sunstein, 2005).”
Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., et al. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615577794 · Cited by 236 · cited 1×
“Other future research includes training the best forecasters, so-called superforecasters (Mellers et al., 2015) with more advanced cognitive debiasing modules to see if training can push the upper limit of forecasting performance even further.”
BackgroundSelf-citationIndividual Differences in Framing and Conjunction Effects
E. Stanovich, K., & West, R. F. (1998) · Thinking & Reasoning · doi:10.1080/135467898394094 · Cited by 229
Does the teaching of probability improve probabilistic intuitions?: An exploratory research study
Fischbein, E., & Gazit, A. (1984) · Educational Studies in Mathematics · doi:10.1007/bf00380436 · Cited by 215 · cited 1×
“A number of studies have shed light on how probability estimates and judgments can be improved (Fischbein & Gazit, 1984; Fischhoff & Bar-Hillel, 1984; Stewart, 2001; Tetlock, 2005; Whitecotton, Sanders & Norris, 1998).”
BackgroundA longitudinal study of the effects of undergraduate training on reasoning.
Lehman, D. R., & Nisbett, R. E. (1990) · Developmental Psychology · doi:10.1037/0012-1649.26.6.952 · Cited by 211 · cited 1×
“…programs, such as undergraduate-level statistical training have also improved probabilistic reasoning (Fong et al., 1986; Lehman et al., 1988; Lehman & Nisbett, 1990; 1 Returning to the Iraq WMD example, nowhere in the intelligence community’s assessments of the probability of Iraq’s…”
Inoculation and Counterexplanation: Debiasing Techniques in the Perseverance of Social Theories
Anderson, C. A. (1982) · Social Cognition · doi:10.1521/soco.1982.1.2.126 · Cited by 175 · cited 2×
“However, past work suffers from at least six sets of limitations: 1) overreliance on student subjects who are often neither intrinsically nor extrinsically motivated to master the task (Anderson, 1982; Petty & Cacioppo, 1984; Sears, 1986); 2) oneshot experimental tasks that limit both subjects’ opportunities to learn…”
Resistance of personal risk perceptions to debiasing interventions.
Weinstein, N. D., & Klein, W. M. (1995) · Health Psychology · doi:10.1037//0278-6133.14.2.132 · Cited by 163 · cited 1×
“A review of debiasing methods for confirmation bias has found mixed effects at best (Lilienfeld et al., 2009; Weinstein & Klein, 1995).”
Focusing techniques: A shortcut to improving probability judgments?
Fischhoff, B., & Bar-Hillel, M. (1984) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(84)90002-3 · Cited by 114 · cited 2×
“A number of studies have shed light on how probability estimates and judgments can be improved (Fischbein & Gazit, 1984; Fischhoff & Bar-Hillel, 1984; Stewart, 2001; Tetlock, 2005; Whitecotton, Sanders & Norris, 1998).”
Robust analogizing and the outside view: two empirical tests of case‐based decision making
Lovallo, D., Clarke, C., & Camerer, C. (2012) · Strategic Management Journal · doi:10.1002/smj.962 · Cited by 110 · cited 1×
“Finally, teaching people reference-class forecasting reduces base-rate neglect more than calling attention to the bias itself (Case, Fantino & Goodie, 1999; Fischhoff & Bar-Hillel, 1984; Fly-vbjerg, 2008; Kahneman & Tversky, 1977; Lovallo, Clarke & Camerer, 2012).”
On the study of statistical intuitions
Kahneman, D. J., & Tversky, A. (1982) · Cambridge University Press eBooks · doi:10.1017/cbo9780511809477.035 · Cited by 109
Debiasing the framing effect: The effect of warning and involvement
Cheng, F.-F., & Wu, C.-S. (2010) · Decision Support Systems · doi:10.1016/j.dss.2010.04.002 · Cited by 105 · cited 1×
“For example, describing common reasoning patterns that lead to biased judgments and explaining how to avoid them is more effective than just informing people about biases (Cheng & Wu, 2010; Ross, Lepper & Hubbard, 1975).”
BackgroundFacilitating Normative Judgments of Conditional Probability: Frequency or Nested Sets?
Yamagishi, K. (2003) · Experimental Psychology (formerly Zeitschrift für Experimentelle Psychologie) · doi:10.1026//1618-3169.50.2.97 · Cited by 105 · cited 1×
“Researchers have also shown that displaying probabilities as “nested sets” or visualizing data distributions can improve probability judgments (Mellers & McGraw, 1999; Yamagishi, 2003).”
BackgroundImproving Reliability of Judgmental Forecasts
Stewart, T. R. (2001) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-0-306-47630-3_5 · Cited by 104 · cited 1×
“A number of studies have shed light on how probability estimates and judgments can be improved (Fischbein & Gazit, 1984; Fischhoff & Bar-Hillel, 1984; Stewart, 2001; Tetlock, 2005; Whitecotton, Sanders & Norris, 1998).”
BackgroundStone, E. R., & Opel, R. B. (2000) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2910 · Cited by 101 · cited 3×
“Several studies have found that providing feedback in probabilistic reasoning tasks improves resolution and calibration (Benson & Önkal, 1992; Lichten-stein & Fischhoff, 1980; Stone & Opel, 2000).”
Forecasting Tournaments: Tools for Increasing Transparency and Improving the Quality of Debate
Tetlock, P. E., Mellers, B. A., Rohrbaugh, N., & Chen, E. (2014) · Current Directions in Psychological Science · doi:10.1177/0963721414534257 · Cited by 98
Self-citationThe effects of feedback and training on the performance of probability forecasters
Benson, P. G., & Önkal, D. (1992) · International Journal of Forecasting · doi:10.1016/0169-2070(92)90066-i · Cited by 92 · cited 2×
“Several studies have found that providing feedback in probabilistic reasoning tasks improves resolution and calibration (Benson & Önkal, 1992; Lichten-stein & Fischhoff, 1980; Stone & Opel, 2000).”
Rational animals, irrational humans
渡辺, 茂. (2009) · 慶應義塾大学出版会 eBooks · Cited by 92
Bilmes, L. J. (2013) · The Economics of Peace and Security Journal · doi:10.15355/epsj.9.1.5 · Cited by 87 · cited 1×
“The United States would continue its involvement in the country for over a decade at an estimated cost between $4 and $6 trillion and thousands of casualties, numbers which underscore the dangers of over-confident “slam-dunk” assessments of ambiguous evidence (Bilmes, 2014).”
Evidential Impact of Base Rates
Tversky, A., & Kahneman, D. (1981) · doi:10.21236/ada099501 · Cited by 80
Jolls, C., & Sunstein, C. R. (2005) · National Bureau of Economic Research · doi:10.3386/w11738 · Cited by 78
Cognitive Aids and Debiasing Methods: CAN Cognitive Pills Cure Cognitive Ills?
Keren, G. B. (1990) · Advances in psychology · doi:10.1016/s0166-4115(08)61341-2 · Cited by 76 · cited 1×
“Process-based techniques are designed to override the erroneous tendencies of some simple heuristics (Keren, 1990).”
MethodMacchi, L. (2000) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.2000.2895 · Cited by 75 · cited 1×
“Problem format also impacts the extent to which subjects consider the prior probabilities of events (i.e., base-rate neglect) (Barbey & Sloman, 2007; Macchi, 2000) and the degree to which they correctly process information (Radecki & Medow, 2007; Roy & Lerch, 1996).”
BackgroundConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
Self-citationOvercoming Ineffective Mental Representations in Base-Rate Problems
Roy, M. C., & Lerch, F. J. (1996) · Information Systems Research · doi:10.1287/isre.7.2.233 · Cited by 72 · cited 1×
“Problem format also impacts the extent to which subjects consider the prior probabilities of events (i.e., base-rate neglect) (Barbey & Sloman, 2007; Macchi, 2000) and the degree to which they correctly process information (Radecki & Medow, 2007; Roy & Lerch, 1996).”
BackgroundHow to improve Bayesian reasoning: Comment on Gigerenzer and Hoffrage (1995).
Mellers, B. A., & McGraw, A. P. (1999) · Psychological Review · doi:10.1037/0033-295x.106.2.417 · Cited by 66 · cited 1×
“Researchers have also shown that displaying probabilities as “nested sets” or visualizing data distributions can improve probability judgments (Mellers & McGraw, 1999; Yamagishi, 2003).”
BackgroundSelf-citationImproving Predictive Accuracy with a Combination of Human Intuition and Mechanical Decision Aids
Whitecotton, S. M., Sanders, D. E., & Norris, K. B. (1998) · Organizational Behavior and Human Decision Processes · doi:10.1006/obhd.1998.2809 · Cited by 63 · cited 1×
“A number of studies have shed light on how probability estimates and judgments can be improved (Fischbein & Gazit, 1984; Fischhoff & Bar-Hillel, 1984; Stewart, 2001; Tetlock, 2005; Whitecotton, Sanders & Norris, 1998).”
How to improve Bayesian reasoning: Comment on Gigerenzer and Hoffrage (1995).
Mellers, B. A., & McGraw, A. P. (1999) · Psychological Review · doi:10.1037//0033-295x.106.2.417 · Cited by 59
Self-citationKardes, F. R., Posavac, S. S., Silvera, D. H., Cronley, M. L., Sanbonmatsu, D. M., Schertzer, S. M. B., Miller, F. M., Herr, P. M., & Chandrashekaran, M. (2006) · Journal of Business Research · doi:10.1016/j.jbusres.2006.01.016 · Cited by 51 · cited 2×
“…and not designed to help with problems that activate multiple biases (Arkes, Faust, Guil-mette & Hart, 1988; Hafenbrack, Kinias & Barsade, 2014; Kardes et al., 2006); 5) multifaceted and lengthy educational interventions, such as statistics courses, that are high in ecological validity…”
Metacognitive Training to Reduce Diagnostic Errors: Ready for Prime Time?
Graber, M. (2003) · Academic Medicine · doi:10.1097/00001888-200308000-00004 · Cited by 46 · cited 2×
“Recognition is only the first step and activating deliberate overrides requires self-awareness of error potential, a meta-cognitive skill (Croskerry, 2003; Flavell, 1979; Graber, 2003; Nelson & Narens, 1994).”
Base-rate training without case cues reduces base-rate neglect
Case, D. A., Fantino, E., & Goodie, A. S. (1999) · Psychonomic Bulletin & Review · doi:10.3758/bf03212337 · Cited by 37 · cited 1×
“Finally, teaching people reference-class forecasting reduces base-rate neglect more than calling attention to the bias itself (Case, Fantino & Goodie, 1999; Fischhoff & Bar-Hillel, 1984; Fly-vbjerg, 2008; Kahneman & Tversky, 1977; Lovallo, Clarke & Camerer, 2012).”
An Experimental Study of the Effectiveness of Three Debiasing Techniques*
Büyükkurt, B. K., & Büyükkurt, M. D. (1991) · Decision Sciences · doi:10.1111/j.1540-5915.1991.tb01262.x · Cited by 19 · cited 2×
“…researchers’ opportunities to assess whether experimentally induced gains were sustainable over time or whether they just briefly cued better thinking (Büyükkurt & Büyükkurt, 1991; Koriat, Lichtenstein & Fischhoff, 1980); 3) brief training modules, often as short as 10–15 minutes, that afforded…”
Cognitive debiasing through sparklines in clinical data displays.
Radecki, R. P., & Medow, M. A. (2007) · PubMed · Cited by 13
Editorial: Improving Bayesian Reasoning: What Works and Why?
Mandel, D. R., & Navarrete, G. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.01872 · Cited by 12 · cited 1×
“Mandel and Navarrete (2015) underscore this point in their recent review.”
BackgroundEffects of training and instruction on analytic and belief-based reasoning processes
Neilens, H. L., Handley, S. J., & Newstead, S. E. (2009) · Thinking & Reasoning · doi:10.1080/13546780802535865 · Cited by 11 · cited 1×
“Informing people of biases is simply not enough (Neilens, Handley & Newstead, 2009; Wegener & Petty, 1997).”
Cirincione, J. (2014) · Routledge eBooks · doi:10.4324/9780203950401 · Cited by 10
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
Macchi, L. (1999) · PsycEXTRA Dataset · doi:10.1037/e683312011-166
Verification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2 · cited 1×
“The traditional Brier scoring rule was used to measure accuracy (Brier, 1950).”
Method
Research works printed but not linked: 2
- Gigerenzer, G., Todd, P.M., & the ABC Research Group. (1999). Simple Heuristics That Make Us Smart. New York, NY: Oxford University Press.
- Surowiecki, J. (2005). The wisdom of crowds . New York, NY: Anchor.
Cited by
Five things you should know about cost overrun
Flyvbjerg, B., Ansar, A., Budzier, A., Buhl, S., Cantarelli, C., Garbuio, M., Glenting, C., Holm, M. S., Lovallo, D., Lunn, D., et al. (2018) · Transportation Research Part A: Policy and Practice · doi:10.1016/j.tra.2018.07.013 · Cited by 238
Expert biases in technology foresight. Why they are a problem and how to mitigate them
Bonaccorsi, A., Apreda, R., & Fantoni, G. (2020) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2019.119855 · Cited by 160
Regression to the tail: Why the Olympics blow up
Flyvbjerg, B., Budzier, A., & Lunn, D. (2020) · Environment and Planning A Economy and Space · doi:10.1177/0308518x20958724 · Cited by 163
“2008, Batselier and Vanhoucke 2016 and 2017, Chang et al. 2016, Horne 2007). Further de-biasing should be carried out, following Kahneman et al. (2011) and Flyvbjerg (2013). Based on the dataset presented above,”
The Cost-Benefit Fallacy: Why Cost-Benefit Analysis Is Broken and How to Fix It
Flyvbjerg, B., & Bester, D. W. (2021) · Journal of Benefit-Cost Analysis · doi:10.1017/bca.2021.9 · Cited by 98
Restructuring structured analytic techniques in intelligence
Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1400230 · Cited by 97
Self-citation
Metadata
- Authors
- Chang, W., Chen, E., Mellers, B., & Tetlock, P.
- Affiliations
- University of Pennsylvania (US)
- Corresponding author
- Chang, Welton
- Venue
- Judgment and Decision Making
- Publisher
- Cambridge University Press (CUP)
- Published
- 2016-09
- Volume
- 11
- Issue
- 5
- Pages
- 509-526
- ISSN
- 1930-2975 (ISSN-L)
- OpenAlex
- W4313425124
- Cited by
- 89
- References
- 103
- Citations by year
- 2026: 7 · 2025: 8 · 2024: 17 · 2023: 10 · 2022: 12 · 2021: 16 · 2020: 7 · 2019: 5 · 2018: 5 · 2017: 2
- FWCI
- 4.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Judgment and Decision Making (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Intelligence Advanced Research Projects Activity (D11PC20061)
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- geopolitical forecasting, judgment accuracy, probability judgment, heuristics and biases, forecasting tournaments, intelligence community, cognitive ability, practice effects, Brier score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.