CoChat
Sign Up
Cited by 61Forecasting Techniques and ApplicationsIntelligence, Security, War StrategyInternational Relations and Foreign Policy

The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament

Friedman, Jeffrey A, Baker, Joshua D, Mellers, Barbara A, Tetlock, Philip E, Zeckhauser, Richard·2018·International Studies Quarterly
DOI 10.1093/isq/sqx078
Open this paper in CoChat — free account

Abstract

Scholars, practitioners, and pundits often leave their assessments of uncertainty vague when debating foreign policy, arguing that clearer probability estimates would provide arbitrary detail instead of useful insight. We provide the first systematic test of this claim using a data set containing 888,328 geopolitical forecasts. We find that coarsening numeric probability assessments in a manner consistent with common qualitative expressions—including expressions currently recommended for use by intelligence analysts—consistently sacrifices predictive accuracy. This finding does not depend on extreme probability estimates, short time horizons, particular scoring rules, or individual attributes that are difficult to cultivate. At a practical level, our analysis indicates that it would be possible to make foreign policy discourse more informative by supplementing natural language-based descriptions of uncertainty with quantitative probability estimates. More broadly, our findings advance long-standing debates over the nature and limits of subjective judgment when assessing social phenomena, showing how explicit probability assessments are empirically justifiable even in domains as complex as world politics.

AI snapshot

Written from the paper’s full text.

TL;DR

Using nearly 900,000 forecasts from the Good Judgment Project, the paper tests whether making probability judgments more precise improves accuracy. It finds that rounding numeric forecasts into common qualitative categories consistently worsens predictive accuracy, while education, numeracy, and cognitive style do little to explain who can benefit from precision.

Key findings

  • Across 888,328 forecasts, coarsening numeric probabilities into two bins, as with estimative verbs, made Brier scores 31% worse on average; the median rounding error was 22%. The weakest group, untrained solo forecasters, had an average rounding error of 15%, while superforecasters exceeded 500% on average.
  • Rounding into three confidence levels degraded accuracy by more than 10% on average and substantially more for high performers. Seven-step words of estimative probability recovered some but not all of the loss; every subgroup showed consistent losses under the Director of National Intelligence lexicon. The mean threshold of estimative precision was 6.1 bins (SD 4.4), and thresholds exceeded seven bins for 42% of questions.
  • The losses were not driven by extreme probabilities, short time horizons, the scoring rule, or question content. Superforecasters showed reliable returns to precision across all seven probability categories under both Brier and logarithmic scoring.
  • Individual-level returns to precision were predicted by cultivable factors such as number of questions, probabilistic training, and group collaboration, but not by selection factors such as education, numeracy, or cognitive style. Superforecasters could reliably parse their judgments into 11.3 bins on average (SD 5.6).

Methods

  • The study uses Good Judgment Project data from IARPA-sponsored geopolitical forecasting tournaments, 2011–2015. The sample includes 1,832 individuals who registered at least 25 predictions in a tournament year, producing 888,328 forecasts on 380 binary questions. Participants were mostly male (83%) and US citizens (74%), with an average age of 40; 64% held a bachelor’s degree and 57% had postgraduate training.
  • The main analysis measures predictive accuracy with Brier scores, then rounds numeric forecasts to the midpoints of bins corresponding to qualitative expressions: estimative verbs (2 bins), confidence levels (3 bins), words of estimative probability (7 bins), and integer percentages (101 bins). It calculates proportional rounding errors and aggregates at the question level, using paired-sample t-tests and Wilcoxon signed-rank tests.
  • Robustness checks use logarithmic scoring, time horizons, question types, and probability segments. Individual-level variation is examined with ordinary least squares regressions predicting each forecaster’s threshold of estimative precision from cultivation and selection variables.

Limitations

  • The paper states four main caveats: the response scale was not randomized, so the data are not a true experimental comparison of numeric versus verbal formats; time horizons were limited to questions resolvable during the study, averaging 76 days (SD 80), so multiyear forecasts are not covered; only future events were assessed, not current or past states; and respondents volunteered, so they are not a representative sample of foreign policy analysts.
  • The text also makes evident that aggregating to the question level reduces the maximum sample from 888,328 forecasts to 380 questions, and the paper acknowledges that the “number of questions” predictor may partly reflect statistical power rather than experience alone.
  • The paper notes that IARPA required questions to be worded precisely enough for later resolution, which is an exception to the ecological validity of the question list.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/isq/sqx078

Cites

74 of 74 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

4 of 29 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    If we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time.

    “Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Rieber 2004 source
    • cites Tetlock 2005, 13

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Mandel and Barnes 2014, 10985 source
  • Premisecitation

    Geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.

    “Mandel and Barnes (2014) provide evidence that geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Mandel and Barnes (2014) source
  • Premisecitation

    When analysts use terms like high confidence, there is no reliable way to know whether they mean probabilities closer to 70 percent or to 90 percent.

    “When analysts use terms like”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Beyth-Merom 1982 source
    • cites Dhami and Wallsten 2005 source

      The Abstract (page unavailable) says that the same probability phrase can refer to different probabilities for different people, supporting the concern about ambiguity. The paper does not establish what analysts mean by “high confidence” or compare 70% with 90%; its Between-Participants Comparisons section also suggests that individual meanings may be translated, so “no reliable way” is too absolute. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    The GJP found that just one hour of randomly assigned training had a substantial, persistent impact on improving forecasters' Brier scores.

    “For example, the GJP found that just one hour of randomly assigned training had a substantial, persistent impact on improving forecasters' Brier scores (Chang et al. 2016).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Chang et al. 2016 source
  • Findingown data

    None of the individual attributes (education, numeracy, cognitive style) predicts substantial variation in returns to precision.

    “In the second-to-last section of this article, we show that none of these attributes predicts substantial variation in returns to precision, especially relative to factors such as skill, effort, and training in probabilistic reasoning.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 36

  • May combine two claims.

    Findingnone given

    Evaluating individual forecasts returns similar estimates of returns to precision, albeit with inappropriate levels of statistical significance.

    “Evaluating individual forecasts returns similar estimates of returns to precision, albeit with inappropriate levels of statistical significance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Forty-nine percent of forecasts in the data set are multiples of 0.10, and twenty-five percent of forecasts are additional multiples of 0.05.

    “Forty-nine percent of forecasts in the data set are multiples of 0.10, and twenty-five percent of forecasts are additional multiples of 0.05.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Model 3 suggests that older respondents might have been able to parse their probability estimates more reliably.

    “Model 3 suggests that older respondents might have been able to parse their probability estimates more reliably.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Respondents' experience making forecasts and their willingness to revise those forecasts consistently predict higher returns to precision (though the latter finding fell short of the p < 0.05 threshold in some models).

    “We also find that respondents' experience making forecasts and their willingness to revise those forecasts consistently predict higher returns to precision (though the latter finding fell short of the p < 0.05 threshold in some models).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    This makes a majority of forecasts worse (and the difference in means more statistically significant) even as average rounding errors decline.

    “This makes a majority of forecasts worse (and the difference in means more statistically significant) even as average rounding errors decline.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    For nonsuperforecasters' most extreme estimates, coarsening degrades accuracy on average but improves it at the median.

    “coarsening these estimates degrade their accuracy on average but improve them at the median”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The National Intelligence Council's current guidelines for expressing uncertainty induce greater variance; rounding errors under them tend to be larger but less consistent.

    “The National Intelligence Council's current guidelines for expressing uncertainty induce greater variance; rounding errors here tend to be larger but also less consistent.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Model 5 returns an R2 value just 0.002 below that of Model 4.

    “Model 5 returns an R 2 value just 0.002 below that of Model 4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    We see no evidence that these returns to precision hinged on extreme forecasts, short time horizons, particular scoring rules, or question content.

    “We see no evidence that these returns to precision hinged on extreme forecasts, short time horizons, particular scoring rules, or question content.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Findingnone given

    The targets for selection add little predictive power to the analysis of individual-level returns to precision.

    “targets for selection add to our analysis of individual-level returns to precision.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Rounding forecasts into seven-step words of estimative probability recovered some, but not all, of the losses from coarsening.

    “Rounding forecasts into seven-step words of estimative probability (WEPs) recovered some, but not all, of these losses.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Shifting GJP forecasts to confidence levels degraded forecast accuracy by more than 10 percent on average and substantially more for high-performing forecasters.

    “On average, this level of imprecision degraded forecast accuracy by more than 10 percent and substantially more for high-performing forecasters.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    We see little evidence that returns to precision belong primarily to forecasters who are especially skilled in quantitative reasoning, who have special educational backgrounds, or who possess particular cognitive styles.

    “The second principal takeaway from this analysis is that we see little evidence that returns to precision belong primarily to forecasters who are especially skilled in quantitative reasoning, who have special educational backgrounds, or who possess particular cognitive styles.”
    • Also stated in: § Conclusion · hedged
      “We also see little indication that the ability to parse probabilities belonged primarily to respondents who possess special educational backgrounds or strong quantitative skills.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    When rounding errors are recalculated using a logarithmic scoring rule, superforecasters exhibit reliable returns to precision in every category and rounding sacrifices predictive accuracy for nonsuperforecasters in every category besides the extremes.

    “When we recalculate rounding errors using a logarithmic scoring rule, 29 we again find that superforecasters exhibit reliable returns to precision in every category and that rounding sacrifices predictive accuracy for nonsuperforecasters in every category besides the extremes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Study design and conductnone given

    The study adopted deliberately conservative statistical assumptions that presumably understate returns to precision across the data.

    “In particular, we highlight how we adopted deliberately conservative statistical assumptions that presumably understate returns to precision across our data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The empirical analysis measures return to precision in geopolitical forecasting, which is a subset of foreign policy analysis writ large.

    “We therefore emphasize that our empirical analysis measures return to precision in geopolitical forecasting, which is a subset of foreign policy analysis writ large.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Study design and conductreasoning

    Rounding off numerical forecasts in a manner consistent with different modes of qualitative expression is the most straightforward way to estimate returns to precision.

    “For these reasons, rounding off numerical forecasts in a manner that is consistent with different modes of qualitative expression is the most straightforward way to estimate returns to precision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    GJP only asked respondents to assess the probability of future events.

    “Third, GJP only asked respondents to assess the probability of future events.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Superforecasters were relatively numerate and relatively knowledgeable about foreign policy, but they were not necessarily experts in particular subjects or methodologies.

    “Generally speaking, superforecasters were relatively numerate and relatively knowledgeable about foreign policy, but they were not necessarily experts in particular subjects or methodologies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    In many cases, shifting a probability estimate by a few percentage points might not matter.

    “In many cases, shifting a probability estimate by a few percentage points might not matter.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Medical professionals, like foreign policy analysts, often prove reluctant to express probabilistic judgments explicitly.

    “Yet medical professionals, like foreign policy analysts, often prove reluctant to express probabilistic judgments explicitly (Braddock, Edwards, Hasenberg, Laidley, and Levinson 1999, 2318).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Braddock, Edwards, Hasenberg, Laidley, and Levinson 1999, 2318 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The application of criminal justice in the United States revolves, in large part, around the ways in which jurors interpret the vague probabilistic standard of guilt beyond a reasonable doubt.

    “The application of criminal justice in the United States revolves, in large part, around the ways in which jurors interpret the vague probabilistic standard of guilt beyond a reasonable doubt (Tillers and Gottfried 2006).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Tillers and Gottfried 2006 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    The Director of National Intelligence's guidelines define remote and almost certain as comprising assessments of 0.01-0.05 and 0.95-0.99, respectively.

    “Though the Director of National Intelligence's guidelines define remote and almost certain as comprising assessments of 0.01-0.05and 0.95-0.99,respectively”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The Director of National Intelligence defines the phrase even chance as representing probabilities between 45 and 55 percent.

    “the Director of National Intelligence defines the phrase even chance as representing probabilities between 45 and 55 percent.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    It is inappropriate to translate uncertainty about quantities into single-point estimates, but it is the proper way to treat uncertainty about probabilities.

    “While it is inappropriate to translate uncertainty about quantities into single-point estimates, it is the proper way to treat uncertainty about probabilities.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ellsberg (1961) source

  • May combine two claims.

    Interpretationown data

    This article shows that it is a mistake to believe that probabilistic reasoning is meaningless in world politics or to think there is no cost to leaving these judgments vague.

    “This article nevertheless shows that it is a mistake to believe that probabilistic reasoning is meaningless in world politics or to think there is no cost to leaving these judgments vague.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Improving assessments of uncertainty will not always improve the quality of decisions.

    “Of course, improving assessments of uncertainty will not always improve the quality of decisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Analyzing means and medians together ensures that outliers do not drive our inferences.

    “Analyzing means and medians together ensures that outliers do not drive our inferences.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    These findings provide additional grounds for optimism that professional forecasters could replicate and potentially exceed the returns to precision shown in GJP's data.

    “These findings provide additional grounds for optimism that professional forecasters could replicate and potentially exceed the returns to precision shown in GJP's data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: GJP forecasters' returns to precision are informative about how professional foreign policy analysts would perform if they made more precise probability estimates.
    • Rests on an assumption: Many national security professionals assess uncertainty on a daily basis over many years.
    • Rests on an assumption: Professional foreign policy analysts have much more opportunity and incentive to refine and revise their forecasts in light of new information than did GJP respondents.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    This article offers a generalizable methodology showing how these disciplines can revisit their own basic skepticism about the value of probabilistic precision.

    “This article offers a generalizable methodology showing how these disciplines can revisit their own basic skepticism about the value of probabilistic precision.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Great scholars such as Popper, Keynes, and Mill have all expressed doubts about the value of assessing subjective probability.

    “Great scholars such as Popper, Keynes, and Mill have all expressed doubts about the value of assessing subjective probability.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Popper

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Keynes

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Mill

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Our methodology can be extended to estimate the value of precision when assessing other quantifiable aspects of uncertainty, such as how much a policy might cost.

    “Our methodology can also be extended to estimate the value of precision when assessing other quantifiable aspects of uncertainty, such as how much a policy might cost.43”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Climate scientists debate the value of precision in communicating predictions to the public.

    “Climate scientists similarly debate the value of precision in communicating predictions to the public (Budescu, Broomell, and Por 2009).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Budescu, Broomell, and Por 2009 source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    It is logically impossible to support a high-stakes decision without believing that its probability of success is large enough to make expected benefits outweigh expected costs.

    “At the broad-est level, it is logically impossible to support a high-stakes decision without believing that its probability of success is large enough to make expected benefits outweigh expected costs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Similar controversies about the value of precision surround assessments of uncertainty in almost any other area of high-stakes decision-making.

    “Finally, while our article focuses on the domain of international relations, similar controversies about the value of precision surround assessments of uncertainty in almost any other area of high-stakes decision-making.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisereasoning

    Our ability to predict world politics is clearly less accurate than we would like it to be.

    “Our ability to predict world politics is clearly less accurate than we would like it to be.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    The histogram is symmetric because predicting that an outcome will occur with probability p implies that the outcome will not occur with probability 1 -p.

    “The histogram is symmetric because predicting that an outcome will occur with probability p implies that the outcome will not occur with probability 1 -p.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    When policy makers encounter these decisions many times over, these shifts will inevitably prove critical in some instances.

    “But when policy makers encounter these decisions many times over, these shifts will inevitably prove critical in some instances.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    Saying that a probability is equally likely to fall anywhere within a range conveys the same expected value as stating that range's midpoint.

    “However, saying that a probability is equally likely to fall anywhere within a range conveys the same expected value as stating that range's midpoint.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Throughout the analysis, the authors control for age, gender, and whether a respondent was designated as a superforecaster in any tournament year.

    “Throughout our analysis, we also control for age, for gender, and for whether a respondent was designated as a superforecaster in any tournament year.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationown data

    The results of our analysis are relevant not only for intelligence analysts and military planners, but also for scholars, pundits, and any other participants in the broader marketplace of ideas.

    “The results of our analysis are relevant not only for intelligence analysts and military planners, but also for scholars, pundits, and any other participants in the broader marketplace of ideas.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    We cannot always know in advance where these differences will be most important.

    “The fact that we cannot always know in advance where these differences will be most important is exactly why analysts should avoid discarding information unnecessarily.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    If foreign policy analysts can reliably parse subjective probability estimates with numeric precision, this suggests that other disciplines may also benefit from scrutinizing their own conventional wisdom about the value of precision when assessing uncertainty.

    “If foreign policy analysts can reliably parse subjective probability estimates with numeric precision, this suggests that other disciplines may also benefit from scrutinizing their own conventional wisdom about the value of precision when assessing uncertainty.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Foreign policy analysis is widely considered an unusually difficult arena for probability assessment.
    • Rests on an assumption: International affairs involve a large number of variables that interact in nonlinear ways within highly specific contexts.
    • Rests on an assumption: Foreign policy analysts generally lack access to broadly accepted theoretical models or to large, well-behaved data sets for grounding their inferences.
    • Rests on an assumption: Analysts in professions such as medicine, law, and climate science often have much stronger bases for defining reference classes, estimating base rates, or employing analytic tools to assist with assessing uncertainty.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Taking the forecasting question as the unit of analysis represents a deliberately conservative approach to statistical analysis.

    “This method represents a deliberately conservative approach to statistical analysis, because it reduces our maximum sample size from 888,328 forecasts to 380 forecasting questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Either interpretation has the same practical implication: the more forecasts analysts make, the more likely it becomes that coarsening those estimates will systematically sacrifice information.

    “Yet, it is important to note that either interpretation has the same practical implication: the more forecasts analysts make, the more likely it becomes that coarsening those estimates will systematically sacrifice information.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    Many scholars and practitioners are deeply skeptical of probability assessment.

    “We have seen throughout this article how many scholars and practitioners are deeply skeptical of probability assessment.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given

    Forecasts are correlated within questions and within individuals who updated forecasts before questions closed for evaluation.

    “However, these forecasts are correlated within questions and within individuals who updated forecasts before questions closed for evaluation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Coarsening probability assessments prevented the best forecasters from reaching their full potential, sacrificing information disproportionately from the sources that produced the most reliable assessments.

    “Coarsening probability assessments thus prevented the best forecasters from reaching their full potential, sacrificing information disproportionately from the sources that produced the most reliable assessments.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given

    Empirical findings from one domain do not directly translate into others.

    “And while empirical findings from one domain do not directly translate into others, foreign policy analysis is widely considered an unusually difficult arena for probability assessment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Even if Aristotle's claim is true, it does not tell us where the frontier of justifiable precision lies in foreign policy analysis or in any other discipline.

    “Yet, even if that is true, it does not tell us where the frontier of justifiable precision lies in foreign policy analysis or in any other discipline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Aristotle argued that justifiable precision declines as questions become more complex.

    “Aristotle himself argued that justifiable precision declines as questions become more complex.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Aristotle

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    That is ultimately an empirical question.

    “That is ultimately an empirical question, and to our knowledge, this article represents the first attempt to address that question directly.”
    Quote in the paperNot a substantial claim
52 of 52

  • The paper relies on this without support.

    No empirical study has systematically addressed the controversy over whether making probability assessments more precise improves their accuracy.

    “To our knowledge, no one has tested this claim directly.”
    “Yet, to date, no empirical study has systematically addressed this controversy.”

  • The paper relies on this without support.

    Analysts find it more difficult to parse probabilities when making forecasts than when assessing uncertainty about the present and past.

    “We expect that analysts should generally find it more difficult to parse probabilities when making forecasts, as forecasting requires assessing imperfect information while accounting for additional uncertainty about how states of the world may change in the future.”

  • The paper relies on this without support.

    GJP forecasters' returns to precision are informative about how professional foreign policy analysts would perform if they made more precise probability estimates.

    “These findings provide additional grounds for optimism that professional forecasters could replicate and potentially exceed the returns to precision shown in GJP's data.”

  • The paper relies on this without support.

    Analytic teamwork is much denser among national security professionals than it was among GJP groups who collaborated online.

    “For example, analytic teamwork is much denser among national security professionals than it was among GJP groups who collaborated online.”

  • The paper relies on this without support.

    Foreign policy organizations have opportunities to train their analysts much more extensively than the simple, one-hour training modules that GJP respondents received.

    “Similarly, foreign policy organizations have opportunities to train their analysts much more extensively than the simple, one-hour training modules that GJP respondents received.”

  • The paper relies on this without support.

    Many national security professionals assess uncertainty on a daily basis over many years.

    “Many national security professionals assess uncertainty on a daily basis over many years.”

  • The paper relies on this without support.

    Professional foreign policy analysts have much more opportunity and incentive to refine and revise their forecasts in light of new information than did GJP respondents.

    “Professional foreign policy analysts also have much more opportunity and incentive to refine and revise their forecasts in light of new information than did GJP respondents, who revised their forecasts less than twice per question, on average.”

  • The paper relies on this without support.

    This article represents the first attempt to address the question of where the frontier of justifiable precision lies directly.

    “That is ultimately an empirical question, and to our knowledge, this article represents the first attempt to address that question directly.”

  • The paper relies on this without support.

    Foreign policy analysis is widely considered an unusually difficult arena for probability assessment.

    “And while empirical findings from one domain do not directly translate into others, foreign policy analysis is widely considered an unusually difficult arena for probability assessment.”

  • The paper relies on this without support.

    International affairs involve a large number of variables that interact in nonlinear ways within highly specific contexts.

    “International affairs involve a large number of variables that interact in nonlinear ways within highly specific contexts.”

  • The paper relies on this without support.

    Foreign policy analysts generally lack access to broadly accepted theoretical models or to large, well-behaved data sets for grounding their inferences.

    “Foreign policy analysts generally lack access to broadly accepted theoretical models or to large, well-behaved data sets for grounding their inferences.”

  • The paper relies on this without support.

    Analysts in professions such as medicine, law, and climate science often have much stronger bases for defining reference classes, estimating base rates, or employing analytic tools to assist with assessing uncertainty.

    “By comparison, analysts in professions such as medicine, law, and climate science often have much stronger bases for defining reference classes, estimating base rates, or employing analytic tools to assist with assessing uncertainty.”
12 of 12

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Model 4 omits a significance marker for the superforecaster coefficient

    Minor

    The Model 4 coefficient divided by its reported standard error is about 8.46, far beyond the table's stated threshold for three stars. Unlike the same variable in the other models, this entry has no significance marker.

  • Model 1 omits a significance marker for age

    Minor

    The Model 1 age coefficient divided by its reported standard error is about 2.43, which meets the table's stated threshold for one star. The entry is unmarked.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 15

  1. Aristotle, 1985, Nicomachean Ethics. Translated by Terence Irwin
  2. Bueno de Mesquita, 2009, The Predictioneer's Game
  3. Connable, 2012, Embracing the Fog of War
  4. Dhami, 2013, Understanding and Communicating Uncertainty in Intelligence Analysis
  5. Gardner, 2011, Future Babble: Why Pundits are Hedgehogs and Foxes Know Best
  6. Johnston, 2005, Analytic Culture in the US Intelligence Community
  7. Marchio, 2014, “The Intelligence Community's Struggle to Express Analytic Uncertainty in the 1970s.”, Studies in Intelligence, 58, 31
  8. McDermott, 2007, “The Psychological Consequences of Terrorism Alerts.”
  9. National Intelligence Council, 2007, Prospects for Iraq's Stability
  10. National Intelligence Council, 2002, Iraq's Continuing Programs for Weapons of Mass Destruction
  11. National Intelligence Council, 2017, Assessing Russian Activities and Intentions in Recent U.S. Elections
  12. Tetlock, 2009, “Reading Tarot on K Street.”, National Interest, 103, 57
  13. US Army, 1997, Field Manual 101–5: Staff Organization and Operations
  14. US Army, 2009, Field Manual 5-0: The Operations Process
  15. US Joint Forces Command, 2006, Commander's Handbook for an Effects-Based Approach to Joint Operations

Cited by

Sorted by citations

Related

Authors
Friedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., & Zeckhauser, R.
Affiliations
  • Dartmouth College (US)
  • University of Pennsylvania (US)
  • Harvard University (US)
ORCID
Venue
International Studies Quarterly
Publisher
Oxford University Press (OUP)
Published
2018-03-19
ISSN
  • 0020-8833 (ISSN-L)
  • 1468-2478
DOI
10.1093/isq/sqx078
OpenAlex
W2792589092
Cited by
58
References
69
Citations by year
2026: 1 · 2025: 16 · 2024: 2 · 2023: 4 · 2022: 6 · 2021: 8 · 2020: 9 · 2019: 5 · 2018: 5 · 2017: 2
FWCI
5.39
Type
journal article
Language
en
Open access
closed
Locations
  • International Studies Quarterly (publishedVersion)
  • Toulouse Capitole Publications (University Toulouse 1 Capitole) (submittedVersion)
Funders
  • Agence Nationale de la Recherche (ANR-17-EURE-0010)
  • Intelligence Advanced Research Projects Activity (D11PC20061)
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
  • Social Sciences › Political Science and International Relations › Intelligence, Security, War Strategy
  • Social Sciences › Political Science and International Relations › International Relations and Foreign Policy
Keywords
probability assessment, geopolitical forecasting, prediction accuracy, uncertainty communication, foreign policy discourse, subjective judgment, intelligence analysis, scoring rules

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review