The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
“Accuracy of Forecasts in Strategic Intelligence.”
Open accessAbstract
Significance: Forecasting is a vital part of strategic intelligence, offering policy makers indications about probable future conditions and aiding sound decision making. Nevertheless, there has not been a concerted effort to study the accuracy of intelligence forecasts over a large set of assessments. This study applied quantitative measures of forecasting accuracy commonly used in other areas of expert forecasting to over 1,500 strategic intelligence forecasts spanning approximately 6 y of intelligence production by a strategic assessment unit. The findings revealed a high level of discrimination and calibration skill. We also show how calibration may be improved through postforecast transformations. The study illustrates the utility of proactively applying scoring rules to intelligence forecasts as a means of outcome-based quality control.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes 2014, 10985
The cited source says what the claim attributes to it.
In the citing paper
“Mandel and Barnes (2014) provide evidence that geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes (2014)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Chaiken, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.39.5.752 · Cited by 5,516
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177
Heuristics and Biases: The Psychology of Intuitive Judgment
Bottom, W. P., Gilovich, T. D., Griffin, D., & Kahneman, D. J. (2004) · Academy of Management Review · doi:10.2307/20159081 · Cited by 3,583
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,227
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 2,046
Evaluating the predictive performance of habitat models developed using logistic regression
Pearce, J., & Ferrier, S. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00322-7 · Cited by 2,004
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Knowing with certainty: The appropriateness of extreme confidence.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.3.4.552 · Cited by 1,428
Expert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260
Do those who know more also know more about how much they know?*1
Lichtenstein, S., & Fischhoff, B. (1977) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(77)90001-0 · Cited by 1,107
Overconfidence as a Cause of Diagnostic Error in Medicine
Berner, E. S., & Graber, M. L. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.01.001 · Cited by 1,034
Kean, T. H., & Hamilton, L. H. (2004) · Cited by 808
Accountability: A Social Check on the Fundamental Attribution Error
Tetlock, P. E. (1985) · Social Psychology Quarterly · doi:10.2307/3033683 · Cited by 777
Accountability and judgment processes in a personality prediction task.
Tetlock, P. E., & Kim, J. I. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.52.4.700 · Cited by 563
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556
WGBH-Boston, B. (1990) · PsycEXTRA Dataset · doi:10.1037/e550512004-001 · Cited by 524
Gilovich, T. D. (2002) · Cited by 513
Swets, J. A. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.2.181 · Cited by 508
Accountability and judgment processes in a personality prediction task.
Tetlock, P. E., & Kim, J. I. (1987) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.52.4.700 · Cited by 416
Wirtz, J. J. (2004) · Political Science Quarterly · doi:10.1002/j.1538-165x.2004.tb01293.x · Cited by 412
Subjectively weighted utility: A descriptive extension of the expected utility model
Karmarkar, U. S. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90039-9 · Cited by 362
Probability Forecasting in Meteorology
Murphy, A. H., & Winkler, R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478075 · Cited by 346
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312
Facing uncertainty in the game of bridge: A calibration study
Keren, G. B. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90047-1 · Cited by 302
Comparing the calibration and coherence of numerical and verbal probability judgments
Wallsten, T., Budescu, D., & Zwick, R. (1993) · Management Science · doi:10.1287/mnsc.39.2.176 · Cited by 252
Fischhoff, B., & de Bruin, W. B. (1999) · Journal of Behavioral Decision Making · doi:10.1002/(sici)1099-0771(199906)12:2 · Cited by 217
Does having to justify one's judgments change the nature of the judgment process?
Hagafors, R., & Brehmer, B. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90122-8 · Cited by 182
Measures of Discrimination Skill in Probabilistic Judgment
Yaniv, I., Yates, J., & Smith, J. K. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.3.611 · Cited by 181
The calibration of expert judgment: heuristics and biases beyond the laboratory
Koehler, D. J., Brenner, L., & Griffin, D. (2002) · Heuristics and Biases · doi:10.1017/cbo9780511808098.041 · Cited by 146
Two Reasons to Make Aggregated Probability Forecasts More Extreme
Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., & Ungar, L. H. (2014) · Decision Analysis · doi:10.1287/deca.2014.0293 · Cited by 130
Analyzing intelligence : origins, obstacles, and innovations
George, R. Z., & Bruce, J. B. (2008) · Cited by 118
Insightful or wishful: Lawyers' ability to predict case outcomes.
Goodman‐Delahunty, J., Granhag, P. A., Hartwig, M., & Loftus, E. F. (2010) · Psychology Public Policy and Law · doi:10.1037/a0019060 · Cited by 106
Forecast aggregation via recalibration
Turner, B. M., Mark Steyvers, B., Merkle, E. C., Budescu, D. V., & Wallsten, T. S. (2013) · Machine Learning · doi:10.1007/s10994-013-5401-4 · Cited by 76
Intelligent management of intelligence agencies: Beyond accountability ping-pong.
Tetlock, P. E., & Mellers, B. A. (2011) · American Psychologist · doi:10.1037/a0024285 · Cited by 73
(2007) · Cited by 65
Probabilistic Coherence Weighting for Optimizing Expert Forecasts
Karvetski, C. W., Olson, K. C., Mandel, D. R., & Twardy, C. R. (2013) · Decision Analytics · doi:10.1287/deca.2013.0279 · Cited by 49
Self-citationCommunicating Uncertainty in Intelligence and Other Professions
Weiss, C. (2007) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850600701649312 · Cited by 40
NOAA Data Buoy Office Programs
Hamilton, G. D. (1980) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1980)061 · Cited by 33
“Intelligence Analysis and Judgmental Calibration.”
RIEBER, S. (2004) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850600490273431 · Cited by 32
A model-based approach for the analysis of the calibration of probability judgments
Budescu, D. V., & Johnson, T. R. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500004277 · Cited by 23
Subjective recalibration of advisors' probability estimates
Shlomi, Y., & Wallsten, T. S. (2010) · Psychonomic Bulletin & Review · doi:10.3758/pbr.17.4.492 · Cited by 16
A cautionary note on global recalibration
Kadane, J. B., & Fischhoff, B. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500004460 · Cited by 12
Research works printed but not linked: 6
- T Fingar Intelligence Analysis: Behavioral and Social Scientific Foundations, eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC), pp. 3–27 (2011).
- , eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC Intelligence Analysis: Behavioral and Social Scientific Foundations, , 338 pp. (2011).
- ; National Research Council Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary (Natl Acad Press, Washington, DC, , 114 pp. (2010).
- ; National Research Council Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social Sciences (Natl Acad Press, Washington, DC, , 102 pp. (2011).
- 10.1037/0022-3514.39.5.752 · DOI · Paper page
- HR Arkes, J Kajdasz Intelligence Analysis: Behavioral and Social Scientific Foundations, eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC), pp. 143–168 (2011).
Cited by
Sorted by citations
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
“Likewise, transformation rules that improve calibration, an important facet of forecasting skill, have been recently documented (Baron et al., 2014), including in use with intelligence forecasts (Mandel & Barnes, 2014).”
BackgroundSelf-citationVerbal probabilities: Very likely to be somewhat more confusing than numbers
Wintle, B., Fraser, H. S., Wills, B. C., Nicholson, A. E., & Fidler, F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213522 · Cited by 70
Insights into the accuracy of social scientists’ forecasts of societal change
Collaborative, T. F., Grossmann, I., Rotella, A., Hutcherson, C. A. C., Sharpinskyi, K., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X., Kara-Yakoubian, M., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01517-1 · Cited by 78
Self-citationConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
“Current systems rely primarily on qualitative assessments of probabilities and risk (Mandel and Barnes 2014).”
BackgroundRestructuring structured analytic techniques in intelligence
Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1400230 · Cited by 74
Self-citation
Metadata
- Authors
- Mandel, D. R., & Barnes, A.
- Affiliations
- Defence Research and Development Canada (CA)
- Privy Council Office (CA)
- ORCID
- Mandel, David R.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2014-07-14
- Volume
- 111
- Issue
- 30
- Pages
- 10984-10989
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4121776
- OpenAlex
- W2085900183
- Cited by
- 101
- References
- 44
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 12 · 2023: 12 · 2022: 5 · 2021: 9 · 2020: 7 · 2019: 11 · 2018: 7 · 2017: 13 · 2016: 8 · 2015: 5 · 2014: 3
- FWCI
- 62.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada (249537-07)
- Topics
- Social Sciences › Political Science and International Relations › Intelligence, Security, War Strategy
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Business, Management and Accounting › Strategy and Management › Competitive and Knowledge Intelligence
- Keywords
- strategic intelligence, forecast accuracy, forecast calibration, underconfidence, analyst expertise, forecast informativeness, intelligence reports, policy decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
