“Accuracy of Forecasts in Strategic Intelligence.”
Open accessAbstract
Significance: Forecasting is a vital part of strategic intelligence, offering policy makers indications about probable future conditions and aiding sound decision making. Nevertheless, there has not been a concerted effort to study the accuracy of intelligence forecasts over a large set of assessments. This study applied quantitative measures of forecasting accuracy commonly used in other areas of expert forecasting to over 1,500 strategic intelligence forecasts spanning approximately 6 y of intelligence production by a strategic assessment unit. The findings revealed a high level of discrimination and calibration skill. We also show how calibration may be improved through postforecast transformations. The study illustrates the utility of proactively applying scoring rules to intelligence forecasts as a means of outcome-based quality control.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time.
“Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes 2014, 10985
Geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.
“Mandel and Barnes (2014) provide evidence that geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes (2014)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879 · see citing works
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650 · see citing works
Chaiken, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.39.5.752 · Cited by 5,516 · see citing works
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177 · see citing works
Heuristics and Biases: The Psychology of Intuitive Judgment
Bottom, W. P., Gilovich, T. D., Griffin, D., & Kahneman, D. J. (2004) · Academy of Management Review · doi:10.2307/20159081 · Cited by 3,579 · see citing works
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,187 · see citing works
Evaluating the predictive performance of habitat models developed using logistic regression
Pearce, J., & Ferrier, S. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00322-7 · Cited by 2,004 · see citing works
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,970 · see citing works
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967 · see citing works
Knowing with certainty: The appropriateness of extreme confidence.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.3.4.552 · Cited by 1,428 · see citing works
Expert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260 · see citing works
Do those who know more also know more about how much they know?*1
Lichtenstein, S., & Fischhoff, B. (1977) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(77)90001-0 · Cited by 1,107 · see citing works
Overconfidence as a Cause of Diagnostic Error in Medicine
Berner, E. S., & Graber, M. L. (2008) · The American Journal of Medicine · doi:10.1016/j.amjmed.2008.01.001 · Cited by 1,034 · see citing works
Kean, T. H., & Hamilton, L. H. (2004) · Cited by 804 · see citing works
Accountability: A Social Check on the Fundamental Attribution Error
Tetlock, P. E. (1985) · Social Psychology Quarterly · doi:10.2307/3033683 · Cited by 777 · see citing works
Accountability and judgment processes in a personality prediction task.
Tetlock, P. E., & Kim, J. I. (1987) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.52.4.700 · Cited by 563 · see citing works
Two further applications of a model for binary regression
COX, D. R. (1958) · Biometrika · doi:10.1093/biomet/45.3-4.562 · Cited by 556 · see citing works
WGBH-Boston, B. (1990) · PsycEXTRA Dataset · doi:10.1037/e550512004-001 · Cited by 528 · see citing works
Gilovich, T. D. (2002) · Cited by 513 · see citing works
Swets, J. A. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.99.2.181 · Cited by 508 · see citing works
Accountability and judgment processes in a personality prediction task.
Tetlock, P. E., & Kim, J. I. (1987) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.52.4.700 · Cited by 416 · see citing works
Wirtz, J. J. (2004) · Political Science Quarterly · doi:10.1002/j.1538-165x.2004.tb01293.x · Cited by 412 · see citing works
Subjectively weighted utility: A descriptive extension of the expected utility model
Karmarkar, U. S. (1978) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(78)90039-9 · Cited by 362 · see citing works
Probability Forecasting in Meteorology
Murphy, A. H., & Winkler, R. (1984) · Journal of the American Statistical Association · doi:10.1080/01621459.1984.10478075 · Cited by 346 · see citing works
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312 · see citing works
Facing uncertainty in the game of bridge: A calibration study
Keren, G. B. (1987) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(87)90047-1 · Cited by 302 · see citing works
Comparing the calibration and coherence of numerical and verbal probability judgments
Wallsten, T., Budescu, D., & Zwick, R. (1993) · Management Science · doi:10.1287/mnsc.39.2.176 · Cited by 252 · see citing works
Fischhoff, B., & de Bruin, W. B. (1999) · Journal of Behavioral Decision Making · doi:10.1002/(sici)1099-0771(199906)12:2 · Cited by 212 · see citing works
Does having to justify one's judgments change the nature of the judgment process?
Hagafors, R., & Brehmer, B. (1983) · Organizational Behavior and Human Performance · doi:10.1016/0030-5073(83)90122-8 · Cited by 182 · see citing works
Measures of Discrimination Skill in Probabilistic Judgment
Yaniv, I., Yates, J., & Smith, J. K. (1991) · Psychological Bulletin · doi:10.1037/0033-2909.110.3.611 · Cited by 181 · see citing works
The calibration of expert judgment: heuristics and biases beyond the laboratory
Koehler, D. J., Brenner, L., & Griffin, D. (2002) · Heuristics and Biases · doi:10.1017/cbo9780511808098.041 · Cited by 146 · see citing works
Misinterpretations of Precipitation Probability Forecasts
Murphy, A. H., Lichtenstein, S., Fischhoff, B., & Winkler, R. L. (1980) · Bulletin of the American Meteorological Society · doi:10.1175/1520-0477(1980)061 · Cited by 132 · see citing works
Two Reasons to Make Aggregated Probability Forecasts More Extreme
Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E., & Ungar, L. H. (2014) · Decision Analysis · doi:10.1287/deca.2014.0293 · Cited by 130 · see citing works
Analyzing intelligence : origins, obstacles, and innovations
George, R. Z., & Bruce, J. B. (2008) · Cited by 118 · see citing works
Insightful or wishful: Lawyers' ability to predict case outcomes.
Goodman‐Delahunty, J., Granhag, P. A., Hartwig, M., & Loftus, E. F. (2010) · Psychology Public Policy and Law · doi:10.1037/a0019060 · Cited by 106 · see citing works
Intelligent management of intelligence agencies: Beyond accountability ping-pong.
Tetlock, P. E., & Mellers, B. A. (2011) · American Psychologist · doi:10.1037/a0024285 · Cited by 71 · see citing works
Forecast aggregation via recalibration
Turner, B. M., Steyvers, M., Merkle, E., Budescu, D., & Wallsten, T. (2013) · Machine-mediated learning · doi:10.1007/s10994-013-5401-4 · Cited by 71 · see citing works
(2007) · Cited by 65 · see citing works
Probabilistic Coherence Weighting for Optimizing Expert Forecasts
Karvetski, C. W., Olson, K. C., Mandel, D. R., & Twardy, C. R. (2013) · Decision Analytics · doi:10.1287/deca.2013.0279 · Cited by 49 · see citing works
Self-citationCommunicating Uncertainty in Intelligence and Other Professions
Weiss, C. (2007) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850600701649312 · Cited by 40 · see citing works
“Intelligence Analysis and Judgmental Calibration.”
RIEBER, S. (2004) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850600490273431 · Cited by 32 · see citing works
A model-based approach for the analysis of the calibration of probability judgments
Budescu, D. V., & Johnson, T. R. (2011) · Judgment and Decision Making · doi:10.1017/s1930297500004277 · Cited by 23 · see citing works
Subjective recalibration of advisors' probability estimates
Shlomi, Y., & Wallsten, T. S. (2010) · Psychonomic Bulletin & Review · doi:10.3758/pbr.17.4.492 · Cited by 16 · see citing works
A cautionary note on global recalibration
Kadane, J. B., & Fischhoff, B. (2013) · Judgment and Decision Making · doi:10.1017/s1930297500004460 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- [2]2. Fingar T (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations, eds Fischhoff B, Chauvin C (Natl Acad Press, Washington, DC), pp 3-27.
- [11]11. Fischhoff B, Chauvin C, eds (2011) Intelligence Analysis: Behavioral and Social Scien- tific Foundations (Natl Acad Press, Washington, DC), 338 pp.
- [13]13. National Research Council (2010) Field Evaluation in the Intelligence and Counterin- telligence Context: Workshop Summary (Natl Acad Press, Washington, DC), 114 pp.
- [14]14. National Research Council (2011) Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social Sciences (Natl Acad Press, Washington, DC), 102 pp.
- [22]22. Charba JP, Klein WH (1980) Skill in precipitation forecasting in the National Weather Service. Bull Am Meteorol Soc 61(12):1546-1555.
- [35]35. Turner BM, Steyvers M, Merkle EC, Budescu DV, Wallsten TS (2014) Forecast aggre- gation via recalibration. Mach Learn 95(3):261-289.
- [47]47. Arkes HR, Kajdasz J (2011) Intelligence Analysis: Behavioral and Social Scientific Foundations, eds Fischhoff B, Chauvin C (Natl Acad Press, Washington, DC), pp 143-168.
- [48]48. Mellers B, et al. (2014) Psychological strategies for winning geopolitical forecasting tournaments. Psychol Sci 25(5):1106-1115.
Cited by
Sorted by citations
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104 · see citing works
“Likewise, transformation rules that improve calibration, an important facet of forecasting skill, have been recently documented (Baron et al., 2014), including in use with intelligence forecasts (Mandel & Barnes, 2014).”
BackgroundSelf-citationVerbal probabilities: Very likely to be somewhat more confusing than numbers
Wintle, B., Fraser, H. S., Wills, B. C., Nicholson, A. E., & Fidler, F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213522 · Cited by 70 · see citing works
Insights into the accuracy of social scientists’ forecasts of societal change
Collaborative, T. F., Grossmann, I., Rotella, A., Hutcherson, C. A. C., Sharpinskyi, K., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X., Kara-Yakoubian, M., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01517-1 · Cited by 75 · see citing works
Self-citationConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74 · see citing works
“Current systems rely primarily on qualitative assessments of probabilities and risk (Mandel and Barnes 2014).”
BackgroundRestructuring structured analytic techniques in intelligence
Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1400230 · Cited by 74 · see citing works
Self-citationImproving the communication of uncertainty in climate science and intelligence analysis
Ho, E., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015) · Behavioral Science & Policy · doi:10.1353/bsp.2015.0015 · Cited by 73 · see citing works
Self-citationCommunicating uncertainty using words and numbers
Dhami, M. K., & Mandel, D. R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.002 · Cited by 69 · see citing works
Self-citationFriedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., & Zeckhauser, R. (2018) · International Studies Quarterly · doi:10.1093/isq/sqx078 · Cited by 58 · see citing works
“Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
BackgroundUse and communication of probabilistic forecasts
Raftery, A. E. (2016) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.11302 · Cited by 67 · see citing works
“In a rare counterexample, Mandel and Barnes [56] found”
BackgroundWisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. (2024) · Science Advances · doi:10.1126/sciadv.adp1528 · Cited by 130 · see citing works
“For further non-preregistered analyses, we conduct calibration analyses using the Murphy decomposition ( 54 , 55 ) to provide data on how well-calibrated the models are in this context, i.e., how reliably their probability estimates match the fraction of real outcomes.”
Geopolitical Forecasting Skill in Strategic Intelligence
Mandel, D. R., & Barnes, A. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2055 · Cited by 44 · see citing works
Self-citationFriedman, J. A., Lerner, J. S., & Zeckhauser, R. (2017) · International Organization · doi:10.1017/s0020818317000352 · Cited by 43 · see citing works
Instruction in information structuring improves Bayesian judgment in intelligence analysts
Mandel, D. R. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00387 · Cited by 43 · see citing works
“Although one study has found that strategic intelligence forecasts showed good discrimination and calibration (Mandel and Barnes, 2014), the extent to which analytical judgments are coherent has not been addressed in an intelligence analyst sample.”
BackgroundSelf-citationHow generalizable is good judgment? A multi-task, multi-benchmark study
Mellers, B. A., Baker, J. D., Chen, E., Mandel, D. R., & Tetlock, P. E. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500006240 · Cited by 43 · see citing works
“Mellers et al. (2015b) examined correlations between correspondence a measure of intelligence in forecasters from the geopolitical tournaments discussed earlier.”
BackgroundSelf-citationMathematically aggregating experts’ predictions of possible futures
Hanea, A. M., Wilkinson, D. P., McBride, M. F., Lyon, A., van Ravenzwaaij, D., Thorn, F. S., Gray, C. T., Mandel, D. R., Willcox, A., Gould, E., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256919 · Cited by 38 · see citing works
“Intelligence analysts might seek predictions of the outcome of an important election, or the price of oil at some point in the future (e.g., [4]).”
Self-citationAccuracy of Intelligence Forecasts From the Intelligence Consumer’s Perspective
Mandel, D. R. (2015) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215602907 · Cited by 37 · see citing works
“Using the Mandel and Barnes (2014) forecast data set in conjunction with data from a new experiment on the interpretation of the linguistic probabilities used in those forecasts, the present research addressed the challenge of inferring forecast accuracy from the intelligence consumer’s perspective.”
BackgroundMethodResultInfluentialSelf-citationBoosting intelligence analysts’ judgment accuracy: What works, what fails?
Mandel, D. R., Karvetski, C. W., & Dhami, M. K. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500006628 · Cited by 35 · see citing works
“…have shown that analysts’ forecasts tend to be underconfident, and that the calibration of intelligence units can be improved post-judgment through an organizational recalibration process that “extremizes” overly-cautious forecasts (Mandel & Barnes, 2014; Baron et al., 2014; Turner et al., 2014).”
BackgroundSelf-citationImproving information evaluation for intelligence production
Irwin, D., & Mandel, D. R. (2019) · Intelligence & National Security · doi:10.1080/02684527.2019.1569343 · Cited by 35 · see citing works
Self-citationRethinking the training of intelligence analysts
Chang, W., & Tetlock, P. E. (2016) · Intelligence & National Security · doi:10.1080/02684527.2016.1147164 · Cited by 32 · see citing works
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32 · see citing works
Self-citationForecasting forecaster accuracy: Contributions of past performance and individual differences
Himmelstein, M., Atanasov, P. D., & Budescu, D. V. (2021) · Judgment and Decision Making · doi:10.1017/s1930297500008597 · Cited by 28 · see citing works
“…true trait level, and E to residual error, is given as: - = ) + (1) This equation is mirrored in the literature on forecasting, which partitions measures of average individual accuracy into skill components perturbed by residual error (Mandel & Barnes, 2014; Murphy, 1973; Wallsten & Budescu, 1983).”
BackgroundKori, B. W., Muathe, S. M., & Maina, S. M. (2020) · International Business Research · doi:10.5539/ibr.v13n10p130 · Cited by 27 · see citing works
Towards an evidence-based approach to communicating uncertainty in intelligence analysis
Dhami, M. K. (2018) · Intelligence & National Security · doi:10.1080/02684527.2017.1394252 · Cited by 20 · see citing works
“32 For instance, Mandel and Barnes (2014) used a widely accepted quantitative procedure involving Brier scores (which can be partitioned into components measuring calibration and discrimination) to assess the accuracy of analysts’ forecasts.”
MethodCorrecting Judgment Correctives in National Security Intelligence
Mandel, D. R., & Tetlock, P. E. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02640 · Cited by 21 · see citing works
“, 2014) and in actual strategic intelligence forecasts (Mandel and Barnes, 2014).”
BackgroundSelf-citationThe occasional maverick of analytic tradecraft
Mandel, D. R. (2020) · Intelligence & National Security · doi:10.1080/02684527.2020.1723830 · Cited by 21 · see citing works
Self-citationFailure dynamics of the global risk networkPreprint
Szymanski, B. K., Lin, X., Asztalos, A., & Sreenivasan, S. (2016) · Cited by 22 · see citing works
The wisdom of many in few: Finding individuals who are as wise as the crowd.
Himmelstein, M., Budescu, D. V., & Ho, E. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001340 · Cited by 19 · see citing works
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 20 · see citing works
Self-citationFailure dynamics of the global risk network
Szymanski, B. K., Lin, X., Asztalos, A., & Sreenivasan, S. (2015) · Scientific Reports · doi:10.1038/srep10998 · Cited by 19 · see citing works
An IRT forecasting model: linking proper scoring rules to item response theory
Bo, Y. E., Budescu, D. V., Lewis, C., Tetlock, P. E., & Mellers, B. A. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500005647 · Cited by 17 · see citing works
“Second, some forecasts relate to outcomes of events whose “ground truth” is hard to determine (e.g., Armstrong, 2001; Lehner, Micheslson, Adelma & Goodman, 2012; Mandel & Barnes, 2014; Tetlock, 2005).”
BackgroundTracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study
Mandel, D. R., & Irwin, D. (2021) · Futures & Foresight Science · doi:10.1002/ffo2.98 · Cited by 16 · see citing works
Self-citationSchneider, G., Hadar, M., & Bosler, N. (2017) · Journal of Peace Research · doi:10.1177/0022343316683437 · Cited by 16 · see citing works
How the “wisdom of the inner crowd” can boost accuracy of confidence judgments.
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2020) · Decision · doi:10.1037/dec0000119 · Cited by 15 · see citing works
Comparative evaluation of the forecast accuracy of analysis reports and a prediction market
Stastny, B. J., & Lehner, P. E. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500007105 · Cited by 14 · see citing works
Till, C., Haverkamp, J., White, D., & Bhaduri, B. (2018) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512916679038 · Cited by 9 · see citing works
“They also need to fully understand the degree of uncertainty involved [6, 7] and question anything that they do not understand.”
Mandel, D. R., & Irwin, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0248424 · Cited by 10 · see citing works
“At least, this is suggested by work showing substantial underconfidence in intelligence analysts’ probabilistic forecasts [31,32].”
BackgroundSelf-citationTetlock, P. E., & Mellers, B. A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1412524111 · Cited by 11 · see citing works
Strategic intelligence as an art and a science: creating and using conceptual frameworks
Shapira, I. (2019) · Intelligence & National Security · doi:10.1080/02684527.2019.1681135 · Cited by 9 · see citing works
Mandel, D. R. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500004320 · Cited by 9 · see citing works
“I will note, however, that efforts could have been made to establish inter-rater reliability on ground truth (Mandel & Barnes, 2014), which is important because the “reality” used to score forecast accuracy was judged by subject-matter experts.”
BackgroundInfluentialSelf-citationPerformance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
Collins, R. N., Mandel, D. R., & Budescu, D. V. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_7 · Cited by 16 · see citing works
Self-citationWeighted Brier score decompositions for topically heterogenous forecasting tournaments
Merkle, E. C., & Hartman, R. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500007099 · Cited by 11 · see citing works
Using scenarios to forecast outcomes of a refugee crisis
Wicke, L., Dhami, M. K., Önkal, D., & Belton, I. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.05.017 · Cited by 8 · see citing works
Learning to Predict Proliferation
Miller, N. L. (2021) · International Organization · doi:10.1017/s0020818321000345 · Cited by 7 · see citing works
“Friedman 2019; Friedman, Lerner, and Zeckhauser 2017; Friedman and Zeckhauser 2016; Mandel and Barnes 2014; and Tetlock 2005.”
BackgroundResultWilcox, J. E., & Mandel, D. R. (2024) · Intelligence & National Security · doi:10.1080/02684527.2024.2304934 · Cited by 7 · see citing works
“90 If this is the case, then it may be premature to recommend ACH to analysts, regardless of whether it provides benefits to other populations (That said, to be fair, we do not know how many of the analysts in Mandel and Barnes (2014) study used ACH or may have been impacted by it somehow.”
Self-citationUse and Communication of Probabilistic ForecastsPreprint
Raftery, A. E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.4812 · Cited by 7 · see citing works
“In a rare counterexample, Mandel and Barnes [56] found”
BackgroundAssessing public forecasts to encourage accountability: The case of MIT’s Technology Review
Funk, J. L. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183038 · Cited by 6 · see citing works
Credences and Trustworthiness: a Calibrationist Account
Wilcox, J. E. (2024) · Review of Philosophy and Psychology · doi:10.1007/s13164-024-00724-1 · Cited by 6 · see citing works
“In general, then, miscalibration currently appears to be pervasive—although there are also some notable exceptions (Mandel and Barnes 2014, 2018; Mandel and Irwin 2021).”
Adopting and improving a new forecasting paradigm
Speigel, I. (2021) · Intelligence & National Security · doi:10.1080/02684527.2021.1946955 · Cited by 6 · see citing works
Automated calibration training for forecasters
Stone, E. R., Luu, J., Costello, C., & Somerville, A. H. (2023) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2334 · Cited by 6 · see citing works
Crowdsourcing Accurately and Robustly Predicts Supreme Court DecisionsPreprint
Katz, D. M., Bommarito, M. J., & Blackman, J. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3085710 · Cited by 6 · see citing works
Weighted Brier score decompositions for topically heterogenous forecasting tournamentsPreprint
Merkle, E. C. (2018) · doi:10.31219/osf.io/p6wk5 · Cited by 6 · see citing works
“These restrictive requirements are frequently violated in applied forecasting activities and competitions (see, e.g., Tetlock, 2005; Mandel & Barnes, 2014).”
BackgroundScience and Technology Forecasting
Rejeski, D. W., Pauwels, E., & Koo, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-07052-0_13 · Cited by 5 · see citing works
How the “Wisdom of the Inner Crowd” Can Boost Accuracy of Confidence JudgmentsPreprint
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2019) · doi:10.31234/osf.io/ngweh · Cited by 5 · see citing works
The effect of calibration training on the calibration of intelligence analysts' judgments
Kelly, M. O., & Mandel, D. R. (2024) · Applied Cognitive Psychology · doi:10.1002/acp.4236 · Cited by 4 · see citing works
Self-citationIsaksen, B. G. M., & McNaught, K. R. (2019) · Journal of Risk Research · doi:10.1080/13669877.2018.1474245 · Cited by 4 · see citing works
“Notable exceptions, concerning the communication of uncertainty in particular, are provided by Dhami (2017), Ho et al. (2015) and Mandel and Barnes (2014).”
BackgroundA mathematical model for evaluation of intelligence products value
Korać, D., & Cica, D. (2018) · Journal of Information and Optimization Sciences · doi:10.1080/02522667.2018.1427027 · Cited by 4 · see citing works
“Also, there is a significant number of authors such as Mandel and Barnes [32] that have dealt with a similar approach to the development of the communication standard that is used in generating the intelligence evaluations.”
BackgroundFriedman, J. A., Lerner, J. S., & Zeckhauser, R. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2782598 · Cited by 4 · see citing works
Intelligence, Science and the Ignorance HypothesisPreprint
Mandel, D. R. (2021) · doi:10.31234/osf.io/m2fn8 · Cited by 4 · see citing works
Self-citationInsights into accuracy of social scientists' forecasts of societal changePreprint
Grossmann, I., Rotella, A., Hutcherson, C. A. C., Sharpinskyi, C., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X., Kara-Yakoubian, M., Mandel, D. R., et al. (2022) · doi:10.31234/osf.io/wdxsb · Cited by 4 · see citing works
Self-citationStrategic Intelligence: A Semantic Leadership Perspective
Paiuc, D., Săniuță, A., & Parincu, A. M. T. (2024) · Encyclopedia · doi:10.3390/encyclopedia4020050 · Cited by 4 · see citing works
“Such intelligence can help leaders and major stakeholders anticipate future events, avoid strategic surprises, and make informed decisions [5].”
BackgroundHow do we know if an intelligence analytic product is good?
Riehle, K. P. (2025) · Intelligence & National Security · doi:10.1080/02684527.2025.2468051 · Cited by 3 · see citing works
Science and Technology Forecasting
Rejeski, D. W., Pauwels, E., & Koo, J. (2014) · Springer eBooks · doi:10.1007/978-3-319-04033-2_13-1 · Cited by 3 · see citing works
Towards a data-driven military. A multi-disciplinary perspective
Tinga, T., Homborg, A. M., Rijsdijk, C., Pijpers, P. B. M. J., Voskuijl, M., & Beeres, R. J. M. (2023) · Leiden University Press eBooks · doi:10.24415/9789087284084 · Cited by 3 · see citing works
Terrorism-Centric Behavior Recognition and Adversarial Threat Forecasting
Regens, J. L., Mould, N. A., Jensen, C. J., & Edger, D. N. (2016) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850607.2015.1083317 · Cited by 3 · see citing works
Innovation Capabilities and the role of Strategic Intelligence
Reichert, F. M., Janissek-Muniz, R., Heinze, M., Cainelli, A., & Zawislak, P. A. (2020) · Journal of the Association for Information Systems · Cited by 3 · see citing works
The Occasional Maverick of Analytic TradecraftPreprint
Mandel, D. R. (2019) · doi:10.31234/osf.io/atjve · Cited by 3 · see citing works
Self-citationWhen alternative hypotheses shape your beliefs: Context effects in probability judgmentsPreprint
Cai, X., & Pleskac, T. J. (2021) · Cognition · doi:10.1016/j.cognition.2022.105306 · Cited by 3 · see citing works
Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence
Martin, M. J., & Mandel, D. R. (2024) · Futures & Foresight Science · doi:10.1002/ffo2.199 · Cited by 2 · see citing works
Self-citationCharacterizing, Analyzing, and Communicating Uncertainty1
Morgan, M. G. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316882665.009 · Cited by 2 · see citing works
Predicting Known Terrorist Event Outcomes and the Illusion of Self‐Assessed Expertise*
Regens, J. L., Mould, N. A., & Foster, C. (2019) · Social Science Quarterly · doi:10.1111/ssqu.12601 · Cited by 2 · see citing works
Mandel, D. R., & Irwin, D. (2022) · doi:10.31234/osf.io/7hx2c · Cited by 2 · see citing works
Self-citationStrategic Intelligence: A Semantic Leadership PerspectivePreprint
Paiuc, D., Săniuță, A., & Parincu, A. M. T. (2024) · Preprints.org · doi:10.20944/preprints202404.0690.v1 · Cited by 2 · see citing works
Morgan, M. G. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316882665.010 · Cited by 1 · see citing works
Applying Information Theory to Validate Commanders' Critical Information Requirements
Timms, M. A. C., Mandel, D. R., & Nelson, J. D. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003396307-18 · Cited by 1 · see citing works
Self-citationA Roadmap for the Future Research of the White Stork Ciconia ciconia Population in Poland
Tryjanowski, P., Białas, J. T., Jankowiak, Ł., Jerzak, L., Mielczarek, P., Nowak, M. K., Profus, P., Siekiera, J., Tobółka, M., Walasz, K., et al. (2024) · Polish Journal of Ecology · doi:10.3161/15052249pje2024.72.1.004 · Cited by 1 · see citing works
“2012, Mandel and Barnes 2014).”
Sajduk, B. (2022) · Politeja · doi:10.12797/politeja.19.2022.77.05 · Cited by 1 · see citing works
Al-Janabi, W. R., & Kurdi, B. H. A. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-12382-5_88 · Cited by 1 · see citing works
Friedman, J. A. (2024) · Critical Review · doi:10.1080/08913811.2024.2433878 · Cited by 1 · see citing works
“Mandel and Barnes (2014) and Miller (2022) have additionally demonstrated that political forecasts made by intelligence analysts carry considerable predictive validity.”
Studies past and future of the past and future: Commentary on Schoemaker 2020
Mandel, D. R. (2020) · Futures & Foresight Science · doi:10.1002/ffo2.39 · Cited by 1 · see citing works
Self-citationPredicting the future of society
Salganik, M. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01535-7 · Cited by 1 · see citing works
The effectiveness of risk assessments in informing decision makers
Peace, C. (2019) · doi:10.26686/wgtn.17142020 · Cited by 1 · see citing works
Applying Information Theory to Validate Commanders’ Critical Information RequirementsPreprint
Timms, M. A. C., Mandel, D. R., & Nelson, J. D. (2023) · doi:10.31234/osf.io/n8xka · Cited by 1 · see citing works
“For instance, while bias-awareness training for analysts often focuses on the problem of overconfidence, there is evidence that analysts’ anticipatory estimates are often underconfident (Mandel & Barnes, 2014, 2018).”
Self-citationWhen expert predictions failPreprint
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · doi:10.31234/osf.io/my24e · Cited by 1 · see citing works
Self-citationEffect of Calibration Training on the Calibration of Intelligence Analysts’ JudgmentsPreprint
Kelly, M. O., & Mandel, D. R. (2023) · doi:10.31234/osf.io/p82nx · Cited by 1 · see citing works
Self-citationHashem, Y. R., & Al-Sheikli, A. I. (2021) · Journal of Economics and Administrative Sciences · doi:10.33095/jeas.v27i125.2064
Some Widely Used Analysis Tools and Topics
Morgan, M. G. (2017)
UK and US policies for communicating probability in intelligence analysis: A reviewPreprint
Dhami, M. K., & Mandel, D. R. (2020) · doi:10.31234/osf.io/kuyhb
Self-citationMandel, D. R., & Irwin, D. (2020) · doi:10.31234/osf.io/6atw5
Self-citationWhat We Think: The Accuracy of Our Judgments
Wilcox, J. E. (2022) · SpringerBriefs in psychology · doi:10.1007/978-3-031-19205-0_3
A response to Mandel’s (2019) commentary on
Lehner, P., & Stastny, B. J. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500004915
“And importantly, as shown by Mandel and Barnes (2014), it is very feasible to track and measure forecasting accuracy.”
BackgroundCarr, P., Ebersole, E. K., & Sturtz, J. L. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3234328
“For calibration of confidence, a number of scoring methods are available [17], [45].”
BackgroundMethodNarrative AbductionPreprint
van Gelder, T., & Wilcox, J. E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790721
Duke, M. C., Bolsinger, D., & Landon-Murray, M. (2024) · Intelligence & National Security · doi:10.1080/02684527.2024.2432771
CIA’s evaluation of analytic quality: can it fortify tradecraft?
Levine, R. L., & George, R. Z. (2025) · Intelligence & National Security · doi:10.1080/02684527.2024.2432770
Harper, E. (2025) · Critique Open Research & Review · doi:10.55640/corr-v03i03-02
Algorithm Perception When Using Threat Intelligence in Vulnerability Risk Assessment
van Gerwen, S., Papotti, A., Tuma, K., & Massacci, F. (2026) · Risk Analysis · doi:10.1111/risa.70178 · Cited by 1 · see citing works
Mandel, D. R. (2026) · Intelligence & National Security · doi:10.1080/02684527.2026.2635695
Self-citationPhillips, L. D. (2026) · doi:10.1017/9781009622899.024
Calibration of Retrospective and Prospective Life Satisfaction Judgments Around the World
Mandel, D. R., Meyers, E. A., & Omorogbe, P. (2026) · Futures & Foresight Science · doi:10.1002/ffo2.70045
Self-citationMouynes, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7114978
- Authors
- Mandel, D. R., & Barnes, A.
- Affiliations
- Defence Research and Development Canada (CA)
- Privy Council Office (CA)
- ORCID
- Mandel, David R.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2014-07-14
- Volume
- 111
- Issue
- 30
- Pages
- 10984-10989
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4121776
- OpenAlex
- W2085900183
- Cited by
- 101
- References
- 44
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 12 · 2023: 12 · 2022: 5 · 2021: 9 · 2020: 7 · 2019: 11 · 2018: 7 · 2017: 13 · 2016: 8 · 2015: 5 · 2014: 3
- FWCI
- 62.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada (249537-07)
- Topics
- Social Sciences › Political Science and International Relations › Intelligence, Security, War Strategy
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Business, Management and Accounting › Strategy and Management › Competitive and Knowledge Intelligence
- Keywords
- strategic intelligence, forecast accuracy, forecast calibration, underconfidence, analyst expertise, forecast informativeness, intelligence reports, policy decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
