The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
“Accuracy of Forecasts in Strategic Intelligence.”
Open accessAbstract
Significance: Forecasting is a vital part of strategic intelligence, offering policy makers indications about probable future conditions and aiding sound decision making. Nevertheless, there has not been a concerted effort to study the accuracy of intelligence forecasts over a large set of assessments. This study applied quantitative measures of forecasting accuracy commonly used in other areas of expert forecasting to over 1,500 strategic intelligence forecasts spanning approximately 6 y of intelligence production by a strategic assessment unit. The findings revealed a high level of discrimination and calibration skill. We also show how calibration may be improved through postforecast transformations. The study illustrates the utility of proactively applying scoring rules to intelligence forecasts as a means of outcome-based quality control.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes 2014, 10985
The cited source says what the claim attributes to it.
In the citing paper
“Mandel and Barnes (2014) provide evidence that geopolitical forecasts in Canadian intelligence reports are surprisingly well calibrated.”
From The Value of Precision in Probability Assessment: Evidence from a Large-Scale Geopolitical Forecasting Tournament (2018) · cites this paper as Mandel and Barnes (2014)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A new look at the statistical model identification
Akaike, H. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100705 · Cited by 51,879
Judgment under Uncertainty: Heuristics and Biases
Tversky, A., & Kahneman, D. (1975) · Utility, Probability, and Human Decision Making · doi:10.1007/978-94-010-1834-0_8 · Cited by 22,650
Chaiken, S. (1980) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.39.5.752 · Cited by 5,516
ANew Look at the Statistical Model Identification
AKAlKE, H. (2011) · Cited by 4,177
Heuristics and Biases: The Psychology of Intuitive Judgment
Bottom, W. P., Gilovich, T. D., Griffin, D., & Kahneman, D. J. (2004) · Academy of Management Review · doi:10.2307/20159081 · Cited by 3,583
Clinical versus statistical prediction: A theoretical analysis and a review of the evidence.
Meehl, P. E. (1954) · University of Minnesota Press eBooks · doi:10.1037/11281-000 · Cited by 2,227
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 2,046
Evaluating the predictive performance of habitat models developed using logistic regression
Pearce, J., & Ferrier, S. (2000) · Ecological Modelling · doi:10.1016/s0304-3800(00)00322-7 · Cited by 2,004
Calibration of probabilities: the state of the art to 1980
Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982) · Judgment under Uncertainty · doi:10.1017/cbo9780511809477.023 · Cited by 1,967
Knowing with certainty: The appropriateness of extreme confidence.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.3.4.552 · Cited by 1,428
Research works printed but not linked: 6
- T Fingar Intelligence Analysis: Behavioral and Social Scientific Foundations, eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC), pp. 3–27 (2011).
- , eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC Intelligence Analysis: Behavioral and Social Scientific Foundations, , 338 pp. (2011).
- ; National Research Council Field Evaluation in the Intelligence and Counterintelligence Context: Workshop Summary (Natl Acad Press, Washington, DC, , 114 pp. (2010).
- ; National Research Council Intelligence Analysis for Tomorrow: Advances from the Behavioral and Social Sciences (Natl Acad Press, Washington, DC, , 102 pp. (2011).
- 10.1037/0022-3514.39.5.752 · DOI · Paper page
- HR Arkes, J Kajdasz Intelligence Analysis: Behavioral and Social Scientific Foundations, eds B Fischhoff, C Chauvin (Natl Acad Press, Washington, DC), pp. 143–168 (2011).
Cited by
Sorted by citations
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
“Likewise, transformation rules that improve calibration, an important facet of forecasting skill, have been recently documented (Baron et al., 2014), including in use with intelligence forecasts (Mandel & Barnes, 2014).”
BackgroundSelf-citationVerbal probabilities: Very likely to be somewhat more confusing than numbers
Wintle, B., Fraser, H. S., Wills, B. C., Nicholson, A. E., & Fidler, F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0213522 · Cited by 70
Insights into the accuracy of social scientists’ forecasts of societal change
Collaborative, T. F., Grossmann, I., Rotella, A., Hutcherson, C. A. C., Sharpinskyi, K., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X., Kara-Yakoubian, M., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01517-1 · Cited by 78
Self-citationConfidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
“Current systems rely primarily on qualitative assessments of probabilities and risk (Mandel and Barnes 2014).”
BackgroundRestructuring structured analytic techniques in intelligence
Chang, W., Berdini, E., Mandel, D. R., & Tetlock, P. E. (2017) · Intelligence & National Security · doi:10.1080/02684527.2017.1400230 · Cited by 74
Self-citationImproving the communication of uncertainty in climate science and intelligence analysis
Ho, E., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015) · Behavioral Science & Policy · doi:10.1353/bsp.2015.0015 · Cited by 73
Self-citationCommunicating uncertainty using words and numbers
Dhami, M. K., & Mandel, D. R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.03.002 · Cited by 69
Self-citationFriedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., & Zeckhauser, R. (2018) · International Studies Quarterly · doi:10.1093/isq/sqx078 · Cited by 58
“Thus, if we take a large body of cases where analysts estimated that their conclusions were 70 percent likely to be true, we can see whether those conclusions were actually true roughly 70 percent of the time (Rieber 2004; Tetlock 2005, 13; Mandel and Barnes 2014, 10985).”
BackgroundUse and communication of probabilistic forecasts
Raftery, A. E. (2016) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.11302 · Cited by 76
“In a rare counterexample, Mandel and Barnes [56] found”
BackgroundWisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy
Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S., & Tetlock, P. E. (2024) · Science Advances · doi:10.1126/sciadv.adp1528 · Cited by 117
“For further non-preregistered analyses, we conduct calibration analyses using the Murphy decomposition ( 54 , 55 ) to provide data on how well-calibrated the models are in this context, i.e., how reliably their probability estimates match the fraction of real outcomes.”
Geopolitical Forecasting Skill in Strategic Intelligence
Mandel, D. R., & Barnes, A. (2017) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2055 · Cited by 44
Self-citationFriedman, J. A., Lerner, J. S., & Zeckhauser, R. (2017) · International Organization · doi:10.1017/s0020818317000352 · Cited by 43
Instruction in information structuring improves Bayesian judgment in intelligence analysts
Mandel, D. R. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00387 · Cited by 43
“Although one study has found that strategic intelligence forecasts showed good discrimination and calibration (Mandel and Barnes, 2014), the extent to which analytical judgments are coherent has not been addressed in an intelligence analyst sample.”
BackgroundSelf-citationHow generalizable is good judgment? A multi-task, multi-benchmark study
Mellers, B. A., Baker, J. D., Chen, E., Mandel, D. R., & Tetlock, P. E. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500006240 · Cited by 43
“Mellers et al. (2015b) examined correlations between correspondence a measure of intelligence in forecasters from the geopolitical tournaments discussed earlier.”
BackgroundSelf-citationMathematically aggregating experts’ predictions of possible futures
Hanea, A. M., Wilkinson, D. P., McBride, M. F., Lyon, A., van Ravenzwaaij, D., Thorn, F. S., Gray, C. T., Mandel, D. R., Willcox, A., Gould, E., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0256919 · Cited by 38
“Intelligence analysts might seek predictions of the outcome of an important election, or the price of oil at some point in the future (e.g., [4]).”
Self-citationAccuracy of Intelligence Forecasts From the Intelligence Consumer’s Perspective
Mandel, D. R. (2015) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/2372732215602907 · Cited by 37
“Using the Mandel and Barnes (2014) forecast data set in conjunction with data from a new experiment on the interpretation of the linguistic probabilities used in those forecasts, the present research addressed the challenge of inferring forecast accuracy from the intelligence consumer’s perspective.”
BackgroundMethodResultInfluentialSelf-citationBoosting intelligence analysts’ judgment accuracy: What works, what fails?
Mandel, D. R., Karvetski, C. W., & Dhami, M. K. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500006628 · Cited by 35
“…have shown that analysts’ forecasts tend to be underconfident, and that the calibration of intelligence units can be improved post-judgment through an organizational recalibration process that “extremizes” overly-cautious forecasts (Mandel & Barnes, 2014; Baron et al., 2014; Turner et al., 2014).”
BackgroundSelf-citationImproving information evaluation for intelligence production
Irwin, D., & Mandel, D. R. (2019) · Intelligence & National Security · doi:10.1080/02684527.2019.1569343 · Cited by 35
Self-citationRethinking the training of intelligence analysts
Chang, W., & Tetlock, P. (2016) · Intelligence & National Security · doi:10.1080/02684527.2016.1147164 · Cited by 31
Can cancer researchers accurately judge whether preclinical reports will reproduce?
Benjamin, D., Mandel, D. R., & Kimmelman, J. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002212 · Cited by 32
Self-citationForecasting forecaster accuracy: Contributions of past performance and individual differences
Himmelstein, M., Atanasov, P. D., & Budescu, D. V. (2021) · Judgment and Decision Making · doi:10.1017/s1930297500008597 · Cited by 28
“…true trait level, and E to residual error, is given as: - = ) + (1) This equation is mirrored in the literature on forecasting, which partitions measures of average individual accuracy into skill components perturbed by residual error (Mandel & Barnes, 2014; Murphy, 1973; Wallsten & Budescu, 1983).”
BackgroundKori, B. W., Muathe, S. M., & Maina, S. M. (2020) · International Business Research · doi:10.5539/ibr.v13n10p130 · Cited by 38
Towards an evidence-based approach to communicating uncertainty in intelligence analysis
Dhami, M. K. (2018) · Intelligence & National Security · doi:10.1080/02684527.2017.1394252 · Cited by 20
“32 For instance, Mandel and Barnes (2014) used a widely accepted quantitative procedure involving Brier scores (which can be partitioned into components measuring calibration and discrimination) to assess the accuracy of analysts’ forecasts.”
MethodCorrecting Judgment Correctives in National Security Intelligence
Mandel, D. R., & Tetlock, P. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.02640 · Cited by 24
“, 2014) and in actual strategic intelligence forecasts (Mandel and Barnes, 2014).”
BackgroundSelf-citationThe occasional maverick of analytic tradecraft
Mandel, D. R. (2020) · Intelligence & National Security · doi:10.1080/02684527.2020.1723830 · Cited by 21
Self-citationFailure dynamics of the global risk networkPreprint
Szymanski, B. K., Lin, X., Asztalos, A., & Sreenivasan, S. (2016) · Cited by 22
The wisdom of many in few: Finding individuals who are as wise as the crowd.
Himmelstein, M., Budescu, D. V., & Ho, E. (2023) · Journal of Experimental Psychology General · doi:10.1037/xge0001340 · Cited by 19
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.005 · Cited by 20
Self-citationFailure dynamics of the global risk network
Szymanski, B. K., Lin, X., Asztalos, A., & Sreenivasan, S. (2015) · Scientific Reports · doi:10.1038/srep10998 · Cited by 21
An IRT forecasting model: linking proper scoring rules to item response theory
Bo, Y. E., Budescu, D. V., Lewis, C., Tetlock, P. E., & Mellers, B. A. (2017) · Judgment and Decision Making · doi:10.1017/s1930297500005647 · Cited by 17
“Second, some forecasts relate to outcomes of events whose “ground truth” is hard to determine (e.g., Armstrong, 2001; Lehner, Micheslson, Adelma & Goodman, 2012; Mandel & Barnes, 2014; Tetlock, 2005).”
BackgroundTracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study
Mandel, D. R., & Irwin, D. (2021) · Futures & Foresight Science · doi:10.1002/ffo2.98 · Cited by 16
Self-citationSchneider, G., Hadar, M., & Bosler, N. (2017) · Journal of Peace Research · doi:10.1177/0022343316683437 · Cited by 17
How the “wisdom of the inner crowd” can boost accuracy of confidence judgments.
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2020) · Decision · doi:10.1037/dec0000119 · Cited by 15
Comparative evaluation of the forecast accuracy of analysis reports and a prediction market
Stastny, B. J., & Lehner, P. E. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500007105 · Cited by 14
Till, C., Haverkamp, J., White, D., & Bhaduri, B. (2018) · The Journal of Defense Modeling and Simulation Applications Methodology Technology · doi:10.1177/1548512916679038 · Cited by 9
“They also need to fully understand the degree of uncertainty involved [6, 7] and question anything that they do not understand.”
Mandel, D. R., & Irwin, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0248424 · Cited by 10
“At least, this is suggested by work showing substantial underconfidence in intelligence analysts’ probabilistic forecasts [31,32].”
BackgroundSelf-citationTetlock, P., & Mellers, B. (2014) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.1412524111 · Cited by 10
Strategic intelligence as an art and a science: creating and using conceptual frameworks
Shapira, I. (2019) · Intelligence & National Security · doi:10.1080/02684527.2019.1681135 · Cited by 9
Mandel, D. R. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500004320 · Cited by 9
“I will note, however, that efforts could have been made to establish inter-rater reliability on ground truth (Mandel & Barnes, 2014), which is important because the “reality” used to score forecast accuracy was judged by subject-matter experts.”
BackgroundInfluentialSelf-citationPerformance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
Collins, R. N., Mandel, D. R., & Budescu, D. V. (2023) · International series in management science/operations research/International series in operations research & management science · doi:10.1007/978-3-031-30085-1_7 · Cited by 16
Self-citationWeighted Brier score decompositions for topically heterogenous forecasting tournaments
Merkle, E. C., & Hartman, R. (2018) · Judgment and Decision Making · doi:10.1017/s1930297500007099 · Cited by 11
Using scenarios to forecast outcomes of a refugee crisis
Wicke, L., Dhami, M. K., Önkal, D., & Belton, I. (2019) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2019.05.017 · Cited by 8
Learning to Predict Proliferation
Miller, N. L. (2021) · International Organization · doi:10.1017/s0020818321000345 · Cited by 7
“Friedman 2019; Friedman, Lerner, and Zeckhauser 2017; Friedman and Zeckhauser 2016; Mandel and Barnes 2014; and Tetlock 2005.”
BackgroundResultWilcox, J. E., & Mandel, D. R. (2024) · Intelligence & National Security · doi:10.1080/02684527.2024.2304934 · Cited by 14
“90 If this is the case, then it may be premature to recommend ACH to analysts, regardless of whether it provides benefits to other populations (That said, to be fair, we do not know how many of the analysts in Mandel and Barnes (2014) study used ACH or may have been impacted by it somehow.”
Self-citationUse and Communication of Probabilistic ForecastsPreprint
Raftery, A. E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.4812 · Cited by 7
“In a rare counterexample, Mandel and Barnes [56] found”
BackgroundAssessing public forecasts to encourage accountability: The case of MIT’s Technology Review
Funk, J. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183038 · Cited by 3
Credences and Trustworthiness: a Calibrationist Account
Wilcox, J. (2024) · Review of Philosophy and Psychology · doi:10.1007/s13164-024-00724-1 · Cited by 3
“In general, then, miscalibration currently appears to be pervasive—although there are also some notable exceptions (Mandel and Barnes 2014, 2018; Mandel and Irwin 2021).”
Adopting and improving a new forecasting paradigm
Speigel, I. (2021) · Intelligence & National Security · doi:10.1080/02684527.2021.1946955 · Cited by 6
Automated calibration training for forecasters
Stone, E. R., Luu, J., Costello, C., & Somerville, A. H. (2023) · Journal of Behavioral Decision Making · doi:10.1002/bdm.2334 · Cited by 6
Crowdsourcing Accurately and Robustly Predicts Supreme Court DecisionsPreprint
Katz, D. M., Bommarito, M. J., & Blackman, J. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3085710 · Cited by 10
Weighted Brier score decompositions for topically heterogenous forecasting tournamentsPreprint
Merkle, E. C. (2018) · doi:10.31219/osf.io/p6wk5 · Cited by 6
“These restrictive requirements are frequently violated in applied forecasting activities and competitions (see, e.g., Tetlock, 2005; Mandel & Barnes, 2014).”
BackgroundScience and Technology Forecasting
Rejeski, D. W., Pauwels, E., & Koo, J. (2016) · Springer eBooks · doi:10.1007/978-3-319-07052-0_13 · Cited by 5
How the “Wisdom of the Inner Crowd” Can Boost Accuracy of Confidence JudgmentsPreprint
Litvinova, A., Herzog, S. M., Kall, A. A., Pleskac, T. J., & Hertwig, R. (2019) · doi:10.31234/osf.io/ngweh · Cited by 5
The effect of calibration training on the calibration of intelligence analysts' judgments
Kelly, M. O., & Mandel, D. R. (2024) · Applied Cognitive Psychology · doi:10.1002/acp.4236 · Cited by 4
Self-citationIsaksen, B. G. M., & McNaught, K. R. (2019) · Journal of Risk Research · doi:10.1080/13669877.2018.1474245 · Cited by 4
“Notable exceptions, concerning the communication of uncertainty in particular, are provided by Dhami (2017), Ho et al. (2015) and Mandel and Barnes (2014).”
BackgroundA mathematical model for evaluation of intelligence products value
Korać, D., & Cica, D. (2018) · Journal of Information and Optimization Sciences · doi:10.1080/02522667.2018.1427027 · Cited by 4
“Also, there is a significant number of authors such as Mandel and Barnes [32] that have dealt with a similar approach to the development of the communication standard that is used in generating the intelligence evaluations.”
BackgroundFriedman, J. A., Lerner, J. S., & Zeckhauser, R. (2015) · SSRN Electronic Journal · doi:10.2139/ssrn.2782598 · Cited by 2
Intelligence, Science and the Ignorance HypothesisPreprint
Mandel, D. R. (2021) · doi:10.31234/osf.io/m2fn8 · Cited by 4
Self-citationInsights into accuracy of social scientists' forecasts of societal changePreprint
Grossmann, I., Rotella, A., Hutcherson, C. A. C., Sharpinskyi, C., Varnum, M. E. W., Achter, S., Dhami, M. K., Guo, X., Kara-Yakoubian, M., Mandel, D. R., et al. (2022) · doi:10.31234/osf.io/wdxsb · Cited by 4
Self-citationStrategic Intelligence: A Semantic Leadership Perspective
Paiuc, D., Săniuță, A., & Parincu, A. M. T. (2024) · Encyclopedia · doi:10.3390/encyclopedia4020050 · Cited by 4
“Such intelligence can help leaders and major stakeholders anticipate future events, avoid strategic surprises, and make informed decisions [5].”
BackgroundHow do we know if an intelligence analytic product is good?
Riehle, K. P. (2025) · Intelligence & National Security · doi:10.1080/02684527.2025.2468051 · Cited by 3
Science and Technology Forecasting
Rejeski, D. W., Pauwels, E., & Koo, J. (2014) · Springer eBooks · doi:10.1007/978-3-319-04033-2_13-1 · Cited by 3
Towards a data-driven military. A multi-disciplinary perspective
Tinga, T., Homborg, A. M., Rijsdijk, C., Pijpers, P. B. M. J., Voskuijl, M., & Beeres, R. J. M. (2023) · Leiden University Press eBooks · doi:10.24415/9789087284084 · Cited by 3
Terrorism-Centric Behavior Recognition and Adversarial Threat Forecasting
Regens, J. L., Mould, N. A., Jensen, C. J., & Edger, D. N. (2016) · International Journal of Intelligence and CounterIntelligence · doi:10.1080/08850607.2015.1083317 · Cited by 3
Innovation Capabilities and the role of Strategic Intelligence
Reichert, F. M., Janissek-Muniz, R., Heinze, M., Cainelli, A., & Zawislak, P. A. (2020) · Journal of the Association for Information Systems · Cited by 3
The Occasional Maverick of Analytic TradecraftPreprint
Mandel, D. R. (2019) · doi:10.31234/osf.io/atjve · Cited by 3
Self-citationWhen alternative hypotheses shape your beliefs: Context effects in probability judgmentsPreprint
Cai, X., & Pleskac, T. J. (2021) · Cognition · doi:10.1016/j.cognition.2022.105306 · Cited by 3
Calibration Feedback With the Practical Scoring Rule Does Not Improve Calibration of Confidence
Martin, M. J., & Mandel, D. R. (2024) · Futures & Foresight Science · doi:10.1002/ffo2.199 · Cited by 2
Self-citationCharacterizing, Analyzing, and Communicating Uncertainty1
Morgan, M. G. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316882665.009 · Cited by 2
Predicting Known Terrorist Event Outcomes and the Illusion of Self‐Assessed Expertise*
Regens, J. L., Mould, N. A., & Foster, C. (2019) · Social Science Quarterly · doi:10.1111/ssqu.12601 · Cited by 2
Mandel, D. R., & Irwin, D. (2022) · doi:10.31234/osf.io/7hx2c · Cited by 2
Self-citationStrategic Intelligence: A Semantic Leadership PerspectivePreprint
Paiuc, D., Săniuță, A., & Parincu, A. M. T. (2024) · Preprints.org · doi:10.20944/preprints202404.0690.v1 · Cited by 2
Morgan, M. G. (2017) · Cambridge University Press eBooks · doi:10.1017/9781316882665.010 · Cited by 1
Applying Information Theory to Validate Commanders' Critical Information Requirements
Timms, M. A. C., Mandel, D. R., & Nelson, J. D. (2024) · Chapman & Hall/CRC eBooks · doi:10.1201/9781003396307-18 · Cited by 1
Self-citationA Roadmap for the Future Research of the White Stork Ciconia ciconia Population in Poland
Tryjanowski, P., Białas, J. T., Jankowiak, Ł., Jerzak, L., Mielczarek, P., Nowak, M. K., Profus, P., Siekiera, J., Tobółka, M., Walasz, K., et al. (2024) · Polish Journal of Ecology · doi:10.3161/15052249pje2024.72.1.004 · Cited by 1
“2012, Mandel and Barnes 2014).”
Sajduk, B. (2022) · Politeja · doi:10.12797/politeja.19.2022.77.05 · Cited by 1
Al-Janabi, W. R., & Kurdi, B. H. A. (2023) · Studies in computational intelligence · doi:10.1007/978-3-031-12382-5_88 · Cited by 1
Friedman, J. A. (2024) · Critical Review · doi:10.1080/08913811.2024.2433878 · Cited by 1
“Mandel and Barnes (2014) and Miller (2022) have additionally demonstrated that political forecasts made by intelligence analysts carry considerable predictive validity.”
Studies past and future of the past and future: Commentary on Schoemaker 2020
Mandel, D. R. (2020) · Futures & Foresight Science · doi:10.1002/ffo2.39 · Cited by 1
Self-citationPredicting the future of society
Salganik, M. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01535-7 · Cited by 1
The effectiveness of risk assessments in informing decision makers
Peace, C. (2019) · doi:10.26686/wgtn.17142020 · Cited by 1
Applying Information Theory to Validate Commanders’ Critical Information RequirementsPreprint
Timms, M. A. C., Mandel, D. R., & Nelson, J. D. (2023) · doi:10.31234/osf.io/n8xka · Cited by 1
“For instance, while bias-awareness training for analysts often focuses on the problem of overconfidence, there is evidence that analysts’ anticipatory estimates are often underconfident (Mandel & Barnes, 2014, 2018).”
Self-citationWhen expert predictions failPreprint
Grossmann, I., Varnum, M. E. W., Hutcherson, C. A. C., & Mandel, D. R. (2023) · doi:10.31234/osf.io/my24e · Cited by 1
Self-citationEffect of Calibration Training on the Calibration of Intelligence Analysts’ JudgmentsPreprint
Kelly, M. O., & Mandel, D. R. (2023) · doi:10.31234/osf.io/p82nx · Cited by 1
Self-citationHashem, Y. R., & Al-Sheikli, A. I. (2021) · Journal of Economics and Administrative Sciences · doi:10.33095/jeas.v27i125.2064
Some Widely Used Analysis Tools and Topics
Morgan, M. G. (2017)
UK and US policies for communicating probability in intelligence analysis: A reviewPreprint
Dhami, M. K., & Mandel, D. R. (2020) · doi:10.31234/osf.io/kuyhb
Self-citationMandel, D. R., & Irwin, D. (2020) · doi:10.31234/osf.io/6atw5
Self-citationWhat We Think: The Accuracy of Our Judgments
Wilcox, J. E. (2022) · SpringerBriefs in psychology · doi:10.1007/978-3-031-19205-0_3
A response to Mandel’s (2019) commentary on
Lehner, P., & Stastny, B. J. (2019) · Judgment and Decision Making · doi:10.1017/s1930297500004915
“And importantly, as shown by Mandel and Barnes (2014), it is very feasible to track and measure forecasting accuracy.”
BackgroundCarr, P., Ebersole, E. K., & Sturtz, J. L. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3234328
“For calibration of confidence, a number of scoring methods are available [17], [45].”
BackgroundMethodNarrative AbductionPreprint
van Gelder, T., & Wilcox, J. E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4790721
Duke, M. C., Bolsinger, D., & Landon-Murray, M. (2024) · Intelligence & National Security · doi:10.1080/02684527.2024.2432771
CIA’s evaluation of analytic quality: can it fortify tradecraft?
Levine, R. L., & George, R. Z. (2025) · Intelligence & National Security · doi:10.1080/02684527.2024.2432770
Harper, E. (2025) · Critique Open Research & Review · doi:10.55640/corr-v03i03-02
Algorithm Perception When Using Threat Intelligence in Vulnerability Risk Assessment
van Gerwen, S., Papotti, A., Tuma, K., & Massacci, F. (2026) · Risk Analysis · doi:10.1111/risa.70178 · Cited by 1
Mandel, D. R. (2026) · Intelligence & National Security · doi:10.1080/02684527.2026.2635695
Self-citationPhillips, L. D. (2026) · doi:10.1017/9781009622899.024
Calibration of Retrospective and Prospective Life Satisfaction Judgments Around the World
Mandel, D. R., Meyers, E. A., & Omorogbe, P. (2026) · Futures & Foresight Science · doi:10.1002/ffo2.70045
Self-citationMouynes, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7114978
Metadata
- Authors
- Mandel, D. R., & Barnes, A.
- Affiliations
- Defence Research and Development Canada (CA)
- Privy Council Office (CA)
- ORCID
- Mandel, David R.
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2014-07-14
- Volume
- 111
- Issue
- 30
- Pages
- 10984-10989
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC4121776
- OpenAlex
- W2085900183
- Cited by
- 101
- References
- 44
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 12 · 2023: 12 · 2022: 5 · 2021: 9 · 2020: 7 · 2019: 11 · 2018: 7 · 2017: 13 · 2016: 8 · 2015: 5 · 2014: 3
- FWCI
- 62.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada (249537-07)
- Topics
- Social Sciences › Political Science and International Relations › Intelligence, Security, War Strategy
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Social Sciences › Business, Management and Accounting › Strategy and Management › Competitive and Knowledge Intelligence
- Keywords
- strategic intelligence, forecast accuracy, forecast calibration, underconfidence, analyst expertise, forecast informativeness, intelligence reports, policy decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
