“Improving Intelligence Analysis with Decision Science.”
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Intelligence analysis plays a vital role in policy decision making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabilistic forecasting and uncertainty communication, drawing attention to findings that could be used to reform intelligence processes and contribute to more effective intelligence oversight. We recommend that the intelligence community (IC) regularly and quantitatively monitor its forecasting accuracy to better understand how well it is achieving its functions. We also recommend that the IC use decision science to improve these functions (namely, forecasting and communication of intelligence estimates made under conditions of uncertainty). In the case of forecasting, decision research offers suggestions for improvement that involve interventions on data (e.g., transforming forecasts to debias them) and behavior (e.g., via selection, training, and effective team structuring). In the case of uncertainty communication, the literature suggests that current intelligence procedures, which emphasize the use of verbal probabilities, are ineffective. The IC should, therefore, leverage research that points to ways in which verbal probability use may be improved as well as exploring the use of numerical probabilities wherever feasible.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Expert political judgment: how good is it? how can we know?
(2006) · Choice Reviews Online · doi:10.5860/choice.43-3634 · Cited by 1,260 · cited 1×
“Second, political experts have been shown to be overconfident in their forecasts of geopolitical events, and they are easily outperformed by all but the most basic statistical models (Tetlock, 2005).”
BackgroundSelf-citationSuperforecasting: the art and science of prediction
(2016) · Choice Reviews Online · doi:10.5860/choice.196400 · Cited by 919 · cited 1×
“Finally, assignment of the best forecasters into so-called “superteams”, allowing elite performers to interact online with each other, yields additional improvements to forecasting accuracy (Mellers et al., 2015b; Tetlock & Gardner, 2015).”
BackgroundSwets, J. A. (1986) · Psychological Bulletin · doi:10.1037//0033-2909.99.2.181 · Cited by 508
Verbal probabilities: Ambiguous, context-dependent, or both?
Brun, W., & Teigen, K. H. (1988) · Organizational Behavior and Human Decision Processes · doi:10.1016/0749-5978(88)90036-2 · Cited by 334 · cited 1×
“First, people receiving communications about uncertain estimates prefer them to be expressed numerically even though communicators prefer to use words to convey uncertainty (Brun & Teigen, 1988; Murphy, Lichtenstein, Fischhoff, & Winkler, 1980; Wallsten, Budescu, Zwick, & Kemp, 1993).”
BackgroundOffice of the Director of National Intelligence
Senate, C. (2016) · govinfo.gov · Cited by 322
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312 · cited 2×
“Forecasting can also be improved by using well-coordinated teamwork that allows forecasters to share information, debate rationales, and motivate each other to perform well (Mellers et al., 2014).”
BackgroundSelf-citationThe interpretation of IPCC probabilistic statements around the world
Budescu, D. V., Por, H.-H., Broomell, S. B., & Smithson, M. (2014) · Nature Climate Change · doi:10.1038/nclimate2194 · Cited by 291
Preferences and reasons for communicating probabilistic information in verbal or numerical terms
Wallsten, T. S., Budescu, D. V., Zwick, R., & Kemp, S. M. (1993) · Bulletin of the Psychonomic Society · doi:10.3758/bf03334162 · Cited by 291
Decisions based on numerically and verbally expressed uncertainties.
Budescu, D. V., Weinberg, S., & Wallsten, T. S. (1988) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037//0096-1523.14.2.281 · Cited by 252 · cited 1×
“Third, and more problematic, individuals vary greatly in the ranges and best estimates they assign to probability terms (Budescu, Weinberg, & Wallsten, 1988; Dhami & Wallsten, 2005).”
BackgroundIdentifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions
Mellers, B., Stone, E., Murray, T., Minster, A., Rohrbaugh, N., Bishop, M., Chen, E., Baker, J., Hou, Y., Horowitz, M., et al. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615577794 · Cited by 236 · cited 2×
“Finally, assignment of the best forecasters into so-called “superteams”, allowing elite performers to interact online with each other, yields additional improvements to forecasting accuracy (Mellers et al., 2015b; Tetlock & Gardner, 2015).”
BackgroundSelf-citation
Research works printed but not linked: 9
- 10.1037/0096-1523.14.2.281 · doi:10.1037/0096-1523.14.2.281 · Paper page
- Dhami M. K., 2013, Understanding and communicating uncertainty in intelligence analysis
- Ho E., Behavioral Science & Policy
- 10.1037/0033-2909.114.1.162 · doi:10.1037/0033-2909.114.1.162 · Paper page
- Mandel D. R., 2014, A quantitative assessment of the quality of strategic intelligence forecasts
- National Research Council, 2011, Intelligence analysis for tomorrow: Advances from the behavioral and social sciences
- Pool R., 2010, Field evaluation in the intelligence and counterintelligence context: Workshop summary
- 10.1037/0033-2909.99.2.181 · doi:10.1037/0033-2909.99.2.181 · Paper page
- 10.1037/0033-2909.110.3.611 · doi:10.1037/0033-2909.110.3.611 · Paper page
Cited by
Complex Problem Solving: What It Is and What It Is Not
Dörner, D., & Funke, J. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01153 · Cited by 269
“…life-streaming (Wendt, 2017), a medium in which streamers generate a video stream of think-aloud data in computer-gaming; political decision-making (Dhami et al., 2015) that demonstrates error-proneness in groups; historical case studies (Dörner and Güss, 2011) that give insights into the thinking…”
BackgroundFischhoff, B., & Broomell, S. B. (2020) · Annual Review of Psychology · doi:10.1146/annurev-psych-010419-050747 · Cited by 163
“Given the central role of expert judgment in policy analyses (see below), these results have important practical implications (see also Dhami et al. 2015, Morgan 2017, O’Hagan et al. 2006).”
BackgroundRoutledge Handbook of Bounded Rationality
Mercier, H., & Sperber, D. (2020) · Routledge eBooks · doi:10.4324/9781315658353 · Cited by 116
Felin, T., Lakhani, K. R., & Tushman, M. L. (2017) · Strategic Organization · doi:10.1177/1476127017706610 · Cited by 106
“While expert judgment of course remains important in many settings (cf. Mollick and Nanda, 2015), even naive participants in crowd-related activities have been shown, particularly in the aggregate, to beat experts (Dhami et al., 2015; Tetlock and Gardner, 2016).”
BackgroundImproving the communication of uncertainty in climate science and intelligence analysis
Ho, E. H., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015) · Behavioral Science & Policy · doi:10.1177/237946151500100206 · Cited by 73
Self-citation
Metadata
- Authors
- Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E.
- Affiliations
- Middlesex University London (GB)
- Defence Research and Development Canada (CA)
- University of Pennsylvania (US)
- Corresponding author
- Dhami, Mandeep K.
- Venue
- Perspectives on Psychological Science
- Publisher
- SAGE Publications
- Published
- 2015-11
- Volume
- 10
- Issue
- 6
- Pages
- 753-757
- ISSN
- 1745-6916 (ISSN-L)
- 1745-6924
- OpenAlex
- W2338024782
- Cited by
- 104
- References
- 29
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 11 · 2023: 13 · 2022: 8 · 2021: 8 · 2020: 18 · 2019: 14 · 2018: 6 · 2017: 11 · 2016: 4 · 2015: 4
- FWCI
- 54.01
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Perspectives on Psychological Science (publishedVersion)
- PubMed (publishedVersion)
- ScholarlyCommons (University of Pennsylvania) (submittedVersion)
- Middlesex University Research Repository (Middlesex University Of London) (submittedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Intelligence, Security, War Strategy
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Forecasting Techniques and Applications
- Keywords
- intelligence analysis, decision science, probabilistic forecasting, uncertainty communication, intelligence community, verbal probability, numerical probability, intelligence oversight, intelligence estimates
- MeSH terms
- Decision Making, Forecasting, Humans, Probability, Public Policy, United States, Uncertainty, United States Department of Defense
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: “Improving Intelligence Analysis with Decision Science.”. https://app.cochat.ai/papers/10.1177/1745691615598511
- The paper
- Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015). “Improving Intelligence Analysis with Decision Science.”. Perspectives on Psychological Science. https://doi.org/10.1177/1745691615598511
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.