Intelligent management of intelligence agencies: Beyond accountability ping-pong.
The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The intelligence community (IC) is asked to predict outcomes that may often be inherently unpredictable-and is blamed for the inevitable forecasting failures, be they false positives or false negatives. To move beyond blame games of accountability ping-pong that incentivize bureaucratic symbolism over substantive reform, it is necessary to reach bipartisan agreements on performance indicators that are transparent enough to reassure clashing elites (to whom the IC must answer) that estimates have not been politicized. Establishing such transideological credibility requires (a) developing accuracy metrics for decoupling probability and value judgments; (b) using the resulting metrics as criterion variables in validity tests of the IC's selection, training, and incentive systems; and (c) institutionalizing adversarial collaborations that conduct level-playing-field tests of clashing perspectives.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The case for motivated reasoning.
Kunda, Z. (1990) · Psychological Bulletin · doi:10.1037/0033-2909.108.3.480 · Cited by 9,022
Kluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882
Perception and Misperception in International Politics
Jervis, R. (2017) · doi:10.1515/9781400885114 · Cited by 3,199
Conditions for intuitive expertise: A failure to disagree.
Kahneman, D., & Klein, G. (2009) · American Psychologist · doi:10.1037/a0016755 · Cited by 2,443
Accounting for the effects of accountability.
Lerner, J. S., & Tetlock, P. E. (1999) · Psychological Bulletin · doi:10.1037/0033-2909.125.2.255 · Cited by 2,383 · cited 1×
“Ping-pong accountability pressures the IC to focus its collective analytic resources on defensive bolstering (justifying previous judgments) and strategic attitude shifting (providing the answers policy makers want; Lerner & Tetlock, 1999).”
BackgroundSelf-citationTetlock, P. E., Kristel, O. V., Elson, S. B., Green, M. C., & Lerner, J. S. (2000) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.78.5.853 · Cited by 1,400
Self-citationThe Impact of Accountability on Judgment and Choice: Toward A Social Contingency Model
Tetlock, P. E. (1992) · Advances in Experimental Social Psychology · doi:10.1016/s0065-2601(08)60287-7 · Cited by 926
Self-citationOutcome Bias in Decision Evaluation
Baron, J., & Hershey, J. C. (1988) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.54.4.569 · Cited by 873 · cited 1×
“tapraid4/z2n-amepsy/z2n-amepsy/z2n00611/z2n3783d11z xppws Sϭ1 7/8/11 6:43 Art: 2011-2363 4 rather than by how rigorously the decisions were made (Baron & Hershey, 1988).”
BackgroundVarieties of Confirmation Bias
Klayman, J. (1995) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60315-1 · Cited by 755
Simultaneous over- and underconfidence: the role of error in judgment processes
Erev, I., Wallsten, T. S., & Budescu, D. V. (1994) · Psychological Review · doi:10.1037/0033-295x.101.3.519 · Cited by 644
Cited by
Psychological Strategies for Winning a Geopolitical Forecasting Tournament
Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., et al. (2014) · Psychological Science · doi:10.1177/0956797614524255 · Cited by 312
Self-citationBalancing on the Creative Highwire: Forecasting the Success of Novel Ideas in Organizations
Berg, J. M. (2016) · Administrative Science Quarterly · doi:10.1177/0001839216642211 · Cited by 252
The individual risk assessment of terrorism.
Monahan, J. (2012) · Psychology, Public Policy, and Law · doi:10.1037/a0025792 · Cited by 220
“Improving Intelligence Analysis with Decision Science.”
Dhami, M. K., Mandel, D. R., Mellers, B. A., & Tetlock, P. E. (2015) · Perspectives on Psychological Science · doi:10.1177/1745691615598511 · Cited by 104
“Such an approach would address the aforementioned key challenges and also strengthen the IC’s accountability processes, enabling it to better protect itself from the adverse consequences of blame-games that ensue following significant errors (Tetlock & Mellers, 2011).”
BackgroundSelf-citation“Accuracy of Forecasts in Strategic Intelligence.”
Mandel, D. R., & Barnes, A. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1406138111 · Cited by 101
Confidence Calibration in a Multiyear Geopolitical Forecasting Competition
Moore, D. A., Swift, S. A., Minster, A., Mellers, B., Ungar, L., Tetlock, P., Yang, H. H. J., & Tenney, E. R. (2017) · Management Science · doi:10.1287/mnsc.2016.2525 · Cited by 74
“Intelligence analysts aware of their accountability to a political establishment prone to seeking blame when things go wrong may be skittish about making their forecasts clear enough to be tested and scored (Tetlock and Gardner 2015, Tetlock and Mellers 2011).”
BackgroundSelf-citationFriedman, J. A., Baker, J. D., Mellers, B. A., Tetlock, P. E., & Zeckhauser, R. (2018) · International Studies Quarterly · doi:10.1093/isq/sqx078 · Cited by 58
Self-citation
Metadata
- Authors
- Tetlock, P. E., & Mellers, B. A.
- Venue
- American Psychologist
- Published
- 2011-09
- Volume
- 66
- Issue
- 6
- Pages
- 542-554
- ISSN
- 0003-066X (ISSN-L)
- 1935-990X
- OpenAlex
- W2074932245
- Cited by
- 71
- References
- 15
- FWCI
- 40.08
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.