Seven deadly sins of contemporary quantitative political analysis
Open accessThe DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
A combination of technological change, methodological drift and a certain degree of intellectual sloth, particularly with respect to philosophy of science, has allowed contemporary quantitative political analysis to accumulate a series of dysfunctional habits that have rendered much of contemporary research more or less meaningless. I identify these ‘seven deadly sins’ as: Garbage can models that ignore the effects of collinearity; Pre-scientific explanation in the absence of prediction; Excessive reanalysis of a small number of datasets; Using complex methods without understanding the underlying assumptions; Interpreting frequentist statistics as if they were Bayesian; A linear statistical monoculture that fails to consider alternative structures; Confusing statistical controls and experimental controls. The answer to these problems is not to abandon quantitative approaches, but rather engage in solid, thoughtful, original work driven by an appreciation of both theory and data. The article closes with suggestions for changes in current practice that might serve to ameliorate some of these problems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
The Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170
(2011) · Cited by 14,317
Pattern Recognition and Machine Learning
(2007) · Kybernetes · doi:10.1108/03684920710743466 · Cited by 8,265
Designing Social Inquiry: Scientific Inference in Qualitative Research
King, G., Keohane, R. O., & Verba, S. (1994) · Princeton University Press · doi:10.1515/9781400821211 · Cited by 7,225
Hainmueller, J. (2012) · Political Analysis · doi:10.1093/pan/mpr025 · Cited by 6,026
Chiang, L. H., Russell, E. L., & Braatz, R. D. (2001) · Advanced Textbooks in Control and Signal Processing · doi:10.1007/978-1-4471-0347-9_3 · Cited by 5,003
Hoeting, J. A., Madigan, D., Raftery, A. E., & Volinsky, C. T. (1999) · Statistical Science · doi:10.1214/ss/1009212519 · Cited by 4,118 · cited 1×
“Bayesian approaches are common in the consistently top-ranked – by impact factor – Political Analysis (for example Hoeting et al., 1999; Lock & Gelman, 2010; Montgomery & Nyhan, 2010; Alvarez, Bailey & Katz, 2011; Grimmer, 2011; Montgomery, Hollenbach & Ward, 2012) but not in more massmarket venues…”
BackgroundCausal Inference without Balance Checking: Coarsened Exact Matching
Iacus, S. M., King, G., & Porro, G. (2012) · Political Analysis · doi:10.1093/pan/mpr013 · Cited by 3,563
Studies in the Logic of Explanation
Hempel, C. G., & Oppenheim, P. (1948) · Philosophy of Science · doi:10.1086/286983 · Cited by 3,174
Main Trends in Recent Philosophy: Two Dogmas of Empiricism
Quine, W. V. (1951) · The Philosophical Review · doi:10.2307/2181906 · Cited by 3,029
van Orman Quine, W. (1976) · Can Theories be Refuted? · doi:10.1007/978-94-010-1863-0_2 · Cited by 2,709
Taking Time Seriously: Time-Series-Cross-Section Analysis with a Binary Dependent Variable
Beck, N., Katz, J. N., & Tucker, R. (1998) · American Journal of Political Science · doi:10.2307/2991857 · Cited by 2,471
The Statistical Analysis of Time Series.
Anderson, T. W. (1995) · Biometrics · doi:10.2307/2532976 · Cited by 2,017
The robust beauty of improper linear models in decision making.
Dawes, R. M. (1979) · American Psychologist · doi:10.1037/0003-066x.34.7.571 · Cited by 1,966
The Oxford Handbook of Political Methodology
(2008) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199286546.001.0001 · Cited by 1,362
The Statistical Analysis of Time Series.
Priestley, M. B., & Anderson, T. W. (1972) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344686 · Cited by 1,269
Fox, J. (1991) · doi:10.4135/9781412985604 · Cited by 1,008
Statistical Models: Theory and Practice
(2006) · Technometrics · doi:10.1198/tech.2006.s403 · Cited by 991
A Global Model for Forecasting Political Instability
Goldstone, J. A., Bates, R. H., Epstein, D. L., Gurr, T. R., Lustik, M. B., Marshall, M. G., Ulfelder, J., & Woodward, M. (2010) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00426.x · Cited by 861
Huff, D., & Geis, I. (1954) · Cited by 762
The cult of statistical significance: how the standard error costs us jobs, justice, and lives
(2008) · Choice Reviews Online · doi:10.5860/choice.46-0372 · Cited by 713
Statistical Models: Theory and Practice
Freedman, D. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9781139165495 · Cited by 635
Statistics of Deadly Quarrels.
N., M., & Richardson, L. F. (1961) · Journal of the American Statistical Association · doi:10.2307/2282320 · Cited by 621
Bayesian Methods: A Social and Behavioral Sciences Approach
Gill, J. (2002) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420057478 · Cited by 579
Statistics of Deadly Quarrels, 1809-1949: Version 1
Richardson, L. F. (1984) · ICPSR Data Holdings · doi:10.3886/icpsr05407.v1 · Cited by 572
Assessing the Liberal Peace with Alternative Specifications: Trade Still Reduces Conflict
Oneal, J. R., & Russett, B. (1999) · Journal of Peace Research · doi:10.1177/0022343399036004003 · Cited by 534
The Phantom Menace: Omitted Variable Bias in Econometric Research
Clarke, K. A. (2005) · Conflict Management and Peace Science · doi:10.1080/07388940500339183 · Cited by 484
The perils of policy by p-value: Predicting civil conflicts
Ward, M. D., Greenhill, B. D., & Bakke, K. M. (2010) · Journal of Peace Research · doi:10.1177/0022343309356491 · Cited by 476
Fashionable nonsense: postmodern intellectuals' abuse of science
(1999) · Choice Reviews Online · doi:10.5860/choice.37-0896 · Cited by 469
T OWARD A N EW P OLITICAL M ETHODOLOGY: Microfoundations and ART
Achen, C. H. (2002) · Annual Review of Political Science · doi:10.1146/annurev.polisci.5.112801.080943 · Cited by 452
Rethinking Social Inquiry: Diverse Tools, Shared Standards
(2010) · doi:10.5040/9798216385530 · Cited by 405
Grasping the Democratic Peace: Principles for a Post-Cold War World
Russett, B. (1994) · Princeton University Press eBooks · doi:10.2307/j.ctt7rqf6 · Cited by 358
(2011) · Choice Reviews Online · doi:10.5860/choice.49-0826 · Cited by 340
The Insignificance of Null Hypothesis Significance Testing
Gill, J. (1999) · Political Research Quarterly · doi:10.1177/106591299905200309 · Cited by 338
Regression Diagnostics: An Introduction.
Trustrum, K., & Fox, J. (1993) · The Statistician · doi:10.2307/2348998 · Cited by 338
State Failure Task Force Report: Phase II Findings
Esty, D. C., Goldstone, J. Α., Gurr, T. R., Harff, B., Levy, M. A., Dabelko, G. D., Surko, P. T., & Unger, A. N. (1999) · Cited by 332
Expert Political Judgment: How Good Is It? How Can We Know?
Tetlock, P. E. (2018) · Princeton University Press eBooks · doi:10.1515/9781400888818 · Cited by 277
Improving Forecasts of State Failure
King, G., & Zeng, L. (2001) · World Politics · doi:10.1017/s0043887100019171 · Cited by 271
Crisis Early Warning and Decision Support: Contemporary Approaches and Thoughts on Future Research
O’Brien, S. P. (2010) · International Studies Review · doi:10.1111/j.1468-2486.2009.00914.x · Cited by 268
(1994) · Long Range Planning · doi:10.1016/0024-6301(94)90248-8 · Cited by 267
Expert Political Judgment: How Good Is It? How Can We Know?
Tetlock, P. E. (2009) · Princeton University Press eBooks · doi:10.1515/9781400830312 · Cited by 232
Some Remarks on the Typological Procedures in Social Research
Lazarsfeld, P. F., & Sozialforschung, I. F. (1937) · Zeitschrift für Sozialforschung · doi:10.5840/zfs193761137 · Cited by 231
The Statistical Analysis of Time Series
Anderson, T. W. (1994) · Wiley Series in Probability and Statistics · doi:10.1002/9781118186428 · Cited by 199
The Significance Test Controversy: A Reader
Henkel, R. E. (2006) · CERN Document Server (European Organization for Nuclear Research) · Cited by 190
Bayesian Model Averaging: Theoretical Developments and Practical Applications
Montgomery, J. M., & Nyhan, B. (2010) · Political Analysis · doi:10.1093/pan/mpq001 · Cited by 180
McKeown, T. J. (1999) · International Organization · doi:10.1162/002081899550841 · Cited by 179
Real Time, Time Series Forecasting of Inter- and Intra-State Political Conflict
Brandt, P. T., Freeman, J. R., & Schrodt, P. A. (2011) · Conflict Management and Peace Science · doi:10.1177/0738894210388125 · Cited by 167
Self-citation(1991) · Choice Reviews Online · doi:10.5860/choice.28-3527 · Cited by 166
The Significance Test Controversy: A Reader.
Cushing, R. G., Morrison, D. E., & Henckel, R. E. (1973) · Contemporary Sociology · doi:10.2307/2064182 · Cited by 165
Predicting Conflict in Space and Time
Weidmann, N. B., & Ward, M. D. (2010) · Journal of Conflict Resolution · doi:10.1177/0022002710371669 · Cited by 162
Improving Predictions using Ensemble Bayesian Model Averaging
Montgomery, J. M., Hollenbach, F. M., & Ward, M. D. (2012) · Political Analysis · doi:10.1093/pan/mps002 · Cited by 158
Preventive measures : building risk assessment and crisis early warning systems
Davies, J. L., & Gurr, T. R. (1998) · Medical Entomology and Zoology · Cited by 150
Fashionable Nonsense: Postmodern Intellectuals' Abuse of Science
Sokal, A. D., Bricmont, J., & Mermin, N. D. (1999) · Physics Today · doi:10.1063/1.882506 · Cited by 149
An Introduction to Bayesian Inference via Variational Approximations
Grimmer, J. (2010) · Political Analysis · doi:10.1093/pan/mpq027 · Cited by 134
Staff, S. (2011) · Science · doi:10.1126/science.331.6018.692 · Cited by 123
Orme‐Johnson, D. W., Alexander, C. N., Davies, J. L., Chandler, H. M., & Larimore, W. E. (1988) · Journal of Conflict Resolution · doi:10.1177/0022002788032004009 · Cited by 108
Statistical Models and Causal Inference: A Dialogue with the Social Sciences
Freedman, D. L., Collier, D., Sekhon, J. S., & Stark, P. B. (2009) · Cited by 99
Statistical Models and Causal Inference: A Dialogue with the Social Sciences
Felson, J. (2011) · Contemporary Sociology A Journal of Reviews · doi:10.1177/0094306110391764k · Cited by 96
Cluster-Based Early Warning Indicators for Political Change in the Contemporary Levant
Schrodt, P. A., & Gerner, D. J. (2000) · American Political Science Review · doi:10.2307/2586209 · Cited by 95
Self-citationSerbian Compliance or Defiance in Kosovo? Statistical Analysis and Real-Time Predictions
Pevehouse, J. C. W., & Goldstein, J. S. (1999) · Journal of Conflict Resolution · doi:10.1177/0022002799043004007 · Cited by 87
Gerring, J. (2008) · The Journal of Politics · doi:10.1017/s0022381607080218 · Cited by 84
Statistical Models and Causal Inference: A Dialogue with the Social Sciences
Freedman, D. A. (2009) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815874 · Cited by 82
(2007) · Journal of Classification · doi:10.1007/s00357-007-0015-9 · Cited by 71
Forecasting in international relations : theory, methods, problems, prospects
Choucri, N., & Robinson, T. W. (1978) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 66
Exploring the Past, Anticipating the Future: A Symposium
Schneider, G., Gleditsch, N. P., & Carey, S. C. (2010) · International Studies Review · doi:10.1111/j.1468-2486.2009.00909.x · Cited by 61
Beyond the Linear Frequentist Orthodoxy
Schrodt, P. A. (2006) · Political Analysis · doi:10.1093/pan/mpj013 · Cited by 52
Self-citationBayesian Combination of State Polls and Election Forecasts
Lock, K. F., & Gelman, A. (2010) · Political Analysis · doi:10.1093/pan/mpq002 · Cited by 44
How Not to Lie with Statistics: Avoiding Common Mistakes in Quantitative Political Science
King, G. (1986) · American Journal of Political Science · doi:10.2307/2111095 · Cited by 41
An Empirical Bayes Approach to Estimating Ordinal Treatment Effects
Alvarez, R. M., Bailey, D., & Katz, J. N. (2011) · Political Analysis · doi:10.1093/pan/mpq033 · Cited by 39
Theodoridis, S., & Koutroumbas, K. (2009) · Elsevier eBooks · doi:10.1016/b978-1-59749-272-0.50006-2 · Cited by 34
Clustering Algorithms II: Hierarchical Algorithms
Theodoridis, S., & Koutroumbas, K. (2009) · Elsevier eBooks · doi:10.1016/b978-1-59749-272-0.50015-3 · Cited by 23
A Methodological Critique of a Test of the Effects of the Maharishi Technology of the Unified Field
Schrodt, P. A. (1990) · Journal of Conflict Resolution · doi:10.1177/0022002790034004008 · Cited by 12
Self-citationExplanation and Prediction by Covering Laws
Hempel, C. G., & Fetzer, J. H. (2001) · Oxford University Press eBooks · doi:10.1093/oso/9780195121360.003.0005 · Cited by 12
Clarke, K. A. (2012) · Conflict Management and Peace Science · doi:10.1177/0738894211433170 · Cited by 9
Scientific Prediction versus Crystal Ball Gazing: Can the Unknown be Known?
Vincent, J. E. (1980) · International Studies Quarterly · doi:10.2307/2600255 · Cited by 6
Bayesian Methods in Political Science: Introduction to the Virtual Issue
Gill, J. (2012) · Political Analysis · doi:10.1017/s1047198700014273 · Cited by 6
Cited by
Leaving theory behind: Why simplistic hypothesis testing is bad for International Relations
Mearsheimer, J. J., & Walt, S. M. (2013) · European Journal of International Relations · doi:10.1177/1354066113494320 · Cited by 412
Integrating explanation and prediction in computational social science
Hofman, J. M., Watts, D. J., Athey, S., Garip, F., Griffiths, T. L., Kleinberg, J., Margetts, H., Mullainathan, S., Salganik, M. J., Vazire, S., et al. (2021) · Nature · doi:10.1038/s41586-021-03659-0 · Cited by 371
Can Structural Conditions Explain the Onset of Nonviolent Uprisings?
Chenoweth, E., & Ulfelder, J. (2017) · Journal of Conflict Resolution · doi:10.1177/0022002715576574 · Cited by 349
Why Partisans Do Not Sort: The Constraints on Political Segregation
Mummolo, J., & Nall, C. (2017) · The Journal of Politics · doi:10.1086/687569 · Cited by 269
Common Sense and Sociological Explanations
Watts, D. J. (2014) · American Journal of Sociology · doi:10.1086/678271 · Cited by 176
Metadata
- Authors
- Schrodt, P. A.
- Venue
- Journal of Peace Research
- Publisher
- Oxford University Press (OUP)
- Published
- 2013-10-11
- Volume
- 51
- Issue
- 2
- Pages
- 287-300
- ISSN
- 0022-3433 (ISSN-L)
- 1460-3578
- OpenAlex
- W2117564857
- Cited by
- 264
- References
- 78
- FWCI
- 129.44
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.