"Truth" Is Stranger than Prediction, More Questionable than Causal Inference
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
Robert Luskin's article in this issue provides a useful service by appropriately qualifying several points I made in my 1986 American Journal of Political Science article. Whereas I focused on how to avoid common mistakes in quantitative political science, Luskin clarifies ways to extract some useful information from usually problematic statistics: correlation coefficients, standardized coefficients, and especially RZ. Since these three statistics are very closely related (and indeed deterministic functions of one another in some cases), I focus in this discussion primarily on R2, the most widely used and abused. Luskin also widens the discussion to various kinds of specification tests, a general issue I also address. In fact, as Beck (1991) reports, a large number of formal specification tests are just functions of R2, with differences among them primarily due to how much each statistic penalizes one for including extra parameters and fewer\nobservations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Instead, out-of-sample predictive validity is the appropriate measure [66,68-72].”
From Modeling causes of death: an integrated approach using CODEm (2012) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In-sample fit is not a robust measure of prediction when data are sparse or missing.”
From Modeling causes of death: an integrated approach using CODEm (2012) · cites this paper as [68]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The R -Squared: Some Straight Talk
Lewis‐Beck, M. S., & Skalaban, A. (1990) · Political Analysis · doi:10.1093/pan/2.1.153 · Cited by 101 · cited 1×
“'In King (1991) I responded to a related critique of my 1986 AJPS article by Lewis-Beck and Skalaban (1991).”
BackgroundAbusus Non Tollit Usum: Standardized Coefficients, Correlations, and R 2 s
Luskin, R. C. (1991) · American Journal of Political Science · doi:10.2307/2111505 · Cited by 100
How Not to Lie with Statistics: Avoiding Common Mistakes in Quantitative Political Science
King, G. (1986) · American Journal of Political Science · doi:10.2307/2111095 · Cited by 41
Self-citation
Research works printed but not linked: 2
- -. 1989. Unifying Political Methodology: The Likelihood Theory of Statistical Inference. New York: Cambridge University Press.
- Luskin, Robert. 1991. "Abusus Non Tollit Usum: Standardized Coefficients, Correlations, and R²s." American Journal of Political Science 35: 1030–44.
Cited by
Modeling causes of death: an integrated approach using CODEm
Foreman, K. J., Lozano, R., Lopez, A. D., & Murray, C. J. (2012) · Population Health Metrics · doi:10.1186/1478-7954-10-1 · Cited by 564
The Dangers of Extreme Counterfactuals
King, G., & Zeng, L. (2006) · Political Analysis · doi:10.1093/pan/mpj004 · Cited by 540
“Post-treatment variable bias is a large and often overlooked component of bias in estimating causal effects in political science (see King 1991; King et al. 1994, pp. 173ff).”
BackgroundSelf-citationRace as a Bundle of Sticks: Designs that Estimate Effects of Seemingly Immutable Characteristics
Sen, M., & Wasow, O. (2016) · Annual Review of Political Science · doi:10.1146/annurev-polisci-032015-010015 · Cited by 422
“Although perhaps unsatisfactory to many applied researchers, the most appropriate initial approach is to drop any post-treatment variables from an analysis (King 1991; King, Keohane, and Verba 1994; King and Zeng 2006; Gelman and Hill 2007).”
BackgroundThe Effect of Political Trust on the Presidential Vote, 1968–96
Hetherington, M. J. (1999) · American Political Science Review · doi:10.2307/2585398 · Cited by 363
How to compare cross-lagged associations in a multilevel autoregressive model.
Schuurman, N. K., Ferrer, E., de Boer-Sonnenschein, M., & Hamaker, E. L. (2016) · Psychological Methods · doi:10.1037/met0000062 · Cited by 315
Neighborhood Poverty and African American Politics
Cohen, C. J., & Dawson, M. C. (1993) · American Political Science Review · doi:10.2307/2939041 · Cited by 311
When Can History Be Our Guide? The Pitfalls of Counterfactual Inference
KING, G., & ZENG, L. (2007) · International Studies Quarterly · doi:10.1111/j.1468-2478.2007.00445.x · Cited by 256
Self-citationThe contribution of critical realism as an underpinning philosophy for OR/MS and systems
Mingers, J. (2000) · Journal of the Operational Research Society · doi:10.1057/palgrave.jors.2601033 · Cited by 235
Exploiting a Rare Communication Shift to Document the Persuasive Power of the News Media
Ladd, J. M., & Lenz, G. S. (2009) · American Journal of Political Science · doi:10.1111/j.1540-5907.2009.00377.x · Cited by 230
Women's autonomy and its impact on fertility
Abadian, S. (1996) · World Development · doi:10.1016/s0305-750x(96)00075-7 · Cited by 148
Gazal‐Ayal, O., & Sulitzeanu‐Kenan, R. (2010) · Journal of Empirical Legal Studies · doi:10.1111/j.1740-1461.2010.01183.x · Cited by 137
“The analyses begin by estimating unconditional models that estimate the relative amount of variation at the decision and judge the estimate of the randomly assigned causal effect – in our case the ethnic identity of the judge (see: King 1991, pp. 1050-1).”
Atkinson, C. (2006) · International Studies Quarterly · doi:10.1111/j.1468-2478.2006.00412.x · Cited by 157
The Importance of Concurrence: The Impact of Bicameralism on Government Formation and Duration
Druckman, J. N., & THIES, M. F. (2002) · American Journal of Political Science · doi:10.2307/3088432 · Cited by 137
Explaining use of information in public policymaking
Oh, C. H., & Rich, R. F. (1996) · Knowledge, technology & policy/Knowledge in society/Knowledge, technology, & policy · doi:10.1007/bf02832231 · Cited by 117
Face Value? Experimental Evidence that Candidate Appearance Influences Electoral Choice
Ahler, D. J., Citrin, J., Dougal, M. C., & Lenz, G. (2016) · Political Behavior · doi:10.1007/s11109-016-9348-6 · Cited by 94
Chiozza, G., & Goemans, H. E. (2004) · Journal of Peace Research · doi:10.1177/0022343304044475 · Cited by 90
King, G. (2006) · PS Political Science & Politics · doi:10.1017/s1049096506060252 · Cited by 102
Self-citationScholasticism in Political Science
Mead, L. M. (2010) · Perspectives on Politics · doi:10.1017/s1537592710001192 · Cited by 79
“4 Beck and Katz 1995, King 1991, Nagler 1991.”
It Is Not Only What You Say, It Is Also How You Say It: The Strategic Use of Campaign Sentiment
Crabtree, C. D., Golder, M., Gschwend, T., & Indridason, I. (2019) · The Journal of Politics · doi:10.1086/707613 · Cited by 97
Bensch, G., & Peters, J. (2013) · Land Economics · doi:10.3368/le.89.4.676 · Cited by 97
Economic Openness and Territorial Politics in China
Sheng, Y. (2010) · Cambridge University Press eBooks · doi:10.1017/cbo9780511761966 · Cited by 74
The role of faculty in reducing academic dishonesty among engineering students
Chirikov, I., Shmeleva, E., & Loyalka, P. (2020) · Studies in Higher Education · doi:10.1080/03075079.2019.1616169 · Cited by 65
Skipworth, S. A., Garner, A., & Dettrey, B. J. (2010) · Politics & Policy · doi:10.1111/j.1747-1346.2010.00262.x · Cited by 56
Consumer guilt and green banking services
Burhanudin, B., Ronny, R., & Sihotang, E. T. (2020) · International Journal of Consumer Studies · doi:10.1111/ijcs.12602 · Cited by 82
Barrington, L. W. (2002) · European Journal of Political Research · doi:10.1111/1475-6765.00019 · Cited by 54
Transfers of Governmental Power: The Meaning of Time Dependence
Alt, J. Ε., & King, G. (1994) · Comparative Political Studies · doi:10.1177/0010414094027002002 · Cited by 49
Self-citationThe Effects of Malapportionment on Cabinet Inclusion: Subnational Evidence from India
Bhavnani, R. R. (2015) · British Journal of Political Science · doi:10.1017/s0007123415000587 · Cited by 47
Bargaining on the U.S. Supreme Court: Justices' Responses to Majority Opinion Drafts
Spriggs, J. F., Maltzman, F., & Wahlbeck, P. J. (1999) · The Journal of Politics · doi:10.2307/2647513 · Cited by 43
Sheng, Y. (2007) · Comparative Political Studies · doi:10.1177/0010414005285752 · Cited by 46
Explaining the impact of policy information on policy-making
Oh, C. H. (1997) · Knowledge and policy · doi:10.1007/bf02912505 · Cited by 39
The Importance of Concurrence: The Impact of Bicameralism on Government Formation and DurationPreprint
Druckman, J. N., & THIES, M. F. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1158644 · Cited by 35
Environmental Scarcity and International Conflict
Stalley, P. (2003) · Conflict Management and Peace Science · doi:10.1177/073889420302000202 · Cited by 42
Sheng, Y. (2008) · Studies in Comparative International Development · doi:10.1007/s12116-008-9023-y · Cited by 41
Do Hegemons Distribute Private Goods?
Bussmann, M., & Oneal, J. R. (2007) · Journal of Conflict Resolution · doi:10.1177/0022002706296178 · Cited by 27
“In a test not reported in the table, both measures of satisfaction were clearly insignificant when proximity, major-power status, the years of peace, and the size of the international system were held constant.”
BackgroundBurhanudin, B., Ronny, R., & Sihotang, E. T. (2019) · Procedia Computer Science · doi:10.1016/j.procs.2019.11.132 · Cited by 31
Sustainable development, sustainable decisions and the precautionary principle
Paterson, J. B. (2007) · Natural Hazards · doi:10.1007/s11069-006-9071-4 · Cited by 23
“Experience reveals that predictions even 10 years out are just as likely to be wildly wide of the mark as to be anywhere near it. And disagreement among experts on the use and abuse of statistics does little to reassure either policymakers or public (for example Luskin 1991; King 1991 ).”
BackgroundWhy Process Matters for Causal Inference
Glynn, A., & Quinn, K. M. (2011) · Political Analysis · doi:10.1093/pan/mpr021 · Cited by 23
“…variables in an analysis of total effects) with the standard warnings about posttreatment bias (which is induced by adjusting for a posttreatment variable in a regression or matching analysis; Rosenbaum and Rubin 1983; King 1991; King, Keohane, and Verba 1994; Pearl 2009, Section 3.3.1).”
MethodInfluentialGanghof, S. (2005) · Zeitschrift für Internationale Beziehungen · doi:10.5771/0946-7165-2005-1-7 · Cited by 21
Information Searching in Governmental Bureaucracies: An Integrated Model
Oh, C. H. (1996) · The American Review of Public Administration · doi:10.1177/027507409602600103 · Cited by 14
It’s Not Only What you Say, It’s Also How You Say It: The Strategic Use of Campaign SentimentPreprint
Crabtree, C. D., Golder, M., Gschwend, T., & Indridason, I. (2018) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/g2sd6 · Cited by 12
Mingers, J. (2006) · Cited by 11
The effects of malapportionment on economic development
Bhavnani, R. R. (2021) · PLoS ONE · doi:10.1371/journal.pone.0259150 · Cited by 14
“Controlling for post-treatment variables would bias the estimated effect of malapportionment, since malapportionment might work precisely through these variables [27, 28].”
Strategic Outrage: the Politics of Presidential Scandal
Nyhan, B. J. (2009) · DukeSpace (Duke University) · Cited by 14
An Event Count Model for Studying Health Services Utilization
Chi, C. (1998) · Medical Care · doi:10.1097/00005650-199812000-00003 · Cited by 9
Bensch, G., & Peters, J. (2012) · SSRN Electronic Journal · doi:10.2139/ssrn.1978710 · Cited by 9
The Dangers of Extreme Counterfactuals
King, G., & Zeng, L. (2006) · Cited by 8
Self-citationEthnicity, Insurgency, and Civil War Revisited ∗
Quinn, K., Hechter, M., & Wibbels, E. (2003) · Cited by 119
Ladd, J. M., & Lenz, G. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1189811 · Cited by 7
Friedman, J. R., Graetz, N., & Gakidou, E. E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0208019 · Cited by 7
Rejoinder: Can We Weight Models by Their Probability of Being True?
Muñoz, J., & Young, C. (2018) · Sociological Methodology · doi:10.1177/0081175018796841 · Cited by 5
“O’Brien notes that “not all independent variables in the model should be controlled” (this volume, p. 35) and details pitfalls of overcontrolling with intervening or collider variables (see also Clarke, Kenkel, and Rueda 2018; Elwert and Winship 2014; Heckman and Navarro-Lozano 2004; King 1991).”
BackgroundPerspektiven und Potenziale der vergleichenden Politischen Ökonomie
Kittel, B. (2003) · VS Verlag für Sozialwissenschaften eBooks · doi:10.1007/978-3-322-80884-4_15 · Cited by 12
Causal Explanation and Multi-Method Research in the Social SciencesPreprint
Kuehn, D., & Rohlfing, I. (2009) · SSRN Electronic Journal · Cited by 10
Problems with and Solutions for Two-Dimensional Models of Continuous Dependent Variables
Goodrich, B. (2005) · Cited by 5
Conceptualization, measurement and effects of positional issues in the Canadian electoral context
Dufresne, Y., & Ouellet, C. (2019) · Heliyon · doi:10.1016/j.heliyon.2019.e01453 · Cited by 4
“…between proponents of fully specified models, measured by a high R2, and those favouring specific causal inferences by including only control variables that are theoretically prior to the focal independent variables and correlated with both the dependent and independent variables (King, 1991).”
BackgroundLeft/Right Asymmetries in a Multidimensional Universe: Citizens, Activists, and Parties
Cochrane, C. (2010) · TSpace (University of Toronto) · Cited by 4
Reconciling Race and Causation: Designs to Estimate Effects of Seemingly Immutable Characteristics
Sen, M., & Wasow, O. (2014) · Cited by 3
The Methodology of Presidential ResearchPreprint
King, G. (2008) · SSRN Electronic Journal · doi:10.2139/ssrn.1156481 · Cited by 3
Self-citationDid the German aviation tax have a lasting effect on passenger numbers?
Helmers, V., & van der Werf, E. (2025) · Transportation Research Part D Transport and Environment · doi:10.1016/j.trd.2024.104570 · Cited by 2
Las teorías de sistemas en el estudio de la cultura política
Montiel, Á. G. L. (2008) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) · Cited by 1
FOREIGN POLICY AND IDEOLOGICAL VOTING IN THE U.S. SENATE
Sulfaro, V. A. (1996) · Southeastern Political Review · doi:10.1111/j.1747-1346.1996.tb00431.x · Cited by 1
Is the Road to Bad Inference Paved with Good Intentions? An Audit of Vote Intention Survey Items
Arceneaux, K. (2001) · Cited by 1
Optimizing Residential Segregation and Urban Crime
Sykes, B. L., & Young, J. T. N. (2008)
Proximity, Politics and Policy Attitudes in the North American Context
Gravelle, T. B. (2016) · Open Access at Essex (University of Essex)
The Political Causes and Effects of Fiscal Adjustments
Donnelly, P. T. (2016) · eScholarship (California Digital Library)
Did the German Aviation Tax Have a Lasting Effect on Passenger Numbers?Preprint
Helmers, V., & van der Werf, E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4892451
Uno studio comparato sulle disuguaglianze di salute : Italia e Francia
Ornaghi, A. (2014) · theses.fr (ABES) · doi:10.70675/3a863cb2z7093z4edbz9598z5090d6e904ef
Metadata
- Authors
- King, G.
- Affiliations
- Harvard University (US)
- Corresponding author
- King, Gary
- ORCID
- King, Gary
- Venue
- American Journal of Political Science
- Publisher
- JSTOR
- Published
- 1991-11
- Volume
- 35
- Issue
- 4
- Pages
- 1047
- ISSN
- 0092-5853 (ISSN-L)
- OpenAlex
- W1994998176
- Cited by
- 67
- References
- 3
- Citations by year
- 2025: 1 · 2024: 1 · 2021: 1 · 2020: 1 · 2019: 4 · 2018: 3 · 2016: 5 · 2015: 1 · 2014: 3 · 2013: 1 · 2012: 2
- FWCI
- 6.34
- Type
- journal article
- Language
- en
- Open access
- green
- nrs.harvard.edu
- Licence
- cc-by
- Locations
- American Journal of Political Science (publishedVersion)
- Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Social Sciences › Political Science and International Relations › Political Science Research and Education
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Statistical Methods and Models
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Keywords
- R-squared, R2, correlation coefficient, standardized coefficients, specification tests, model fit statistics, causal inference
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.