The DOI matches its Crossref record; 109 of the 110 references it cites were found in a registry or an index.
What Would It Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences
Abstract
We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin’s causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Using the konfound framework [31,32], we found that nullifying the estimated effects would require replacing approximately 50% of observations or an omitted confounder with correlations exceeding 0.25 with both exposure and outcome.”
Citations
Cites
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779 · cited 1×
“Heckman (1979) also established a relationship between bias due to nonrandom assignment to treatment conditions and bias due to sample selection.”
BackgroundThe Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170 · cited 1×
“For education research to inform policy, it should emerge through debate among a community of scholars about the relative merits of different studies (e.g., Greco, 2009; Habermas, 1987; Kuhn, 1962; Kvanvig, 2003; Sosa, 2007).”
Background(2002) · Cited by 13,211
(1979) · Medical Entomology and Zoology · Cited by 11,252
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 4×
“As a foundation, we draw on Rubin’s causal model (RCM; Rubin, 1974) to express concerns about bias in terms of characteristics of unobserved data.”
BackgroundMethodInfluentialJournal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Quasi-experimentation: Design and analysis issues for field settings
Shrout, P. E. (1980) · Evaluation and Program Planning · doi:10.1016/0149-7189(80)90063-4 · Cited by 6,010
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 4×
“Consistent with this, Hong and Raudenbush (2005), reported how their estimate would have changed if it was reduced by an omitted variable comparable with their most important covariate.”
BackgroundMethodInfluential
Research works printed but not linked: 15
- Alexander K. L., 1998, Psychology in Schools, 9, 410
- Bogatz G. A., 1972, The impact of “sesame street” on children’s first school experience
- 10.3102/0162373712461933 · DOI
- 10.3102/00028312027003557 · doi:10.3102/00028312027003557 · Paper page
- Greco J., 2009, Epistemic value, 313
- Harvard Education Letter, 1986, Harvard Education Letter, 2, 1
- Holmes C. T., 1989, Flunking grades, 16
- 10.3102/00346543054002225 · doi:10.3102/00346543054002225 · Paper page
- Manski C., 1990, American Economic Review Papers and Proceedings, 80, 319
- 10.3102/01623737014002101 · doi:10.3102/01623737014002101 · Paper page
- Roderick M., 1999, Ending social promotion: Results from the first two years
- Shepard L. A., 1989, Flunking grades
- 10.1002/(SICI)1520-6807(199810)35:4<404::AID-PITS14>3.0.CO;2-L · doi:10.1002/(sici)1520-6807(199810)35:4<404::aid-pits14>3.0.co;2-l
- SRA/McGraw-Hill, 2001, Technical report performance assessments
- US Department of Health and Human Services, 2000, Trends in the well-being of America’s children and youth
Cited by
Making sense of sensitivity: extending omitted variable bias
Cinelli, C., & Hazlett, C. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12348 · Cited by 1,088
Xu, D., Zhou, K. Z., & Du, F. (2019) · Academy of Management Journal · doi:10.5465/amj.2016.0749 · Cited by 510
Busenbark, J. R., Yoon, H. (., Gamache, D. L., & Withers, M. C. (2022) · Journal of Management · doi:10.1177/01492063211006458 · Cited by 457
“…to highlight that the ITCV is useful to examine the causal inference and statistical significance of relationships between independent and dependent variables, but it does not address inaccuracies of the parameter estimate itself (Frank, 2000; Frank et al., 2013; Larcker & Rusticus, 2010).”
BackgroundMethodInfluentialNonprice incentives and energy conservation
Asensio, O. I., & Delmas, M. A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1401880112 · Cited by 422
Wolske, K. S., Stern, P. C., & Dietz, T. (2017) · Energy Research & Social Science · doi:10.1016/j.erss.2016.12.023 · Cited by 406
Metadata
- Authors
- Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M.
- Affiliations
- Michigan State University (US)
- Arizona State University (US)
- Pacific Metrics Corporation
- University of Cincinnati (US)
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2013-12
- Volume
- 35
- Issue
- 4
- Pages
- 437-460
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2114805009
- Cited by
- 406
- References
- 110
- Citations by year
- 2026: 67 · 2025: 55 · 2024: 45 · 2023: 50 · 2022: 58 · 2021: 27 · 2020: 37 · 2019: 15 · 2018: 22 · 2017: 11 · 2016: 9 · 2015: 7 · 2014: 3
- FWCI
- 5.57
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Evaluation and Policy Analysis (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- causal inference, inference robustness, reading achievement, kindergarten retention, randomized experiments, observational studies, research design, scientific discourse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: What Would It Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. https://app.cochat.ai/papers/10.3102/0162373713493129
- The paper
- Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013). What Would It Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/0162373713493129
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.