What Would It Take to Change an Inference? Using Rubin’s Causal Model to Interpret the Robustness of Causal Inferences
The DOI matches its Crossref record; 109 of the 110 references it cites were found in a registry or an index.
Abstract
We contribute to debate about causal inferences in educational research in two ways. First, we quantify how much bias there must be in an estimate to invalidate an inference. Second, we utilize Rubin’s causal model to interpret the bias necessary to invalidate an inference in terms of sample replacement. We apply our analysis to an inference of a positive effect of Open Court Curriculum on reading achievement from a randomized experiment, and an inference of a negative effect of kindergarten retention on reading achievement from an observational study. We consider details of our framework, and then discuss how our approach informs judgment of inference relative to study design. We conclude with implications for scientific discourse.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Using the konfound framework [31,32], we found that nullifying the estimated effects would require replacing approximately 50% of observations or an omitted confounder with correlations exceeding 0.25 with both exposure and outcome.”
Citations
Cites
The Structure of Scientific Revolutions
Kuhn, T. S., & Schlegel, R. (1963) · Physics Today · doi:10.1063/1.3050879 · Cited by 35,538
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779 · cited 1×
“Heckman (1979) also established a relationship between bias due to nonrandom assignment to treatment conditions and bias due to sample selection.”
BackgroundThe Structure of Scientific Revolutions: 50th Anniversary Edition
Kuhn, T. S., & Hacking, I. (2012) · doi:10.7208/chicago/9780226458144.001.0001 · Cited by 21,170 · cited 1×
“For education research to inform policy, it should emerge through debate among a community of scholars about the relative merits of different studies (e.g., Greco, 2009; Habermas, 1987; Kuhn, 1962; Kvanvig, 2003; Sosa, 2007).”
Background(2002) · Cited by 13,211
(1979) · Medical Entomology and Zoology · Cited by 11,252
Statistical Methods for Research Workers
(1945) · American Journal of Clinical Pathology · doi:10.1093/ajcp/15.11.552b · Cited by 10,880
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 4×
“As a foundation, we draw on Rubin’s causal model (RCM; Rubin, 1974) to express concerns about bias in terms of characteristics of unobserved data.”
BackgroundMethodInfluentialJournal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Quasi-experimentation: Design and analysis issues for field settings
Shrout, P. E. (1980) · Evaluation and Program Planning · doi:10.1016/0149-7189(80)90063-4 · Cited by 6,010
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 4×
“Consistent with this, Hong and Raudenbush (2005), reported how their estimate would have changed if it was reduced by an omitted variable comparable with their most important covariate.”
BackgroundMethodInfluentialHabermas, J. (1968) · Cited by 4,683
Langenberg, D. N. (2000) · Cited by 4,305
Journal of the American Statistical Association
Fildes, R. (1989) · International Journal of Forecasting · doi:10.1016/0169-2070(89)90106-4 · Cited by 3,874
Effects of Teachers’ Mathematical Knowledge for Teaching on Student Achievement
Hill, H. C., Rowan, B., & Ball, D. L. (2005) · American Educational Research Journal · doi:10.3102/00028312042002371 · Cited by 2,812
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Rosenbaum, P. R. (2002) · Springer Series in Statistics · doi:10.1007/978-1-4757-3692-2 · Cited by 2,394
Politics, markets, and America's schools
(1991) · Choice Reviews Online · doi:10.5860/choice.28-3405 · Cited by 1,837 · cited 1×
“…Beyond, Coleman, Hoffer, and Kilgore (1982) estimated that students attending Catholic schools had higher achievement than similar students attending public schools leading to an inference that Catholic schools educate students better than public schools (Chubb & Moe, 1990; Coleman et al., 1982).”
BackgroundBecker, H. S. (1967) · Social Problems · doi:10.2307/799147 · Cited by 1,746 · cited 1×
“As a result, debate about the general bases for causal inferences in the social sciences dates back to the 1900s (e.g., Becker, 1967; Rubin, 1974; Thorndike & Woodworth, 1901; see Abbott, 1998 or Oakley, 1998, for reviews), with some heated as in the Cronbach versus Campbell exchanges of the 1980s…”
BackgroundScientific Research in Education
(2002) · National Academies Press eBooks · doi:10.17226/10236 · Cited by 1,723
Identification, Inference and Sensitivity Analysis for Causal Mediation Effects
Imai, K., Keele, L., & Yamamoto, T. (2010) · Statistical Science · doi:10.1214/10-sts321 · Cited by 1,628
Statistical Methods for Research Workers.
Hotelling, H., & Fisher, R. A. (1930) · Journal of the American Statistical Association · doi:10.2307/2278215 · Cited by 1,529
How Large Are Teacher Effects?
Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737026003237 · Cited by 1,493
(2007) · Cited by 1,428
- Statistical Models for Epidemiology
(2013) · Bayesian Methods in Epidemiology · doi:10.1201/b15382-9 · Cited by 1,346
The influence of improvement in one mental function upon the efficiency of other functions. (I).
Woodworth, R. S., & Thorndike, E. L. (1901) · Psychological Review · doi:10.1037/h0074898 · Cited by 1,223 · cited 1×
“As a result, debate about the general bases for causal inferences in the social sciences dates back to the 1900s (e.g., Becker, 1967; Rubin, 1974; Thorndike & Woodworth, 1901; see Abbott, 1998 or Oakley, 1998, for reviews), with some heated as in the Cronbach versus Campbell exchanges of the 1980s…”
BackgroundNONPARAMETRIC BOUNDS ON TREATMENT EFFECTS
Manski, C. F. (1989) · American Economic Review · Cited by 1,220
Significant Benefits: The High/Scope Perry Preschool Study Through Age 27
Schweinhart, L. J., Barnes, H., Weikart, D. P., & Ypsilanti, M. (1993) · Cited by 1,217
Counterfactuals and Causal Inference: Methods and Principles for Social Research
Morgan, S. L., & Winship, C. (2007) · doi:10.1017/cbo9780511804564 · Cited by 1,146
Rosenbaum, P. R., & Rubin, D. B. (1983) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1983.tb01242.x · Cited by 1,124 · cited 2×
“This is similar to the strong assumption of ignorability (Rosenbaum & Rubin, 1983)—that other factors can be ignored conditioning on the observed factors.”
BackgroundDesigning Evaluations of Educational and Social Programs.
Hendrickson, L., & Cronbach, L. J. (1983) · Contemporary Sociology · doi:10.2307/2068975 · Cited by 1,042 · cited 6×
“…external validity, decompose the combined population treatment effect, δ, into two components: δ p , the treatment effect for the population potentially sampled, and δ p′ , the treatment effect for the population not potentially sampled (Cronbach, 1982; Fisher & Sir, 1930/1970; Frank & Min, 2007).”
BackgroundMethodInfluentialHirano, K., & Imbens, G. W. (2001) · Health Services and Outcomes Research Methodology · doi:10.1023/a:1020371312283 · Cited by 1,039 · cited 1×
“Technically, this can be dealt with by conducting dual sensitivity (Gastwirth, Krieger, & Rosenbaum, 1998) or by characterizing sensitivity analysis in terms of the product of the two relationships (Frank, 2000; Hirano & Imbens, 2001; Imai, Keele, & Yamamoto, 2010).”
BackgroundImpact of a Confounding Variable on a Regression Coefficient
FRANK, K. A. (2000) · Sociological Methods & Research · doi:10.1177/0049124100029002001 · Cited by 1,032 · cited 3×
“Technically, this can be dealt with by conducting dual sensitivity (Gastwirth, Krieger, & Rosenbaum, 1998) or by characterizing sensitivity analysis in terms of the product of the two relationships (Frank, 2000; Hirano & Imbens, 2001; Imai, Keele, & Yamamoto, 2010).”
BackgroundSelf-citationThe Value of Knowledge and the Pursuit of Understanding
Kvanvig, J. L. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511498909 · Cited by 998
High school achievement : public, Catholic, and private schools compared
Coleman, J. S., Hoffer, T. B., & Kilgore, S. B. (1982) · Basic Books · Cited by 969
Assessing the impact of planned social change
Campbell, D. T. (1979) · Evaluation and Program Planning · doi:10.1016/0149-7189(79)90048-x · Cited by 957
Encyclopedia of educational research
(1992) · Choice Reviews Online · doi:10.5860/choice.30-1262 · Cited by 905
THE ESTIMATION OF CAUSAL EFFECTS FROM OBSERVATIONAL DATA
Winship, C., & Morgan, S. L. (1999) · Annual Review of Sociology · doi:10.1146/annurev.soc.25.1.659 · Cited by 890
Identification, Inference, and Sensitivity Analysis for Causal Mediation Effects
Imai, K., & Keele, L. J. (2008) · Cited by 887
Heckman, J. (1989) · National Bureau of Economic Research · doi:10.3386/w2861 · Cited by 882
Understanding Instrumental Variables in Models with Essential Heterogeneity
Heckman, J. J., Urzua, S., & Vytlacil, E. (2006) · Review of Economics and Statistics · doi:10.1162/rest.88.3.389 · Cited by 874
Assessing the sensitivity of regression results to unmeasured confounders in observational studies
Lin, D. Y., Psaty, B. M., & Kronmal, R. A. (1998) · Biometrics · doi:10.2307/2533848 · Cited by 742 · cited 1×
“449 robustness of inferences from two other observational studies: Morgan’s (2001) inference of a Catholic school effect on achievement (building on Coleman et al.”
BackgroundPolitics, Markets and America's Schools
Chubb, J. E., & Moe, T. M. (1991) · British Journal of Sociology of Education · doi:10.1080/0142569910120306 · Cited by 742
The value of knowledge and the pursuit of understanding
(2004) · Choice Reviews Online · doi:10.5860/choice.41-4578 · Cited by 690 · cited 1×
“For education research to inform policy, it should emerge through debate among a community of scholars about the relative merits of different studies (e.g., Greco, 2009; Habermas, 1987; Kuhn, 1962; Kvanvig, 2003; Sosa, 2007).”
BackgroundHarding, D. J. (2003) · American Journal of Sociology · doi:10.1086/379217 · Cited by 644 · cited 1×
“Hedges and O’Muircheartaigh (2011) used the estimates from a particular propensity stratum to generalize to a corresponding unstudied population.”
BackgroundSosa, E. (2007) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199297023.001.0001 · Cited by 614
Ichino, A., Mealli, F., & Nannicini, T. (2008) · Journal of Applied Econometrics · doi:10.1002/jae.998 · Cited by 603
Meta-analysis of Grade Retention Research: Implications for Practice in the 21st Century
Jimerson, S. R. (2001) · School Psychology Review · doi:10.1080/02796015.2001.12086124 · Cited by 597 · cited 5×
“…there have been considerable studies and syntheses of retention effects (e.g., Alexander et al., 2003; Holmes, 1989; Holmes & Matthews, 1984; Jimerson, 2001; Karweit, 1992; Reynolds, 1992; Roderick, Bryk, Jacobs, TA B L E 1 Q ua nt if yi ng th e R ob us tn es s of I nf er en ce s F ro m…”
BackgroundInfluentialSlavin, R. E. (2008) · Educational Researcher · doi:10.3102/0013189x08314117 · Cited by 560
Answers and Questions about Class Size: A Statewide Experiment
Finn, J. D., & Achilles, C. M. (1990) · American Educational Research Journal · doi:10.2307/1162936 · Cited by 546 · cited 5×
“For example, one might use estimated effects of small classes in Tennessee (Finn & Achilles, 1990) for certain strata based on background characteristics to project a score for California.”
BackgroundResultInfluentialThe Use of Propensity Scores to Assess the Generalizability of Results from Randomized Trials
Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00673.x · Cited by 524
The importance of covariate selection in controlling for selection bias in observational studies.
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010) · Psychological Methods · doi:10.1037/a0018719 · Cited by 490
Formal mode of statistical inference for causal effects
Rubin, D. B. (1990) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(90)90077-8 · Cited by 483 · cited 6×
“…be observed if unit i were in the control condition and therefore not exposed to the treatment (e.g., being promoted to the first grade).1 If SUTVA (Rubin, 1986, 1990) holds—that there are no spillover effects of treatments from one unit to another— then the causal mechanisms are independent…”
BackgroundInfluentialUnbiased Estimation of Certain Correlation Coefficients
Olkin, I., & Pratt, J. W. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706717 · Cited by 480 · cited 1×
“One could also adjust the estimated correlation for estimation bias in small samples (e.g., Olkin & Pratt, 1958).”
BackgroundShadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479
1. The Scientific Model of Causality
Heckman, J. J. (2005) · Sociological Methodology · doi:10.1111/j.0081-1750.2006.00164.x · Cited by 401 · cited 3×
“In particular, schools that had approached SRA/McGraw-Hill prior to Borman et al.’s study may have had substantive reasons for believing OCR would work particularly well for them (e.g., Heckman, 2005).”
BackgroundInfluentialInference for Non-random Samples
Copas, J. B., & Li, H. G. (1997) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00055 · Cited by 392 · cited 1×
“…omitted variables that could alter an inference either because they affected selection into the sample or assignment to treatments or both (e.g., Copas & Li, 1997; Holland, 1989; Lin, Psaty, & Kronmal, 1998; Robins, Rotnisky, & Scharfstein, 2000; Rosenbaum, 1986, 2002; Rosenbaum & Rubin, 1983;…”
BackgroundExperimental Evaluation of the Effects of a Research-Based Preschool Mathematics Curriculum
Clements, D. H., & Sarama, J. (2008) · American Educational Research Journal · doi:10.3102/0002831207312908 · Cited by 388
An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schooling
Altonji, J. G., Elder, T. E., & Taber, C. R. (2005) · Journal of Human Resources · doi:10.3368/jhr.xl.4.791 · Cited by 364 · cited 1×
“Altonji, Elder, and Taber (2005) and Altonji, Conley, Elder, and Taber (2010) bounded estimated effects by drawing on information associated with observed confounds (cf. Manski, 1990, who bounds based on the maximum value of the outcome).”
BackgroundRobins, J. M., Rotnitzky, A., & Scharfstein, D. O. (2000) · The IMA Volumes in Mathematics and its Applications · doi:10.1007/978-1-4612-1284-3_1 · Cited by 363
The Effects of Nonpromotion on Elementary and Junior High School Pupils: A Meta-Analysis
Holmes, C. T., & Matthews, K. M. (1984) · Review of Educational Research · doi:10.2307/1170303 · Cited by 340 · cited 1×
“…and importance of retention, there have been considerable studies and syntheses of retention effects (e.g., Alexander et al., 2003; Holmes, 1989; Holmes & Matthews, 1984; Jimerson, 2001; Karweit, 1992; Reynolds, 1992; Roderick, Bryk, Jacobs, TA B L E 1 Q ua nt if yi ng th e R ob us tn es s of I…”
BackgroundOn the Success of Failure: A Reassessment of the Effects of Retention in the Primary School Grades
Alexander, K. L., Entwisle, D. R., & Dauber, S. L. (2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511500091 · Cited by 290
Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics
Hong, G., & Raudenbush, S. W. (2005) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737027003205 · Cited by 280 · cited 15×
“Hong and Raudenbush (2005) did not use the “Gold Standard” of random assignment to treatment conditions (e.g., Eisenhart & Towne, 2008; U.S. Department of Education, 2002).”
BackgroundMethodInfluentialCounterfactuals, Causal Effect Heterogeneity, and the Catholic School Effect on Learning
Morgan, S. L. (2001) · Sociology of Education · doi:10.2307/2673139 · Cited by 269 · cited 2×
“For example, we call attention to whether the observational studies controlled for pretests (as did Hong and Raudenbush, 2005, as well as Morgan, 2001) which have recently been found to be critical in reducing bias in educational studies (e.g., Shadish, Clark, & Steiner, 2008; Steiner, Cook, &…”
BackgroundMethodEngel, M., Claessens, A., & Finch, M. A. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712461850 · Cited by 259
ABBOTT, A. (1998) · Sociological Methods & Research · doi:10.1177/0049124198027002002 · Cited by 258 · cited 3×
“…bases for causal inferences in the social sciences dates back to the 1900s (e.g., Becker, 1967; Rubin, 1974; Thorndike & Woodworth, 1901; see Abbott, 1998 or Oakley, 1998, for reviews), with some heated as in the Cronbach versus Campbell exchanges of the 1980s (e.g., Cook & Campbell, 1979;…”
BackgroundInfluentialCook, T. D. (2002) · Educational Evaluation and Policy Analysis · doi:10.3102/01623737024003175 · Cited by 257 · cited 2×
“Education is fundamentally a pragmatic enterprise (e.g., National Research Council, 2002; Raudenbush, 2005), with the ultimate goal of educational research to inform choices about curricula, pedagogy, practices, or school organization (e.g., Bulterman-Bos, 2008; Cook, 2002).”
BackgroundMethodAn Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schools
Altonji, J., Elder, T., & Taber, C. (2002) · National Bureau of Economic Research · doi:10.3386/w9358 · Cited by 256
Contestation and Change in National Policy on “Scientifically Based” Education Research
Eisenhart, M., & Towne, L. (2003) · Educational Researcher · doi:10.3102/0013189x032007031 · Cited by 228 · cited 3×
“(p. 170) Given concerns about inferences from observational studies, several institutions, such as the What Works Clearinghouse (Eisenhart & Towne, 2008) and the U.S. Department of Education’s National Center for Education Research (NCER), have drawn on the medical model to call for a sound, scientifically rigorous ba…”
BackgroundMethodGrade Retention and School Adjustment: An Explanatory Analysis
Reynolds, A. J. (1992) · Educational Evaluation and Policy Analysis · doi:10.2307/1164496 · Cited by 180 · cited 1×
“…studies and syntheses of retention effects (e.g., Alexander et al., 2003; Holmes, 1989; Holmes & Matthews, 1984; Jimerson, 2001; Karweit, 1992; Reynolds, 1992; Roderick, Bryk, Jacobs, TA B L E 1 Q ua nt if yi ng th e R ob us tn es s of I nf er en ce s F ro m R an do m iz ed E xp er im en ts S…”
Learning from Attempts to Improve Schooling: The Contribution of Methodological Diversity
Raudenbush, S. W. (2005) · Educational Researcher · doi:10.3102/0013189x034005025 · Cited by 171 · cited 12×
“Hong and Raudenbush (2005) did not use the “Gold Standard” of random assignment to treatment conditions (e.g., Eisenhart & Towne, 2008; U.S. Department of Education, 2002).”
BackgroundMethodInfluentialThe Impact of Dual Enrollment on College Degree Attainment: Do Low-SES Students Benefit?
An, B. P. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712461933 · Cited by 153
Experimentation and social interventions: a forgotten but important history
Oakley, A. (1998) · BMJ · doi:10.1136/bmj.317.7167.1239 · Cited by 150 · cited 1×
“…causal inferences in the social sciences dates back to the 1900s (e.g., Becker, 1967; Rubin, 1974; Thorndike & Woodworth, 1901; see Abbott, 1998 or Oakley, 1998, for reviews), with some heated as in the Cronbach versus Campbell exchanges of the 1980s (e.g., Cook & Campbell, 1979; Cronbach, 1982).”
BackgroundSteiner, P. M., Cook, T. D., & Shadish, W. R. (2011) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998610375835 · Cited by 148
Dropping out of High School in the United States: An Observational Study
Rosenbaum, P. R. (1986) · Journal of Educational Statistics · doi:10.2307/1165073 · Cited by 140 · cited 4×
“For example, Rosenbaum (1986) wrote, The inclusion of an omitted variable U, which is as predictive as the most predictive covariate in the short list of covariates excluding the pretest, would have a relatively modest impact on the estimated effect [of dropout on cognitive test scores] unless that…”
BackgroundInfluentialTeaching Statistical Inference for Causal Effects in Experiments and Observational Studies
Rubin, D. B. (2004) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986029003343 · Cited by 127 · cited 1×
“Rubin (2004) would differentiate possible outcomes using Y(1) for the treatment and Y(0) for the control, but the parenthetical expressions become awkward when part of a larger function.”
BackgroundDual and simultaneous sensitivity analysis for matched pairs
Gastwirth, J. (1998) · Biometrika · doi:10.1093/biomet/85.4.907 · Cited by 117
Will a Clinical Approach Make Education Research More Relevant for Practice?
Bulterman-Bos, J. A. (2008) · Educational Researcher · doi:10.3102/0013189x08325555 · Cited by 116 · cited 1×
“Education is fundamentally a pragmatic enterprise (e.g., National Research Council, 2002; Raudenbush, 2005), with the ultimate goal of educational research to inform choices about curricula, pedagogy, practices, or school organization (e.g., Bulterman-Bos, 2008; Cook, 2002).”
BackgroundSaunders, W. M., & Marcelletti, D. J. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712461849 · Cited by 100
Quick Analysis for Busy Decision Makers
Winer, M. D., Behn, R. D., & Vaupel, J. W. (1983) · Journal of Policy Analysis and Management · doi:10.2307/3324047 · Cited by 97
Yuan, K., Le, V.-N., McCaffrey, D. F., Marsh, J. A., Hamilton, L. S., Stecher, B. M., & Springer, M. G. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462625 · Cited by 92
Complex Systems View of Educational Policy Research
Maroulis, S. J., Guimerà, R., Petry, H., Stringer, M. J., Gómez, L. M., Amaral, L. A. N., & Wilensky, U. J. (2010) · Science · doi:10.1126/science.1195153 · Cited by 91 · cited 1×
“For example, one can use agent-based models to explore the dynamic implications of changes in treatment assignments (e.g., Maroulis et al., 2010).”
MethodSelf-citationShaping Up the Practice of Null Hypothesis Significance Testing
Wainer, H., & Robinson, D. H. (2003) · Educational Researcher · doi:10.3102/0013189x032007022 · Cited by 90
Alexander, K. L., & Pallas, A. M. (1983) · Sociology of Education · doi:10.2307/2112546 · Cited by 89 · cited 1×
“Indeed, in a critique of the Coleman findings, Alexander and Pallas (1983) noted that the single greatest burden of school effects research is to distinguish convincingly between outcome differences that reflect simply differences in the kinds of students who attend various schools from differences…”
BackgroundTransportability Across Studies: A Formal Approach
Pearl, J., & Bareinboim, E. (2011) · Defense Technical Information Center (DTIC) · doi:10.21236/ada557437 · Cited by 83
Can High Schools Reduce College Enrollment Gaps With a New Counseling Model?
Stephan, J. L., & Rosenbaum, J. E. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462624 · Cited by 79
Why have Educational Evaluators Chosen Not to Do Randomized Experiments?
Cook, T. D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254764 · Cited by 76 · cited 3×
“This is because it is rare when an educational researcher can randomly sample participants from the desired target population and then also randomly assign participants to meaningful treatment conditions (Cook, 2003).”
BackgroundQuick Analysis for Busy Decision Makers, by Robert D. Behn and James W. Vaupel
Bloom, H. S. (1984) · Political Science Quarterly · doi:10.2307/2150422 · Cited by 74 · cited 1×
“Equally important, our framework enables researchers to identify a “switch point” (Behn & Vaupel, 1982) where the bias is large enough to undo one’s belief about an effect (e.g., from inferring an effect to inferring no effect).”
BackgroundDoes NBPTS Certification Affect the Number of Colleagues a Teacher Helps With Instructional Matters?
Frank, K. A., Sykes, G., Anagnostopoulos, D., Cannata, M., Chard, L., Krause, A., & McCrory, R. (2008) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373707313781 · Cited by 71 · cited 1×
“After doing so, one could apply our framework to either estimate (e.g., Frank et al., 2008).”
MethodSelf-citationShager, H. M., Schindler, H. S., Magnuson, K. A., Duncan, G. J., Yoshikawa, H., & Hart, C. M. D. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712462453 · Cited by 65
Nomi, T. (2012) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712453869 · Cited by 63
Lorence, J., Dworkin, A. G., Toenjes, L. A., & Hill, A. N. (2002) · Brookings papers on education policy · doi:10.1353/pep.2002.0014 · Cited by 61
van der Vleuten, C. (1995) · AJP Advances in Physiology Education · doi:10.1152/advances.1995.269.6.s3 · Cited by 59
Effects of Two Scientific Inquiry Professional Development Interventions on Teaching Practice
Grigg, J., Kelly, K. A., Gamoran, A., & Borman, G. D. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712461851 · Cited by 55
The Differential Effects of Repeating Kindergarten
Burkam, D. T., LoGerfo, L. F., Ready, D. D., & Lee, V. E. (2007) · Journal of Education for Students Placed at Risk (JESPAR) · doi:10.1080/10824660701261052 · Cited by 50
A Multisite Cluster Randomized Field Trial of Open Court Reading
Borman, G. D., Dowling, N. M., & Schneck, C. (2008) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373708326283 · Cited by 48 · cited 16×
“Furthermore, Borman et al. carefully attended to sample attrition in their analyses. Even given Borman et al.’s (2008) attention to their sample, there may be important concerns about the external validity of Borman et al.’s inference. In particular, schools that had approached SRA/McGraw-Hill prior to Borman et al.’s…”
BackgroundMethodInfluentialThe Academic Effects of Summer Instruction and Retention in New York City
Mariano, L. T., & Martorell, P. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712454327 · Cited by 46 · cited 1×
“The use of superscripts is as in Winship and Morgan (1999). 2.”
MethodThe Conventional Wisdom About Group Mean Scores
Brennan, R. L. (1995) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1995.tb00473.x · Cited by 42 · cited 1×
“The reliability of the Terra Nova comprehensive reading test used by Borman et al. (2008) was very high (.84-.93 across Grades 3–6—SRA/McGraw-Hill, 2001), and would likely be higher at the classroom level given fairly large sample sizes per classroom (see Brennan, 1995, for a discussion).”
MethodWISC-III profiles for children with and without learning disabilities
Dickerson Mayes, S., Calhoun, S. L., & Crowell, E. W. (1998) · Psychology in the Schools · doi:10.1002/(sici)1520-6807(199810)35:4<309::aid-pits1>3.0.co;2-v · Cited by 40
10. Indices of Robustness for Sample Representation
Frank, K., & Min, K.-S. (2007) · Sociological Methodology · doi:10.1111/j.1467-9531.2007.00186.x · Cited by 38 · cited 4×
“The correlation was obtained from the ratio of the estimate to its standard error: r t df t = + 2 , and the threshold was obtained using the critical value of the t distribution: r t df t # = critical critical+ 2 (Frank & Min, 2007).”
BackgroundMethodInfluentialSelf-citationMethods for Using Selection on Observed Variables to Address Selection on Unobserved VariablesPreprint
Elder, T. E., Conley, T. G., Taber, C. R., & Altonji, J. G. (2009) · RePEc: Research Papers in Economics · Cited by 37
Bozick, R., & Dalton, B. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712453870 · Cited by 36
Greco, J. (2015) · Routledge eBooks · doi:10.4324/9780203839065.ch21 · Cited by 35
Scharfstein, D. O., & Irizarry, R. A. (2003) · Biometrics · doi:10.1111/1541-0420.00070 · Cited by 34 · cited 2×
“Typical sensitivity analyses are expressed in terms of the properties of the omitted variables that could alter an inference either because they affected selection into the sample or assignment to treatments or both (e.g., Copas & Li, 1997; Holland, 1989; Lin, Psaty, & Kronmal, 1998; Robins, Rotnisky, & Scharfstein, 2…”
BackgroundCarlson, D. E., Cowen, J. M., & Fleming, D. J. (2012) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712461852 · Cited by 33
Knowledge and Human Interests.
Geuss, R., Habermas, J., & Shapiro, J. J. (1975) · The Journal of Philosophy · doi:10.2307/2024974 · Cited by 26
Shepard, L. A., Smith, M. L., & Marion, S. F. (1998) · Psychology in the Schools · Cited by 22
Lorence, J., Dworkin, A. G., Toenjes, L. A., et al. (2002) · Brookings papers on education policy · Cited by 20
Miller, S., & Connolly, P. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373712452628 · Cited by 17
Holland, P. W. (1989) · Journal of the American Statistical Association · doi:10.1080/01621459.1989.10478849 · Cited by 16 · cited 1×
“…that could alter an inference either because they affected selection into the sample or assignment to treatments or both (e.g., Copas & Li, 1997; Holland, 1989; Lin, Psaty, & Kronmal, 1998; Robins, Rotnisky, & Scharfstein, 2000; Rosenbaum, 1986, 2002; Rosenbaum & Rubin, 1983; Scharfstein, 2002).”
Background
Research works printed but not linked: 16
- Alexander K. L., 1998, Psychology in Schools, 9, 410
- Bogatz G. A., 1972, The impact of “sesame street” on children’s first school experience
- 10.3102/0162373712461933 · DOI · Paper page
- 10.3102/00028312027003557 · doi:10.3102/00028312027003557 · Paper page
- Greco J., 2009, Epistemic value, 313
- Harvard Education Letter, 1986, Harvard Education Letter, 2, 1
- Holmes C. T., 1989, Flunking grades, 16
- 10.3102/00346543054002225 · doi:10.3102/00346543054002225 · Paper page
- Manski C., 1990, American Economic Review Papers and Proceedings, 80, 319
- 10.3102/01623737014002101 · doi:10.3102/01623737014002101 · Paper page
- Roderick M., 1999, Ending social promotion: Results from the first two years
- 10.3102/10769986011003207 · doi:10.3102/10769986011003207 · Paper page
- Shepard L. A., 1989, Flunking grades
- 10.1002/(SICI)1520-6807(199810)35:4<404::AID-PITS14>3.0.CO;2-L · doi:10.1002/(sici)1520-6807(199810)35:4<404::aid-pits14>3.0.co;2-l
- SRA/McGraw-Hill, 2001, Technical report performance assessments
- US Department of Health and Human Services, 2000, Trends in the well-being of America’s children and youth
Cited by
Making sense of sensitivity: extending omitted variable bias
Cinelli, C., & Hazlett, C. (2020) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/rssb.12348 · Cited by 1,088
Xu, D., Zhou, K. Z., & Du, F. (2019) · Academy of Management Journal · doi:10.5465/amj.2016.0749 · Cited by 510
Busenbark, J. R., Yoon, H. (., Gamache, D. L., & Withers, M. C. (2022) · Journal of Management · doi:10.1177/01492063211006458 · Cited by 457
“…to highlight that the ITCV is useful to examine the causal inference and statistical significance of relationships between independent and dependent variables, but it does not address inaccuracies of the parameter estimate itself (Frank, 2000; Frank et al., 2013; Larcker & Rusticus, 2010).”
BackgroundMethodInfluentialNonprice incentives and energy conservation
Asensio, O. I., & Delmas, M. A. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1401880112 · Cited by 422
Wolske, K. S., Stern, P. C., & Dietz, T. (2017) · Energy Research & Social Science · doi:10.1016/j.erss.2016.12.023 · Cited by 406
Metadata
- Authors
- Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M.
- Affiliations
- Michigan State University (US)
- Arizona State University (US)
- Pacific Metrics Corporation
- University of Cincinnati (US)
- Venue
- Educational Evaluation and Policy Analysis
- Published
- 2013-12
- Volume
- 35
- Issue
- 4
- Pages
- 437-460
- ISSN
- 0162-3737 (ISSN-L)
- 1935-1062
- OpenAlex
- W2114805009
- Cited by
- 406
- References
- 110
- Citations by year
- 2026: 67 · 2025: 55 · 2024: 45 · 2023: 50 · 2022: 58 · 2021: 27 · 2020: 37 · 2019: 15 · 2018: 22 · 2017: 11 · 2016: 9 · 2015: 7 · 2014: 3
- FWCI
- 5.57
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Educational Evaluation and Policy Analysis (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- causal inference, inference robustness, reading achievement, kindergarten retention, randomized experiments, observational studies, research design, scientific discourse
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.