Why have Educational Evaluators Chosen Not to Do Randomized Experiments?
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
This article analyzes the reasons that have been adduced within the community of educational evaluators for not doing randomized experiments. The objections vary in cogency. Those that have most substance are not insurmountable, however, and strategies are mentioned for dealing with them. However, the objections are serious enough, and the remedies partial enough, that it seems hardly warranted to call experiments the “gold standard” of causal inference. Yet even if they are not perfect in research practice, this article shows how they are logically and empirically superior to all currently known alternatives. The article particularly addresses the objection that school personnel will not accept experiments. It shows that hundreds of them have been done there by researchers with backgrounds in psychology and public health who study the prevention of unhealthy behaviors. But experiments are much rarer among researchers trained in education who study changing academic performance. Reasons are adduced for this difference in academic culture within school-based research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · cited 2×
“Control groups generated through random assignment provide the best counterfactual to describe what would have happened to students in a treatment group if they had not been exposed to the treatment (Rubin 1974; Holland 1986).”
BackgroundIdentification of Causal Effects Using Instrumental Variables
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476902 · Cited by 4,230
Ontological Relativity and Other Essays
Quine, W. V. (1969) · doi:10.7312/quin92204 · Cited by 3,593
Does matching overcome lalonde’s critique of nonexperimental estimators?
A. Smith, J., & E. Todd, P. (2005) · Journal of Econometrics · doi:10.1016/j.jeconom.2004.04.011 · Cited by 3,176
Characterizing Selection Bias Using Experimental Data
Heckman, J., Ichimura, H., Smith, J., & Todd, P. (1998) · Econometrica · doi:10.2307/2999630 · Cited by 2,286
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033 · cited 1×
“However, there have been three instances where a series of experiments on a topic was contrasted with a series of nonexperiments on the same topic (Lipsey and Wilson 1993; Bloom et al. 2002; Glazerman et al. 2002).”
BackgroundOlds, D. L. (1997) · JAMA · doi:10.1001/jama.1997.03550080047038 · Cited by 1,372 · cited 1×
“…the Abcedarian Project (Ramey and Campbell 1991; Campbell and Ramey 1995), the Comprehensive Child Development Program (Goodson et al. 2000), Olds’s home nurse visiting program (Olds et al. 1997), Early Head Start (Raikes and Love 2002), and even the new Head Start study (Cook and Puma 2002).”
BackgroundExperimental Estimates of Education Production Functions
Krueger, A. B. (1999) · The Quarterly Journal of Economics · doi:10.1162/003355399556052 · Cited by 1,120
Primary prevention mental health programs for children and adolescents: A meta-analytic review
Durlak, J. A., & Wells, A. M. (1997) · American Journal of Community Psychology · doi:10.1023/a:1024654026646 · Cited by 963 · cited 1×
“For instance, school-based prevention studies are designed to improve student health; to prevent school violence; or to reduce teen use of tobacco, drugs, and alcohol (e.g., Cook, Anson, and Walchli 1993; Peters and McMahon 1996; Durlak and Wells 1997a, 1997b, 1998).”
Background
Cited by
Frank, K. A., Maroulis, S. J., Duong, M. Q., & Kelcey, B. M. (2013) · Educational Evaluation and Policy Analysis · doi:10.3102/0162373713493129 · Cited by 406
“This is because it is rare when an educational researcher can randomly sample participants from the desired target population and then also randomly assign participants to meaningful treatment conditions (Cook, 2003).”
BackgroundStudents as Experimental Participants: A Defense of the 'Narrow Data Base'
Druckman, J. N., & Kam, C. D. (2009) · SSRN Electronic Journal · doi:10.2139/ssrn.1498843 · Cited by 350
Sharif Ishak, S. I. Z., Chin, Y. S., Mohd. Taib, M. N., Chan, Y. M., & Mohd. Shariff, Z. (2020) · BMC Pediatrics · doi:10.1186/s12887-020-02023-x · Cited by 46
Survey research methods in evaluation and case-control studies
Kalton, G., & Piesse, A. (2007) · Statistics in Medicine · doi:10.1002/sim.2796 · Cited by 27
Metadata
- Authors
- Cook, T. D.
- Venue
- The ANNALS of the American Academy of Political and Social Science
- Publisher
- SAGE Publications
- Published
- 2003-09
- Volume
- 589
- Issue
- 1
- Pages
- 114-149
- ISSN
- 0002-7162 (ISSN-L)
- 1552-3349
- OpenAlex
- W2170966242
- Cited by
- 76
- References
- 9
- FWCI
- 2.61
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Why have Educational Evaluators Chosen Not to Do Randomized Experiments? https://app.cochat.ai/papers/10.1177/0002716203254764
- The paper
- Cook, T. D. (2003). Why have Educational Evaluators Chosen Not to Do Randomized Experiments? The ANNALS of the American Academy of Political and Social Science. https://doi.org/10.1177/0002716203254764
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.