Challenges and Dilemmas in Implementing Random Assignment in Educational Research
Abstract
In the contentious debate over the use of randomized clinical trials (RCTs) in education research, little attention has been given to how methodological issues arise and/or are resolved in the implementation of random assignment. Following a review of the methodological-theoretical literature on RCTs and alternative research designs, this article analyzes qualitative and quantitative data on three years of random assignment in a multi-site literacy intervention for adolescents. Practical lessons are drawn about how researcher practices and learning by school staff contributed to random assignment compliance. Consideration of challenges encountered in implementing random assignment suggests that 1) researcher communication with program staff improves compliance, but may not overcome the need for learning through experience; 2) in keeping with arguments in favor of random assignment-based research, random assignment may control for diverse selection processes that would undermine valid treatment-control comparisons in some non-RCT research designs; and 3) in keeping with arguments against random assignment-based research, random assignment may lead to samples dissimilar to target populations. In concluding, the broader limitations of research that focus on a priori causes and quantifiable effects are noted.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Students who are assigned to the control group may underperform due to a resentful awareness of not being included.
“students who are assigned to the control group may underperform due to a resentful awareness of not being included (Ong-Dean et al., 2011).”
From Outcomes and Processes in the Meyerhoff Scholars Program: STEM PhD Completion, Sense of Community, Perceived Program Benefit, Science Identity, and Research Self-Efficacy (2016) · cites this paper as Ong-Dean et al., 2011
Cites
(1963) · Medical Entomology and Zoology · Cited by 15,085 · see citing works
(2012) · American Economic Review · doi:10.1257/aer.102.2.i · Cited by 14,047 · see citing works
(1979) · Medical Entomology and Zoology · Cited by 11,252 · see citing works
(2014) · SAGE Publications Ltd eBooks · doi:10.4135/9781446294406.n148 · Cited by 7,272 · see citing works
Quasi-experimentation: Design and analysis issues for field settings
Shrout, P. E. (1980) · Evaluation and Program Planning · doi:10.1016/0149-7189(80)90063-4 · Cited by 6,091 · see citing works
Neufeld, A., & Ryan, A. G. (1988) · Canadian Journal of Program Evaluation · doi:10.3138/cjpe.3.006 · Cited by 5,906 · see citing works
Designing Evaluations of Educational and Social Programs.
Hendrickson, L., & Cronbach, L. J. (1983) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068975 · Cited by 1,035 · see citing works
Comment: Educational research: The hardest science of all
Berliner, D. C. (2002) · Educational Researcher · doi:10.3102/0013189x031008018 · Cited by 772 · see citing works
Statistical analysis and optimal design for cluster randomized trials.
Raudenbush, S. W. (1997) · Psychological Methods · doi:10.1037/1082-989x.2.2.173 · Cited by 636 · see citing works
Cartwright, N. D. (2007) · BioSocieties · doi:10.1017/s1745855207005029 · Cited by 588 · see citing works
Accounting for No-Shows in Experimental Evaluation Designs
Bloom, H. S. (1984) · Evaluation Review · doi:10.1177/0193841x8400800205 · Cited by 532 · cited 2× · see citing works
“A common method of estimating TOT effects is the Bloom adjustment (Bloom, 1984). This adjustment, which assumes no effect of the treatment on ‘‘noshows’’ (i.e., treatment-assigned subjects who do not participate in the treatment), increases the estimated treatment effect in proportion to the number of no-shows. Howeve…”
MethodShadish, W. R., Clark, M. H., & Steiner, P. M. (2008) · Journal of the American Statistical Association · doi:10.1198/016214508000000733 · Cited by 479 · see citing works
Nonexperimental Versus Experimental Estimates of Earnings Impacts
Glazerman, S., Levy, D. M., & Myers, D. (2003) · The ANNALS of the American Academy of Political and Social Science · doi:10.1177/0002716203254879 · Cited by 352 · see citing works
Comment: Culture, Rigor, and Science in Educational Research
Erickson, F., & Gutiérrez, K. D. (2002) · Educational Researcher · doi:10.3102/0013189x031008021 · Cited by 206 · see citing works
Social Experiments: Some Developments Over the Past Fifteen Years
Cook, T. D., & Shadish, W. R. (1994) · Annual Review of Psychology · doi:10.1146/annurev.ps.45.020194.002553 · Cited by 187 · see citing works
Toward Reform of Program Evaluation: Aims, Methods, and Institutional Arrangements.
Cain, P. S., & Cronbach, L. J. (1982) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068404 · Cited by 167 · see citing works
Toward reform of program evaluation: Aims, methods, and institutional arrangements
Weiss, C. H. (1981) · Evaluation and Program Planning · doi:10.1016/0149-7189(81)90038-0 · Cited by 152 · see citing works
Using Cluster Random Assignment to Measure Program Impacts
Bloom, H. S., Bos, J. M., & Lee, S.-W. (1999) · Evaluation Review · doi:10.1177/0193841x9902300405 · Cited by 121 · see citing works
Wolff, N. (2000) · The Journal of Mental Health Policy and Economics · doi:10.1002/1099-176x(200006)3:2 · Cited by 119 · see citing works
Objecting to the objections to using random assignment in educational research
Cook, T. D., & Payne, M. R. (2001) · Cited by 83 · see citing works
A quasi‐sampling theory of the generalization of causal relationships
Cook, T. D. (1993) · New Directions for Program Evaluation · doi:10.1002/ev.1638 · Cited by 79 · see citing works
Contemporary Thinking About Causation in Evaluation: A Dialogue With Tom Cook and Michael Scriven
Cook, T. D., Scriven, M. S., Coryn, C. L. S., & Evergreen, S. D. H. (2009) · American Journal of Evaluation · doi:10.1177/1098214009354918 · Cited by 66 · see citing works
The politics of random assignment: implementing studies and impacting policy
Gueron, J. M. (2008) · Journal of Children s Services · doi:10.1108/17466660200800003 · Cited by 35 · see citing works
Missing the target and missing the point: ‘successful’ random assignment but misleading results
Goldkamp, J. S. (2008) · Journal of Experimental Criminology · doi:10.1007/s11292-008-9052-6 · Cited by 15 · see citing works
Jenney, B., & Lohr, S. L. (2012) · Cited by 2 · see citing works
Jenney, B., & Lohr, S. (2009) · The Annals of Applied Statistics · doi:10.1214/08-aoas216 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Maton, K. I., Beason, T. S., Godsay, S., Sto. Domingo, M. R., Bailey, T. C., Sun, S., & Hrabowski, F. A. (2016) · CBE—Life Sciences Education · doi:10.1187/cbe.16-01-0062 · Cited by 147 · see citing works
Robinson, K., & Webber, M. (2012) · The British Journal of Social Work · doi:10.1093/bjsw/bcs025 · Cited by 105 · see citing works
Thomas, G. (2016) · Harvard Educational Review · doi:10.17763/1943-5045-86.3.390 · Cited by 62 · see citing works
Flowers, C. P., Test, D. W., Povenmire-Kirk, T., Diegelmann, K. M., Bunch-Crump, K. R., Kemp-Inman, A., & Goodnight, C. I. (2017) · The Journal of Special Education · doi:10.1177/0022466917720764 · Cited by 43 · see citing works
Pant, H. A. (2014) · Zeitschrift für Erziehungswissenschaft · doi:10.1007/s11618-014-0510-9 · Cited by 42 · see citing works
Evaluating Methods for Estimating Program Effects
Reichardt, C. S. (2011) · American Journal of Evaluation · doi:10.1177/1098214011398954 · Cited by 31 · see citing works
A Comprehensive Inclusion Program for Kindergarten Children With Autism Spectrum Disorder
Sainato, D. M., Morrison, R. S., Jung, S., Axe, J. B., & Nixon, P. A. (2015) · Journal of Early Intervention · doi:10.1177/1053815115613836 · Cited by 30 · see citing works
Kigobe, J., Noortgate, W. V. D., Ligembe, N. N., Ogondiek, M. W., Ghesquière, P., & Leeuwen, K. G. V. (2021) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2021.1931998 · Cited by 26 · see citing works
Farquharson, K., & Murphy, K. A. (2016) · Frontiers in Psychology · doi:10.3389/fpsyg.2016.00419 · Cited by 22 · see citing works
Scanlan, J. N., Berry, B., Wells, K., & Somerville, J. (2022) · Australian Occupational Therapy Journal · doi:10.1111/1440-1630.12837 · Cited by 20 · see citing works
Mujtaba, S. J., Kigobe, J., & Leeuwen, K. G. V. (2024) · Teaching and Teacher Education · doi:10.1016/j.tate.2023.104459 · Cited by 11 · see citing works
Lewis, K. M., DuBois, D. L., Ji, P. Y., Day, J. P., Silverthorn, N., Bavarian, N., Vuchinich, S., Acock, A. C., Malloy, M., Schure, M. B., et al. (2016) · Evaluation & the Health Professions · doi:10.1177/0163278716673688 · Cited by 9 · see citing works
The effects of teaching mental calculation in the development of mathematical abilities
Ruiz, C., & Balbi, A. (2018) · The Journal of Educational Research · doi:10.1080/00220671.2018.1519689 · Cited by 7 · see citing works
Leow, C., Wen, X., & Korfmacher, J. (2014) · Applied Developmental Science · doi:10.1080/10888691.2014.977995 · Cited by 6 · see citing works
YİĞİTER, M. S. (2026) · Current Psychology · doi:10.1007/s12144-025-08576-3 · Cited by 3 · see citing works
Liu, Q., Dong, Z., & Han, H. (2025) · Social Psychology of Education · doi:10.1007/s11218-024-09995-z · Cited by 3 · see citing works
Claravall, E. B., Irey, R. C., & Shen, M. (2026) · Journal of Mixed Methods Research · doi:10.1177/15586898261422330 · Cited by 2 · see citing works
Mayasari, S., Fawziah Zahrawati, B., Ramli, N., Zurahmah, Z., & Minasyan, S. (2025) · AL MA ARIEF Jurnal Pendidikan Sosial dan Budaya · doi:10.35905/almaarief.v7i2.14649 · Cited by 2 · see citing works
Investigating the impact of argumentation on high school students’ achievement in probability
Chitera, J., Gudyanga, R., & Dziva, D. (2025) · Advanced Research Journal · doi:10.71350/3062192515 · Cited by 1 · see citing works
Towards More Effective Teacher Professional Development Initiatives
Hooks, L. S. (2015) · ScholarWorks (Walden University)
Motivation: how to tame the elephant in the classroom?
Prince, A. (2014)
THE RELATIONSHIP BETWEEN CHALLENGE COURSES AND THE
Luna, C. (2018) · UNM’s Digital Repository (University of New Mexico)
Schwartz, S. J. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780190095918.003.0003
Fields, D., & Asbury, K. (2025) · International Journal of Educational Research · doi:10.1016/j.ijer.2025.102856
Olson, D. J., Doyle, P., & Jiang, Y. (2026) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2026.100338
- Authors
- Ong-Dean, C., Huie Hofstetter, C., & Strick, B. R.
- Affiliations
- University of California San Diego (US)
- University of California, San Diego, CA, USA
- Corresponding author
- Ong-Dean, Colin
- ORCID
- Ong-Dean, Colin
- Venue
- American Journal of Evaluation
- Publisher
- SAGE Publications
- Published
- 2010-08-30
- Volume
- 32
- Issue
- 1
- Pages
- 29-49
- ISSN
- 1098-2140 (ISSN-L)
- 1557-0878
- OpenAlex
- W2146317635
- Cited by
- 26
- References
- 27
- Citations by year
- 2026: 3 · 2025: 4 · 2024: 1 · 2022: 1 · 2021: 3 · 2018: 2 · 2017: 1 · 2016: 4 · 2015: 2 · 2014: 3 · 2012: 1
- FWCI
- 0.52
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- American Journal of Evaluation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Evaluation and Performance Assessment
- Social Sciences › Education › Educational Research and Methods
- Keywords
- random assignment, research implementation challenges, literacy intervention, adolescents, selection bias, sample representativeness, experiential learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
