Assessing Intervention Effects in the Presence of Missing Scores
Open accessAbstract
Due to repeated observations of an outcome behavior in N-of-1 or single-case design (SCD) intervention studies, the occurrence of missing scores is inevitable in such studies. Approximately 21% of SCD articles published in five reputable journals between 2015 and 2019 exhibited evidence of missing scores. Missing rates varied by designs, with the highest rate (24%) found in multiple baseline/probe designs. Missing scores cause difficulties in data analysis. And inappropriate treatments of missing scores lead to consequences that threaten internal validity and weaken generalizability of intervention effects reported in SCD research. In this paper, we comprehensively review nine methods for treating missing SCD data: the available data method, six single imputations, and two model-based methods. The strengths, weaknesses, assumptions, and examples of these methods are summarized. The available data method and three single imputation methods are further demonstrated in assessing an intervention effect at the class and students’ levels. Assessment results are interpreted in terms of effect sizes, statistical significances, and visual analysis of data. Differences in results among the four methods are noted and discussed. The extensive review of problems caused by missing scores and possible treatments should empower researchers and practitioners to account for missing scores effectively and to support evidence-based interventions vigorously. The paper concludes with a discussion of contingencies for implementing the nine methods and practical strategies for managing missing scores in single-case intervention studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A clear consensus to address missing data in SCEDs does not yet exist nor does it feature under the SCRIBE reporting guidelines or risk of bias tools for SCEDs.
“Moreover, a clear consensus to address missing data in SCEDs does not yet exist nor does it feature under the SCRIBE reporting guidelines or risk of bias tools for SCEDs [83,84,85,86].”
From Brief and Telehealth Acceptance and Commitment Therapy (ACT) Interventions for Stress in Inflammatory Bowel Disease (IBD): A Series of Single Case Experimental Design (SCED) Studies (2022) · cites this paper as [86]
The Introduction and Discussion and Conclusions say that SCRIBE 2016 and the WWC handbooks offer no recommendations for viable methods of handling missing data in single-case designs. But the paper also says SCRIBE defines missing data and requires reporting its reasons and how it was handled, so missing data is not absent from those reporting guidelines. The cited passages do not establish the claim about risk-of-bias tools specifically; page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · see citing works
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Multiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · see citing works
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473 · see citing works
Multiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072 · see citing works
Schafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949 · see citing works
The Use of Single-Subject Research to Identify Evidence-Based Practice in Special Education
Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005) · Exceptional Children · doi:10.1177/001440290507100203 · Cited by 3,368 · see citing works
Multiple Imputation After 18+ Years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.2307/2291635 · Cited by 2,686 · see citing works
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389 · see citing works
Principled missing data methods for researchers
Dong, Y., & Peng, C.-Y. J. (2013) · SpringerPlus · doi:10.1186/2193-1801-2-222 · Cited by 2,208 · see citing works
Self-citationAllison, P. (2002) · doi:10.4135/9781412985079 · Cited by 2,050 · see citing works
Edgington, E. S., Edgington, E., & Onghena, P. (1964) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2169 · Cited by 2,034 · see citing works
How can I deal with missing data in my study?
Bennett, D. A. (2001) · Australian and New Zealand Journal of Public Health · doi:10.1111/j.1467-842x.2001.tb00294.x · Cited by 1,656 · see citing works
Single-case experimental designs: a systematic review of published research and current standards
Smith, J. D. (2012) · Psychological Methods · doi:10.1037/a0029312 · Cited by 630 · cited 5× · see citing works
“Sci. 2021, 11, 76. https://doi.org/10.3390/educsci11020076 https://www.mdpi.com/journal/education In a review of SCD standards and 409 empirical studies published in refereed journals between 2000 and 2010, Smith [11] lamented, “SCDs undeniably present researchers with a complex array of methodological and research de…”
BackgroundInfluentialAdding Missing-Data-Relevant Variables to FIML-Based Structural Equation Models
Graham, J. A. (2003) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1001_4 · Cited by 621 · see citing works
Handbook of Statistical Modeling for the Social and Behavioral Sciences
Arminger, G., Clogg, C. C., & Sobel, M. E. (1995) · Springer eBooks · doi:10.1007/978-1-4899-1292-3 · Cited by 488 · see citing works
Horton, N., & Kleinman, K. (2007) · American Statistician · doi:10.1198/000313007x172556 · Cited by 476 · see citing works
Bootstrap inference when using multiple imputation
Schomaker, M., & Heumann, C. (2018) · Statistics in Medicine · doi:10.1002/sim.7654 · Cited by 466 · see citing works
Kellam, S. G., Rebok, G. W., Ialongo, N. S., & Mayer, L. S. (1994) · Journal of Child Psychology and Psychiatry · doi:10.1111/j.1469-7610.1994.tb01161.x · Cited by 407 · see citing works
Characteristics of single-case designs used to assess intervention effects in 2008
Shadish, W. R., & Sullivan, K. J. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0111-y · Cited by 338 · see citing works
A standardized mean difference effect size for single case designs
Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2012) · Research Synthesis Methods · doi:10.1002/jrsm.1052 · Cited by 308 · see citing works
An Improved Rank Correlation Effect Size Statistic for Single-Case Designs: Baseline Corrected Tau
Tarlow, K. R. (2016) · Behavior Modification · doi:10.1177/0145445516676750 · Cited by 289 · see citing works
Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014) · Journal of School Psychology · doi:10.1016/j.jsp.2013.11.005 · Cited by 270 · see citing works
Design Specification Issues in Time-Series Intervention Models
Huitema, B. E., & McKean, J. W. (2000) · Educational and Psychological Measurement · doi:10.1177/00131640021970358 · Cited by 250 · see citing works
Advances in Missing Data Methods and Implications for Educational Research
Peng, C. J., Harwell, M. R., Liou, S.-M., & Ehman, L. H. (2006) · Emerald Group Publishing Limited eBooks · doi:10.1108/978-1-60752-578-320251005 · Cited by 224 · see citing works
Self-citationA phase1 study of stereotactic gene delivery of AAV2-NGF for Alzheimer’s disease
Rafii, M. S., Baumann, T. L., Bakay, R. A. E., Ostrove, J. M., Siffert, J., Fleisher, A. S., Herzog, C. D., Barba, D., Pay, M., Salmon, D. P., et al. (2014) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2013.09.004 · Cited by 223 · see citing works
Tate, R. L., Perdices, M., Rosenkoetter, U., McDonald, S., Togher, L., Shadish, W., Horner, R., Kratochwill, T., Barlow, D. H., Kazdin, A., et al. (2016) · Archives of Scientific Psychology · doi:10.1037/arc0000027 · Cited by 219 · see citing works
Single-case experimental designs. Evaluating interventions in research and clinical practice
Kazdin, A. E. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2018.11.015 · Cited by 217 · see citing works
McDonald, S., Quinn, F., Vieira, R., O’Brien, N., White, M., Johnston, D. W., & Sniehotta, F. F. (2017) · Health Psychology Review · doi:10.1080/17437199.2017.1316672 · Cited by 200 · see citing works
Considerations for the Systematic Analysis and Use of Single-Case Research
Horner, R. H., Swaminathan, H., Sugai, G. M., & Smolkowski, K. (2012) · Education and Treatment of Children · doi:10.1353/etc.2012.0011 · Cited by 172 · see citing works
Lambert, M. C., Cartledge, G., Heward, W. L., & Lo, Y. (2006) · Journal of Positive Behavior Interventions · doi:10.1177/10983007060080020701 · Cited by 140 · see citing works
Goodwin, L. D., & Goodwin, W. L. (1985) · Educational Researcher · doi:10.3102/0013189x014002005 · Cited by 131 · see citing works
An R package for single-case randomization tests
Bulté, I., & Onghena, P. (2008) · Behavior Research Methods · doi:10.3758/brm.40.2.467 · Cited by 113 · see citing works
Reichow, B., Barton, E. E., & Maggin, D. M. (2018) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2018.05.008 · Cited by 107 · see citing works
Long, A. C. J., Miller, F. G., & Upright, J. J. (2018) · School Psychology · doi:10.1037/spq0000305 · Cited by 67 · see citing works
Simulation modelling analysis for small sets of single-subject data collected over time
Borckardt, J. J., & Nash, M. R. (2014) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2014.895390 · Cited by 58 · see citing works
Can the theory of planned behaviour predict the physical activity behaviour of individuals?
Hobbs, N., Dixon, D. L., Johnston, M., & Howie, K. (2012) · Psychology and Health · doi:10.1080/08870446.2012.716838 · Cited by 55 · see citing works
Cosbey, J., & Muldoon, D. M. (2016) · Journal of Autism and Developmental Disorders · doi:10.1007/s10803-016-2977-0 · Cited by 49 · see citing works
Measuring and reducing college students' procrastination.
Perrin, C. J., Miller, N. D., Haberlin, A. T., Ivy, J. W., Meindl, J. N., & Neef, N. (2011) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.2011.44-463 · Cited by 47 · see citing works
Harrison, J. R., Soares, D. A., Rudzinski, S., & Johnson, R. (2019) · Review of Educational Research · doi:10.3102/0034654319857038 · Cited by 45 · see citing works
Smith, J. D., Eichler, W. C., Norman, K. R., & Smith, S. R. (2014) · Journal of Personality Assessment · doi:10.1080/00223891.2014.955917 · Cited by 43 · see citing works
McDonald, S., Vieira, R., & Johnston, D. W. (2020) · Health Psychology and Behavioral Medicine · doi:10.1080/21642850.2019.1711096 · Cited by 43 · see citing works
Goodwin, L. D., & Goodwin, W. L. (1985) · Educational Researcher · doi:10.2307/1174902 · Cited by 42 · see citing works
Pole, N., Ablon, J. S., & O’Connor, L. E. (2008) · Journal of Counseling Psychology · doi:10.1037/0022-0167.55.2.221 · Cited by 36 · see citing works
Smith, J. D., Borckardt, J. J., & Nash, M. R. (2011) · Behavior Therapy · doi:10.1016/j.beth.2011.10.001 · Cited by 35 · see citing works
Mason, M. J. (1999) · Measurement and Evaluation in Counseling and Development · doi:10.1080/07481756.1999.12068976 · Cited by 34 · see citing works
Handling missing data in randomization tests for single-case experiments: A simulation study
De, T. K., Michiels, B., Tanious, R., & Onghena, P. (2020) · Behavior Research Methods · doi:10.3758/s13428-019-01320-3 · Cited by 31 · see citing works
Martínez, J. R., Waters, C. L., Conroy, M. A., & Reichow, B. (2019) · Topics in Early Childhood Special Education · doi:10.1177/0271121419839136 · Cited by 30 · see citing works
Nyman, S. R., Goodwin, K., Kwaśnicka, D., & Callaway, A. J. (2015) · Psychology and Health · doi:10.1080/08870446.2015.1088014 · Cited by 28 · see citing works
Publication trends for single‐case methodology in school psychology: A systematic review
Radley, K. C., Dart, E. H., Fischer, A. J., & Collins, T. A. (2020) · Psychology in the Schools · doi:10.1002/pits.22359 · Cited by 28 · see citing works
Dealing with missing data by EM in single-case studies
Chen, L., Feng, Y., Wu, P.-J., & Peng, C. J. (2019) · Behavior Research Methods · doi:10.3758/s13428-019-01210-8 · Cited by 27 · see citing works
Self-citationQuinn, F. X., Johnston, M., & Johnston, D. W. (2013) · Psychology and Health · doi:10.1080/08870446.2013.814773 · Cited by 26 · see citing works
Computing Tools for Implementing Standards for Single-Case Designs
Chen, L.-T., Peng, C., & Chen, M. (2015) · Behavior Modification · doi:10.1177/0145445515603706 · Cited by 25 · see citing works
Self-citationCausal Analysis of Self-tracked Time Series Data Using a Counterfactual Framework for N-of-1 Trials*
Daza, E. J. (2018) · Methods of Information in Medicine · doi:10.3414/me16-02-0044 · Cited by 23 · see citing works
Handling missing data in single-case studies
Peng, C.-Y. J., & Chen, L.-T. (2018) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1525133280 · Cited by 22 · see citing works
Self-citationHidden Bias in the Use of Archival Data
Dworkin, R. J. (1987) · Evaluation & the Health Professions · doi:10.1177/016327878701000202 · Cited by 21 · see citing works
Accounting for baseline trends in intervention studies: Methods, effect sizes, and software
Chen, L., Wu, P.-J., & Peng, C. J. (2019) · Cogent Psychology · doi:10.1080/23311908.2019.1679941 · Cited by 19 · see citing works
Self-citationStiegler, J. R., Molde, H., & Schanche, E. (2017) · Clinical Psychology & Psychotherapy · doi:10.1002/cpp.2144 · Cited by 18 · see citing works
Five direct behavior rating multi-item scales: Sensitivity to the effects of classroom interventions
Matta, M., Volpe, R. J., Briesch, A. M., & Owens, J. S. (2020) · Journal of School Psychology · doi:10.1016/j.jsp.2020.05.002 · Cited by 15 · see citing works
Ratanavivan, W., & Ricard, R. J. (2018) · Journal of Counseling & Development · doi:10.1002/jcad.12223 · Cited by 14 · see citing works
Improved analyses of single cases: Dynamic multilevel analysis
Chiu, M. M., & Roberts, C. A. (2016) · Developmental Neurorehabilitation · doi:10.3109/17518423.2015.1119904 · Cited by 9 · see citing works
Power, Sample Size Estimation, and Early Stopping Rules
Franklin, R. D., Allison, D. B., & Gorman, B. S. (2014) · doi:10.4324/9781315806402-16 · Cited by 9 · see citing works
Analysis of Incomplete Multivariate Data (Monographs on Statistics and Applied Probability, No. 72)
Gadbury, G. L., & Schafer, J. L. (2000) · Journal of the American Statistical Association · doi:10.2307/2669497 · Cited by 8 · see citing works
Skills Training for Pregnant and Parenting Adolescents
Hall, J. A. (1995) · PsycEXTRA Dataset · doi:10.1037/e495732006-012 · Cited by 8 · see citing works
Flagged first, then by citations
1 of the 66 references couldn’t be matched to a known paper.
Research works printed but not linked: 3
- Busk, P.L.; Marascuilo, L.A. Autocorrelation in single-subject research: A counterargument to the myth of no autocorrelation. Behav. Assess. 1988, 10, 229-242.
- Buu, A. Analysis of Longitudinal Data with Missing Values: A Methodological Comparison. Ph.D. Thesis, Indiana University, Bloomington, IN, USA, 1999.
- Carpenter, J.R.; Goldstein, H. Multiple imputation in MLwiN. Multilevel Model. Newsl. 2004, 16, 9-18.
Cited by
Sorted by citations
Matko, K., Sedlmeier, P., & Bringmann, H. C. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.672301 · Cited by 37 · see citing works
Gruber, R., Gauthier-Gagné, G., Voutou, D., Somerville, G., Saha, S., & Boursier, J. (2021) · Child and Adolescent Psychiatry and Mental Health · doi:10.1186/s13034-021-00399-x · Cited by 33 · see citing works
Lavelle, J., Storan, D., Eswara Murthy, V., De Dominicis, N., Mulcahy, H. E., & McHugh, L. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11102757 · Cited by 23 · see citing works
Lebrault, H., Martini, R., Manolov, R., Chavanne, C., Krasny‐Pacini, A., & Chevignard, M. (2023) · Developmental Medicine & Child Neurology · doi:10.1111/dmcn.15759 · Cited by 17 · see citing works
Cheng, C. K. C., Chow, E. Y. H., Lam, K. C. K., & Lee, J. H. Y. (2023) · The International Journal of Management Education · doi:10.1016/j.ijme.2023.100887 · Cited by 16 · see citing works
McMahon, K., Cassiello-Robbins, C. F., Greenleaf, A. S., Guetta, R. E., Frazer-Abel, E., Kelley, L., & Rosenthal, M. Z. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1294571 · Cited by 15 · see citing works
Tanious, R., & Manolov, R. (2022) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2022.2035774 · Cited by 12 · see citing works
Rothschild, L. B., Ratto, A. B., Kenworthy, L., Hardy, K. K., Verbalis, A., Pugliese, C. E., Strang, J. F., Safer‐Lichtenstein, J., Anthony, B. J., Anthony, L. G., et al. (2022) · Journal of Clinical Psychology · doi:10.1002/jclp.23309 · Cited by 12 · see citing works
Aydın, O. (2023) · Behavior Modification · doi:10.1177/01454455231212265 · Cited by 11 · see citing works
How to assess and take into account trend in single-case experimental design data
Manolov, R., Lebrault, H., & Krasny‐Pacini, A. (2023) · Neuropsychological Rehabilitation · doi:10.1080/09602011.2023.2190129 · Cited by 8 · see citing works
Aydın, O. (2024) · Behavior Modification · doi:10.1177/01454455241226879 · Cited by 8 · see citing works
Applying Peer Tutoring to Spelling with Elementary-Aged Students
Lundberg, A. R., Petersen‐Brown, S., Houlihan, D., Panahon, C. J., & Wagner, D. (2022) · Journal of Applied School Psychology · doi:10.1080/15377903.2022.2052218 · Cited by 7 · see citing works
Hall, G. J., Putzeys, S., Kratochwill, T. R., & Levin, J. R. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09962-2 · Cited by 5 · see citing works
Analyzing data in single-case experimental designs: Objectives and available software options
Manolov, R., & Rochat, L. (2024) · Journal of Behavioral and Cognitive Therapy · doi:10.1016/j.jbct.2024.100511 · Cited by 5 · see citing works
Stapleton, A., Stynes, G., Cassidy, S., & McHugh, L. (2023) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.06.011 · Cited by 4 · see citing works
Stynes, G., Stapleton, A., MOORE, B. J., Russell, T., O’Connor, M., Richardson, N., Ruiz, F. J., & McHugh, L. (2025) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2025.100932 · Cited by 2 · see citing works
Kühl, E., Deković, M., Thomaes, S. C. E., Koning, I. M., & Liber, J. M. (2024) · Journal of Applied Research in Intellectual Disabilities · doi:10.1111/jar.13250 · Cited by 1 · see citing works
King, S., Rijkeboer, M. M., Winkens, I., Tuts, N., Ford, M. E., Frankenmolen, N. L., Tanious, R., van der Naalt, J., & Heugten, C. M. V. (2025) · Behaviour Research and Therapy · doi:10.1016/j.brat.2025.104941 · Cited by 1 · see citing works
Stapleton, A., Moore, B., Stynes, G., Richardson, N., Russell, T., & McHugh, L. (2026) · Advancements in Agricultural Development · doi:10.37433/aad.v7i2.657 · Cited by 1 · see citing works
Sypré, S., Onghena, P., Verschueren, K., Vansteenkiste, M., & Soenens, B. (2025) · Gifted Child Quarterly · doi:10.1177/00169862251388967
Aydın, O. (2026) · Journal of Applied Behavior Analysis · doi:10.1002/jaba.70070
- Authors
- Peng, C.-Y. J., & Chen, L.-T.
- Affiliations
- National Taiwan University (TW)
- University of Nevada, Reno (US)
- Corresponding author
- Chen, Li-Ting
- Venue
- Education Sciences
- Publisher
- MDPI AG
- Published
- 2021-02-14
- Volume
- 11
- Issue
- 2
- Pages
- 76
- ISSN
- 2227-7102 (ISSN-L)
- OpenAlex
- W3132929059
- Cited by
- 21
- References
- 66
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 5 · 2022: 4 · 2021: 2
- FWCI
- 2.99
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Education Sciences (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Scholarly Activities Funds of College of Education at University of Nevada, Reno. (No number assigned)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Psychology › Developmental and Educational Psychology › Behavioral and Psychological Studies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- missing data, single-case experimental design, N-of-1 design, intervention effect estimation, single imputation, multiple baseline design, probe design, internal validity, effect size, data analysis methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
