A randomization test wrapper for synthesizing single-case experiments using multilevel models: a Monte Carlo simulation study
Open accessAbstract
Multilevel models (MLMs) have been proposed in single-case research, to synthesize data from a group of cases in a multiple-baseline design (MBD). A limitation of this approach is that MLMs require several statistical assumptions that are often violated in single-case research. In this article we propose a solution to this limitation by presenting a randomization test (RT) wrapper for MLMs that offers a nonparametric way to evaluate treatment effects, without making distributional assumptions or an assumption of random sampling. We present the rationale underlying the proposed technique and validate its performance (with respect to Type I error rate and power) as compared to parametric statistical inference in MLMs, in the context of evaluating the average treatment effect across cases in an MBD. We performed a simulation study that manipulated the numbers of cases and of observations per case in a dataset, the data variability between cases, the distributional characteristics of the data, the level of autocorrelation, and the size of the treatment effect in the data. The results showed that the power of the RT wrapper is superior to the power of parametric tests based on F distributions for MBDs with fewer than five cases, and that the Type I error rate of the RT wrapper is controlled for bimodal data, whereas this is not the case for traditional MLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To avoid making distributional and random sampling assumptions, the randomisation test wrapper for multilevel models will be used to synthesise data and evaluate treatment effects.
“Finally, to avoid making distributional and random sampling assumptions, the randomisation test wrapper for multilevel models will be used to synthesise the data from the whole group of cases and evaluate treatment effects.69”
From Development, evaluation and implementation of a digital behavioural health treatment for chronic pain: study protocol of the multiphase DAHLIA project (2022) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013) · doi:10.1201/b16018 · Cited by 27,276 · see citing works
<b>lmerTest</b> Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346 · see citing works
Hierarchical Linear Models: Applications and Data Analysis Methods.
Longford, N. T., Bryk, A. S., & Raudenbush, S. W. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2075823 · Cited by 18,768 · see citing works
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638 · see citing works
Continuous Univariate Distributions.
Omori, Y., Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995) · Journal of the American Statistical Association · doi:10.2307/2291547 · Cited by 9,085 · see citing works
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 6,830 · see citing works
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · see citing works
Gelman, A. (2006) · Bayesian Analysis · doi:10.1214/06-ba117a · Cited by 4,050 · see citing works
SOME CURRENT DIMENSIONS OF APPLIED BEHAVIOR ANALYSIS<sup>1</sup>
Baer, D. M., Wolf, M. M., & Risley, T. R. (1968) · Journal of Applied Behavior Analysis · doi:10.1901/jaba.1968.1-91 · Cited by 3,935 · see citing works
Single Case Experimental Designs: Strategies for Studying Behavior Change
Barlow, D. H., Nock, M. K., & Hersen, M. (1976) · Cited by 3,036 · see citing works
Single-Case Research Designs: Methods for Clinical and Applied Settings
Kazdin, A. E. (1982) · Cited by 2,852 · see citing works
Single-case research designs: Methods for clinical and applied settings
Hayes, S. C. (1983) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/0005-7916(83)90018-6 · Cited by 2,677 · see citing works
Maximum Likelihood Approaches to Variance Component Estimation and to Related Problems
Harville, D. A. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10480998 · Cited by 2,667 · see citing works
Edgington, E. S., Edgington, E., & Onghena, P. (1964) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess2169 · Cited by 2,034 · see citing works
Self-citationBayesian Approaches to Clinical Trials and Health‐Care Evaluation
Spiegelhalter, D. J., Abrams, K. R., & Myles, J. P. (2003) · Wiley eBooks · doi:10.1002/0470092602 · Cited by 1,675 · see citing works
Prior distributions for variance parameters in hierarchical modelsPreprint
Gelman, A. (2004) · RePEc: Research Papers in Economics · Cited by 1,558 · see citing works
The unicorn, the normal curve, and other improbable creatures.
Micceri, T. (1989) · Psychological Bulletin · doi:10.1037/0033-2909.105.1.156 · Cited by 1,487 · see citing works
Single-Case Designs Technical Documentation.
Kratochwill, T. R., Hitchcock, J. H., Horner, R., Levin, J., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010) · Cited by 1,345 · see citing works
Halekoh, U., & Højsgaard, S. (2014) · Journal of Statistical Software · doi:10.18637/jss.v059.i09 · Cited by 1,324 · see citing works
Single case experimental designs: Strategieṡ for studying behavior change
Matson, J. L. (1985) · Applied Research in Mental Retardation · doi:10.1016/0270-3092(85)90012-8 · Cited by 1,193 · see citing works
Continuous Univariate Distributions.Vol. 1
Kocherlakota, S., Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995) · Biometrics · doi:10.2307/2533292 · Cited by 861 · see citing works
Strategy for intention to treat analysis in randomised trials with missing outcome data
White, I. R., Horton, N. J., Carpenter, J. R., statistics, R. I. M. A. S., & Pocock, S. (2011) · BMJ · doi:10.1136/bmj.d40 · Cited by 849 · see citing works
Robustness issues in multilevel regression analysis
Maas, C. J. M., & Hox, J. J. (2004) · Statistica Neerlandica · doi:10.1046/j.0039-0402.2003.00252.x · Cited by 837 · see citing works
Satterthwaite, F. E. (1941) · Psychometrika · doi:10.1007/bf02288586 · Cited by 832 · see citing works
A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder
Fabiano, G. A., Pelham, W. E., Coles, E. K., Gnagy, E. M., Chronis-Tuscano, A., & O'Connor, B. C. (2009) · Clinical Psychology Review · doi:10.1016/j.cpr.2008.11.001 · Cited by 588 · see citing works
The use of multiple imputation for the analysis of missing data.
Sinharay, S., Stern, H. S., & Russell, D. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.317 · Cited by 560 · see citing works
Kratochwill, T. R., & Levin, J. R. (2010) · Psychological Methods · doi:10.1037/a0017736 · Cited by 497 · see citing works
Fitzmaurice, G. M., Davidian, M., Molenberghs, G., & Molenberghs, G. (2008) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420011579 · Cited by 474 · see citing works
An improved approximation to the precision of fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. (2009) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2008.12.013 · Cited by 458 · see citing works
Baltagi, B. H. (2009) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.1111/j.1467-985x.2009.00614_9.x · Cited by 397 · see citing works
Confidence Intervals on Variance Components
Burdick (1992) · CRC Press eBooks · doi:10.1201/9781482277142 · Cited by 351 · see citing works
Characteristics of single-case designs used to assess intervention effects in 2008
Shadish, W. R., & Sullivan, K. J. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0111-y · Cited by 338 · see citing works
Heyvaert, M., Saenen, L., Campbell, J. M., Maes, B., & Onghena, P. (2014) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2014.06.017 · Cited by 285 · see citing works
Self-citationFai, A. H.-T., & Cornelius, P. L. (1996) · Journal of Statistical Computation and Simulation · doi:10.1080/00949659608811740 · Cited by 255 · see citing works
N-of-1 Trials in the Medical Literature: A Systematic Review
Gabler, N. B., Duan, N., Vohra, S., & Kravitz, R. L. (2011) · Medical Care · doi:10.1097/mlr.0b013e318215d90d · Cited by 226 · see citing works
Gettinger, M., & Kratochwill, T. R. (1987) · Springer eBooks · doi:10.1007/978-1-4899-0041-8_7 · Cited by 224 · see citing works
Ferron, J. M., Bell, B. A., Hess, M. R., Rendina‐Gobioff, G., & Hibbard, S. T. (2009) · Behavior Research Methods · doi:10.3758/brm.41.2.372 · Cited by 197 · see citing works
Combining single-case experimental data using hierarchical linear models.
Noortgate, W. V. D., & Onghena, P. (2003) · School Psychology Quarterly · doi:10.1521/scpq.18.3.325.22577 · Cited by 195 · see citing works
Self-citationA multilevel meta-analysis of single-subject experimental design studies
Van den Noortgate, W., & Onghena, P. (2008) · Evidence-Based Communication Assessment and Intervention · doi:10.1080/17489530802505362 · Cited by 192 · see citing works
Self-citationHierarchical linear models for the quantitative integration of effect sizes in single-case research
Noortgate, W. V. D., & Onghena, P. (2003) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03195492 · Cited by 190 · see citing works
Self-citationCustomization of Pain Treatments: Single-Case Design and Analysis
Onghena, P., & Edgington, E. S. (2005) · The Clinical Journal of Pain · doi:10.1097/00002508-200501000-00007 · Cited by 188 · see citing works
Self-citationAddressing Data Sparseness in Contextual Population Research
Clarke, P. J., & Wheaton, B. (2007) · Sociological Methods & Research · doi:10.1177/0049124106292362 · Cited by 174 · see citing works
Bayesian methods for the analysis of small sample multilevel data with a complex variance structure.
Baldwin, S., & Fellingham, G. (2013) · Psychological methods · doi:10.1037/a0030642 · Cited by 154 · see citing works
Shadish, W. R., Kyse, E. N., & Rindskopf, D. M. (2013) · Psychological Methods · doi:10.1037/a0032964 · Cited by 151 · see citing works
Statistical reform: Evidence‐based practice, meta‐analyses, and single subject designs
Jenson, W. R., Clark, E., Kircher, J. C., & Kristjansson, S. D. (2007) · Psychology in the Schools · doi:10.1002/pits.20240 · Cited by 130 · see citing works
Statistical Power of Randomization Tests Used with Multiple-Baseline Designs
Ferron, J. M., & Sentovich, C. (2002) · The Journal of Experimental Education · doi:10.1080/00220970209599504 · Cited by 125 · see citing works
Statistical inference : the distribution-free approach
Edgington, E. S. (1969) · McGraw-Hill eBooks · Cited by 108 · see citing works
Randomization tests for multiple-baseline designs: An extension of the SCRT-R package
Bulté, I., & Onghena, P. (2009) · Behavior Research Methods · doi:10.3758/brm.41.2.477 · Cited by 103 · see citing works
Self-citationRegulated randomization: A potentially sharper analytical tool for the multiple-baseline design.
Koehler, M. J., & Levin, J. R. (1998) · Psychological Methods · doi:10.1037/1082-989x.3.2.206 · Cited by 102 · see citing works
Heyvaert, M., & Onghena, P. (2014) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2013.10.002 · Cited by 98 · see citing works
Self-citationFerron, J. M., Farmer, J. L., & Owens, C. M. (2010) · Behavior Research Methods · doi:10.3758/brm.42.4.930 · Cited by 94 · see citing works
Randomization tests for multiple-baseline designs.
Wampold, B. E., & Worsham, N. L. (1986) · Behavioral Assessment · Cited by 93 · see citing works
Randomized single-subject experimental designs
Edgington, E. S. (1996) · Behaviour Research and Therapy · doi:10.1016/0005-7967(96)00012-5 · Cited by 89 · see citing works
Three-Level Analysis of Single-Case Experimental Data: Empirical Validation
Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Noortgate, W. V. D. (2013) · The Journal of Experimental Education · doi:10.1080/00220973.2012.745470 · Cited by 88 · see citing works
The Aggregation of single-case results using hierarchical linear models
Van den Noortgate, W., & Onghena, P. (2007) · The Behavior Analyst Today · doi:10.1037/h0100613 · Cited by 86 · see citing works
Self-citationMoeyaert, M., Ugille, M., Ferron, J. M., et al. (2013) · Multivariate Behavioral Research · doi:10.1080/00273171.2013.816621 · Cited by 82 · see citing works
Heyvaert, M., Maes, B., Noortgate, W. V. D., Kuppens, S., & Onghena, P. (2011) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2011.10.010 · Cited by 82 · see citing works
Self-citationCombining statistics for multiple-baseline AB and replicated ABAB designs across subjects.
Marascuilo, L. A., & Busk, P. L. (1988) · Behavioral Assessment · Cited by 77 · see citing works
Assessing the robustness of randomization tests: examples from behavioural studies
Peres‐Neto, P. R., & Olden, J. D. (2001) · Animal Behaviour · doi:10.1006/anbe.2000.1576 · Cited by 75 · see citing works
Comparison of randomization-test procedures for single-case multiple-baseline designs
Levin, J. R., Ferron, J. M., & Gafurov, B. S. (2016) · Developmental Neurorehabilitation · doi:10.1080/17518423.2016.1197708 · Cited by 72 · see citing works
Estimating causal effects from multiple-baseline studies: Implications for design and analysis.
Ferron, J. M., Moeyaert, M., Noortgate, W. V. D., & Beretvas, S. N. (2014) · Psychological Methods · doi:10.1037/a0037038 · Cited by 69 · see citing works
Multilevel modeling of single-case data: A comparison of maximum likelihood and Bayesian estimation.
Moeyaert, M., Rindskopf, D., Onghena, P., & Noortgate, W. V. D. (2017) · Psychological Methods · doi:10.1037/met0000136 · Cited by 67 · see citing works
Self-citationAnalyzing Single-Case Data: The Power of Randomization Tests
Ferron, J. M., & Ware, W. (1995) · The Journal of Experimental Education · doi:10.1080/00220973.1995.9943820 · Cited by 64 · see citing works
Multilevel meta-analysis of single-subject experimental designs: A simulation study
Ugille, M., Moeyaert, M., Beretvas, S. N., Ferron, J. M., & Noortgate, W. V. D. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0213-1 · Cited by 60 · see citing works
Bayesian and likelihood methods for fitting multilevel models with complex level-1 variation
Browne, W. J., Draper, D., Goldstein, H., & Rasbash, J. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00058-5 · Cited by 60 · see citing works
Violations of Assumptions in School-Based Single-Case Data
Solomon, B. G. (2013) · Behavior Modification · doi:10.1177/0145445513510931 · Cited by 58 · see citing works
Single‐subject designs in special education: advantages and limitations
Alnahdi, G. H. (2013) · Journal of Research in Special Educational Needs · doi:10.1111/1471-3802.12039 · Cited by 53 · see citing works
Poncet, A., Courvoisier, D. S., Combescure, C., & Perneger, T. V. (2016) · Methodology · doi:10.1027/1614-2241/a000110 · Cited by 49 · see citing works
Variance Heterogeneity in Published Psychological Research
Ruscio, J., & Roche, B. (2011) · Methodology · doi:10.1027/1614-2241/a000034 · Cited by 46 · see citing works
Levin, J. R., Ferron, J. M., & Gafurov, B. S. (2017) · Journal of School Psychology · doi:10.1016/j.jsp.2017.02.003 · Cited by 46 · see citing works
Educational/Psychological Intervention Research
Levin, J. R., O’Donnell, A. M., & Kratochwill, T. R. (2003) · Handbook of Psychology · doi:10.1002/0471264385.wei0722 · Cited by 45 · see citing works
Franco, J. H., Davis, B. L., & Davis, J. L. (2013) · American Journal of Speech-Language Pathology · doi:10.1044/1058-0360(2012/10-0103 · Cited by 42 · see citing works
Testing the Intervention Effect in Single-Case Experiments: A Monte Carlo Simulation Study
Heyvaert, M., Moeyaert, M., Verkempynck, P., Van den Noortgate, W., Vervloet, M., Ugille, M., & Onghena, P. (2017) · The Journal of Experimental Education · doi:10.1080/00220973.2015.1123667 · Cited by 42 · see citing works
Self-citationOne by One: Accumulating Evidence by using Meta-Analytical Procedures for Single-Case Experiments
Onghena, P., Michiels, B., Jamshidi, L., Moeyaert, M., & Noortgate, W. V. D. (2017) · Brain Impairment · doi:10.1017/brimp.2017.25 · Cited by 40 · see citing works
Self-citationRandomization and Data-Analysis Items in Quality Standards for Single-Case Experimental Studies
Heyvaert, M., Wendt, O., Noortgate, W. V. D., & Onghena, P. (2014) · The Journal of Special Education · doi:10.1177/0022466914525239 · Cited by 39 · see citing works
Self-citationRandomization tests for changing criterion designs
Onghena, P., Tanious, R., De, T. K., & Michiels, B. (2019) · Behaviour Research and Therapy · doi:10.1016/j.brat.2019.01.005 · Cited by 32 · see citing works
Self-citationRandomization, Masking, and Allocation Concealment
(2017) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315305110 · Cited by 29 · see citing works
Handling missing data in single-case studies
Peng, C.-Y. J., & Chen, L.-T. (2018) · Journal of Modern Applied Statistical Methods · doi:10.22237/jmasm/1525133280 · Cited by 22 · see citing works
RegRand: Statistical software for the multiple-baseline design
Koehler, M. J., & Levin, J. R. (2000) · Behavior Research Methods, Instruments, & Computers · doi:10.3758/bf03207807 · Cited by 19 · see citing works
Pustejovsky, J. E., Rindskopf, D. M., Boyajian, J. G., et al. (2014) · American Psychological Association eBooks · doi:10.1037/14376-009 · Cited by 17 · see citing works
Keller, B. (2012) · Psychometrika · doi:10.1007/s11336-012-9249-5 · Cited by 14 · see citing works
Randomization and the Randomization Test: Two Sides of the Same Coin
Onghena, P. (2017) · Chapman & Hall/CRC eBooks · doi:10.1201/9781315305110-13 · Cited by 14 · see citing works
Self-citationMichiels, B., Heyvaert, M., & Onghena, P. (2017) · Behavior Research Methods · doi:10.3758/s13428-017-0885-7 · Cited by 12 · see citing works
Self-citationStatistical reform in school psychology research: A synthesis
Swaminathan, H., & Rogers, H. J. (2007) · Psychology in the Schools · doi:10.1002/pits.20246 · Cited by 8 · see citing works
Tyrrell, P. N., Corey, P., Feldman, B. M., et al. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.11.006 · Cited by 7 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Cassell, D. L. (2002). A randomization-test wrapper for SAS® PROCs. SAS User's Group International Proceedings, 27, 251. Retrieved from http://www.lexjansen.com/wuss/2002/WUSS02023.pdf
- Johnson, N. L., Kotz, S., & Balakrishnan, N. (1995). Continuous univar- iate distributions, Vol. 2 (2nd). New York: Wiley.
- Onghena, P. (2005). Single-case designs. In B. Everitt & D. Howell (Eds.), Encyclopedia of statistics in behavioral science, vol. 4 (pp. 1850-1854). Chichester, UK: Wiley.
- a000034 · DOI
Cited by
Sorted by citations
A Priori Justification for Effect Measures in Single-Case Experimental Designs
Manolov, R., Moeyaert, M., & Fingerhut, J. E. (2022) · Perspectives on Behavior Science · doi:10.1007/s40614-021-00282-2 · Cited by 53 · see citing works
Bartels, S. L., Johnsson, S. I., Boersma, K., Flink, I., McCracken, L. M., Petersson, S., Christie, H. L., Feldman, I., Simons, L. E., Onghena, P., et al. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059152 · Cited by 23 · see citing works
Self-citationManolov, R., & Onghena, P. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02230-1 · Cited by 5 · see citing works
Self-citation
- Authors
- Michiels, B., Tanious, R., De, T. K., & Onghena, P.
- Affiliations
- KU Leuven (BE)
- Corresponding author
- Onghena, Patrick
- Venue
- Behavior Research Methods
- Published
- 2019-07-03
- Volume
- 52
- Issue
- 2
- Pages
- 654-666
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W2953414565
- Cited by
- 23
- References
- 85
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 4 · 2023: 7 · 2022: 5 · 2021: 3 · 2020: 2
- FWCI
- 1.95
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- Funders
- Research Foundation – Flanders (G.0593.14)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- permutation tests, multilevel models, single subject research, multiple baseline design, nonparametric inference, treatment effect estimation, type I error rate, statistical power, bimodal data, autocorrelation, small sample size
- MeSH terms
- Computer Simulation, Monte Carlo Method, Random Allocation, Models, Statistical, Statistical Distributions, Multilevel Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
