Multiple Imputation of Missing Data for Multilevel Models: Simulations and Recommendations
The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Multiple imputation (MI) is one of the principled methods for dealing with missing data. In addition, multilevel models have become a standard tool for analyzing the nested data structures that result when lower level units (e.g., employees) are nested within higher level collectives (e.g., work groups). When applying MI to multilevel data, it is important that the imputation model takes the multilevel structure into account. In the present paper, based on theoretical arguments and computer simulations, we provide guidance using MI in the context of several classes of multilevel models, including models with random intercepts, random slopes, cross-level interactions (CLIs), and missing data in categorical and group-level variables. Our findings suggest that, oftentimes, several approaches to MI provide an effective treatment of missing data in multilevel research. Yet we also note that the current implementations of MI still have room for improvement when handling missing data in explanatory variables in models with random slopes and CLIs. We identify areas for future research and provide recommendations for research practice along with a number of step-by-step examples for the statistical software R.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Mixed models can handle unbalanced data by making use of available information when outcome data is available at one time point and missing at others, thus preserving sample size [58, 59].”
From Benchmark findings from a veteran electronic patient-reported outcomes evaluation from a chronic pain management telehealth program (2024) · cites this paper as [58]
Citations
Cites
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Multilevel analysis : an introduction to basic and advanced multilevel modeling
Snijders, T. A. B., & Bosker, R. (1999) · Cited by 13,906
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870 · cited 1×
“Here, we consider the JM approach implemented in the pan package (Schafer & Yucel, 2002) and the FCS approach known as “multiple imputation by chained equations” implemented in the mice package (van Buuren & Groothuis-Oudshoorn, 2011) in the statistical software R (R Core Team, 2016).”
MethodData Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
(2010) · Cited by 7,034
Missing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163
WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility
Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000) · Statistics and Computing · doi:10.1023/a:1008929526011 · Cited by 5,368
A general multilevel SEM framework for assessing multilevel mediation
Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010) · Psychological Methods · doi:10.1037/a0020141 · Cited by 3,796
Journal of Statistical Software
Leeuw, J. D. (2009) · WIREs Computational Statistics · doi:10.1002/wics.10 · Cited by 3,599
Handbook of Structural Equation Modeling
Howard, A. L. (2013) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705511.2013.769397 · Cited by 3,440
How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory
Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007) · Prevention Science · doi:10.1007/s11121-007-0070-9 · Cited by 2,795
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389 · cited 1×
“For example, researchers should try to include auxiliary variables in the imputation model, that is, variables that are related to either the occurrence of missing data or the variables with missing data themselves (e.g., Collins et al., 2001; Graham, 2009; Schafer & Graham, 2002).”
BackgroundMultiple Imputation of Missing Values
Royston, P. (2004) · The Stata Journal: Promoting communications on statistics and Stata · doi:10.1177/1536867x0400400301 · Cited by 2,340 · cited 1×
“This is commonly referred to as “passive imputation” (Royston, 2004; van Buuren, 2012).”
BackgroundCentering Decisions in Hierarchical Linear Models: Implications for Research in Organizations
Hofmann, D. (1998) · Journal of Management · doi:10.1016/s0149-2063(99)80077-4 · Cited by 2,109
Flexible Imputation of Missing Data
van Buuren, S. (2012) · doi:10.1201/b11826 · Cited by 2,022 · cited 1×
“This is commonly referred to as “passive imputation” (Royston, 2004; van Buuren, 2012).”
BackgroundMissing Data: Five Practical Guidelines
Newman, D. A. (2014) · Organizational Research Methods · doi:10.1177/1094428114548590 · Cited by 1,411
Eby, L. T. D. T., Hurst, C. S., & Butts, M. M. (2008) · Cited by 926
Multiple Imputation and its Application
Carpenter, J. R., & Kenward, M. G. (2013) · doi:10.1002/9781119942283 · Cited by 913 · cited 5×
“…approach is different, it has been argued that the two approaches imply similar covariance structures at the individual and the group level and can be used interchangeably (e.g., Carpenter & Kenward, 2013, p. 220; Lüdtke et al., 2017; Mistler, 2015; however, see also Resche-Rigon & White, 2016).”
BackgroundMethodInfluentialThe Effect of Different Forms of Centering in Hierarchical Linear Models
Kreft, I. G. G., de Leeuw, J., & Aiken, L. S. (1995) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3001_1 · Cited by 850
Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008) · Psychological Methods · doi:10.1037/a0012869 · Cited by 823
Self-citationHandbook of Advanced Multilevel Analysis
(2011) · Routledge eBooks · doi:10.4324/9780203848852 · Cited by 743
Adding Missing-Data-Relevant Variables to FIML-Based Structural Equation Models
Graham, J. W. (2003) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem1001_4 · Cited by 619
What Improves with Increased Missing Data Imputations?
Bodner, T. E. (2008) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705510802339072 · Cited by 606
Planned missing data designs in psychological research.
Graham, J. W., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.323 · Cited by 573
Bartlett, J. W., Seaman, S. R., White, I. R., Carpenter, J. R., & Initiative*, F. T. A. D. N. (2015) · Statistical Methods in Medical Research · doi:10.1177/0962280214521348 · Cited by 485
7. Multilevel Latent Class Models
Vermunt, J. K. (2003) · Sociological Methodology · doi:10.1111/j.0081-1750.2003.t01-1-00131.x · Cited by 460
8. How to Impute Interactions, Squares, and other Transformed Variables
von Hippel, P. T. (2009) · Sociological Methodology · doi:10.1111/j.1467-9531.2009.01215.x · Cited by 419
REALCOM-IMPUTE Software for Multilevel Multiple Imputation with Mixed Response Types
Carpenter, J. R., Goldstein, H., & Kenward, M. G. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i05 · Cited by 395
Computational Strategies for Multivariate Linear Mixed-Effects Models With Missing Values
Schafer, J. L., & Yucel, R. M. (2002) · Journal of Computational and Graphical Statistics · doi:10.1198/106186002760180608 · Cited by 380
Mixed Effects Models for Complex Data
Wu, L. (2009) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420074086 · Cited by 310
Multiple imputation by chained equations for systematically and sporadically missing multilevel data
Resche-Rigon, M., & White, I. R. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280216666564 · Cited by 218
Enders, C. K., Mistler, S. A., & Keller, B. T. (2016) · Psychological Methods · doi:10.1037/met0000063 · Cited by 207
Multiple Imputation in Multivariate Problems When the Imputation and Analysis Models Differ
Schafer, J. L. (2003) · Statistica Neerlandica · doi:10.1111/1467-9574.00218 · Cited by 202
Multilevel models with multivariate mixed response types
Goldstein, H., Carpenter, J., Kenward, M. G., & Levin, K. A. (2009) · Statistical Modelling · doi:10.1177/1471082x0800900301 · Cited by 202
Seaman, S. R., Bartlett, J. W., & White, I. R. (2012) · BMC Medical Research Methodology · doi:10.1186/1471-2288-12-46 · Cited by 177
Multiple imputation of missing data in multilevel designs: A comparison of different strategies.
Lüdtke, O., Robitzsch, A., & Grund, S. (2017) · Psychological Methods · doi:10.1037/met0000096 · Cited by 150
Self-citationErler, N. S., Rizopoulos, D., van Rosmalen, J., Jaddoe, V. W. V., Franco, O. H., & Lesaffre, E. M. E. H. (2016) · Statistics in Medicine · doi:10.1002/sim.6944 · Cited by 137
9. Multiple Imputation of Incomplete Categorical Data Using Latent Class Analysis
Vermunt, J. K., van Ginkel, J. R., van der Ark, L. A., & Sijtsma, K. (2008) · Sociological Methodology · doi:10.1111/j.1467-9531.2008.00202.x · Cited by 128
Goldstein, H., Carpenter, J. R., & Browne, W. J. (2014) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/rssa.12022 · Cited by 121
Multiple Imputation of Multilevel Missing Data: An Introduction to the R Package pan
Grund, S., Lüdtke, O., & Robitzsch, A. (2016) · Sage Open · doi:10.1177/2158244016668220 · Cited by 120
Self-citationEnders, C. K. (2008) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1080/10705510802154307 · Cited by 119
Imputation Strategies for Missing Continuous Outcomes in Cluster Randomized Trials
Taljaard, M., Donner, A., & Klar, N. (2008) · Biometrical Journal · doi:10.1002/bimj.200710423 · Cited by 110
McNeish, D. (2016) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998615621299 · Cited by 101
The handbook of multilevel theory, measurement, and analysis.
(2019) · American Psychological Association eBooks · doi:10.1037/0000115-000 · Cited by 96
Multiple Imputation of Multilevel Missing Data—Rigor Versus Simplicity
Drechsler, J. (2015) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998614563393 · Cited by 94
Andridge, R. R. (2011) · Biometrical Journal · doi:10.1002/bimj.201000140 · Cited by 89
Stubbendick, A. L., & Ibrahim, J. G. (2003) · Biometrics · doi:10.1111/j.0006-341x.2003.00131.x · Cited by 80
A Latent Cluster-Mean Approach to the Contextual Effects Model With Missing Data
Shin, Y., & Raudenbush, S. W. (2010) · Journal of Educational and Behavioral Statistics · doi:10.3102/1076998609345252 · Cited by 78
Treatment Of Missing Data At The Second Level Of Hierarchical Linear Models
Gibson, N. M., & Olejnik, S. F. (2003) · Educational and Psychological Measurement · doi:10.1177/0013164402250987 · Cited by 69
Yucel, R. M. (2008) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2008.0038 · Cited by 69
Multiple imputation of multilevel missing data: An introduction to the R package pan
Grund, S., Lüdtke, O., & Robitzsch, A. (2016) · arXiv (Cornell University) · doi:10.25656/01:12692 · Cited by 67
Self-citationAguinis, H., & Culpepper, S. A. (2015) · Organizational Research Methods · doi:10.1177/1094428114563618 · Cited by 60
Grund, S., Lüdtke, O., & Robitzsch, A. (2016) · Behavior Research Methods · doi:10.3758/s13428-015-0590-3 · Cited by 58
Self-citationCheung, M. W.-L. (2007) · Organizational Research Methods · doi:10.1177/1094428106295499 · Cited by 53
Moderation analysis with missing data in the predictors.
Zhang, Q., & Wang, L. (2016) · Psychological Methods · doi:10.1037/met0000104 · Cited by 50
Kim, S., Sugar, C. A., & Belin, T. R. (2015) · Statistics in Medicine · doi:10.1002/sim.6435 · Cited by 41
Multiple Imputation of Squared Terms
Vink, G., & van Buuren, S. (2013) · Sociological Methods & Research · doi:10.1177/0049124113502943 · Cited by 26
Research works printed but not linked: 10
- Allison P. D., 2001, Missing data
- Allison P. D., 2012, Proceedings of the SAS Global Forum
- Gottfredson N. C., 2016, Prevention Science
- Grund S., Handbook for multilevel theory, measurement, and analysis
- 10.1177/014920639802400504 · doi:10.1177/014920639802400504 · Paper page
- Hox J. J., 2016, Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications, 39
- Meng X.-L., 1994, Statistical Science, 9
- Mistler S. A., 2013, Proceedings of the SAS Global Forum
- Mistler S. A. (2015). Multilevel multiple imputation: An examination of competing methods (Doctoral dissertation). Retrieved from http://repository.asu.edu/
- Muthén L. K., 2012, Mplus user’s guide
Cited by
Methodological guidance paper: High-quality meta- analysis in a systematic review
Pigott, T. D., & Polanin, J. R. (2020) · Review of Educational Research · doi:10.3102/0034654319877153 · Cited by 568
Evaluation of Pressing Issues in Ecological Momentary Assessment
Stone, A. A., Schneider, S., & Smyth, J. M. (2023) · Annual Review of Clinical Psychology · doi:10.1146/annurev-clinpsy-080921-083128 · Cited by 200
jomo: A Flexible Package for Two-level Joint Modelling Multiple Imputation
Quartagno, M., Grund, S., & Carpenter, J. (2019) · The R Journal · doi:10.32614/rj-2019-028 · Cited by 140
Self-citationmiceadds: Some Additional Multiple Imputation Functions, Especially for 'mice'
Robitzsch, A., & Grund, S. (2014) · doi:10.32614/cran.package.miceadds · Cited by 110
Self-citationPilot randomized trial of self-guided virtual reality exposure therapy for social anxiety disorder
Zainal, N. H., Chan, W. W., Saxena, A. P., Taylor, C. B., & Newman, M. G. (2021) · Behaviour Research and Therapy · doi:10.1016/j.brat.2021.103984 · Cited by 84
Metadata
- Authors
- Grund, S., Lüdtke, O., & Robitzsch, A.
- Affiliations
- Leibniz Institute for Science and Mathematics Education (DE)
- Corresponding author
- Grund, Simon
- Venue
- Organizational Research Methods
- Publisher
- SAGE Publications
- Published
- 2017-05-15
- Volume
- 21
- Issue
- 1
- Pages
- 111-149
- ISSN
- 1094-4281 (ISSN-L)
- 1552-7425
- OpenAlex
- W2615970617
- Semantic Scholar
- e06f85dc773d8ae1f4e21764feecabcb66b2684a
- Cited by
- 224
- References
- 60
- Citations by year
- 2026: 34 · 2025: 24 · 2024: 38 · 2023: 34 · 2022: 27 · 2021: 27 · 2020: 23 · 2019: 13 · 2018: 3
- FWCI
- 7.45
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
- Locations
- Organizational Research Methods (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
- Keywords
- multiple imputation, missing data, multilevel models, nested data, random slopes, cross-level interaction, random intercepts, categorical variables, R implementation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.