Cited by 1,411Statistical Methods and Bayesian InferencePsychometric Methodologies and TestingMeta-analysis and systematic reviews

The DOI matches its Crossref record; 64 of the 66 references it cites were found in a registry or an index.

Missing Data: Five Practical Guidelines

Newman, Daniel A.·2014·Organizational Research Methods

DOI 10.1177/1094428114548590

Open this paper in CoChat — free account

Abstract

Missing data (a) reside at three missing data levels of analysis (item-, construct-, and person-level), (b) arise from three missing data mechanisms (missing completely at random, missing at random, and missing not at random) that range from completely random to systematic missingness, (c) can engender two missing data problems (biased parameter estimates and inaccurate hypothesis tests/inaccurate standard errors/low power), and (d) mandate a choice from among several missing data treatments (listwise deletion, pairwise deletion, single imputation, maximum likelihood, and multiple imputation). Whereas all missing data treatments are imperfect and are rooted in particular statistical assumptions, some missing data treatments are worse than others, on average (i.e., they lead to more bias in parameter estimates and less accurate hypothesis tests). Social scientists still routinely choose the more biased and error-prone techniques (listwise and pairwise deletion), likely due to poor familiarity with and misconceptions about the less biased/less error-prone techniques (maximum likelihood and multiple imputation). The current user-friendly review provides five easy-to-understand practical guidelines, with the goal of reducing missing data bias and error in the reporting of research results. Syntax is provided for correlation, multiple regression, and structural equation modeling with missing data.

Integrity signals

Registry

Crossref · 10.1177/1094428114548590

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Sample Selection Bias as a Specification Error
Heckman, J. J. (1979) · Econometrica · doi:10.2307/1912352 · Cited by 28,779
Bayesian Data Analysis
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Statistical Analysis With Missing Data
Lahiff, M., Little, R. J. A., & Rubin, D. B. (1989) · Journal of the American Statistical Association · doi:10.2307/2289883 · Cited by 17,385
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
Inference and missing data
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Statistical Analysis with Missing Data
Little, R. J. A., & Rubin, D. B. (2002) · Wiley Series in Probability and Statistics · doi:10.1002/9781119013563 · Cited by 8,473
Untitled work
(2010) · Cited by 7,034
Mail and Telephone Surveys: The Total Design Method.
Babbie, E. R., & Dillman, D. A. (1982) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2068433 · Cited by 6,859
Missing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163
Analysis of Incomplete Multivariate Data
Schafer, J. L. (1997) · doi:10.1201/9781439821862 · Cited by 6,012
Analysis of Incomplete Multivariate Data
Booth, D. E., & Schafer, J. L. (2000) · Technometrics · doi:10.2307/1271466 · Cited by 5,627
Mail and Telephone Surveys: The Total Design Method.
Osmond, M. W., & Dillman, D. A. (1979) · Social Forces · doi:10.2307/2577403 · Cited by 5,023
Maximum likelihood estimation from incomplete data via the EM algorithm
Dempster, A. P. (1977) · Journal of the Royal Statistical Society Series A (Statistics in Society) · Cited by 4,543
The Belmont Report: ethical principles and guidelines for the protection of human subjects of research
Eckstein, S. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511550089.028 · Cited by 3,514
Development of a measure of workplace deviance.
Bennett, R. J., & Robinson, S. L. (2000) · Journal of Applied Psychology · doi:10.1037/0021-9010.85.3.349 · Cited by 3,084
How Many Imputations are Really Needed? Some Practical Clarifications of Multiple Imputation Theory
Graham, J. W., Olchowski, A. E., & Gilreath, T. D. (2007) · Prevention Science · doi:10.1007/s11121-007-0070-9 · Cited by 2,795
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389
Statistical Analysis with Missing Data.
Chatfield, C., Little, R. J. A., & Rubin, D. B. (1988) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982783 · Cited by 2,203
Missing Data
Allison, P. (2002) · doi:10.4135/9781412985079 · Cited by 2,050
Statistical Strategies for Small Sample Research
Hoyle, R. H. (1999) · Cited by 1,896
Missing Data in Educational Research: A Review of Reporting Practices and Suggestions for Improvement
Peugh, J., & Enders, C. K. (2004) · Review of Educational Research · doi:10.3102/00346543074004525 · Cited by 1,107
The Heckman Correction for Sample Selection and Its Critique
Puhani, P. A. (2000) · Journal of Economic Surveys · doi:10.1111/1467-6419.00104 · Cited by 1,105
A Primer on Maximum Likelihood Algorithms Available for Use With Missing Data
Enders, C. K. (2001) · Structural Equation Modeling: A Multidisciplinary Journal · doi:10.1207/s15328007sem0801_7 · Cited by 1,027
Pattern-Mixture Models for Multivariate Incomplete Data
Little, R. J. A. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10594302 · Cited by 1,026
The impact of nonnormality on full information maximum-likelihood estimation for structural equation models with missing data.
Enders, C. K. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.352 · Cited by 942
Statistical and Methodological Myths and Urban Legends: Doctrine, Verity and Fable in the Organizational and Social Sciences
Eby, L. T. D. T., Hurst, C. S., & Butts, M. M. (2008) · Cited by 926
Models for Sample Selection Bias
Winship, C., & Mare, R. D. (1992) · Annual Review of Sociology · doi:10.1146/annurev.so.18.080192.001551 · Cited by 812
Missing Data: A Gentle Introduction
McKnight, P. E., McKnight, K., Sidani, S., & Figueredo, A. J. (2007) · Cited by 782
Handbook of Advanced Multilevel Analysis
Hox, J. J., & Roberts, J. (2011) · Routledge eBooks · doi:10.4324/9780203848852 · Cited by 751
Missing Data in Likert Ratings: A Comparison of Replacement Methods
Downey, R. G., & King, C. V. (1998) · The Journal of General Psychology · doi:10.1080/00221309809595542 · Cited by 657
Longitudinal Modeling with Randomly and Systematically Missing Data: A Simulation of Ad Hoc, Maximum Likelihood, and Multiple Imputation Techniques
Newman, D. A. (2003) · Organizational Research Methods · doi:10.1177/1094428103254673 · Cited by 626 · cited 5×
“…article are built upon statistical theory (see reviews by Allison, 2002; Enders, 2001b, 2010; Dempster, Laird, & Rubin, 1977; Little & Rubin, 2002; Newman, 2003; Rubin, 1976, 1987; Schafer, 1997; Schafer & Graham, 2002), but the guidelines themselves are practical guidelines and not intended to…”
BackgroundInfluentialSelf-citation
Adding Missing-Data-Relevant Variables to FIML-Based Structural Equation Models
Graham, J. A. (2003) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1001_4 · Cited by 621
What (Not) to Expect When Surveying Executives
Cycyota, C. S., & Harrison, D. A. (2006) · Organizational Research Methods · doi:10.1177/1094428105280770 · Cited by 586
Planned missing data designs in psychological research.
Graham, J. A., Taylor, B. J., Olchowski, A. E., & Cumsille, P. E. (2006) · Psychological Methods · doi:10.1037/1082-989x.11.4.323 · Cited by 575
Understanding Mail Survey Response Behavior: A Meta-Analysis
Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991) · Public Opinion Quarterly · doi:10.1086/269284 · Cited by 566
Response Rates in Organizational Science, 1995–2008: A Meta-analytic Review and Guidelines for Survey Researchers
Anseel, F., Lievens, F., Schollaert, E., & Chorągwicka, B. (2010) · Journal of Business and Psychology · doi:10.1007/s10869-010-9157-6 · Cited by 430
Missing Data in Multiple Item Scales: A Monte Carlo Analysis of Missing Data Techniques
Roth, P. L., Switzer, F. S., & Switzer, D. M. (1999) · Organizational Research Methods · doi:10.1177/109442819923001 · Cited by 344 · cited 3×
“This is like the practice that Roth, Switzer, and Switzer (1999) recommend, which they referred to as ‘‘mean substitution across items (and within an individual)’’ or ‘‘meanperson imputation’’ (pp. 214, 222; also see Downey & King, 1998), although using the technique as I am describing it here…”
BackgroundMethod
Treatments of Missing Data: A Monte Carlo Comparison of RBHDI, Iterative Stochastic Regression Imputation, and Expectation-Maximization
Gold, M. S., & Bentler, P. M. (2000) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem0703_1 · Cited by 332
Personnel Selection: Test and Measurement Techniques
Smith, P. C., & Thorndike, R. L. (1951) · Industrial and Labor Relations Review · doi:10.2307/2518516 · Cited by 309
Statistical power in criterion-related validation studies.
Schmidt, F. L., Hunter, J. E., & Urry, V. W. (1976) · Journal of Applied Psychology · doi:10.1037/0021-9010.61.4.473 · Cited by 260
Response Rates in HRM/OB Survey Research: Norms and Correlates, 1990-1994
Roth, P. L., & BeVier, C. A. (1998) · Journal of Management · doi:10.1016/s0149-2063(99)80055-5 · Cited by 237
On the performance of multiple imputation for multivariate data with small sample size
Graham, J. A., Schafer, J. L., & Hoyle, R. H. (1999) · Cited by 235
Profiling Active and Passive Nonrespondents to an Organizational Survey.
Rogelberg, S. G., Conway, J. M., Sederburg, M. E., Spitzmueller, C., Aziz, S., & Knight, W. E. (2003) · Journal of Applied Psychology · doi:10.1037/0021-9010.88.6.1104 · Cited by 210
Science and Ethics in Conducting, Analyzing, and Reporting Psychological Research
Rosenthal, R. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00646.x · Cited by 208
Personnel Selection: Test and Measurement Techniques.
Cronbach, L. J. (1949) · Journal of Educational Psychology · doi:10.1037/h0051818 · Cited by 192
Selection Modeling Versus Mixture Modeling with Nonignorable Nonresponse
Glynn, R. J., Laird, N. M., & Rubin, D. B. (1986) · Springer eBooks · doi:10.1007/978-1-4612-4976-4_10 · Cited by 178
Influence of imputation and EM methods on factor analysis when item nonresponse in questionnaire data is nonignorable
Bernaards, C. A., & Sijtsma, K. (2000) · Multivariate Behavioral Research · doi:10.1207/s15327906mbr3503_03 · Cited by 177
Estimation for the Multiple Factor Model when Data are Missing
Finkbeiner, C. T. (1979) · Psychometrika · doi:10.1007/bf02296204 · Cited by 153
Multiple Imputation of Multilevel Data
van Buuren, S. (2015) · Routledge eBooks · doi:10.4324/9780203848852.ch10 · Cited by 149
Using an EM Covariance Matrix to Estimate Structural Equation Models With Missing Data: Choosing an Adjusted Sample Size to Improve the Accuracy of Inferences
Enders, C. K., & Peugh, J. (2004) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1101_1 · Cited by 129
More Statistical and Methodological Myths and Urban Legends
Kirby, R. S. (2016) · Annals of Epidemiology · doi:10.1016/j.annepidem.2016.07.010 · Cited by 112
UNDERSTANDING MAIL SURVEY RESPONSE BEHAVIOR
Meta-Analysis, A., Yammarino, F. J., Skinner, S. J., & Childers, T. L. (2004) · Cited by 108
Pairwise deletion for missing data in structural equation models: Nonpositive definite matrices, parameter estimates, goodness of fit, and adjusted sample sizes
Marsh, H. W. (1998) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705519809540087 · Cited by 104
A Two-Stage Approach to Missing Data: Theory and Application to Auxiliary Variables
Savalei, V., & Bentler, P. M. (2009) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1080/10705510903008238 · Cited by 99
How Do Missing Data Bias Estimates of Within-Group Agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * , and ICC to Systematic Nonresponse
Newman, D., & Sin, H.-P. (2009) · Organizational Research Methods · doi:10.1177/1094428106298969 · Cited by 94
Self-citation
Missing data techniques and low response rates: The role of systematic nonresponse parameters
Newman, D. A. (2008) · Cited by 88
Self-citation
Estimating interaction effects with incomplete predictor variables.
Enders, C. K., Baraldi, A. N., & Cham, H. (2014) · Psychological Methods · doi:10.1037/a0035314 · Cited by 82
A Comparison of Maximum-Likelihood and Asymptotically Distribution-Free Methods of Treating Incomplete Nonnormal Data
Gold, M. S., Bentler, P. M., & Kim, K. H. (2003) · Structural Equation Modeling A Multidisciplinary Journal · doi:10.1207/s15328007sem1001_3 · Cited by 68
“If you treat me right, I reciprocate”: examining the role of exchange in organizational survey response
Spitzmueller, C., Glenn, D. M., Barr, C. D., Rogelberg, S. G., & Daniel, P. (2006) · Journal of Organizational Behavior · doi:10.1002/job.363 · Cited by 51
Systematic Data Loss in HRM Settings: A Monte Carlo Analysis
Switzer, F. S., Roth, P. L., & Switzer, D. M. (1998) · Journal of Management · doi:10.1177/014920639802400605 · Cited by 45
Choosing the best method for local validity estimation: Relative accuracy of meta-analysis versus a local study versus Bayes-analysis.
Newman, D. A., Jacobs, R. R., & Bartram, D. (2007) · Journal of Applied Psychology · doi:10.1037/0021-9010.92.5.1394 · Cited by 41
Self-citation
Implications of empirical Bayes meta-analysis for test validation.
Brannick, M. Τ. (2001) · Journal of Applied Psychology · doi:10.1037//0021-9010.86.3.468 · Cited by 33

Flagged first, then by citations

2 of the 66 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

10.1037/0021-9010.86.3.468 · DOI
10.1201/b16018 · DOI · Paper page
Mistler S. A., 2013, Proceedings of the SAS Global Forum
10.1177/014920639802400107 · DOI

Sorted by citations

The murky distinction between self-concept and self-efficacy: Beware of lurking jingle-jangle fallacies.
Marsh, H. W., Pekrun, R., Parker, P. D., Murayama, K., Guo, J., Dicke, T., & Arens, A. K. (2019) · Journal of Educational Psychology · doi:10.1037/edu0000281 · Cited by 535
Normalizing the Use of Single-Item Measures: Validation of the Single-Item Compendium for Organizational Psychology
Matthews, R. A., Pineault, L., & Hong, Y.-H. (2022) · Journal of Business and Psychology · doi:10.1007/s10869-022-09813-3 · Cited by 528
Green supply chain management in Chinese firms: Innovative measures and the moderating role of quick response technology
Li, G., Li, L., Choi, T., & SETHI, S. P. (2019) · Journal of Operations Management · doi:10.1002/joom.1061 · Cited by 439
Why Ordinal Variables Can (Almost) Always Be Treated as Continuous Variables: Clarifying Assumptions of Robust Continuous and Ordinal Factor Analysis Estimation Methods
Robitzsch, A. (2020) · Frontiers in Education · doi:10.3389/feduc.2020.589965 · Cited by 396
Using multi‐item psychometric scales for research and practice in human resource management
Robinson, M. A. (2017) · Human Resource Management · doi:10.1002/hrm.21852 · Cited by 367

See all 1,411

Authors: Newman, D. A.
Affiliations: University of Illinois Urbana-Champaign (US)
Corresponding author: Newman, Daniel A.
ORCID: Newman, Daniel A.
Venue: Organizational Research Methods
Publisher: SAGE Publications
Published: 2014-09-26
Volume: 17
Issue: 4
Pages: 372-411
ISSN: 1094-4281 (ISSN-L)
1552-7425
DOI: 10.1177/1094428114548590
OpenAlex: W2047813468
Cited by: 1,411
References: 66
Citations by year: 2026: 188 · 2025: 225 · 2024: 234 · 2023: 174 · 2022: 153 · 2021: 141 · 2020: 96 · 2019: 68 · 2018: 50 · 2017: 34 · 2016: 29 · 2015: 17 · 2014: 1
FWCI: 28.54
Type: journal article
Language: en
Open access: closed
Licence: journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
Locations: Organizational Research Methods (publishedVersion)
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords: missing data, missing data mechanisms, missing completely at random, missing at random, missing not at random, listwise deletion, pairwise deletion, single imputation, maximum likelihood estimation, multiple imputation, missing data bias, statistical power, structural equation modeling, multiple linear regression, correlation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Missing Data: Five Practical Guidelines

Abstract

Overview

Integrity signals

Verification

Claims the paper makes

Claims other papers make about this one

Citations

Cites

Cited by

Related

Metadata

Record

About these checks