The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Practical and statistical issues in missing data for longitudinal patient-reported outcomes
Abstract
Patient-reported outcomes are increasingly used in health research, including randomized controlled trials and observational studies. However, the validity of results in longitudinal studies can crucially hinge on the handling of missing data. This paper considers the issues of missing data at each stage of research. Practical strategies for minimizing missingness through careful study design and conduct are given. Statistical approaches that are commonly used, but should be avoided, are discussed, including how these methods can yield biased and misleading results. Methods that are valid for data which are missing at random are outlined, including maximum likelihood methods, multiple imputation and extensions to generalized estimating equations: weighted generalized estimating equations, generalized estimating equations with multiple imputation, and doubly robust generalized estimating equations. Finally, we discuss the importance of sensitivity analyses, including the role of missing not at random models, such as pattern mixture, selection, and shared parameter models. We demonstrate many of these concepts with data from a randomized controlled clinical trial on renal cancer patients, and show that the results are dependent on missingness assumptions and the statistical approach.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Multiple Imputation for Nonresponse in Surveys
Rubin, D. B. (1987) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316696 · Cited by 20,701
Longitudinal data analysis using generalized linear models
LIANG, K.-Y., & ZEGER, S. L. (1986) · Biometrika · doi:10.1093/biomet/73.1.13 · Cited by 18,550
Missing data: our view of the state of the art
Schafer, J. L., & Graham, J. W. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.2.147 · Cited by 11,638
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925
Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gotzsche, P. C., Devereaux, P. J., Elbourne, D., Egger, M., & Altman, D. G. (2010) · BMJ · doi:10.1136/bmj.c869 · Cited by 8,883
The functional assessment of cancer therapy scale: development and validation of the general measure
Cella, D. F., Tulsky, D. S., Gray, G., Sarafian, B., Linn, E., Bonomi, A., Silberman, M., Yellen, S. B., Winicour, P., & Brannon, J. (1993) · Journal of Clinical Oncology · doi:10.1200/jco.1993.11.3.570 · Cited by 6,020
Fitzmaurice, G. M., Laird, N. M., & Ware, J. H. (2011) · Wiley Series in Probability and Statistics · doi:10.1002/9781119513469 · Cited by 4,639
Small sample inference for fixed effects from restricted maximum likelihood
Kenward, M. G., & Roger, J. H. (1997) · Biometrics · doi:10.2307/2533558 · Cited by 4,552 · cited 1×
“All mixed models used the Kenward-Roger degrees of freedom, which performs well for both small and large samples.(27) All generalized estimating equations (GEEs) assumed an independent working correlation matrix.”
MethodSchafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949
Analysis of serial measurements in medical research.
Matthews, J. N., Altman, D. G., Campbell, M. J., & Royston, P. (1990) · BMJ · doi:10.1136/bmj.300.6719.230 · Cited by 3,136
Linear Mixed Models for Longitudinal Data
Molenberghs, G., & Verbeke, G. (2000) · Springer Series in Statistics · doi:10.1007/978-1-4419-0300-6 · Cited by 2,646
A comparison of inclusive and restrictive strategies in modern missing data procedures.
Collins, L. M., Schafer, J. L., & Kam, C.-M. (2001) · Psychological Methods · doi:10.1037/1082-989x.6.4.330 · Cited by 2,389
Statistical Analysis of Correlated Data Using Generalized Estimating Equations: An Orientation
Hanley, J. A. (2003) · American Journal of Epidemiology · doi:10.1093/aje/kwf215 · Cited by 2,185
Doubly robust estimation in missing data and causal inference models
Bang, H., & Robins, J. M. (2005) · Biometrics · doi:10.1111/j.1541-0420.2005.00377.x · Cited by 1,998
Goodness of fit tests for the multiple logistic regression model
Hosmer, D. W., & Lemesbow, S. (1980) · Communications in Statistics - Theory and Methods · doi:10.1080/03610928008827941 · Cited by 1,871
Best practices for missing data management in counseling psychology
Schlomer, G. L., Bauman, S., & Card, N. A. (2010) · Journal of Counseling Psychology · doi:10.1037/a0018082 · Cited by 1,835
Analysis of Semiparametric Regression Models for Repeated Outcomes in the Presence of Missing Data
Robins, J. M., Rotnitzky, A., & Zhao, L. P. (1995) · Journal of the American Statistical Association · doi:10.1080/01621459.1995.10476493 · Cited by 1,595
Gueorguieva, R., & Krystal, J. H. (2004) · Archives of General Psychiatry · doi:10.1001/archpsyc.61.3.310 · Cited by 1,529
Principal Stratification in Causal Inference
Frangakis, C. E., & Rubin, D. B. (2002) · Biometrics · doi:10.1111/j.0006-341x.2002.00021.x · Cited by 1,511
(2003) · Health and Quality of Life Outcomes · doi:10.1186/1477-7525-1-79 · Cited by 1,506
Informative Drop-Out in Longitudinal Data Analysis
Diggle, P., & Kenward, M. G. (1994) · Applied Statistics · doi:10.2307/2986113 · Cited by 1,468
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077
Missing Data in Clinical Studies
Molenberghs, G., & Kenward, M. G. (2007) · Wiley eBooks · doi:10.1002/9780470510445 · Cited by 962
Kang, J. D. Y., & Schafer, J. L. (2007) · Statistical Science · doi:10.1214/07-sts227 · Cited by 797
Horton, N. J., & Kleinman, K. P. (2007) · The American Statistician · doi:10.1198/000313007x172556 · Cited by 786
Burton, P., Gurrin, L., & Sly, P. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980615)17:11<1261::aid-sim846>3.0.co;2-z · Cited by 635
Wood, A. M., White, I. R., & Thompson, S. G. (2004) · Clinical Trials · doi:10.1191/1740774504cn032oa · Cited by 543
Fundamentals of Clinical Trials
Friedman, L. M., Furberg, C. D., & DeMets, D. L. (2010) · Springer eBooks · doi:10.1007/978-1-4419-1586-3 · Cited by 528
Missing data methods in longitudinal studies: a review
Ibrahim, J. G., & Molenberghs, G. (2009) · TEST · doi:10.1007/s11749-009-0138-x · Cited by 512
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474
Analyzing incomplete longitudinal clinical trial data
Molenberghs, G. (2004) · Biostatistics · doi:10.1093/biostatistics/5.3.445 · Cited by 463
Jeličić, H., Phelps, E., & Lerner, R. M. (2009) · Developmental Psychology · doi:10.1037/a0015665 · Cited by 433
Handling drop‐out in longitudinal studies
Hogan, J. W., Roy, J., & Korkontzelou, C. (2004) · Statistics in Medicine · doi:10.1002/sim.1728 · Cited by 404
Modelling Progression of CD4-Lymphocyte Count and Its Relationship to Survival Time
Gruttola, V. D., & Tu, X. M. (1994) · Biometrics · doi:10.2307/2533439 · Cited by 384
Clinical Trials: A Methodologic Perspective
Piantadosi, S. (2005) · Wiley Series in Probability and Statistics · doi:10.1002/0471740136 · Cited by 366
Quality of reporting of observational longitudinal research.
Tooth, L. (2005) · American Journal of Epidemiology · doi:10.1093/aje/kwi042 · Cited by 363
Willke, R. J., Burke, L. B., & Erickson, P. (2004) · Controlled Clinical Trials · doi:10.1016/j.cct.2004.09.003 · Cited by 305
Missing data in randomised controlled trials: a practical guide
Carpenter, J. R., & Kenward, M. (2007) · LSHTM Research Online (London School of Hygiene and Tropical Medicine) · Cited by 285
Motzer, R. J., Murphy, B. A., Bacik, J., Schwartz, L. H., Nanus, D. M., Mariani, T., Loehrer, P., Wilding, G., Fairclough, D. L., Cella, D., et al. (2000) · Journal of Clinical Oncology · doi:10.1200/jco.2000.18.16.2972 · Cited by 264
Self-citationA Comparison of Multiple Imputation and Doubly Robust Estimation for Analyses with Missing Data
Carpenter, J. R., Kenward, M. G., & Vansteelandt, S. (2006) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2006.00407.x · Cited by 255
Design and Analysis of Quality of Life Studies in Clinical Trials
Fairclough, D. L. (2010) · Chapman & Hall/CRC eBooks · doi:10.1201/9781420061185 · Cited by 228
Self-citationMissing quality of life data in cancer clinical trials: serious problems and challenges
Bernhard, J., Cella, D. F., Coates, A. S., Fallowfield, L. J., Ganz, P. A., Moinpour, C. M., Mosconi, P., Osoba, D., Simes, R. J., & Hürny, C. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980315/15)17:5/7<679::aid-sim814>3.0.co;2-x · Cited by 216
Functional Assessment of Cancer Therapy (FACT-G): Non-response to individual questions
Fairclough, D. L., & Cella, D. F. (1996) · Quality of Life Research · doi:10.1007/bf00433916 · Cited by 210
Self-citationMissing outcomes in randomized trials: addressing the dilemma.
Altman, D. G. (2009) · PubMed Central · Cited by 192
Longitudinal Data with Follow-up Truncated by Death: Match the Analysis Method to Research Aims
Kurland, B. F., Johnson, L. L., Egleston, B. L., & Diehr, P. H. (2009) · Statistical Science · doi:10.1214/09-sts293 · Cited by 186
On the performance of random‐coefficient pattern‐mixture models for non‐ignorable drop‐out
Demirtas, H., & Schafer, J. L. (2003) · Statistics in Medicine · doi:10.1002/sim.1475 · Cited by 177
Shared parameter models for the joint analysis of longitudinal data and event times
Vonesh, E. F., Greene, T., & Schluchter, M. D. (2006) · Statistics in Medicine · doi:10.1002/sim.2249 · Cited by 176
Vach, W., & Blettner, M. (1991) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a116164 · Cited by 175
Precision of Health-Related Quality-of-Life Data Compared With Other Clinical Measures
Hahn, E. A., Cella, D., Chassany, O., Fairclough, D. L., Wong, G. Y., & Hays, R. D. (2007) · Mayo Clinic Proceedings · doi:10.4065/82.10.1244 · Cited by 160
Self-citationAnalysing changes in the presence of informative right censoring caused by death and withdrawal
Wu, M. C., & Bailey, K. (1988) · Statistics in Medicine · doi:10.1002/sim.4780070134 · Cited by 158
(2004) · Statistics in Medicine · doi:10.1002/sim.1647 · Cited by 143
Mallinckrodt, C. H., Clark, W. S., Carroll, R. J., & Molenberghs, G. (2003) · Journal of Biopharmaceutical Statistics · doi:10.1081/bip-120019265 · Cited by 138
Hu, W., Li, G., & Li, N. (2009) · Statistics in Medicine · doi:10.1002/sim.3562 · Cited by 126
Choice of the primary analysis in longitudinal clinical trials
Mallinckrodt, C. H., Watkin, J. G., Molenberghs, G., & Carroll, R. J. (2004) · Pharmaceutical Statistics · doi:10.1002/pst.124 · Cited by 121
Au, H., Ringash, J. G., Brundage, M., Palmer, M. K., Richardson, H., & Meyer, R. M. (2010) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1586/erp.10.15 · Cited by 117
Fielding, S., MacLennan, G., Cook, J. A., & Ramsay, C. R. (2008) · Trials · doi:10.1186/1745-6215-9-51 · Cited by 105
Thiébaut, R., Jacqmin‐Gadda, H., Babiker, A. G., & Commenges, D. (2004) · Statistics in Medicine · doi:10.1002/sim.1923 · Cited by 99
Nevalainen, J., Kenward, M. G., & Virtanen, S. M. (2009) · Statistics in Medicine · doi:10.1002/sim.3731 · Cited by 91
An approach to joint analysis of longitudinal measurements and competing risks failure time data
Elashoff, R. M., Li, G., & Li, N. (2006) · Statistics in Medicine · doi:10.1002/sim.2749 · Cited by 84
Fairclough, D. L. (1997) · Statistics in Medicine · Cited by 83
Self-citationAnalysis and Interpretation of Results Based on Patient-Reported Outcomes
Sloan, J. A., Dueck, A. C., Erickson, P., Guess, H. A., Revicki, D. A., & Santanello, N. C. (2007) · Value in Health · doi:10.1111/j.1524-4733.2007.00273.x · Cited by 64
GEE with Gaussian Estimation of the Correlations When Data Are Incomplete
Lipsitz, S. R., Molenberghs, G., Fitzmaurice, G. M., & Ibrahim, J. G. (2000) · Biometrics · doi:10.1111/j.0006-341x.2000.00528.x · Cited by 63
Including deaths when measuring health status over time.
Diehr, P. H., Patrick, D. L., Hedrick, S. C., Rothman, M. L., Grembowski, D. E., Raghunathan, T. E., & Beresford, S. A. A. (1995) · PubMed · Cited by 61
Learning to Live With Missing Quality-of-Life Data in Advanced-Stage Disease Trials
Donaldson, G. W., & Moinpour, C. M. (2005) · Journal of Clinical Oncology · doi:10.1200/jco.2005.07.022 · Cited by 59
Doubly robust generalized estimating equations for longitudinal data
Seaman, S. R., & Copas, A. J. (2009) · Statistics in Medicine · doi:10.1002/sim.3520 · Cited by 59
Harris, A. H. S., Reeder, R. N., & Hyun, J. K. (2009) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2009.04.007 · Cited by 51
Doubly Robust and Multiple-Imputation-Based Generalized Estimating Equations
Ayele, B. T., Molenberghs, G., Sotto, C., & Kenward, M. G. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2011.550096 · Cited by 38
Analysis of messy longitudinal data from a randomized clinical trial
Qian, W., Parmar, M., Sambrook, R., Fayers, P. M., Girling, D. J., & Stephens, R. J. (2000) · Statistics in Medicine · Cited by 36
Harris, A. H. S., Reeder, R. N., & Hyun, J. K. (2011) · The Journal of Psychology · doi:10.1080/00223980.2011.555431 · Cited by 28
Correction of bias from non‐random missing longitudinal data using auxiliary information
Wang, C., & Hall, C. B. (2009) · Statistics in Medicine · doi:10.1002/sim.3821 · Cited by 25
Baker, S. G. (2005) · Biostatistics · doi:10.1093/biostatistics/kxi038 · Cited by 19
Analysing longitudinal continuous quality of life data with dropout
Curran, D., Molenberghs, G., Aaronson, N. K., Fosså, S. D., & Sylvester, R. J. (2002) · Statistical Methods in Medical Research · doi:10.1191/0962280202sm270ra · Cited by 14
Genetic effects, gene-lifestyle interactions, and type 2 diabetes
Qi, L. (2007) · Open Medicine · doi:10.2478/s11536-007-0051-1 · Cited by 9
Harris, A. H. S., Reeder, R. N., & Hyun, J. K. (2009) · The Open Public Health Journal · doi:10.2174/1874944500902010044 · Cited by 6
Multiple imputation of missing blood pressure covariates in survival analysis
van Buuren, S., Boshuizen, H. C., & Knook, D. L. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990330)18:6<681::aid-sim71>3.0.co;2-r
Cited by
Overman, M. J., Lonardi, S., Wong, K. Y. M., Lenz, H.-J., Gelsomino, F., Aglietta, M., Morse, M. A., Van Cutsem, E., McDermott, R., Hill, A., et al. (2018) · Journal of Clinical Oncology · doi:10.1200/jco.2017.76.9901 · Cited by 2,079
Mercieca-Bebber, R., King, M. T., Calvert, M. J., Stockler, M. R., & Friedlander, M. (2018) · Patient Related Outcome Measures · doi:10.2147/prom.s156279 · Cited by 659
“It is impossible to determine the true missing data mechanism, but information about the clinical status of the patient or the reasons for missing data, particularly whether these reasons are related to the patient’s worsening illness or not, can assist researchers in deducing the underlying mechanism.(52) A review of…”
Handling missing data in RCTs; a review of the top medical journals
Bell, M. L., Fiero, M., Horton, N. J., & Hsu, C.-H. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-118 · Cited by 395
“One trial performed worst-case imputation, LOCF, and multiple imputation.”
BackgroundMethodSelf-citationCoens, C., Pe, M., Dueck, A. C., Sloan, J., Basch, E., Calvert, M., Campbell, A., Cleeland, C., Cocks, K., Collette, L., et al. (2020) · The Lancet Oncology · doi:10.1016/s1470-2045(19)30790-9 · Cited by 323
Should multiple imputation be the method of choice for handling missing data in randomized trials?
Sullivan, T. R., White, I. R., Salter, A. B., Ryan, P., & Lee, K. J. (2018) · Statistical Methods in Medical Research · doi:10.1177/0962280216683570 · Cited by 308
Metadata
- Authors
- Bell, M. L., & Fairclough, D. L.
- Venue
- Statistical Methods in Medical Research
- Publisher
- SAGE Publications
- Published
- 2013-02-19
- Volume
- 23
- Issue
- 5
- Pages
- 440-459
- ISSN
- 0962-2802 (ISSN-L)
- 1477-0334
- OpenAlex
- W2016841028
- Cited by
- 285
- References
- 76
- FWCI
- 13.19
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.