Missing Data Matter: An Empirical Evaluation of the Impacts of Missing EHR Data in Comparative Effectiveness Research
Open accessAbstract
Objectives: The impacts of missing data in comparative effectiveness research (CER) using electronic health records (EHRs) may vary depending on the type and pattern of missing data. In this study, we aimed to quantify these impacts and compare the performance of different imputation methods. Materials and Methods: We conducted an empirical (simulation) study to quantify the bias and power loss in estimating treatment effects in CER using EHR data. We considered various missing scenarios and used the propensity scores to control for confounding. We compared the performance of the multiple imputation and spline smoothing methods to handle missing data. Results: When missing data depended on the stochastic progression of disease and medical practice patterns, the spline smoothing method produced results that were close to those obtained when there were no missing data. Compared to multiple imputation, the spline smoothing generally performed similarly or better, with smaller estimation bias and less power loss. The multiple imputation can still reduce study bias and power loss in some restrictive scenarios, eg, when missing data did not depend on the stochastic process of disease progression. Discussion and Conclusion: Missing data in EHRs could lead to biased estimates of treatment effects and false negative findings in CER even after missing data were imputed. It is important to leverage the temporal information of disease trajectory to impute missing values when using EHRs as a data resource for CER and to consider the missing rate and the effect size when choosing an imputation method.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automated tools can track sources and issue real-time alerts for deviations from quality standards, supporting compliance with regulatory requirements for accurate and secure management.
“Automated tools can track sources and issue real-time alerts for deviations from quality standards [75,90], supporting compliance with regulatory requirements for accurate and secure management [18,91]”
From Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles (2025) · cites this paper as [91]
The Introduction (page unavailable) discusses missing EHR data and its effects on comparative effectiveness research. Searches for automated source tracking, real-time quality alerts, and regulatory compliance found no passage in our copy addressing those claims. (LLM, conf=0.74)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106 · see citing works
RUBIN, D. B. (1976) · Biometrika · doi:10.1093/biomet/63.3.581 · Cited by 9,925 · see citing works
Wood, S. N. (2006) · doi:10.1201/9781420010404 · Cited by 4,953 · see citing works
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293 · see citing works
Multiple imputation after 18+ years
Rubin, D. B. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10476908 · Cited by 4,072 · see citing works
Schafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949 · see citing works
Review: a gentle introduction to imputation of missing values.
Donders, A. R. T., van der Heijden, G. J. M. G., Stijnen, T., & Moons, K. G. M. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.014 · Cited by 2,582 · see citing works
Statistical Analysis with Missing Data.
Chatfield, C., Little, R. J. A., & Rubin, D. B. (1988) · Journal of the Royal Statistical Society Series A (Statistics in Society) · doi:10.2307/2982783 · Cited by 2,203 · see citing works
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · see citing works
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,107 · see citing works
Balancing Covariates via Propensity Score Weighting
Li, F., Morgan, K. L., & Zaslavsky, A. M. (2016) · Journal of the American Statistical Association · doi:10.1080/01621459.2016.1260466 · Cited by 1,033 · see citing works
Next-generation phenotyping of electronic health records
Hripcsak, G., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001145 · Cited by 795 · see citing works
Multiple Imputation for Nonresponse in Surveys
Campion, W. M., & Rubin, D. B. (1989) · Journal of Marketing Research · doi:10.2307/3172772 · Cited by 775 · see citing works
Multiple Imputation for Interval Estimation From Simple Random Samples With Ignorable Nonresponse
Rubin, D. B., & Nathaniel Schenker, A. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478280 · Cited by 768 · see citing works
Hersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · see citing works
Comparative Effectiveness Research: A Report From the Institute of Medicine
Sox, H. C., & Greenfield, S. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-3-200908040-00125 · Cited by 525 · see citing works
Strategies for Handling Missing Data in Electronic Health Record Derived Data
Wells, B. J., Chagin, K., Nowacki, A., & Kattan, M. (2013) · eGEMs · doi:10.13063/2327-9214.1035 · Cited by 444 · see citing works
Missing data imputation: focusing on single imputation.
Zhang, Z. (2016) · PubMed Central · doi:10.3978/j.issn.2305-5839.2015.12.38 · Cited by 428 · see citing works
Agniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · see citing works
Lasko, T. A., Denny, J. C., & Levy, M. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066341 · Cited by 313 · see citing works
Optimal error bounds for cubic spline interpolation
Hall, C. A., & Meyer, W. W. (1976) · Journal of Approximation Theory · doi:10.1016/0021-9045(76)90040-x · Cited by 289 · see citing works
Chen, M., Tan, X., & Padman, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa143 · Cited by 287 · see citing works
PEDSnet: a National Pediatric Learning Health System
Forrest, C. B., Margolis, P. A., Bailey, L. C., Marsolo, K. A., Beccaro, M. A. D., Finkelstein, J. A., Milov, D. E., Vieland, V. J., Wolf, B. A., Yu, F., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002743 · Cited by 229 · see citing works
Self-citationGhassemi, M., Pimentel, M. A. F., Naumann, T., Brennan, T. J., Clifton, D. A., Peter Szolovits, B., & Feng, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9209 · Cited by 201 · see citing works
Missing data in medical databases: Impute, delete or classify?
Cismondi, F., Fialho, A. S., Vieira, S. M., Reti, S. R., Sousa, J. M. C., & Finkelstein, S. N. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.01.003 · Cited by 200 · see citing works
MISSING DATA IMPUTATION IN THE ELECTRONIC HEALTH RECORD USING DEEPLY LEARNED AUTOENCODERS
Beaulieu‐Jones, B. K., & Moore, J. H. (2016) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789813207813_0021 · Cited by 194 · see citing works
Natural history and long-term clinical course of Crohn’s disease
Freeman, H. J. (2014) · World Journal of Gastroenterology · doi:10.3748/wjg.v20.i1.31 · Cited by 175 · see citing works
Miriovsky, B. J., Shulman, L. N., & Abernethy, A. P. (2012) · Journal of Clinical Oncology · doi:10.1200/jco.2012.42.8011 · Cited by 160 · see citing works
Zhang, W.-X., Cao, L.-B., Zhao, Y., et al. (2021) · Annals of Translational Medicine · doi:10.21037/atm-20-3623 · Cited by 158 · see citing works
zoo: S3 Infrastructure for Regular and Irregular Time SeriesPreprint
Zeileis, A., & Grothendieck, G. (2005) · arXiv (Cornell University) · doi:10.48550/arxiv.math/0505527 · Cited by 152 · see citing works
Exploiting missing clinical data in Bayesian network modeling for predicting medical problems
Lin, J.-H., & Haug, P. J. (2007) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.06.001 · Cited by 138 · see citing works
Analysis of free text in electronic health records for identification of cancer patient trajectories
Jensen, K., Soguero-Ruíz, C., Mikalsen, K. Ø., Lindsetmo, R., Kouskoumvekaki, I. A., Girolami, M., Skrøvseth, S. O., & Augestad, K. M. (2017) · Scientific Reports · doi:10.1038/srep46226 · Cited by 133 · see citing works
Assessing Missing Data Assumptions in EHR-Based Studies: A Complex and Underappreciated Task
Haneuse, S., Arterburn, D., & Daniels, M. J. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.0184 · Cited by 110 · see citing works
Biases introduced by filtering electronic health records for patients with “complete data.”
Weber, G. M., Adams, W. G., Bernstam, E. V., Bickel, J. P., Fox, K. P., Marsolo, K., Raghavan, V. A., Turchin, A., Zhou, X., Murphy, S. N., et al. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx071 · Cited by 93 · see citing works
Identifying and mitigating biases in EHR laboratory tests
Pivovarov, R., Albers, D. J., Sepulveda, J. L., & Elhadad, N. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.03.016 · Cited by 91 · see citing works
Robust parameter extraction for decision support using multimodal intensive care data
Clifford, G. D., Long, W. J., Moody, G. B., & Peter Szolovits, B. (2008) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2008.0157 · Cited by 85 · see citing works
Taslimitehrani, V., Dong, G., Pereira, N. L., Panahiazar, M., & Pathak, J. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.01.009 · Cited by 83 · see citing works
Thygesen, J. H., Tomlinson, C., Hollings, S., Mizani, M. A., Handy, Á., Akbari, A., Banerjee, A., Cooper, J., Lai, A. G., Li, K., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00091-7 · Cited by 81 · see citing works
Li, T., Vedula, S. S., Scherer, R. W., & Dickersin, K. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-5-201203060-00009 · Cited by 78 · see citing works
Comparative Effectiveness Research: What Kind of Studies Do We Need?
Concato, J., Peduzzi, P. N., Huang, G. D., O’Leary, T. J., & Kupersmith, J. (2010) · Journal of Investigative Medicine · doi:10.2310/jim.0b013e3181e3d2af · Cited by 76 · see citing works
Exploiting time in electronic health record correlations
Hripcsak, G., Albers, D., & Perotte, A. (2011) · J. Am. Medical Informatics Assoc. · doi:10.1136/amiajnl-2011-000463 · Cited by 75 · see citing works
Parameterizing time in electronic health record studies
Hripcsak, G. M., Albers, D. J., & Perotte, A. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu051 · Cited by 75 · see citing works
Xu, Z., Chou, J., Zhang, X. S., Luo, Y., Isakova, T., Adekkanattu, P. M., Ancker, J. S., Jiang, G., Kiefer, R. C., Pacheco, J. A., et al. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103361 · Cited by 69 · see citing works
Comparative effectiveness research: what kind of studies do we need?
Concato, J., Peduzzi, P. N., Huang, G. D., O’Leary, T. J., & Kupersmith, J. (2010) · PubMed · Cited by 68 · see citing works
Correlating electronic health record concepts with healthcare process events
Hripcsak, G. M., & Albers, D. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-001922 · Cited by 64 · see citing works
The Evolving Uses of “Real-World” Data
Basch, E., & Schrag, D. (2019) · JAMA · doi:10.1001/jama.2019.4064 · Cited by 62 · see citing works
Prospective EHR-Based Clinical Trials: The Challenge of Missing Data
Kharrazi, H. H. K., Wang, C., & Scharfstein, D. O. (2014) · Journal of General Internal Medicine · doi:10.1007/s11606-014-2883-0 · Cited by 60 · see citing works
Grundmeier, R. W., Song, L., Ramos, M. J., Fiks, A. G., Elliott, M. N., Fremont, A., Pace, W., Wasserman, R. C., & Localio, R. (2015) · Health Services Research · doi:10.1111/1475-6773.12295 · Cited by 55 · see citing works
Temporal phenotyping by mining healthcare data to derive lines of therapy for cancer
Meng, W., Ou, W., Chandwani, S., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103335 · Cited by 54 · see citing works
Albers, D. J., & Hripcsak, G. (2012) · Chaos: An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.3675621 · Cited by 53 · see citing works
Zhao, J., Zhang, Y., Schlueter, D. J., et al. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103270 · Cited by 50 · see citing works
Adjusting for indirectly measured confounding using large-scale propensity score
Zhang, L., Wang, Z., Schuemie, M. J., Blei, D. M., & Hripcsak, G. M. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104204 · Cited by 47 · see citing works
Albers, D. J., Levine, M. E., Stuart, A. M., Mamykina, L., Gluckman, B. J., & Hripcsak, G. M. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy106 · Cited by 43 · see citing works
Goldstein, B. A., Phelan, M., Pagidipati, N. J., & Peskoe, S. B. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz148 · Cited by 42 · see citing works
Temporal trends of hemoglobin A1c testing
Pivovarov, R., Albers, D. J., Hripcsak, G., Sepulveda, J. L., & Elhadad, N. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002592 · Cited by 41 · see citing works
Exploiting Convolutional Neural Network for Risk Prediction with Medical Feature EmbeddingPreprint
Che, Z., Cheng, Y., Sun, Z., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.07474 · Cited by 41 · see citing works
Albers, D. J., & Hripcsak, G. (2010) · Physics Letters A · doi:10.1016/j.physleta.2009.12.067 · Cited by 36 · see citing works
Role of electronic health records in comparative effectiveness research
Gallego, B., Dunn, A. G., & Coiera, E. W. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.65 · Cited by 33 · see citing works
Informative presence bias in analyses of electronic health records-derived data: a cautionary note
Harton, J., Mitra, N., & Hubbard, R. A. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac050 · Cited by 29 · see citing works
Hu, Z., & Du, D. (2020) · PLoS ONE · doi:10.1371/journal.pone.0237724 · Cited by 29 · see citing works
Learning About Missing Data Mechanisms in Electronic Health Records-based Research
Haneuse, S., Bogart, A., Jazić, I., Westbrook, E. O., Boudreau, D. M., Theis, M. K., Simon, G. E., & Arterburn, D. E. (2015) · Epidemiology · doi:10.1097/ede.0000000000000393 · Cited by 27 · see citing works
High-throughput phenotyping with temporal sequences
Estiri, H., Strasser, Z. H., & Murphy, S. N. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa288 · Cited by 26 · see citing works
Methodological variations in lagged regression for detecting physiologic drug effects in EHR data
Levine, M. E., Albers, D. J., & Hripcsak, G. M. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.08.014 · Cited by 19 · see citing works
A method for the graphical modeling of relative temporal constraints
Mate, S., Bürkle, T., Kapsner, L. A., Toddenroth, D., Kampf, M. O., Sedlmayr, M., Castellanos, I., Prokosch, H., & Kraus, S. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103314 · Cited by 16 · see citing works
Brown, D. W., DeSantis, S., Greene, T. J., Maroufy, V., Yaseen, A., Wu, H.-L., Williams, G., & Swartz, M. (2020) · Statistics in Medicine · doi:10.1002/sim.8540 · Cited by 15 · see citing works
A Computational Method for Learning Disease Trajectories From Partially Observable EHR Data
Oh, W., Steinbach, M. S., Castro, M. R., PETERSON, K. A., Kumar, V., Caraballo, P. J., & Simon, G. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3089441 · Cited by 12 · see citing works
Dagliati, A., Gatta, R., Malovini, A. L., Tibollo, V., Sacchi, L., Cascini, F., Chiovato, L., & Bellazzi, R. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.815674 · Cited by 10 · see citing works
Koffman, L. J., Levis, A. W., Arterburn, D. E., Coleman, K. J., Herrinton, L. J., Cooper, J., Ewing, J. J., Fischer, H. J., Fraser, J. R. P., Johnson, E., et al. (2021) · Obesity Surgery · doi:10.1007/s11695-021-05226-y · Cited by 9 · see citing works
Levine, M. E., Albers, D. J., & Hripcsak, G. M. (2016) · PubMed Central · Cited by 8 · see citing works
Zhou, R., Johnson, K. E., Rousseau, J. F., & Rathouz, P. J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0294892 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- [24]24. Saria S, Duchi A, Koller D. Discovering deformable motifs in continuous time series data. In: Twenty-Second International Joint Conference on Artificial Intelligence; 2011.
- [36]36. Liu C, Wang F, Hu J, Xiong H. Temporal phenotyping from longitudinal electronic health records: a graph based framework. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2015.
- [40]40. Cheng Y, Wang F, Zhang P, Hu J. Risk prediction with electronic health records: a deep learning approach. In: Proceedings of the 2016 SIAM International Conference on Data Mining. SIAM; 2016.
- [50]50. Cismondi F, Fialho AS, Vieira SM, et al. Computational intelligence meth- ods for processing misaligned, unevenly sampled time series containing missing data. In: 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM). IEEE; 2011.
- [52]52. Donders ART, van der Heijden GJMG, Stijnen T, et al. A gentle introduc- tion to imputation of missing values. J Clin Epidemiol 2006; 59 (10): 1087-91.
- [53]53. Norazian MN, Shukri YA, Azam RN, Al Bakri AMM. Estimation of miss- ing values in air pollution data using single imputation techniques. Scien- ceAsia 2008; 34 (3): 341-5.
- [63]63. Zhou R, Johnson. KE, Rousseau JF, Rathouz PJ, N3C Consortium. Com- parative effectiveness of dexamethasone in treatment of hospitalized COVID-19 patients during the first year of the pandemic: the N3C data repository. medRxiv, 2022: p. 2022.10. 22.22281373. https://www.medr- xiv.org/content/medrxiv/early/2022/10/24/2022.10.22.22281373.full. pdf. Accessed February 14, 2023.
Cited by
Sorted by citations
Data Quality in the Age of AI: A Review of Governance, Ethics, and the FAIR Principles
Guillen-Aguinaga, M., Aguinaga-Ontoso, E., Guillen-Aguinaga, L., Guillen-Grima, F., & Aguinaga-Ontoso, I. (2025) · Data · doi:10.3390/data10120201 · Cited by 37 · see citing works
- Authors
- Zhou, Y., Shi, J., Stein, R., Liu, X., Baldassano, R. N., Forrest, C. B., Chen, Y., & Huang, J.
- Affiliations
- Children's Hospital of Philadelphia (US)
- University of Pennsylvania (US)
- Corresponding author
- Zhou, Yizhao
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-04-17
- Volume
- 30
- Issue
- 7
- Pages
- 1246-1256
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC10280351
- OpenAlex
- W4366085127
- Cited by
- 33
- References
- 70
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 6 · 2023: 2
- FWCI
- 6.71
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Patient-Centered Outcomes Research Institute (PCORI) Project Program Awards (ME-2018C3-14899)
- Eunice Kennedy Shriver National Institute of Child Health (R01HD099348)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Keywords
- missing data, electronic health records, comparative effectiveness research, multiple imputation, smoothing splines, imputation method comparison, propensity score, disease progression, temporal data
- MeSH terms
- Computer Simulation, Data Interpretation, Statistical, Research Design, Bias, Comparative Effectiveness Research, Propensity Score
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
