CoChat
Sign Up
Cited by 133Statistical Methods and Bayesian InferenceMachine Learning in HealthcareArtificial Intelligence in Healthcare

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.

Imputation of missing values for electronic health record laboratory data

Open access
Li, Jiang, Yan, Xiaowei S., Chaudhary, Durgesh, Avula, Venkatesh, Mudiganti, Satish, Husby, Hannah, Shahjouei, Shima, Afshar, Ardavan, Stewart, Walter F., Yeasin, Mohammed, Zand, Ramin, Abedi, Vida·2021·npj Digital Medicine
DOI 10.1038/s41746-021-00518-0PMC PMC8505441
Open this paper in CoChat — free accountOpen PDF

Abstract

Laboratory data from Electronic Health Records (EHR) are often used in prediction models where estimation bias and model performance from missingness can be mitigated using imputation methods. We demonstrate the utility of imputation in two real-world EHR-derived cohorts of ischemic stroke from Geisinger and of heart failure from Sutter Health to: (1) characterize the patterns of missingness in laboratory variables; (2) simulate two missing mechanisms, arbitrary and monotone; (3) compare cross-sectional and multi-level multivariate missing imputation algorithms applied to laboratory data; (4) assess whether incorporation of latent information, derived from comorbidity data, can improve the performance of the algorithms. The latter was based on a case study of hemoglobin A1c under a univariate missing imputation framework. Overall, the pattern of missingness in EHR laboratory variables was not at random and was highly associated with patients’ comorbidity data; and the multi-level imputation algorithm showed smaller imputation error than the cross-sectional method.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-021-00518-0

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Data extraction and analysis pipeline. 1a Inspired by a stepwise imputation by observation blocks in longitudinal data from EHR, we extracted the last observation before an event (e.g., stroke or heart failure), and the first observation after the event. Two types of holdouts, random holdout values (HV) and random holdout complete cases (HC), represent two missingness scenarios, MAR and monotone with NMAR, respectively. 1b Outline the tested imputation models and algorithms evaluated by error metrics and the results after repeated multiple imputation (MI). Abbreviations: PMM predictive mean matching, 2lpan Implements the Gibbs sampler for the linear two-level model with homogeneous within-group (patient ID) variances, LMM linear mixed-effects model, FCS fully conditional specification, nRMSE Root Mean Square Error normalized by standard deviation, CR Coverage rate, the proportion of confidence intervals that contain the true value.
Fig. 1 Data extraction and analysis pipeline. 1a Inspired by a stepwise imputation by observation blocks in longitudinal data from EHR, we extracted the last observation before an event (e.g., stroke or heart failure), and the first observation after the event. Two types of holdouts, random holdout values (HV) and random holdout complete cases (HC), represent two missingness scenarios, MAR and monotone with NMAR, respectively. 1b Outline the tested imputation models and algorithms evaluated by error metrics and the results after repeated multiple imputation (MI). Abbreviations: PMM predictive mean matching, 2lpan Implements the Gibbs sampler for the linear two-level model with homogeneous within-group (patient ID) variances, LMM linear mixed-effects model, FCS fully conditional specification, nRMSE Root Mean Square Error normalized by standard deviation, CR Coverage rate, the proportion of confidence intervals that contain the true value.CC BY · PubMed Central
Fig. 2 Exploring missingness pattern and mechanism: level of missingness in GNSIS (n=9037) and HF (=5192). a, b/c, d Missingness pattern, monotone with some degree of randomness, before or after the event was created by R “naniar” package (black represents missing and light gray represents observed). The x-axis is the description of the laboratory variables, which are sorted based on the percentage of missingness. The corresponding laboratory component ID and % missing was labeled on the top row. Some patients were marked as both monotone and arbitrary missing because some providers ordered a specific laboratory test (or a specific panel of tests) for some patients for a variety of reasons under certain circumstances. This would violate the sequentially ordering of laboratory tests and lead to some random missing (arbitrary) for other patients. e, f A fluxplot showing the distribution of laboratory variables before the event is determined by the corresponding influx and outflux values. The laboratory variables in a panel test generally were clustered together. g, h Area plot (identity but not stacking) to show the percent of subjects with repeated (up to three) measures. The x-axis was laboratory variables sorted by percent of missingness (%) before the event.
Fig. 2 Exploring missingness pattern and mechanism: level of missingness in GNSIS (n=9037) and HF (=5192). a, b/c, d Missingness pattern, monotone with some degree of randomness, before or after the event was created by R “naniar” package (black represents missing and light gray represents observed). The x-axis is the description of the laboratory variables, which are sorted based on the percentage of missingness. The corresponding laboratory component ID and % missing was labeled on the top row. Some patients were marked as both monotone and arbitrary missing because some providers ordered a specific laboratory test (or a specific panel of tests) for some patients for a variety of reasons under certain circumstances. This would violate the sequentially ordering of laboratory tests and lead to some random missing (arbitrary) for other patients. e, f A fluxplot showing the distribution of laboratory variables before the event is determined by the corresponding influx and outflux values. The laboratory variables in a panel test generally were clustered together. g, h Area plot (identity but not stacking) to show the percent of subjects with repeated (up to three) measures. The x-axis was laboratory variables sorted by percent of missingness (%) before the event.CC BY · PubMed Central
Fig. 3 This missingness (dummy variables) was highly associated with comorbidity. HbA1C (a) in GNSIS dataset and LDL cholesterol (b) in HF dataset was one of the most valuable laboratory variables used in the prediction model for the outcome of interest in ischemic stroke and/or heart failure. PCA was conducted by R “factoextra“ package. Dim represented the dimensions extracted from the comorbidity matrix using prcomp function; The y-axis represented log-transformed p-value of the significance of the difference (absolute value) in principal component values between observed and missing groups after Welch unpaired t-test.
Fig. 3 This missingness (dummy variables) was highly associated with comorbidity. HbA1C (a) in GNSIS dataset and LDL cholesterol (b) in HF dataset was one of the most valuable laboratory variables used in the prediction model for the outcome of interest in ischemic stroke and/or heart failure. PCA was conducted by R “factoextra“ package. Dim represented the dimensions extracted from the comorbidity matrix using prcomp function; The y-axis represented log-transformed p-value of the significance of the difference (absolute value) in principal component values between observed and missing groups after Welch unpaired t-test.CC BY · PubMed Central
Fig. 4 Mean coverage rate (CR) across all imputation algorithms evaluated for GNSIS and HF datasets. The vertical bar represents 95%CI of the corresponding mean imputed value for 50 holdouts. The horizontal line represents the mean coverage rate of 0.90. a/b represents CR for 50 holdout values (HV) or complete cases (HC), respectively, in GNSIS; c/d represents CR for 50 HV or HC, respectively, in HF.
Fig. 4 Mean coverage rate (CR) across all imputation algorithms evaluated for GNSIS and HF datasets. The vertical bar represents 95%CI of the corresponding mean imputed value for 50 holdouts. The horizontal line represents the mean coverage rate of 0.90. a/b represents CR for 50 holdout values (HV) or complete cases (HC), respectively, in GNSIS; c/d represents CR for 50 HV or HC, respectively, in HF.CC BY · PubMed Central
Fig. 5 Laboratory test, missingness pattern, and imputation algorithm dependent uncertainty propagation during 50 repeated multiple imputation (MI) for holdouts using RMSE. The y-axis was labeled with mean(nRMSE) after the number of MI. This mean of normalized RMSE in y-axis represents the error over the number of runs. The size of the triangle or round dot represents the standard error of RMSE over the number of runs. “L” shaped or inverse “L” shaped distribution of the mean(nRMSE) after 50 runs of MI suggested this uncertainty reached a plateau. However, for some variables (e.g., 13457-7 of LDL or 17856-6 of HgA1C) in HC, the uncertainty went upward but not reached a statistical convergence.
Fig. 5 Laboratory test, missingness pattern, and imputation algorithm dependent uncertainty propagation during 50 repeated multiple imputation (MI) for holdouts using RMSE. The y-axis was labeled with mean(nRMSE) after the number of MI. This mean of normalized RMSE in y-axis represents the error over the number of runs. The size of the triangle or round dot represents the standard error of RMSE over the number of runs. “L” shaped or inverse “L” shaped distribution of the mean(nRMSE) after 50 runs of MI suggested this uncertainty reached a plateau. However, for some variables (e.g., 13457-7 of LDL or 17856-6 of HgA1C) in HC, the uncertainty went upward but not reached a statistical convergence.CC BY · PubMed Central
Fig. 6 Comparing the performance of selected imputation algorithms after 50 repeated multiple imputation for random holdout cases (HC) or values (HV) using nRMSE in GNSIS and HF datasets. Levene’s test showed an equal variance of the nRMSE from two compared imputation algorithms, e.g. 2LPAN-FCS and PMM-FCS. Shapiro-Wilk test showed the normality of the difference for each comparison. An unpaired t-test was conducted to determine the mean difference of nRMSE between two compared imputation algorithms. Only the raw p-value < 0.05/45 (~0.0011) for GNSIS (a) or < 0.05/38 (~0.0013) for HF (b) was considered as statistical significance, which survived the Bonferroni correction for multiple testing. The curve for Pearson’s correlation coefficient of observed values between before and after index date per variable was overlaid to the corresponding dot plot for normalized RMSE.
Fig. 6 Comparing the performance of selected imputation algorithms after 50 repeated multiple imputation for random holdout cases (HC) or values (HV) using nRMSE in GNSIS and HF datasets. Levene’s test showed an equal variance of the nRMSE from two compared imputation algorithms, e.g. 2LPAN-FCS and PMM-FCS. Shapiro-Wilk test showed the normality of the difference for each comparison. An unpaired t-test was conducted to determine the mean difference of nRMSE between two compared imputation algorithms. Only the raw p-value < 0.05/45 (~0.0011) for GNSIS (a) or < 0.05/38 (~0.0013) for HF (b) was considered as statistical significance, which survived the Bonferroni correction for multiple testing. The curve for Pearson’s correlation coefficient of observed values between before and after index date per variable was overlaid to the corresponding dot plot for normalized RMSE.CC BY · PubMed Central
Fig. 7 Over-imputation plot demonstrated the mean and standard error of imputed values for the observed 50 holdouts where imputed values would lie if they were missing in the GNSIS dataset using HbA1c as an example. The lm fit line with 95% CI was superimposed on the scatter plot. The Pearson’s correlation coefficient (R), as well as the significance of this correlation between 50 holdouts and imputed mean values were also present. This R-value represented the optimal correlation coefficient, which could be reached by each MI algorithm under multivariate or univariate setting. The kernel density plots at the margin of the scatter plot represented the corresponding distribution of observed and imputed 50 holdouts. Upper panels represent multivariate missing imputation using PMM or 2LPAN for HV or HC; Lower panels represent univariate missing imputation using 2LPAN with or without PCs derived from a comorbidity matrix as auxiliary variables.
Fig. 7 Over-imputation plot demonstrated the mean and standard error of imputed values for the observed 50 holdouts where imputed values would lie if they were missing in the GNSIS dataset using HbA1c as an example. The lm fit line with 95% CI was superimposed on the scatter plot. The Pearson’s correlation coefficient (R), as well as the significance of this correlation between 50 holdouts and imputed mean values were also present. This R-value represented the optimal correlation coefficient, which could be reached by each MI algorithm under multivariate or univariate setting. The kernel density plots at the margin of the scatter plot represented the corresponding distribution of observed and imputed 50 holdouts. Upper panels represent multivariate missing imputation using PMM or 2LPAN for HV or HC; Lower panels represent univariate missing imputation using 2LPAN with or without PCs derived from a comorbidity matrix as auxiliary variables.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 47 references

Research works printed but not linked: 2

  1. Ford, B. in Incomplete Data in Sample Surveys, Theory and Bibliographies Vol. 2 (Part IV) (eds. W. Madow, H. Nisselson, & I. Olkin) 185–207 (Academic Press, 1983).
  2. Van Buuren, S. & Groothuis-Oudshoorn, K. mice: multivariate imputation by chained equations in R. J. Stat. Softw. 45, 67 (2011). · DOI · Paper page

Cited by

Sorted by citations

See all 133

Related

Metadata

Authors
Li, J., Yan, X. S., Chaudhary, D., Avula, V., Mudiganti, S., Husby, H., Shahjouei, S., Afshar, A., Stewart, W. F., Yeasin, M., Zand, R., & Abedi, V.
Affiliations
  • Geisinger Health System (US)
  • Sutter Health (US)
  • Georgia Institute of Technology (US)
  • University of Memphis (US)
  • Pennsylvania State University (US)
Corresponding author
Abedi, Vida
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2021-10-11
Volume
4
Issue
1
Pages
147-147
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-021-00518-0
PMC
PMC8505441
OpenAlex
W3205484571
Cited by
133
References
47
Citations by year
2026: 30 · 2025: 35 · 2024: 33 · 2023: 23 · 2022: 11 · 2021: 1
FWCI
12.68
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
U.S. Department of Health & Human Services | NIH | Center for Information Technology (R56HL116832)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
missing data imputation, electronic health records, laboratory findings, ischemic stroke, heart failure, missingness patterns, comorbidities, HbA1c, imputation error

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review