Data heterogeneity in federated learning with Electronic Health Records: Case studies of risk prediction for acute kidney injury and sepsis diseases in critical care

Because overall patient distributions are conserved after outlier removal, MICE imputation can provide robust estimation of these values.

“Because overall patient distributions are conserved after outlier removal (due to limited number of values being considered outliers), MICE imputation can provide robust estimation of these values as well [33].”

• cites [33] SK Kwak, 2017, Statistical Data Preparation: Management of missing values and outliers, Korean Journal of Anesthesiology sourceDisputed

  The Trimming section (page unavailable) says excluding outliers decreases variance and can introduce bias, contrary to the claim that the overall distribution is conserved after removal. The Imputation analysis section discusses imputation generally but does not address MICE or establish that it robustly estimates removed outliers. (LLM, conf=0.84)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured

The prediction models developed in this study showed comparable performance with state-of-the-art AKI and sepsis prediction studies.

“Our prediction models have shown comparable performance with state-of-the art AKI and sepsis prediction studies [7].”

• cites [7] X Song, 2020, Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction, Nature Communications sourcePartly supported

  The Discussion (page not provided) describes the source paper’s AKI model as performing better than a state-of-the-art comparator for any AKI prediction and competitively for moderate-to-severe AKI prediction. It does not address sepsis prediction or evaluate the citing authors’ models, so it supports only the AKI benchmark portion of the comparison. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

For both AKI and sepsis prediction tasks, important variables identified by predictive models were consistent with prior studies.

“For both AKI and sepsis prediction tasks, important variables identified by predictive models were consistent with prior studies [7].”

• cites [7] X Song, 2020, Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction, Nature Communications sourcePartly supported

  The Results section on clinical interpretation identifies important variables for AKI prediction, and the Discussion describes some of their associations as known. The cited paper does not report a sepsis prediction task or establish that its important variables were consistent with prior sepsis studies. Page numbers are unavailable in our copy. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

Treatments may vary depending on the etiology and nature of the condition driving sepsis and AKI.

“Thus, treatments may vary depending on the etiology and nature of the condition driving sepsis and AKI [30].”

• cites [30] B. Bilgili, 2014, Sepsis and Acute Kidney Injury. Turkish journal of anaesthesiology and reanimation sourcePartly supported

  The Pathogenesis of sepsis-associated AKI section says appropriate treatment is necessary, and The two-way relationship between sepsis and AKI section says septic and nonseptic AKI have different pathogenic mechanisms (page numbers unavailable). The paper does not say that treatments vary according to the etiology or nature of the condition driving sepsis and AKI. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Age distributions are similar across all sites with most patients being between 50-75 years old.

“Age distributions are also similar across all sites with most patients being between 50–75 years old.”

The African American population varies across sites from 0.02/0.01 at Site 199 to 0.3/0.32 at Site 243.

“The African American population varies across sites from 0.02/0.01 (AKI/sepsis) at Site 199 to 0.3/0.32 at Site 243.”

In both settings, there was a disparity in the number of patients that underwent elective surgery, with proportions ranging from 0.12-0.28.

“In both settings, there was a disparity in the number of patients that underwent elective surgery, with the proportions ranging from 0.12–0.28.”

In the pooled MLP model, creatinine_last of ~4 mg/dL is associated with a 1.5-fold increase in risk of AKI 24h.

“In the pooled MLP model, creatinine_last of ~4 mg/dL is associated with an exp(0.4) = 1.5-fold increase in risk of AKI 24h.”

A bun_last measurement of ~60 mg/dL is associated with a 1.2-fold increase in risk of AKI 24h.

“A bun_last measurement of ~60 mg/dL is associated with a exp(0.2) = 1.2-fold increase in risk of AKI 24h.”

Both AKI 24h and 48h settings' pooled models have relatively fewer features that are only important at a small number of sites compared to the local model framework.

“Both AKI 24h and 48h settings’ pooled models have relatively fewer features that are only important at a small number of sites compared to the local model framework.”

Pooled sepsis LR models achieved AUC 0.731-0.800.

“Pooled sepsis LR models’ performances ranged between 0.731–0.800 across sites”

Federated AKI 48h LR models achieved AUC 0.722-0.835.

“AKI 48h LR models performed within the range of 0.722–0.835”

The weights of the federated models were concentrated around zero compared to pooled models for all settings.

“The weights of the federated models are concentrated around zero as compared to pooled models for all settings.”

In the AKI 24h setting, local models have moderately strong positive correlations with Pearson-correlation coefficients ranging from 0.79-0.84.

“In the AKI 24h setting, local models have moderately strong positive correlations with Pearson-correlation coefficients (PC) ranging from 0.79–0.84.”

Local sepsis LR models achieved AUC 0.771-0.834.

“Sepsis LR models’ performances ranged between 0.771–0.834 across sites”

LR and MLP models performed similarly across all prediction tasks.

“The LR and MLP models performed similarly across all prediction tasks.”

Local AKI 48h LR models achieved AUC 0.680-0.809.

“AKI 48h LR models performed within the range of 0.680–0.809”

Federated models outperformed pooled models.

“Generally, the federated model outperformed the local model and pooled model.”

There is a strong positive correlation (PC ranging from 0.84-0.93) between local UIS and pooled/federated.

“There is a strong positive correlation (PC ranging from 0.84–0.93) between local UIS and pooled/federated.”

Local sepsis MLP models achieved AUC 0.772-0.829.

“MLP models’ performances ranged between 0.772–0.829.”

Pooled models underperformed local models.

“The pooled models underperformed the local model.”

Pooled AKI 48h LR models achieved AUC 0.683-0.744.

“pooled AKI 48h LR models performed within the range of 0.683–0.744”

Patient BMI is similar across sites with most patients having a BMI between 23-34.

“Patient BMI is similar across sites with most patients having a BMI between 23–34.”

Local AKI 24h MLP models achieved AUC 0.677-0.821.

“MLP models performed within a range of 0.677–0.821.”

Federated AKI 24h MLP models achieved AUC 0.732-0.839.

“MLP models performed within a range of 0.732–0.839.”

Local AKI 24h LR models achieved AUC 0.680-0.809.

“AKI 24h LR models performed within the range of 0.680–0.809”

For all settings, there are features that are both universally important and site-specific.

“For all settings, there are features that are both ‘universally important’ and site-specific (i.e., important at only a subset of sites).”

The pooled and federated AKI 24h model shows slightly weaker positive correlations as compared to the local models PC = 0.79, 0.77.

“The pooled and federated AKI 24h model shows slightly weaker positive correlations as compared to the local models PC = 0.79, 0.77.”

For the AKI 24h setting, the federated MLP and LR models consider more medications important than their respective pooled counterparts.

“For the AKI 24h setting, the federated MLP and LR model consider more medications important than their respective pooled counterparts.”

Federated AKI 24h LR models achieved AUC 0.742-0.834.

“AKI 24h LR models performed within the range of 0.742–0.834”

Pooled sepsis MLP models achieved AUC 0.732-0.793.

“MLP models’ performances ranged between 0.732–0.793.”

The relative importance of features at local sites were different.

“However, the relative importance of features at local sites were different.”

For AKI 24h and AKI 48h settings, the proportion of AKI positive patients ranges from 0.06/0.08 to 0.1/0.13.

“For AKI 24h and AKI 48h settings, the proportion of AKI positive patients ranges from 0.06/0.08 (24h/48h) to 0.1/0.13.”

Federated sepsis LR models achieved AUC 0.833-0.862.

“Federated sepsis LR models’ performances ranged between 0.833–0.862 across sites”

The pooled model did not show much improvement over the local models.

“Moreover, the pooled model did not show much improvement over the local models”

Federated learning performed better than pooled models in this study's investigations.

“Though FL performed better than pooled models in our investigations”

For the sepsis setting, the proportion of positive patients ranges from 0.02 to 0.20.

“For the sepsis setting, the proportion of positive patients ranges from 0.02 to 0.20.”

Local AKI 48h MLP models achieved AUC 0.673-0.800.

“MLP models performed within a range of 0.673–0.800.”

LR pooled models showed more consistent performances to local model counterparts in comparison to MLP pooled models.

“LR pooled models showed more consistent performances to local model counterparts in comparison to MLP pooled models.”

Federated sepsis MLP models achieved AUC 0.823-0.861.

“MLP models’ performances ranged between 0.823–0.861.”

Patients show differences in racial breakdowns across sites.

“Patients show differences in racial breakdowns across sites.”

The universally important features across most sites for AKI included creatinine_last, creatinine_min, creatinine_max, administration of Ondansteron, glucose_max, and urineoutput_sum.

“The universally important features across most sites for AKI (both 24h and 48h) included creatinine_last, creatinine_min, creatinine_max, administration of Ondansteron, glucose_max, and urineoutput_sum.”

Federated AKI 48h MLP models achieved AUC 0.72-0.833.

“MLP models performed within a range of 0.72–0.833.”

In the AKI 24h setting, the pooled MLP model identified creatinine_last, hematocrit_last, Furosemide, bg_paco2_min, potassium_max, creatinine_min, sysbp_last, hemoglobin_first, bicarbonate_min, and calcium_last as the top 10 most important variables.

“In the AKI 24h setting, the pooled MLP model identified last measured level of creatinine (creatinine_last), last measured hematocrit level (hematocrit_last), Furosemide, bg_paco2_min, maximum potassium level (potassium_max), minimum creatinine levels (creatinine_min), last measured systolic blood pressure (sysbp_last), hemoglobin_first, minimum bicarbonate level (bicarbonate_min), and last Measur”

Federated models outperformed local models.

“We found models built within a FL framework outperformed local counterparts.”

• Also stated in: § Development of AKI and sepsis prediction models in local sites · hedged

  “Generally, the federated model outperformed the local model and pooled model.”

Federated model frameworks generally outperformed their local counterparts in this study's results.

“In addition, federated model frameworks generally outperform their local counterparts in our results.”

Creatinine and furosemide exposure showed positive associations with AKI.

“creatinine and furosemide exposure showed positive associations with AKI”

Features with higher UIS scores tend to be medications and demographic information.

“features with higher UIS scores tend to be medications and demographic information.”

Within this analysis, differences in features of the hospitals and ICUs were notable.

“Within our analysis, differences in features of the hospitals, and ICUs were notable.”

Lab tests and vital signs were generally universally important features across hospitals.

“lab tests and vital signs were generally universally important features across hospitals”

The feature heterogeneity plots for federated and pooled frameworks showed a decreased amount of uniquely important features.

“The feature heterogeneity plots for federated and pooled frameworks showed a decreased amount of uniquely important features.”

There was disagreement on relative feature importance across sites, as shown by the presence of universally important and uniquely important features.

“The presence of ‘universally important features’ (i.e., features that were considered highly important at most sites) and ’uniquely important features’ (i.e., features that were highly important at a small subset of sites) showed that there was disagreement on relative importance across sites.”

A total of 354 features were available at every hospital site for each patient.

“Taken together, a total of 354 features were available at every hospital site for each patient.”

Federated and pooled models were not successful at decreasing feature importance discrepancies between LR and MLP architectures.

“Our findings also demonstrated that federated and pooled models were not successful at decreasing feature importance discrepancies between LR and MLP architectures”

Many sites did not have any patients admitted to ICU types that other sites had a high proportion of patients within, for example the Medical Surgery ICU or SICU.

“Many sites did not have any patients admitted to ICU types that other sites had a high proportion of patients within, for example the Medical Surgery ICU or SICU.”

The models highlighted a putative disagreement between medication usage at hospitals, creating another source of heterogeneity in model training.

“Our models highlighted this putative disagreement between medication usage at hospitals, creating another source of heterogeneity in model training.”

Different distributions of demographic profiles, medication use, and site information contributed to data heterogeneity.

“The different distributions of demographic profiles, medication use, and site information contributed to data heterogeneity.”

The extent to which federated models outperform local models depends largely on how heterogeneous the patient populations from different hospitals are.

“However, this largely depends on how heterogeneous the patient populations from different hospitals are.”

Communication costs between sites and the central server in the federated framework take a significant amount of time, as evidenced by the time differences between pooled and federated framework.

“Communication costs between sites and the central sever within the federated framework take a significant amount of time, as evidenced by the time differences between pooled and federated framework.”

The lack of improvement of the pooled model over local models could be largely due to cross-site sample heterogeneity.

“this could be largely due to the cross-site sample heterogeneity.”

The federated model may be better at discriminating the key features of patient-level clinical, lab, and demographic information that improves risk prediction.

“These findings also suggest that the federated model may be better at discriminating the key features of patient-level clinical, lab, and demographic information that improves risk prediction.”

Different hospitals, which are specialized at treating different conditions, may have slightly differing medication regimens for treating patients when faced with the same disease.

“different hospitals, which are specialized at treating different conditions, may have slightly differing medication regimens for treating patients when faced with the same disease”

Both federated and pooled frameworks were able to attribute higher importance to features shared across multiple sites.

“This was indicative of both these frameworks being able to attribute higher importance to features shared across multiple sites.”

At these hospitals, different sites treat different conditions.

“At these hospitals, different sites treat different conditions.”

Local frameworks may heavily suffer in generalizability even when population demographics are similar across sites, due to disagreements in medication and treatment administration.

“Taken together, local frameworks may heavily suffer in generalizability even when population demographics are similar across sites, due to disagreements in medication and treatment administration.”