Predicting Clinical Outcomes Across Changing Electronic Health Record Systems
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Existing machine learning methods typically assume consistency in how semantically equivalent information is encoded. However, the way information is recorded in databases differs across institutions and over time, often rendering potentially useful data obsolescent. To address this problem, we map database-specific representations of information to a shared set of semantic concepts, thus allowing models to be built from or transition across different databases. We demonstrate our method on machine learning models developed in a healthcare setting. In particular, we evaluate our method using two different intensive care unit (ICU) databases and on two clinically relevant tasks, in-hospital mortality and prolonged length of stay. For both outcomes, a feature representation mapping EHR-specific events to a shared set of clinical concepts yields better results than using EHR-specific events alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, it is highly unlikely that a mortality prediction system at a hospital in California will work in the same way when implemented half-way around the world, particularly due to site-specific variations32,33.”
From Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency (2020) · cites this paper as [32]
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 6×
“(2) We demonstrate the effectiveness of our approach on two commonly used predictive models and on data from two epochs of EHR systems in the publicly available MIMIC-III dataset.”
MethodInfluentialSelf-citationLe Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773 · cited 6×
“This provides a fair comparison against baseline acuity scores, which commonly use only information from this time period [16].”
BackgroundMethodInfluentialFreebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852 · cited 1×
“Although general domain ontologies (e.g., [3]) and tools for identifying equivalent semantic concepts (e.g., [10]) exist, these tools do not work well with the highly domain-specific vocabulary present in clinical text.”
BackgroundThe unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · cited 11×
“Instead, we map each element directly to concepts in the UMLS ontologies and use this representation for greater generalizability of predictive models.”
MethodInfluentialIncorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008 · cited 1×
“, [10]) exist, these tools do not work well with the highly domain-specific vocabulary present in clinical text.”
BackgroundSavova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · cited 1×
“Using the Clinical Text Analysis Knowledge Extraction System (cTAKES), a frequently used tool for identifying UMLS concepts, we annotate the human-readable item descriptions from both EHR versions in our data [23].”
MethodTransfer learning via dimensionality reduction
Pan, S. J., Kwok, J. T.-Y., & Yang, Q. (2008) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 653
Dong, X. L., & Srivastava, D. (2015) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01853-4 · Cited by 539
Cited by
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
“For example, it is highly unlikely that a mortality prediction system at a hospital in California will work in the same way when implemented half-way around the world, particularly due to site-speci fi c variations 32,33 .”
BackgroundThe role of machine learning in clinical research: transforming the future of evidence generation
Weissler, E. H., Naumann, T., Andersson, T., Ranganath, R., Elemento, O., Luo, Y., Freitag, D. F., Benoit, J., Hughes, M. C., Khan, F., et al. (2021) · Trials · doi:10.1186/s13063-021-05489-x · Cited by 280
“methods for feature aggregation across large temporal changes, such as EHR transitions [70, 74].”
MethodSelf-citationReproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273
A Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed · doi:10.48550/arxiv.1806.00388 · Cited by 248
“cific kind of EHR each uses. Even within the same hospital, transitions from one medical record system to another create non-obviousfeature mapping problems, which themselves warrant machine learning (Gong et al., 2017). This issue will remain until infrastructureto easily test across multiplesites and health systems b…”
Self-citationNestor, B. A., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 71
“While others have considered the automated mapping of clinical data elements with mapping tools (Gong et al., 2017) or learned vector space embeddings (Rajkomar et al.”
BackgroundInfluentialSelf-citationTemporal pointwise convolutional networks for length of stay prediction in the intensive care unit
Rocheteau, E., Liò, P., & Hyland, S. (2021) · Proceedings of the Conference on Health, Inference, and Learning · doi:10.1145/3450439.3451860 · Cited by 56
“, and there is significant positive skew in the labels (shown in Figure 1). To simplify the task, LoS prediction has most frequently appeared as a binary classification task, i.e. long vs. short stays2 [17, 35, 41], or less commonly as a regression task optimised with a mean squared error (MSE) loss function [40, 46]. H…”
BackgroundExpectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J., Whybrow, P. C., Achtyes, E., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 55
“Dataset shifts also occur after changing from one EHR system to another [104].”
BackgroundWhy do probabilistic clinical models fail to transport between sites
Lasko, T. A., Strobl, E. V., & Stead, W. W. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01037-4 · Cited by 64
“Similarly, if semantically equivalent data variables are encoded with different identi fi ers in a new dataset, then the performance of a model that relies on those identi fi ers will suffer 41,42 .”
BackgroundDynamic Prediction of ICU Mortality Risk Using Domain Adaptation
Alves, T., Laender, A., Veloso, A., & Ziviani, N. (2018) · 2018 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata.2018.8621927 · Cited by 45
“Authors in [30] discussed problems due to the lack of consistency in how semantically equivalent information is encoded in different ICU databases.”
BackgroundNestor, B. A., McDermott, M. B. A., Chauhan, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12583 · Cited by 26
“Clinically Aggregated representation: While prior work has sought to learn mappings between similar features[15, 16], we use simple aggregations of highly-present itemid s [6] to create “expert driven” condensed categories 2 .”
MethodResultSelf-citationLearning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU
Suresh, H., Gong, J. J., & Guttag, J. V. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219930 · Cited by 54
“note of "Do Not Resuscitate" (DNR) or "Comfort Measures Only" (CMO). This definition is in contrast to what has been used in prior work, where only mortality was considered (e.g., [15, 17, 18]). Notes of DNR or CMO indicate differences in what clinical interventions will be taken, and our proposed…”
BackgroundResultSelf-citationDeasy, J., Liò, P., & Ercole, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-79142-z · Cited by 52
“Attempts to incorporate a broader set of variables have used pretrained word embeddings from outside of the medical context and have avoided numerical values 22 .”
BackgroundMethodMultitask learning deep neural networks to combine revealed and stated preference data
Wang, S., Wang, Q., & Zhao, J. (2020) · Journal of Choice Modelling · doi:10.1016/j.jocm.2020.100236 · Cited by 45
“This is not a simple question since many different methods can apply [26].”
MethodZhu, R., Tu, X., & Huang, J. X. (2019) · Studies in big data · doi:10.1007/978-3-030-33966-1_13 · Cited by 32
Representation Learning for Electronic Health RecordsPreprint
Weng, W., & Szolovits, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.09248 · Cited by 27
“The bag-of-events method, which utilizes the biomedical concepts and relations in the ontology, is one of the approaches that can standardize the clinical concepts and mitigate such problems (Gong et al., 2017).”
BackgroundMethodSelf-citationKondylakis, H., Axenie, C., Bastola, D. K., Katehakis, D. G., Kouroubali, A., Kurz, D., Larburu, N., Macía, I., Maguire, R., Maramis, C., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/22034 · Cited by 25
“Learning patient disease trajectory for personalized diagnosis [81-84] • Use of discriminative models that exploit conditions on marker histories instead of jointly modeling them • Handling data and choosing specific latent variable models to summarize and extract information from the irregularly sampled and sparse da…”
BackgroundKhope, S. R., & Elias, S. (2023) · Healthcare · doi:10.3390/healthcare11050710 · Cited by 20
“Existing studies 10 of 25 Bao et al. [31] recently worked on a predictive scheme on MIMIC-III data using a capsule network, where the authors have emphasized the benefits of adopting a deep learning scheme. emphasize phenotyping analysis and predict the risk of mortality associated [36]).”
Machine Learning and Visualization in Clinical Decision Support: Current State and Future DirectionsPreprint
Levy-Fix, G., Kuperman, G. J., & Elhadad, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02664 · Cited by 19
Multiple Instance Learning for ECG Risk StratificationPreprint
Shanmugam, D., Blalock, D. W., & Guttag, J. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.00475 · Cited by 16
“, 2018), in-hospital mortality (Gong et al., 2017), and treatment resistance (Perlis, 2013).”
BackgroundSelf-citationFutoma, J., Simons, M. G., Doshi‐Velez, F., & Kamaleswaran, R. (2021) · Critical Care Explorations · doi:10.1097/cce.0000000000000453 · Cited by 15
Du, J., Zeng, D., Li, Z., Liu, J., Lv, M., Chen, L., Zhang, D., & Ji, S. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22697 · Cited by 10
“Traditional machine learning ‐ based methods first extract handcrafted features from the EHR data, and then adopt algorithms such as logistic regression, 11,12 support vector machine, 13,14 and random forest 15 – 17 to train the outcome prediction models.”
MethodDynamic prediction of life-threatening events for patients in intensive care unit
Hu, J., Kang, X., Xu, F.-F., Huang, K., Du, B., & Weng, L. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02026-x · Cited by 10
“applied a logistic regression approach to predict patient outcomes based on the features extracted from the free-text notes in the Electronic Health Records (EHR) [20]; Johnson et al.”
MethodPan, P., Wang, Y., Liu, C., Tu, Y., Cheng, H., Yang, Q., Xie, F., Li, Y., Xie, L., & Liu, Y. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00896-8 · Cited by 9
Nova, F. F., Pfafman, R., Kardys, K., Kerrigan, C., Guha, S., & Pater, J. A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555605 · Cited by 8
“While EHRs are primarily designed for improving healthcare efficiency from an operational standpoint, researchers have also found secondary use for these data in clinical informatics applications, such as prediction of certain health related activities [128], specific health conditions [20, 93], or clinical outcomes […”
BackgroundClinical Artificial Intelligence
McDermott, M. B. A., Nestor, B., & Szolovits, P. (2022) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2022.09.004 · Cited by 7
Self-citationGao, J., Chen, G., O’Rourke, A. P., Caskey, J., Carey, K. A., Oguss, M., Stey, A., Dligach, D., Miller, T., Mayampurath, A., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae071 · Cited by 5
Estimating Risk-Adjusted Hospital Performance
van Weenen, E., & Feuerriegel, S. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9378441 · Cited by 4
“Here literature must address typical challenges that arise in practice such as sparsity, irregularity, and noise [11], [24], [25].”
BackgroundMachine learning for predictive analytics
Kashyap, S., Corey, K., Kansal, A., & Sendak, M. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-820273-9.00003-8 · Cited by 4
Brosula, R. J., Corbin, C. K., & Chen, J. H. (2024) · PubMed Central · Cited by 3
Towards Scalable Structured Data from Clinical Text
Agrawal, M. (2023) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 3
Deep Ordinal Neural Network for Length of Stay Estimation in the Intensive Care Units
Cai, D., Song, M., Sun, C., Zhang, B., Hong, S., & Li, H. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557578 · Cited by 2
“Some research frames the LoS estimation task as a binary classification problem, predicting whether the LoS exceeds the given threshold [10, 11, 23, 25, 38].”
BackgroundData Science in Healthcare Systems: A Comprehensive Review and Future Directions
Olugboja, A. M. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-94953-1_15 · Cited by 2
Feature augmented stacked bagging for length of stay prediction on mimic III
Geethamani, D., & Rangaraj, R. (2022) · International Journal of Health Sciences · doi:10.53730/ijhs.v6ns2.7261 · Cited by 2
“The machine learning problem is converted to a binary classification problem by considering this threshold [27, 28].”
BackgroundTowards Metaclinical Medicine ― The Non-Human Physician as Our Destiny ―
Vardas, P. E., Xintarakou, A., Vardas, E. P., & Tzeis, S. E. (2024) · Circulation Journal · doi:10.1253/circj.cj-24-0760 · Cited by 2
Leveraging text representations for clinical predictive tasks
Naumann, T. (2018) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 2
Self-citationDeep Thompson Sampling for Length of Stay Prediction
Lacerda, A. M., & Pappa, G. L. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533667 · Cited by 1
Satisfying Model Validity for Robust Clinical Machine Learning for Time-Series
Nestor, B. (2023) · TSpace (University of Toronto) · Cited by 1
Improving clinical decisions using correspondences within and across electronic health records
Gong, J. J. (2018) · DSpace@MIT (Massachusetts Institute of Technology)
Self-citationLuo, M., & Yang, X. (2022) · Neural Computing and Applications · doi:10.1007/s00521-021-06869-9
“Methods have also been proposed to overcome the problem of dif-ficult data integration between systems [9].”
BackgroundHeterogenous Patient Populations in the ICUPreprint
Suresh, H., Gong, J. J., & Guttag, J. V. (2018) · DSpace@MIT (Massachusetts Institute of Technology)
Self-citation
Metadata
- Authors
- Gong, J. J., Naumann, T., Szolovits, P., & Guttag, J. V.
- Affiliations
- Massachusetts Institute of Technology (US)
- Massachusetts Institute of Technology, Cambridge, MA, USA
- Venue
- Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2017-08-13
- Pages
- 1497-1505
- OpenAlex
- W2742792846
- Cited by
- 34
- References
- 27
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 1 · 2022: 7 · 2021: 5 · 2020: 5 · 2019: 5 · 2018: 6
- FWCI
- 2.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Human Genome Research Institute (U54-HG007963)
- National Science Foundation (Graduate Research Fellowship Grant No. 1122374)
- National Institute of Biomedical Imaging and Bioengineering (R01-EB017205)
- Intel Corporation (Intel Science and Technology Center for Big Data)
- U.S. National Library of Medicine (2T15LM007092-22)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- electronic health records, semantic interoperability, intensive care unit, in-hospital mortality, prolonged hospital stay, clinical outcome prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.