Predicting Clinical Outcomes Across Changing Electronic Health Record Systems
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Existing machine learning methods typically assume consistency in how semantically equivalent information is encoded. However, the way information is recorded in databases differs across institutions and over time, often rendering potentially useful data obsolescent. To address this problem, we map database-specific representations of information to a shared set of semantic concepts, thus allowing models to be built from or transition across different databases. We demonstrate our method on machine learning models developed in a healthcare setting. In particular, we evaluate our method using two different intensive care unit (ICU) databases and on two clinically relevant tasks, in-hospital mortality and prolonged length of stay. For both outcomes, a feature representation mapping EHR-specific events to a shared set of clinical concepts yields better results than using EHR-specific events alone.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, it is highly unlikely that a mortality prediction system at a hospital in California will work in the same way when implemented half-way around the world, particularly due to site-specific variations32,33.”
From Machine intelligence in healthcare—perspectives on trustworthiness, explainability, usability, and transparency (2020) · cites this paper as [32]
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 6×
“(2) We demonstrate the effectiveness of our approach on two commonly used predictive models and on data from two epochs of EHR systems in the publicly available MIMIC-III dataset.”
MethodInfluentialSelf-citationLe Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773 · cited 6×
“This provides a fair comparison against baseline acuity scores, which commonly use only information from this time period [16].”
BackgroundMethodInfluentialFreebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852 · cited 1×
“Although general domain ontologies (e.g., [3]) and tools for identifying equivalent semantic concepts (e.g., [10]) exist, these tools do not work well with the highly domain-specific vocabulary present in clinical text.”
BackgroundThe unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · cited 11×
“Instead, we map each element directly to concepts in the UMLS ontologies and use this representation for greater generalizability of predictive models.”
MethodInfluentialIncorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008 · cited 1×
“, [10]) exist, these tools do not work well with the highly domain-specific vocabulary present in clinical text.”
BackgroundSavova, G. K., Masanz, J. J., Ogren, P. V., Zheng, J., Sohn, S., Kipper-Schuler, K. C., & Chute, C. G. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.001560 · Cited by 1,967 · cited 1×
“Using the Clinical Text Analysis Knowledge Extraction System (cTAKES), a frequently used tool for identifying UMLS concepts, we annotate the human-readable item descriptions from both EHR versions in our data [23].”
MethodTransfer learning via dimensionality reduction
Pan, S. J., Kwok, J. T.-Y., & Yang, Q. (2008) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 653
Dong, X. L., & Srivastava, D. (2015) · Synthesis Lectures on Data Management · doi:10.1007/978-3-031-01853-4 · Cited by 539
Data integration in the era of omics: current and future challenges
Gomez-Cabrero, D., Abugessaisa, I., Maier, D., Teschendorff, A., Merkenschlager, M., Gisel, A., Ballestar, E., Bongcam-Rudloff, E., Conesa, A., & Tegnér, J. (2014) · BMC Systems Biology · doi:10.1186/1752-0509-8-s2-i1 · Cited by 413 · cited 1×
“This approach enables building models that can leverage data from another database without re-stricting the data to a subset or requiring database integration, a difficult problem [7, 13].”
MethodLIBLINEAR: A Library for Large Linear Classification
FanRong-En, ChangKai-Wei, HsiehCho-Jui, WangXiang-Rui, & LinChih-Jen (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1442794 · Cited by 328 · cited 1×
“We used the scikit-learn LIBLINEAR implementation to train and test all models [9, 21].”
MethodThe Society of Thoracic Surgeons, 1984-1985
(1984) · The Annals of Thoracic Surgery · doi:10.1016/s0003-4975(10)60342-6 · Cited by 327
Portability of an algorithm to identify rheumatoid arthritis in electronic health records
Carroll, R. J., Thompson, W. K., Eyler, A. E., Mandelin, A. M., Cai, T., Zink, R. M., Pacheco, J. A., Boomershine, C. S., Lasko, T. A., Xu, H., et al. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000583 · Cited by 235
Unfolding physiological state: mortality modelling in intensive care units
Ghassemi, M., Naumann, T., Doshi-Velez, F., Brimmer, N., Joshi, R., Rumshisky, A., & Szolovits, P. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623742 · Cited by 201 · cited 2×
“While using the values of lab tests or vital signs would certainly lead to improved predictive performance [11, 16, 17], it would obscure information about how the encodings affect model performance.”
MethodSelf-citationGhassemi, M., Pimentel, M., Naumann, T., Brennan, T., Clifton, D., Szolovits, P., & Feng, M. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9209 · Cited by 192 · cited 1×
“We show the performance against the Simplified Acute Physiology Score II (SAPS-II) [16], a well-established acuity score that is commonly used as a baseline when developing risk models for mortality in the ICU [11, 12, 14] and also uses information from the first 24 hours in the the ICU.”
MethodSelf-citationPersonalized Mortality Prediction Driven by Electronic Medical Data and a Patient Similarity Metric
Lee, J., Maslove, D. M., & Dubin, J. A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0127428 · Cited by 159 · cited 1×
“While using the values of lab tests or vital signs would certainly lead to improved predictive performance [11, 16, 17], it would obscure information about how the encodings affect model performance.”
MethodWiens, J., Guttag, J., & Horvitz, E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002162 · Cited by 123 · cited 2×
“For example, [29] transferred information from other hospitals in the same hospital network to improve risk predictions for a hospital-acquired infection at the hospital of interest.”
BackgroundSelf-citationOntology-Based Data Integration between Clinical and Research Systems
Mate, S., Köpcke, F., Toddenroth, D., Martin, M., Prokosch, H.-U., Bürkle, T., & Ganslandt, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0116656 · Cited by 86 · cited 1×
“Much previous work has developed methods to reconcile health care information with different encodings of variable names by mapping databases to existing clinical vocabularies and ontologies [18, 22, 25].”
MethodReich, C., Ryan, P. B., Stang, P., & Rocca, M. (2012) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2012.05.002 · Cited by 74 · cited 1×
“Much previous work has developed methods to reconcile health care information with different encodings of variable names by mapping databases to existing clinical vocabularies and ontologies [18, 22, 25].”
MethodICU acuity: real-time models versus daily models.
Hug, C. W., & Szolovits, P. (2009) · Europe PMC (PubMed Central) · Cited by 46
Self-citationPractical experience with the maintenance and auditing of a large medical ontology
Baorto, D. M., Li, L., & Cimino, J. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.03.005 · Cited by 41 · cited 1×
“Remapping variables manually is feasible for small changes, but modern EHRs may contain over 100,000 distinct items, and this number continues to grow over time [1, 8].”
BackgroundSophia: A Expedient UMLS Concept Extraction Annotator.
Divita, G., Zeng, Q. T., Gundlapalli, A. V., DuVall, S. L., Nebeker, J. R., & Samore, M. H. (2014) · PubMed · Cited by 39
Methods for automated concept mapping between medical databases
Sun, Y. (2004) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2004.03.003 · Cited by 33 · cited 2×
“Much previous work has developed methods to reconcile health care information with different encodings of variable names by mapping databases to existing clinical vocabularies and ontologies [18, 22, 25].”
MethodAutomated UMLS-Based Comparison of Medical Forms
Dugas, M., Fritz, F., Krumm, R., & Breil, B. (2013) · PLoS ONE · doi:10.1371/journal.pone.0067883 · Cited by 25 · cited 1×
“Remapping variables manually is feasible for small changes, but modern EHRs may contain over 100,000 distinct items, and this number continues to grow over time [1, 8].”
BackgroundChia, C.-C., & Syed, Z. (2011) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.63 · Cited by 15 · cited 1×
“This type of BOE representation has been used previously to construct clinical risk models from structured data [5, 26, 27].”
MethodCardiovascular risk stratification with heart rate topics
Esbroeck, A. V., & Syed, Z. (2012) · Computing in cardiology · Cited by 4
Cited by
Cutillo, C. M., Sharma, K. R., Foschini, L., Kundu, S., Mackintosh, M., Mandl, K. D., MI in Healthcare Workshop Working Group, Beck, T., Collier, E., Colvis, C., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0254-2 · Cited by 337
“For example, it is highly unlikely that a mortality prediction system at a hospital in California will work in the same way when implemented half-way around the world, particularly due to site-speci fi c variations 32,33 .”
BackgroundThe role of machine learning in clinical research: transforming the future of evidence generation
Weissler, E. H., Naumann, T., Andersson, T., Ranganath, R., Elemento, O., Luo, Y., Freitag, D. F., Benoit, J., Hughes, M. C., Khan, F., et al. (2021) · Trials · doi:10.1186/s13063-021-05489-x · Cited by 280
“methods for feature aggregation across large temporal changes, such as EHR transitions [70, 74].”
MethodSelf-citationReproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273
A Review of Challenges and Opportunities in Machine Learning for Health
Ghassemi, M., Naumann, T., Schulam, P., Beam, A. L., Chen, I. Y., & Ranganath, R. (2018) · PubMed · doi:10.48550/arxiv.1806.00388 · Cited by 248
“cific kind of EHR each uses. Even within the same hospital, transitions from one medical record system to another create non-obviousfeature mapping problems, which themselves warrant machine learning (Gong et al., 2017). This issue will remain until infrastructureto easily test across multiplesites and health systems b…”
Self-citationNestor, B. A., McDermott, M. B. A., Boag, W., Berner, G., Naumann, T., Hughes, M. C., Goldenberg, A., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.00690 · Cited by 71
“While others have considered the automated mapping of clinical data elements with mapping tools (Gong et al., 2017) or learned vector space embeddings (Rajkomar et al.”
BackgroundInfluentialSelf-citation
Metadata
- Authors
- Gong, J. J., Naumann, T., Szolovits, P., & Guttag, J. V.
- Affiliations
- Massachusetts Institute of Technology (US)
- Massachusetts Institute of Technology, Cambridge, MA, USA
- Venue
- Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
- Publisher
- ACM
- Published
- 2017-08-13
- Pages
- 1497-1505
- OpenAlex
- W2742792846
- Cited by
- 34
- References
- 27
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 1 · 2022: 7 · 2021: 5 · 2020: 5 · 2019: 5 · 2018: 6
- FWCI
- 2.00
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Human Genome Research Institute (U54-HG007963)
- National Science Foundation (Graduate Research Fellowship Grant No. 1122374)
- National Institute of Biomedical Imaging and Bioengineering (R01-EB017205)
- Intel Corporation (Intel Science and Technology Center for Big Data)
- U.S. National Library of Medicine (2T15LM007092-22)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- electronic health records, semantic interoperability, intensive care unit, in-hospital mortality, prolonged hospital stay, clinical outcome prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.