CoChat
Sign Up
Cited by 235Machine Learning in HealthcareElectronic Health Records SystemsArtificial Intelligence in Healthcare

Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review

Open access
Si, Yuqi, Du, Jingcheng, Li, Zhao, Jiang, Xiaoqian, Miller, Timothy, Wang, Fei, Jim Zheng, W., Roberts, Kirk·2021·Journal of Biomedical Informatics
DOI 10.1016/j.jbi.2020.103671PMC PMC11290708
Open this paper in CoChat — free accountOpen PDF

Abstract

Objectives Patient representation learning refers to learning a dense mathematical representation of a patient that encodes meaningful information from Electronic Health Records (EHRs). This is generally performed using advanced deep learning methods. This study presents a systematic review of this field and provides both qualitative and quantitative analyses from a methodological perspective.Methods We identified studies developing patient representations from EHRs with deep learning methods from MEDLINE, EMBASE, Scopus, the Association for Computing Machinery (ACM) Digital Library, and the Institute of Electrical and Electronics Engineers (IEEE) Xplore Digital Library. After screening 363 articles, 49 papers were included for a comprehensive data collection.Results Publications developing patient representations almost doubled each year from 2015 until 2019. We noticed a typical workflow starting with feeding raw data, applying deep learning models, and ending with clinical outcome predictions as evaluations of the learned representations. Specifically, learning representations from structured EHR data was dominant (37 out of 49 studies). Recurrent Neural Networks were widely applied as the deep learning architecture (Long short-term memory: 13 studies, Gated recurrent unit: 11 studies). Learning was mainly performed in a supervised manner (30 studies) optimized with cross-entropy loss. Disease prediction was the most common application and evaluation (31 studies). Benchmark datasets were mostly unavailable (28 studies) due to privacy concerns of EHR data, and code availability was assured in 20 studies.Discussion & conclusion The existing predictive models mainly focus on the prediction of single diseases, rather than considering the complex mechanisms of patients from a holistic review. We show the importance and feasibility of learning comprehensive representations of patient EHR data through a systematic review. Advances in patient representation learning techniques will be essential for powering patient-level EHR analyses. Future work will still be devoted to leveraging the richness and potential of available EHR data. Reproducibility and transparency of reported results will hopefully improve. Knowledge distillation and advanced learning techniques will be exploited to assist the capability of learning patient representation further.

AI snapshot

Written from the paper’s full text.

TL;DR

This systematic review synthesizes 49 deep-learning studies that learn patient representations from EHRs, finding rapid publication growth and a typical pipeline from raw data to deep models to clinical prediction evaluation. Structured codes, recurrent neural networks, supervised learning, and disease prediction dominate, while limited public benchmarks, code sharing, and reproducibility constrain the field.

Key findings

  • From 363 screened records, 49 papers met inclusion criteria; publications developing patient representations almost doubled each year from 2015 until 2019, with almost 3 times growth in 2017 over 2016.
  • Structured EHR data was dominant (37/49 studies); LSTM was the most common architecture (13 studies), followed by CNN (11) and GRU (11, all structured data); supervised learning was used in 30 studies, and disease prediction was the most common application/evaluation (31 studies).
  • MIMIC-III was the most prominent dataset (22/49 studies), while private local datasets were used in 27/49; benchmark datasets were mostly unavailable (28 studies) due to privacy concerns, and code availability was assured in 20 studies.
  • Attention mechanisms were adopted in 13 studies, hierarchical knowledge graphs in 6 studies, sequential order was considered in 27 studies, and t-SNE was used for visualization in 8 studies.

Methods

  • Systematic review with searches of MEDLINE/PubMed, EMBASE, Scopus, ACM Digital Library, and IEEE Xplore Digital Library, queried on December 31, 2019, using keywords combining deep learning/neural network, patient, representation/embedding/vector, and EHR/EMR terms.
  • Inclusion required focus on patient representation learning, longitudinal EHR data, deep learning or neural network models, and downstream clinical prediction evaluation; exclusions included reviews, non-EHR data, no deep learning, no evaluation, and preprints/posters.
  • Screening began with 363 records (349 from databases, 14 from snowballing), removed 54 duplicates, excluded records at title/abstract and full-text stages, and included 49 papers; data extraction covered variables such as publication date, venue, learning approach, data type, resources, preprocessing, clinical tasks, metrics, interpretability, objective functions, computational resources, and code reproducibility.

Limitations

  • The paper states incomplete retrieval is possible because no MeSH term covers patient representation, requiring librarian collaboration and snowballing; it also notes difficulty distinguishing included studies from end-to-end clinical prediction works, exclusion of preprints, and potential selective bias in publication trends.
  • The paper states the review fails to answer all technical questions because of lack of transparency and reproducibility; many studies claim state-of-the-art results but few can be verified externally, and it remains unclear which approach works best for representing EHR data.
  • The text makes evident that the review period ends in December 2019, with only limited post-2020 works highlighted, and that reporting is uneven—for example, only a few papers reported loss functions, and 29/49 reported computational resources.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.jbi.2020.103671

Cites

153 of 153 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 129 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Unsupervised learning was applied in 11 of the 49 included studies.

    “Unsupervised learning was applied in 11 studies.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Benchmark datasets were mostly unavailable among the 49 included studies (28 studies).

    “Benchmark datasets were mostly unavailable (28 studies) due to privacy concerns of EHR data, and code availability was assured in 20 studies.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Publications developing patient representations almost doubled each year from 2015 until 2019.

    “Publications developing patient representations almost doubled each year from 2015 until 2019.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Disease prediction was the most common application and evaluation among the 49 included studies (31 studies).

    “Disease prediction was the most common application and evaluation (31 studies).”
    Quote in the paperNot a substantial claim
  • Findingown data

    Code availability was assured in 20 of the 49 included studies.

    “Benchmark datasets were mostly unavailable (28 studies) due to privacy concerns of EHR data, and code availability was assured in 20 studies.”
    Quote in the paperNot a substantial claim
  • The quote only partly states this claim. 3
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 63

  • The quote only partly states this claim.

    Findingown data

    Among the 49 included studies, learning was optimized with cross-entropy loss.

    “Learning was mainly performed in a supervised manner (30 studies) optimized with cross-entropy loss.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Studies combining structured codes and unstructured notes did not fully combine raw data from the two modalities into the same model; they only used a subset of data from one modality.

    “Though combining heterogeneous resources is promising, we noticed these studies were not really combining the raw data fully from the two modalities into the same model. Instead, they only used a subset of data from one modality [6,11,54–57].”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [6] Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, Liu PJ, Liu X, Marcus J, Sun M, Sundberg P, Yee H, Zhang K, Zhang Y, Flores G, Duggan GE, Irvine J, Le Q, Litsch K, Mossin A, Tansuwan J, Wang D, Wexler J, Wilson J, Ludwig D, Volchenboum SL, Chou K, Pearson M, Madabushi S, Shah NH, Butte AJ, Howell MD, Cui C, Corrado GS, Dean J, Scalable and accurate deep learning with electronic health records, Npj Digit. Med 1 (2018), 10.1038/s41746-018-0029-1. PMC6550175 31304302 source

      Not checked in the free run.

    • cites [11] Miotto R, Li L, Kidd BA, Dudley JT, Deep patient: An unsupervised representation to predict the future of patients from the electronic health records, Sci. Rep 6 (2016) 26094, 10.1038/srep26094. 27185194 PMC4869115

      Not checked in the free run.

    • cites [54] Zhang E, Robinson R, Pfahringer B, Deep Holistic Representation Learning from EHR, in: 2018. 10.1109/ISMICT.2018.8573698.

      Not checked in the free run.

    • cites [55] Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M, Clinical intervention prediction and understanding with deep neural networks, Mach. Learn. Healthc. Conf (2017) 322– 337.

      Not checked in the free run.

    • cites [56] Bai T, Ch AK, Egleston BL, Vucetic S, EHR phenotyping via jointly embedding medical concepts and words into a unified vector space, BMC Med. Inf. Decis. Mak 18 (2018) 123. http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L625525781. PMC6290514 30537974

      Not checked in the free run.

    • cites [57] Kemp J, Rajkomar A, Dai AM, Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes, ArXiv: 1909.03039 [Cs, Stat]. (2019). http://arxiv.org/abs/1909.03039 (accessed November 22, 2019).

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    The self-supervised tasks included in this review are mainly used for clinical code representation learning, where clinical codes are ordered sequentially and masked codes are predicted using nearby context.

    “The self-supervised tasks included in this review are mainly used for clinical code representation learning, where clinical codes are ordered sequentially, and the codes in between are masked and predicted using the context of clinical codes nearby, similar to the algorithms of skip-gram and CBOW used to learn word embeddings [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Mikolov T, Chen K, Corrado G, Dean J, Efficient estimation of word representations in vector space, ArXiv Preprint ArXiv:1301.3781 (2013).

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Among the 49 included studies, 13 applied LSTM and 11 applied GRU as deep learning architectures.

    “Recurrent Neural Networks were widely applied as the deep learning architecture (Long short-term memory: 13 studies, Gated recurrent unit: 11 studies).”
    • Also stated in: § Study characteristics · plain
      “In total, there are 13 studies (structured: 8; unstructured: 1; combined: 4) that have applied an LSTM model to develop patient representations, which is the most common architecture.”
    • Also stated in: § Study characteristics · plain
      “GRU is also common with 11 studies (all structured EHR data).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among the 49 included studies, 37 used structured codes, 6 used unstructured notes, and 6 used both structured codes and unstructured notes jointly.

    “In terms of patient data types in EHRs, among the 49 studies, 37 papers used structured codes (i.e., diagnosis codes, procedure codes, medication codes) to build the representation, while 6 papers used unstructured notes. The remaining 6 papers used both structured codes and unstructured notes jointly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The volume of studies on patient representation learning grew rapidly each year, almost tripling in 2017 over 2016.

    “The volume of studies is currently growing rapidly each year (almost 3 times in 2017 over 2016).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Self-supervised methods were used by 8 of the 49 included studies.

    “Self-supervised methods were used by 8 studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The search identified 363 papers at the beginning, including 349 from five databases and 14 additional records through snowballing.

    “As a result, 363 papers were queried at the beginning, including 349 articles retrieved from the five databases, and 14 additional records identified through snowballing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    For inputs from multiple resources, input vectors are concatenated before feeding them into the architecture.

    “For inputs from multiple resources, input vectors are concatenated before feeding them into the architecture [14].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14] Xu Y, Biswal S, Deshpande SR, Maher KO, Sun J, RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, New York, NY, USA, 2018, pp. 2565– 2573. 10.1145/3219819.3220051.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Z-score is applied to convert data from continuous variables to discrete one-hot vectors.

    “For instance, z-score is applied to convert data from continuous variables to discrete one-hot vectors [55,57,61].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [55] Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M, Clinical intervention prediction and understanding with deep neural networks, Mach. Learn. Healthc. Conf (2017) 322– 337.

      Not checked in the free run.

    • cites [57] Kemp J, Rajkomar A, Dai AM, Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes, ArXiv: 1909.03039 [Cs, Stat]. (2019). http://arxiv.org/abs/1909.03039 (accessed November 22, 2019).

      Not checked in the free run.

    • cites [61] Suresh H, Gong JJ, Guttag J, Learning tasks for multitask learning: heterogenous patient populations in the ICU, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining - KDD ‘18. (2018) 802– 810. 10.1145/3219819.3219930.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    There was a slight decrease in publication volume from 2018 to 2019.

    “Although there is a slight decrease from 2018 to 2019, we think this is because we exclude preprints at the first step of screening.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    CNNs can be combined with RNNs to learn multifaceted patient representations.

    “CNNs can also be combined with RNNs to learn multifaceted patient representations [14,55,72].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [14] Xu Y, Biswal S, Deshpande SR, Maher KO, Sun J, RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, New York, NY, USA, 2018, pp. 2565– 2573. 10.1145/3219819.3220051.

      Not checked in the free run.

    • cites [55] Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M, Clinical intervention prediction and understanding with deep neural networks, Mach. Learn. Healthc. Conf (2017) 322– 337.

      Not checked in the free run.

    • cites [72] Ma T, Xiao C, Wang F, Health-ATM: A deep architecture for multifaceted patient health record representation and risk prediction, in: 2018, pp. 261– 269. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85045143398&partnerID=40&md5=b63d90606c942e23cb2d49ae4fed27fd.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    One method to simplify clinical notes is to use topic models to represent unstructured resources.

    “One such method is to simplify clinical notes by using topic models to represent unstructured resources [6,11,55].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [6] Rajkomar A, Oren E, Chen K, Dai AM, Hajaj N, Hardt M, Liu PJ, Liu X, Marcus J, Sun M, Sundberg P, Yee H, Zhang K, Zhang Y, Flores G, Duggan GE, Irvine J, Le Q, Litsch K, Mossin A, Tansuwan J, Wang D, Wexler J, Wilson J, Ludwig D, Volchenboum SL, Chou K, Pearson M, Madabushi S, Shah NH, Butte AJ, Howell MD, Cui C, Corrado GS, Dean J, Scalable and accurate deep learning with electronic health records, Npj Digit. Med 1 (2018), 10.1038/s41746-018-0029-1. PMC6550175 31304302 source

      Not checked in the free run.

    • cites [11] Miotto R, Li L, Kidd BA, Dudley JT, Deep patient: An unsupervised representation to predict the future of patients from the electronic health records, Sci. Rep 6 (2016) 26094, 10.1038/srep26094. 27185194 PMC4869115

      Not checked in the free run.

    • cites [55] Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M, Clinical intervention prediction and understanding with deep neural networks, Mach. Learn. Healthc. Conf (2017) 322– 337.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Top_K_Recall or Top_K_Precision are adopted particularly for predicting full sets of diagnoses where K is the number of diseases.

    “Metrics including Top_K_Recall or Top_K_Precision (ranking metrics from information retrieval), are adopted particularly for predicting full sets of diagnoses where K is the number of diseases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The majority of works applied two or more clinical outcomes to evaluate the representations, resulting in sizable overlap in outcomes studied.

    “To show the generalizability of the study, the majority of works have applied two or more clinical outcomes to evaluate the representations, resulting in a number of studies that have sizable overlap in terms of outcomes studied.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    The remaining 22 studies did not consider sequential information, using varied ways of preprocessing patient data.

    “The remaining 22 studies have not considered sequential information, based on varied ways of preprocessing patient data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The 49 papers were categorized into seven main clinical tasks with counts: disease prediction 31, mortality prediction 12, length-of-stay forecasting 8, admission prediction 7, patient subtyping 4, intervention prediction 5, and medical cost forecasting 3.

    “We categorize main clinical tasks for each paper into seven types: disease prediction (n = 31), mortality prediction (12), length-of-stay forecasting (8), admission prediction (7), patient subtyping (4), intervention prediction (5), and medical cost forecasting (3).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For prediction tasks, AUROC, AUPRC, Precision, Recall, F1-measure, and Accuracy are the most used metrics.

    “Typically, for prediction tasks, AUROC, AUPRC, Precision, Recall, F1-measure, and Accuracy are the most used.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Evaluation metrics are mainly dependent on the prediction task.

    “Evaluation metrics are mainly dependent on the prediction task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    For regression tasks predicting a real-valued quantity (e.g., medical cost, readmission forecasting), RMSE, MAPE, and Goodness-of-Fit R-squared are utilized.

    “In addition, for regression tasks that involve predicting a real-valued quantity (i.e., medical cost, readmission forecasting), Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE) and Goodness-of-Fit R-squared are utilized.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Other publicly available datasets used to develop patient representations include PPMI, ADNI, the i2b2 obesity challenge, and eICU Collaborative Database.

    “Other publicly available datasets that have been used to develop patient representations are: the Parkinson Progression Marker Initiative (PPMI) [67,74,94], the Alzheimer’s disease neuroimaging initiative (ADNI) [74,95], the i2b2 obesity challenge [62,63,90,96], and eICU Collaborative Database [42,97].”
    Quote in the paperNot a substantial claim0 of 8 sources checked
    • cites [67] Baytas IM, Xiao C, Zhang X, Wang F, Jain AK, Zhou J, Patient subtyping via time-aware LSTM networks, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2017, pp. 65– 74.

      Not checked in the free run.

    • cites [74] Zhang Y, Zhou H, Li J, Sun W, Chen Y, A Time-Sensitive Hybrid Learning Model for Patient Subgrouping, in: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, de Janeiro Rio, 2018, pp. 1– 8. 10.1109/IJCNN.2018.8488991.

      Not checked in the free run.

    • cites [94] Marek K, Jennings D, Lasch S, Siderowf A, Tanner C, Simuni T, Coffey C, Kieburtz K, Flagg E, Chowdhury S, The Parkinson progression marker initiative (PPMI), Prog. Neurobiol 95 (2011) 629– 635. 21930184 PMC9014725

      Not checked in the free run.

    • cites [95] Mueller SG, Weiner MW, Thal LJ, Petersen RC, Jack C, Jagust W, Trojanowski JQ, Toga AW, Beckett L, The Alzheimer’s disease neuroimaging initiative, Neuroimag. Clin 15 (2005) 869– 877. PMC2376747 16443497 source

      Not checked in the free run.

    • cites [62] Dligach D, Miller T, Learning Patient Representations from Text, in: Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, 2018, pp. 119– 123.

      Not checked in the free run.

    • cites [63] Dligach D, Afshar M, Miller T, Toward a clinical text encoder: pretraining for clinical natural language processing with applications to substance misuse, J. Am. Med. Inform. Assoc (2019) ocz072, 10.1093/jamia/ocz072. PMC6798566 31233140

      Not checked in the free run.

    • cites [90] Liu D, Dligach D, Miller T, Two-stage Federated Phenotyping and Patient Representation Learning, ArXiv: 1908.05596 [Cs]. (2019). http://arxiv.org/abs/1908.05596 (accessed September 20, 2019). PMC8072229 33907350

      Not checked in the free run.

    • cites [96] Uzuner Ö, Recognizing obesity and comorbidities in sparse data, J. Am. Med. Inform. Assoc 16 (2009) 561– 570. 19390096 PMC2705260

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Month-level, year-level, and visit-level predictions are prevalent for long-term prediction tasks.

    “Month-level [69,88], year-level [11,48,66,78,80], and visit-level predictions [31,38,56,59,64,70,71,76] are prevalent for those tasks.”
    Quote in the paperNot a substantial claim0 of 10 sources checked
    • cites [69] Liu J, Zhang Z, Razavian N, Deep EHR: Chronic Disease Prediction Using Medical Notes, ArXiv: 1808.04928 [Cs, Stat]. (2018). http://arxiv.org/abs/1808.04928 (accessed April 9, 2019).

      Not checked in the free run.

    • cites [88] Barbieri S, Kemp J, Perez-Concha O, Kotwal S, Gallagher M, Ritchie A, Jorm L, Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk, Sci Rep. 10 (2020) 1111, 10.1038/s41598-020-58053-z. 31980704 PMC6981230 source

      Not checked in the free run.

    • cites [11] Miotto R, Li L, Kidd BA, Dudley JT, Deep patient: An unsupervised representation to predict the future of patients from the electronic health records, Sci. Rep 6 (2016) 26094, 10.1038/srep26094. 27185194 PMC4869115

      Not checked in the free run.

    • cites [48] Yang K, Li X, Liu H, Mei J, Xie G, Zhao J, Xie B, Wang F, TaGiTeD: Predictive task guided tensor decomposition for representation learning from electronic health records, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI Press, 2017, pp. 2824– 2830.

      Not checked in the free run.

    • cites [66] Zhang XS, Tang F, Dodge HH, Zhou J, Wang F, MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, New York, NY, USA, 2019, pp. 2487– 2495. 10.1145/3292500.3330779. PMC8046258 33859865

      Not checked in the free run.

    • cites [78] Zhang J, Kowsari K, Harrison JH, Lobo JM, Barnes LE, Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record, IEEE Access 6 (2018) 65333– 65346, 10.1109/ACCESS.2018.2875677.

      Not checked in the free run.

    • cites [80] Choi E, Bahadori MT, Sun J, Kulas J, Schuetz A, Stewart W, RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism, Adv. Neural Inf. Process. Syst (2016) 3504– 3512.

      Not checked in the free run.

    • cites [31] Ma F, You Q, Xiao H, Chitta R, Zhou J, Gao J, KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ACM, New York, NY, USA, 2018, pp. 743– 752. 10.1145/3269206.3271701. source

      Not checked in the free run.

    • cites [38] Choi E, Bahadori MT, Schuetz A, Stewart WF, Sun J, Doctor AI: Predicting Clinical Events via Recurrent Neural Networks, ArXiv: 1511.05942 [Cs]. (2015). http://arxiv.org/abs/1511.05942 (accessed April 10, 2019). PMC5341604 28286600

      Not checked in the free run.

    • cites [56] Bai T, Ch AK, Egleston BL, Vucetic S, EHR phenotyping via jointly embedding medical concepts and words into a unified vector space, BMC Med. Inf. Decis. Mak 18 (2018) 123. http://www.embase.com/search/results?subaction=viewrecord&from=export&id=L625525781. PMC6290514 30537974

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    RNNs can assist autoencoders to learn unsupervised representations while improving their ability to capture sequential clinical events.

    “RNNs can assist with autoencoders to learn unsupervised representations while improving their ability to capture sequential clinical events [61,67,74,75].”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [61] Suresh H, Gong JJ, Guttag J, Learning tasks for multitask learning: heterogenous patient populations in the ICU, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining - KDD ‘18. (2018) 802– 810. 10.1145/3219819.3219930.

      Not checked in the free run.

    • cites [67] Baytas IM, Xiao C, Zhang X, Wang F, Jain AK, Zhou J, Patient subtyping via time-aware LSTM networks, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2017, pp. 65– 74.

      Not checked in the free run.

    • cites [74] Zhang Y, Zhou H, Li J, Sun W, Chen Y, A Time-Sensitive Hybrid Learning Model for Patient Subgrouping, in: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, de Janeiro Rio, 2018, pp. 1– 8. 10.1109/IJCNN.2018.8488991.

      Not checked in the free run.

    • cites [75] Lei L, Zhou Y, Zhai J, Zhang L, Fang Z, He P, Gao J, An Effective Patient Representation Learning for Time-series Prediction Tasks Based on EHRs, in: 2019, pp. 885– 892. 10.1109/BIBM.2018.8621542.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Patient representations may be relatively low-dimensional (often less than 500 dimensions).

    “While patient representations may be relatively low-dimensional (often less than 500 dimensions)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    A weighted loss function was applied to account for an imbalanced-class dataset.

    “Aside from traditional cross-entropy loss, a weighted loss function was applied to account for an imbalanced-class dataset [55].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Suresh H, Hunt N, Johnson A, Celi LA, Szolovits P, Ghassemi M, Clinical intervention prediction and understanding with deep neural networks, Mach. Learn. Healthc. Conf (2017) 322– 337.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Visualization techniques are often performed to construct qualitative clusters of patient cohorts to provide an intuitive interpretation of the learned representations.

    “Visualization techniques are often performed to construct qualitative clusters of patient cohorts to provide an intuitive interpretation of the learned representations [102].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [102] Wang F, Kaushal R, Khullar D, Should Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine? Ann. Intern. Med 172 (2020) 59, 10.7326/M19-2548. 31842204 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    24 hours are common as these tasks are immediate and physicians make decisions instantly.

    “Specifically, 24 hours are common as these tasks are immediate and physicians make decisions instantly.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Recurrent Neural Networks were the most common deep learning model.

    “Recurrent Neural Networks were the most common deep learning model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    Among the four visualization methods (t-SNE, MDS, PCA, UMAP), t-SNE has developed into the standard tool in this field due to its capability of revealing clusters in data.

    “Among these four methods, t-SNE has developed into the standard tool in this field due to its capability of revealing clusters in data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The reported deep learning frameworks include Theano (n = 12), TensorFlow (7), PyTorch (6), and Keras (4).

    “The reported deep learning frameworks include Theano (n = 12), TensorFlow (7), PyTorch (6), and Keras (4).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    Providing workflow and computer code associated with publications is becoming increasingly common to enable the reproducibility of the method, if not the full work.

    “Thus, providing workflow and computer code associated with publications is becoming increasingly common to enable the reproducibility of the method, if not the full work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The resources of available EHR data are quite limited due to privacy concerns.

    “We investigated that the resources of available EHR data are quite limited due to privacy concerns.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    Apart from projecting representations, interpretability can be provided by visualization methods including a heatmap of features, visualizing attention weights, and case study.

    “Apart from projecting representations, interpretability can be provided by visualization methods including a heatmap of features [33,67,78], visualizing attention weights [57,72,76,80,88], and case study (explanation by examples) [32].”
    Quote in the paperNot a substantial claim0 of 9 sources checked
    • cites [33] Zhang J, Gong J, Barnes L, HCNN: Heterogeneous Convolutional Neural Networks for Comorbid Risk Prediction with Electronic Health Records, in: Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies, IEEE Press, Piscataway, NJ, USA, 2017, pp. 214– 221. 10.1109/CHASE.2017.80.

      Not checked in the free run.

    • cites [67] Baytas IM, Xiao C, Zhang X, Wang F, Jain AK, Zhou J, Patient subtyping via time-aware LSTM networks, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2017, pp. 65– 74.

      Not checked in the free run.

    • cites [78] Zhang J, Kowsari K, Harrison JH, Lobo JM, Barnes LE, Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record, IEEE Access 6 (2018) 65333– 65346, 10.1109/ACCESS.2018.2875677.

      Not checked in the free run.

    • cites [57] Kemp J, Rajkomar A, Dai AM, Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes, ArXiv: 1909.03039 [Cs, Stat]. (2019). http://arxiv.org/abs/1909.03039 (accessed November 22, 2019).

      Not checked in the free run.

    • cites [72] Ma T, Xiao C, Wang F, Health-ATM: A deep architecture for multifaceted patient health record representation and risk prediction, in: 2018, pp. 261– 269. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85045143398&partnerID=40&md5=b63d90606c942e23cb2d49ae4fed27fd.

      Not checked in the free run.

    • cites [76] Ma F, Chitta R, Zhou J, You Q, Sun T, Gao J, Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ‘17. (2017) 1903– 1911. 10.1145/3097983.3098088.

      Not checked in the free run.

    • cites [80] Choi E, Bahadori MT, Sun J, Kulas J, Schuetz A, Stewart W, RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism, Adv. Neural Inf. Process. Syst (2016) 3504– 3512.

      Not checked in the free run.

    • cites [88] Barbieri S, Kemp J, Perez-Concha O, Kotwal S, Gallagher M, Ritchie A, Jorm L, Benchmarking Deep Learning Architectures for Predicting Readmission to the ICU and Describing Patients-at-Risk, Sci Rep. 10 (2020) 1111, 10.1038/s41598-020-58053-z. 31980704 PMC6981230 source

      Not checked in the free run.

    • cites [32] Wang S, Ren P, Chen Z, Ren Z, Ma J, de Rijke M, Order-free Medicine Combination Prediction with Graph Convolutional Reinforcement Learning, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management - CIKM ‘19, ACM Press, Beijing, China, 2019, pp. 1623– 1632. 10.1145/3357384.3357965.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    8 papers visualized with t-SNE to interpret the representations, and one study visualized with MDS.

    “We observed 8 papers visualized with t-SNE to interpret the representations [14,27,31,60,66,73,75,79], and one study visualized with MDS [70].”
    Quote in the paperNot a substantial claim0 of 9 sources checked
    • cites [14] Xu Y, Biswal S, Deshpande SR, Maher KO, Sun J, RAIM: Recurrent attentive and intensive model of multimodal patient monitoring data, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, New York, NY, USA, 2018, pp. 2565– 2573. 10.1145/3219819.3220051.

      Not checked in the free run.

    • cites [27] Choi E, Bahadori MT, Song L, Stewart WF, Sun J, GRAM: Graph-based Attention Model for Healthcare Representation Learning, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ‘17, ACM Press, Halifax, NS, Canada, 2017, pp. 787– 795. 10.1145/3097983.3098126. PMC7954122 33717639 source

      Not checked in the free run.

    • cites [31] Ma F, You Q, Xiao H, Chitta R, Zhou J, Gao J, KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ACM, New York, NY, USA, 2018, pp. 743– 752. 10.1145/3269206.3271701. source

      Not checked in the free run.

    • cites [60] Tran T, Nguyen TD, Phung D, Venkatesh S, Learning vector representation of medical objects via EMR-driven nonnegative restricted Boltzmann machines (eNRBM), J. Biomed. Inform 54 (2015) 96– 105, 10.1016/j.jbi.2015.01.012. 25661261

      Not checked in the free run.

    • cites [66] Zhang XS, Tang F, Dodge HH, Zhou J, Wang F, MetaPred: Meta-Learning for Clinical Risk Prediction with Limited Patient Electronic Health Records, in: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM, New York, NY, USA, 2019, pp. 2487– 2495. 10.1145/3292500.3330779. PMC8046258 33859865

      Not checked in the free run.

    • cites [73] Si Y, Roberts K, Deep patient representation of clinical notes via multi-task learning for mortality prediction, in: AMIA Jt Summits Transl Sci Proc, 2019, 2019, pp. 779– 788. 31259035 PMC6568068

      Not checked in the free run.

    • cites [75] Lei L, Zhou Y, Zhai J, Zhang L, Fang Z, He P, Gao J, An Effective Patient Representation Learning for Time-series Prediction Tasks Based on EHRs, in: 2019, pp. 885– 892. 10.1109/BIBM.2018.8621542.

      Not checked in the free run.

    • cites [79] Xiao C, Ma T, Dieng AB, Blei DM, Wang F, Readmission prediction via deep contextual embedding of clinical concepts, PLoS ONE 13 (2018) e0195024, 10.1371/journal.pone.0195024. 29630604 PMC5890980

      Not checked in the free run.

    • cites [70] Suo Q, Ma F, Yuan Y, Huai M, Zhong W, Gao J, Zhang A, Deep patient similarity learning for personalized healthcare, IEEE Trans. Nanobiosci 17 (2018) 219– 227, 10.1109/TNB.2018.2837622. 29994534

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    A total of 363 articles were assessed, with 49 studies from 2015 to 2019 meeting the full criteria for review.

    “A total of 363 articles were assessed, with 49 studies from 2015 to 2019 meeting the full criteria for review.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingcitation

    With the pace of deep learning models diffusing into patient representation learning, algorithm and model development continues to enhance in order to adequately consider and model EHR data.

    “With the pace of deep learning models diffusing into patient representation learning, algorithm and model development continues to enhance in order to adequately consider and model EHR data [113].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [113] Wang F, Casalino LP, Khullar D, Deep Learning in Medicine—Promise, Progress, and Challenges, JAMA, Intern Med. 179 (2019) 293, 10.1001/jamainternmed.2018.7117. 30556825

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    Unstructured data are often overlooked when developing patient representations (only 12 out of 49 studies used unstructured data).

    “Unstructured data, additionally, are often overlooked when developing patient representations (only 12 out of 49 studies used unstructured data).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    This has resulted in a wide range of methods that can complement deep learning models to enhance representation power.

    “This has resulted in a wide range of methods that can complement deep learning models to enhance representation power.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    These methods include developing additional components to models for capturing a hierarchical representation from event-level, to visit-level, to patient-level; for encoding the longitudinal factors between clinical events; for incorporating domain knowledge into the representation; and for adopting multiple modalities of EHR data.

    “These methods include developing additional components to models for capturing a hierarchical representation from event-level, to visit-level, to patient-level [81,112]; for encoding the longitudinal factors between clinical events [65,77]; for incorporating domain knowledge into the representation [27,31–33,91]; and for adopting multiple modalities of EHR data [11,14,57].”
    Quote in the paperNot a substantial claim0 of 9 sources checked
    • cites [81] Choi E, Xiao C, Stewart W, Sun J, Mime: Multilevel medical embedding of electronic health records for predictive healthcare, Adv. Neural Inf. Process. Syst (2018) 4547– 4557.

      Not checked in the free run.

    • cites [112] Si Y, Roberts K, Patient Representation Transfer Learning from Clinical Notes based on Hierarchical Attention Network, in: AMIA Joint Summits on Translational Science Proceedings. AMIA Joint Summits on Translational Science, 2020. PMC7233035 32477682

      Not checked in the free run.

    • cites [65] Liu L, Shen J, Zhang M, Wang Z, Tang J, Learning the joint representation of heterogeneous temporal events for clinical endpoint prediction, in: 2018, pp. 109– 116. https://www.scopus.com/inward/record.uri?eid=2-s2.0-85060476955&partnerID=40&md5=e3cfd1382f1464164edc3f0dd4ab7baa.

      Not checked in the free run.

    • cites [77] Liu L, Li H, Hu Z, Shi H, Wang Z, Tang J, Zhang M, Learning Hierarchical Representations of Electronic Health Records for Clinical Outcome Prediction, ArXiv Preprint ArXiv:1903.08652. (2019). PMC7153073 32308854

      Not checked in the free run.

    • cites [27] Choi E, Bahadori MT, Song L, Stewart WF, Sun J, GRAM: Graph-based Attention Model for Healthcare Representation Learning, in: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD ‘17, ACM Press, Halifax, NS, Canada, 2017, pp. 787– 795. 10.1145/3097983.3098126. PMC7954122 33717639 source

      Not checked in the free run.

    • cites [31] Ma F, You Q, Xiao H, Chitta R, Zhou J, Gao J, KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, ACM, New York, NY, USA, 2018, pp. 743– 752. 10.1145/3269206.3271701. source

      Not checked in the free run.

    • cites [32] Wang S, Ren P, Chen Z, Ren Z, Ma J, de Rijke M, Order-free Medicine Combination Prediction with Graph Convolutional Reinforcement Learning, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management - CIKM ‘19, ACM Press, Beijing, China, 2019, pp. 1623– 1632. 10.1145/3357384.3357965.

      Not checked in the free run.

    • cites [33] Zhang J, Gong J, Barnes L, HCNN: Heterogeneous Convolutional Neural Networks for Comorbid Risk Prediction with Electronic Health Records, in: Proceedings of the Second IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies, IEEE Press, Piscataway, NJ, USA, 2017, pp. 214– 221. 10.1109/CHASE.2017.80.

      Not checked in the free run.

    • cites [91] Hosseini A, Chen T, Wu W, Sun Y, Sarrafzadeh M, HeteroMed: Heterogeneous Information Network for Medical Diagnosis, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, Association for Computing Machinery, New York, NY, USA, 2018, pp. 763– 772. 10.1145/3269206.3271805. source

      Not checked in the free run.

  • May combine two claims.

    Findingnone given

    Many studies use diagnosis codes to represent patients, and ignore the real-valued measurements associated with the diagnoses, such as lab test values.

    “For instance, many studies use diagnosis codes to represent patients, and ignore the real-valued measurements associated with the diagnoses, such as lab test values.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Some pilot works have applied advanced methods to model contextualized information into the learning architecture, such as the transformer model.

    “Some pilot works have applied advanced methods to model contextualized information into the learning architecture, such as the transformer model [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I, Attention is all you need, Adv. Neural Inf. Process. Syst (2017) 5998– 6008.

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Although most current work focuses on a single type of data, there are also some pilot works that are trying to harmonize multiple types of data.

    “Although most current work focuses on a single type of data, there are also some pilot works that are trying to harmonize multiple types of data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Findingreasoning

    It is still unclear which approach works best for representing EHR data.

    “However, it is still unclear which approach works best for representing EHR data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Privacy-preserving algorithms and constrained domain adaptation settings are developed to compensate for the tradeoff between privacy and data access to some degree.

    “Privacy-preserving algorithms and constrained domain adaptation settings are developed to compensate for this tradeoff to some degree [124].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [124] Laparra E, Bethard S, Miller TA, Rethinking domain adaptation for machine learning over clinical language, JAMIA Open (2020) ooaa010, 10.1093/jamiaopen/ooaa010. PMC7382626 32734151

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    Prior works have proposed different components in addition to the deep learning models to improve representation ability and predictive performance.

    “Prior works have proposed different components in addition to the deep learning models to improve representation ability and predictive performance.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingreasoning

    Due to the lack of transparency and reproducibility of reported results, many studies in our review claim state-of-the-art results, but few can be verified by external parties.

    “Due to the lack of transparency and reproducibility of reported results, many studies in our review claim state-of-the-art results, but few can be verified by external parties.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given

    Even though MIMIC-III is widely used as a benchmark dataset, not many generic pipelines for benchmarking machine learning studies are available.

    “Even though MIMIC-III is widely used as a benchmark dataset, not many generic pipelines for benchmarking machine learning studies are available.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The existing predictive models mainly focus on the prediction of single diseases, rather than considering the complex mechanisms of patients from a holistic review.

    “The existing predictive models mainly focus on the prediction of single diseases, rather than considering the complex mechanisms of patients from a holistic review.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    A critical element in predictive modeling of EHR data is to effectively convert patient data from the raw EHR format to a machine learning representation.

    “A critical element in predictive modeling of EHR data is to effectively convert patient data from the raw EHR format to a machine learning representation—in other words, to transform patient data to meaningful information that can be further understood algorithmically.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Learning comprehensive representations of patient EHR data is important and feasible.

    “We show the importance and feasibility of learning comprehensive representations of patient EHR data through a systematic review.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    The effectiveness of predictive models for improving disease diagnosis, phenotyping, and prognosis heavily depends on the quality of the feature representation.

    “The effectiveness of predictive models for improving disease diagnosis, phenotyping, and prognosis heavily depends on the quality of this feature representation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Two authors screened the articles under a blind format.

    “Two authors (YS, JD) screened the articles under a blind format.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    At the full-text screening step, each study was read in full by one author to validate the final relevancy.

    “At the full-text screening step performed on Zotero [52], each study was read in full by one author (YS) to validate the final relevancy.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52] Ahmed KM, Al Dhubaib B, Zotero: A bibliographic assistant to researcher, J. Pharmacol. Pharmacotherap 2 (2011) 303. PMC3198533 22025866

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    Both CNN and RNN architectures are utilized to learn local or sequential information from large-scale data and to find the nuance of variance within the data.

    “This result is reasonable since both CNN and RNN architectures are utilized to learn local or sequential information from large-scale data and to find the nuance of variance within the data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    The unique characteristics of EHR data make deep learning a feasible data-driven solution to generate robust representation simultaneously from diverse resources compared with other techniques.

    “Notably, we consider studies using deep learning as one criterion because the unique characteristics of EHR data (as discussed in Introduction) make deep learning a feasible data-driven solution to generate robust representation simultaneously from diverse resources compared with other techniques.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: EHR data have challenging characteristics such as being uncurated, poor-quality, high-dimensional, sparse, heterogeneous, temporal, incomplete, large-scale, and multimodal.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Deep learning models are not mutually exclusive from each other.

    “deep learning models are not mutually exclusive from each other.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Patient representations can be potentially useful for clustering patients based on similarity, which can be used to identify patient subtypes.

    “Rather than focusing on predictive tasks, patient representations can be potentially useful for clustering patients based on similarity, which can be used to identify patient subtypes, among other things.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The best aspects of each deep learning model can be combined into a joint deep learning architecture.

    “The best aspects of each model can be combined into a joint deep learning architecture.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    A large publicly available clinical dataset can push representation performance to a scale similar to that of image processing by giving researchers a common benchmark to build on prior work.

    “a large publicly available clinical dataset can push representation performance to a scale similar to that of image processing by giving researchers a common benchmark to build on prior work.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Since no labels are required, unsupervised learning is more likely to benefit from large EHR datasets.

    “Since no labels are required, unsupervised learning is more likely to benefit from large EHR datasets.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The slight decrease from 2018 to 2019 is because preprints were excluded at the first step of screening.

    “Although there is a slight decrease from 2018 to 2019, we think this is because we exclude preprints at the first step of screening.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Collections of continuous numbers in patient representations can be particularly abstruse and difficult for humans to interpret.

    “these collections of continuous numbers can be particularly abstruse and difficult for humans to interpret [101].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [101] Hinton GE, Salakhutdinov RR, Reducing the dimensionality of data with neural networks, Science 313 (2006) 504– 507. 16873662 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Enhancements in patient representation learning techniques will be continuously growing for powering patient-level EHR analyses.

    “Enhancements in patient representation learning techniques will be continuously growing for powering patient-level EHR analyses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Another major difficulty in data sharing is related to privacy concerns as inappropriate access to sensitive patient data might lead to patients' identity disclosure.

    “Another major difficulty in data sharing is related to privacy concerns as inappropriate access to sensitive patient data might lead to patients’ identity disclosure.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Sharing open-source resources and code would motivate researchers to further explore innovations in a more scientific manner.

    “Thus, we anticipate that sharing open-source resources and code would motivate researchers to further explore innovations in a more scientific manner.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Researchers from diverse communities will leverage the richness and potential of EHR data, and will assist the capability of learning patient representation further.

    “We anticipate that researchers from diverse communities will leverage the richness and potential of EHR data, and will assist the capability of learning patient representation further.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Since learned representations attempt to gain a cohesive picture of a patient's data, these models may very well be more robust to small changes in the data distribution (e.g., when porting a model to a new institution without retraining).

    “Since learned representations attempt to gain a cohesive picture of a patient’s data, these models may very well be more robust to small changes in the data distribution (e.g., when porting a model to a new institution without retraining).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The field of natural language processing has become dominated recently by transformer-based models that essentially follow this approach.

    “the field of natural language processing has become dominated recently by transformer-based models [41] that essentially follow this approach.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41] Devlin J, Chang M-W, Lee K, Toutanova K, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 4171– 4186.

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Analyses and evidence from data extractions of the 49 included papers suggest the importance and feasibility of learning comprehensive representations of patient EHR data.

    “Furthermore, analyses and evidence from data extractions of the 49 included papers suggest the importance and feasibility of learning comprehensive representations of patient EHR data.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    All these data sources can expand the set of features that potentially contribute to fine-grained patient representations.

    “All these data sources can expand the set of features that potentially contribute to fine-grained patient representations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given

    Deep learning-based multimodal representation learning can be exploited to narrow the heterogeneity of EHR data among different modalities.

    “One technique known as deep learning-based multimodal representation learning can be exploited to narrow the heterogeneity of EHR data among different modalities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    It is challenging to differentiate our included studies from numerous works of predicting one clinical outcome end-to-end with deep learning.

    “It is also challenging to differentiate our included studies with numerous works of predicting one clinical outcome end-to-end with deep learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    This might be a barrier for future model development and would slow the pace of improvement.

    “This might be a barrier for future model development and would slow the pace of improvement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Distributed learning and federated learning are paradigms that enable the safe use of data from multiple institutions by only learning mathematical parameters and avoiding any instance that might trace back to a specific patient.

    “Distributed learning and federated learning are such paradigms that enable the safe use of data from multiple institutions by only learning mathematical parameters [125] and avoiding any instance that might trace back to a specific patient.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [125] Koněcný J, McMahan B, Ramage D, Federated Optimization:Distributed Optimization Beyond the Datacenter, ArXiv: 1511.03575 [Cs, Math]. (2015). http://arxiv.org/abs/1511.03575 (accessed May 7, 2020).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    We mitigated the incomplete retrieval issue by working cooperatively with experienced librarians and applying snowballing search strategies from the included publications.

    “We mitigated this issue by working cooperatively with experienced librarians and applying snowballing search strategies from the included publications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    All these limitations would pose a potential threat to selective bias in publication trends.

    “All these limitations would pose a potential threat to selective bias in publication trends.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    These works show the enormous potential to integrate multi-source information to effectively model patient health status.

    “These works show the enormous potential to integrate multi-source information to effectively model patient health status [22].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22] Stein-O’Brien GL, Arora R, Culhane AC, Favorov AV, Garmire LX, Greene CS, Goff LA, Li Y, Ngom A, Ochs MF, Xu Y, Fertig EJ, Enter the Matrix: Factorization Uncovers Knowledge from Omics, Trends Genet. 34 (2018) 790– 805, 10.1016/j.tig.2018.07.003. 30143323 PMC6309559

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The adoption of distributed and federated learning shows that there are emerging technologies to manage privacy concerns and facilitate scientific research without directly sharing data.

    “The adoption of distributed and federated learning shows that there are emerging technologies to manage privacy concerns and facilitate scientific research without directly sharing data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We do not include manuscripts that can only be found as preprints on arXiv or bioRxiv.

    “In addition, we do not include manuscripts [133–136] that can only be found as preprints on arXiv or bioRxiv.”
    Quote in the paperNot a substantial claim0 of 4 sources checked
    • cites [133] Darabi S, Kachuee M, Fazeli S, Sarrafzadeh M, TAPER: Time-Aware Patient EHR Representation, ArXiv: 1908.03971 [Cs, Stat]. (2019). http://arxiv.org/abs/1908.03971 (accessed September 20, 2019). 32287023

      Not checked in the free run.

    • cites [134] Hettige B, Li Y-F, Wang W, Le S, Buntine W, MedGraph: Structural and Temporal Representation Learning of Electronic Medical Records, ArXiv: 1912.03703 [Cs, Stat]. (2020). http://arxiv.org/abs/1912.03703 (accessed May 7, 2020).

      Not checked in the free run.

    • cites [135] Darabi S, Kachuee M, Sarrafzadeh M, Unsupervised Representation for EHR Signals and Codes as Patient Status Vector, ArXiv: 1910.01803 [Cs, Stat]. (2019). http://arxiv.org/abs/1910.01803 (accessed May 7, 2020).

      Not checked in the free run.

    • cites [136] Dubois S, Romano N, Kale DC, Shah N, Jung K, Effective Representations of Clinical Notes, ArXiv: 1705.07025 [Cs, Stat]. (2017). http://arxiv.org/abs/1705.07025 (accessed April 15, 2019).

      Not checked in the free run.

  • May combine two claims.

    Study design and conductreasoning

    We ensured at the full-text review step that all included papers specifically mentioned patient representation or embedding, and that their goal was learning representations at a patient level, not merely a model that maximized performance on a specific disease.

    “To achieve this, we ensured at the full-text review step that all the papers that are included specifically mentioned patient representation or embedding, and that their goal was learning representations at a patient level, not merely a model that maximized performance on a specific disease.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Learning representations from multiple institutions is critical to achieving both scalability and generalizability.

    “Yet learning representations from multiple institutions is critical to achieving both scalability and generalizability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Integrating multi-source information, e.g. genomics and clinical imaging, would be critical for accurate patient representation learning as well as the associated downstream clinical tasks.

    “Therefore, integrating multi-source information, e.g. genomics and clinical imaging, would be critical for accurate patient representation learning as well as the associated downstream clinical tasks [120].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [120] Shilo S, Rossman H, Segal E, Axes of a revolution: challenges and promises of big data in healthcare, Nat. Med 26 (2020) 29– 38, 10.1038/s41591-019-0727-5. 31932803 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductreasoning

    Our review fails to answer all technical questions.

    “Another limitation is that our review fails to answer all technical questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    There is no MeSH term that covers patient representation.

    “There is, unfortunately, no MeSH term that covers patient representation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Deep patient representation learning is a promising direction to acquire powerful, robust, and precise representations.

    “deep patient representation learning is a promising direction to acquire powerful, robust, and precise representations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Deep representation learning has led to a wide variety of innovations in the process of modeling EHR patient data.

    “Deep representation learning has led to a wide variety of innovations in the process of modeling EHR patient data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    Patient representation learning attempts to further address issues related to patient data and promote scientific research.

    “patient representation learning attempts to further address issues related to patient data and promote scientific research.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    A growing number of advanced methods will be continuously developed to learn meaningful patient representations, and these representations will play a greater and greater role in clinical prediction tasks.

    “We believe a growing number of advanced methods will be continuously developed to learn meaningful patient representations, and that these representations will play a greater and greater role in clinical prediction tasks.”
    Quote in the paperNot a substantial claim
88 of 88

  • The paper relies on this without support.

    EHR data have challenging characteristics such as being uncurated, poor-quality, high-dimensional, sparse, heterogeneous, temporal, incomplete, large-scale, and multimodal.

    “This results in the issue that EHR data have many challenging characteristics such as uncurated (data are not carefully chosen and thoughtfully organized or presented), poor-quality (data are rarely subject to data quality audits), high-dimensional (thousands of distinct medical events), sparse (lots of zero values), heterogeneous (drawn from different resources), temporal (data are collected over”

  • The paper relies on this without support.

    The unavailability of benchmark datasets is due to privacy concerns of EHR data.

    “Benchmark datasets were mostly unavailable (28 studies) due to privacy concerns of EHR data, and code availability was assured in 20 studies.”

  • The authors state this as an assumption.

    Deep learning does not necessarily need heavy manual work for preprocessing.

    “Although not many studies emphasized the preprocessing steps used to transfer raw input data into the model, we assume this is because deep learning does not necessarily need heavy manual work for preprocessing.”

  • The authors state this as an assumption.

    A more advanced and robust patient representation would improve the performance of predictive models.

    “The general assumption is that a more advanced and robust patient representation would improve the performance of predictive models.”

  • The authors state this as an assumption.

    Patient subtyping is not like other prediction tasks such as mortality and disease predictions that have straightforward label information.

    “We assume this is because patient subtyping is not like other prediction tasks such as mortality and disease predictions that have straightforward label information.”

  • The authors state this as an assumption.

    There is a trade-off between time and accuracy for mortality prediction, because the decision for this type of demanding task should be up-to-date as well.

    “However, we assume this is a trade-off between time and accuracy, because the decision for this type of demanding task should be up-to-date as well.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Healthcare datasets primarily consist of patient information, and strict regulation of medical privacy encompasses the security of medical records and the confidentiality of conversations between healthcare professionals.

    “Healthcare datasets primarily consist of patient information, and strict regulation of medical privacy, encompassing the security of medical records and the confidentiality of conversations between healthcare professionals [145], [146].”

    From Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond (2024) · cites this paper as [146]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Publication trend reverses in 2019

    Minor

    The abstract says publications almost doubled each year through 2019, but the Results section says they decreased from 2018 to 2019.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations, 2015.
  2. M. Niepert, M. Ahmed, K. Kutzkov, Learning convolutional neural networks for graphs, in: International Conference on Machine Learning, 2016, pp. 2014–2023.
  3. Choi, 2016, RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism, Adv. Neural Inf. Process. Syst., 3504

Cited by

Sorted by citations

Related

Authors
Si, Y., Du, J., Li, Z., Jiang, X., Miller, T., Wang, F., Jim Zheng, W., & Roberts, K.
Affiliations
  • The University of Texas Health Science Center at Houston (US)
  • Boston Children's Hospital (US)
  • Cornell University (US)
Corresponding author
Roberts, Kirk
ORCID
Venue
Journal of Biomedical Informatics
Publisher
Elsevier BV
Published
2021-03
Volume
115
Pages
103671
ISSN
1532-0464 (ISSN-L)
DOI
10.1016/j.jbi.2020.103671
PMC
PMC11290708
OpenAlex
W3092301826
Cited by
235
References
153
Citations by year
2026: 40 · 2025: 54 · 2024: 52 · 2023: 52 · 2022: 25 · 2021: 12
FWCI
0.76
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of Biomedical Informatics (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Cancer Prevention and Research Institute of Texas (RP170668, RR180012)
  • National Institutes of Health (R00LM012104, UL1TR00316701, R01LM012973, U01TR002062, R01AG06674901, R01GM118609)
  • Patient-Centered Outcomes Research Institute (ME-2018C1-10963)
  • National Science Foundation (NSF IIS-1750326)
  • U.S. National Library of Medicine
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
patient representation learning, electronic health records, deep learning, clinical outcome prediction, disease prediction, recurrent neural networks, long short-term memory, gated recurrent unit, reproducibility, benchmark dataset, knowledge distillation
MeSH terms
Deep Learning, Humans, Prognosis, Reproducibility of Results, Neural Networks, Computer, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review