The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records
Open accessPublished version of a preprint
Abstract
Deep learning transformer-based models using longitudinal electronic health records (EHRs) have shown a great success in prediction of clinical diseases or outcomes. Pretraining on a large dataset can help such models map the input space better and boost their performance on relevant tasks through finetuning with limited data. In this study, we present TransformEHR, a generative encoder-decoder model with transformer that is pretrained using a new pretraining objective—predicting all diseases and outcomes of a patient at a future visit from previous visits. TransformEHR’s encoder-decoder framework, paired with the novel pretraining objective, helps it achieve the new state-of-the-art performance on multiple clinical prediction tasks. Comparing with the previous model, TransformEHR improves area under the precision–recall curve by 2% ( p < 0.001) for pancreatic cancer onset and by 24% ( p = 0.007) for intentional self-harm in patients with post-traumatic stress disorder. The high performance in predicting intentional self-harm shows the potential of TransformEHR in building effective clinical intervention systems. TransformEHR is also generalizable and can be easily finetuned for clinical prediction tasks with limited data.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“We compared Trans-formEHR with four strong baseline models: logistic regression, long short-term memory (LSTM) 37 , BERT 22 without pretraining, and BERT 22 pretrained on VHA cohort.”
InfluentialPan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2019) · arXiv · doi:10.48550/arxiv.1910.13461 · Cited by 8,816 · cited 3×
“Although the encoder-decoder framework was originally designed to generate next sentence given previous sentences as context 23,24 , we repurposed the framework for TransformEHR to generate the ICD codes of the next visit given previous EHRs (context).”
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · cited 1×
“AUPRC is the area under PPV and sensitivity curve, and it has shown to be an effective measure for highly imbalanced binary classi fi cation problems, which include self-harm prediction 35,36 .”
MIMIC-IV, a freely accessible electronic health record dataset
Johnson, A. E. W., Bulgarelli, L., Shen, L., Gayles, A., Shammout, A., Horng, S., Pollard, T. J., Hao, S., Moody, B., Gow, B., et al. (2023) · Scientific Data · doi:10.1038/s41597-022-01899-x · Cited by 3,401
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Glypican-1 identifies cancer exosomes and detects early pancreatic cancer
Melo, S. A., Luecke, L. B., Kahlert, C., Fernandez, A. F., Gammon, S. T., Kaye, J., LeBleu, V. S., Mittendorf, E. A., Weitz, J., Rahbari, N., et al. (2015) · Nature · doi:10.1038/nature14581 · Cited by 2,880
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648 · cited 1×
“Speci fi c date of each visit is an important feature inpredictive modeling as importanceof predictor in a visit can vary over time 1,25 – 27 .”
Research works printed but not linked: 5
- Vaswani, A. et al. Attention is All you Need. in Advances in Neural Information Processing Systems 30 (eds. Guyon, I. et al.) 5998–6008 (Curran Associates, Inc., 2017).
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. in Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies 4171–4186 (Association for Computational Linguistics, 2019).
- McDermott, M. B. A., Nestor, B. A., Argaw, P. N. & Kohane, I. S. Event Stream GPT: a data pre-processing and modeling library for generative, pre-trained transformers over continuous-time sequences of complex events. ArXiv abs/2306.11547, (2023). · arXiv
- Pascanu, R., Mikolov, T. & Bengio, Y. On the difficulty of training recurrent neural networks. in International Conference on Machine Learning 1310–1318 (2013).
- Gradus, J. PTSD and death from suicide. Natl Cent. Posttraumatic Stress Disord. 28, 1050–1835 (2017).
Cited by
Large language models for disease diagnosis: a scoping review
Zhou, S., Xu, Z., Zhang, M., Xu, C., Guo, Y., Zhan, Z., Fang, Y., Ding, S., Wang, J., Xu, K., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00011-z · Cited by 243
Learning the natural history of human disease with generative transformers
Shmatko, A., Jung, A. W., Gaurav, K., Brunak, S., Mortensen, L. H., Birney, E., Fitzgerald, T., & Gerstung, M. (2025) · Nature · doi:10.1038/s41586-025-09529-3 · Cited by 110
“Transformer models trained on electronic health records have been used for predicting diagnoses such as pancreatic cancer 26 , self-harm 25 and stroke 24 , as well as non-clinical parameters such as self-esteem 27 .”
The generative era of medical AI
Fahrner, J., Chen, E., Topol, E. J., & Rajpurkar, P. (2025) · Cell · doi:10.1016/j.cell.2025.05.018 · Cited by 144
Simon, B. D., Ozyoruk, K. B., Gelikman, D. G., Harmon, S. A., & Türkbey, B. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242631 · Cited by 73
“22 Transformers are particularly revolutionary, unlike typical recurrent neural networks, in that they employ a parallelized approach, which allows for scalable computation.”
BackgroundTask-Specific Transformer-Based Language Models in Health Care: Scoping Review
Cho, H. N., Jun, T. J., Kim, Y., Kang, H. J., Ahn, I., Gwon, H., Kim, Y., Seo, H., Choi, H., Kim, M., et al. (2024) · JMIR Medical Informatics · doi:10.2196/49724 · Cited by 106
Metadata
- Authors
- Yang, Z., Mitra, A., Liu, W., Berlowitz, D., & Yu, H.
- Affiliations
- University of Massachusetts Amherst (US)
- University of Massachusetts Lowell (US)
- Corresponding author
- Yu, Hong
- Venue
- Nature Communications
- Published
- 2023-11-29
- Volume
- 14
- Issue
- 1
- Pages
- 7857-7857
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC10687211
- OpenAlex
- W4389132345
- Cited by
- 158
- References
- 57
- Citations by year
- 2026: 67 · 2025: 64 · 2024: 27
- FWCI
- 14.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. Department of Health & Human Services | NIH | National Institute of Mental Health (R01MH125027)
- U.S. Department of Health & Human Services | NIH | National Institute on Drug Abuse (R01DA056470)
- U.S. Department of Health & Human Services | NIH | National Institute on Aging (R01AG080670)
- U.S. Department of Veterans Affairs (I01HX003711)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Keywords
- generative models, electronic health records, pre-training objectives, longitudinal clinical data, disease outcome prediction, intentional self-harm, post-traumatic stress disorder
- MeSH terms
- Humans, Pancreatic Neoplasms, Electric Power Supplies, Mental Recall, Stress Disorders, Post-Traumatic, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. https://app.cochat.ai/papers/10.1038/s41467-023-43715-z
- The paper
- Yang, Z., Mitra, A., Liu, W., Berlowitz, D., & Yu, H. (2023). TransformEHR: transformer-based encoder-decoder generative model to enhance prediction of disease outcomes using electronic health records. Nature Communications. https://doi.org/10.1038/s41467-023-43715-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.