Improving clinical outcome predictions using convolution over medical entities with multimodal learning
Open accessAbstract
Early prediction of mortality and length of stay (LOS) of a patient is vital for saving a patient's life and management of hospital resources. Availability of Electronic Health Records (EHR) makes a huge impact on the healthcare domain and there are several works on predicting clinical problems. However, many studies did not benefit from the clinical notes because of the sparse, and high dimensional nature. In this work, we extract medical entities from clinical notes and use them as additional features besides time-series features to improve proposed model predictions. The proposed convolution based multimodal architecture, which not only learns effectively combining medical entities and time-series Intensive Care Unit (ICU) signals of patients but also allows to compare the effect of different embedding techniques such as Word2vec and FastText on medical entities. Results show that the proposed deep multimodal method outperforms all other baseline models including multimodal architectures and improves the mortality prediction performance for Area Under the Receiver Operating Characteristics (AUROC) and Area Under Precision-Recall Curve (AUPRC) by around 3%. For LOS predictions, there is an improvement of around 2.5% over the time-series baseline. The code for the proposed method is available at https://github.com/tanlab/ConvolutionMedicalNer.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A major barrier remains the isolation of modality-specific processing pipelines, where structured and unstructured clinical data are often treated as separate streams rather than synergistically integrated components.
“A major barrier remains the isolation of modality-specific processing pipelines, where structured and unstructured clinical data are often treated as separate streams rather than synergistically integrated components (Bardak and Tan, 2021; Fan et al., 2024).”
From Multimodal neuroimaging and AI integration in cognitive disorders: advances, challenges, and future directions for precision medicine (2026) · cites this paper as Bardak and Tan, 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Vincent, J. L., Moreno, R., Takala, J., Willatts, S., De Mendonça, A., Bruining, H., Reinhart, C. K., Suter, P. M., & Thijs, L. G. (1996) · Intensive Care Medicine · doi:10.1007/bf01709751 · Cited by 11,674 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Distributed Representations of Sentences and DocumentsPreprint
Le, Q. V., & Mikolov, T. (2014) · International Conference on Machine Learning · doi:10.48550/arxiv.1405.4053 · Cited by 10,307 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
Le Gall, J.-R. (1993) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.270.24.2957 · Cited by 7,773 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
The relationship between Precision-Recall and ROC curves
Davis, J., & Goadrich, M. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143874 · Cited by 6,559 · see citing works
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545 · see citing works
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601 · see citing works
(2011) · Cited by 2,752 · see citing works
APACHE—acute physiology and chronic health evaluation: a physiologically based classification system
Knaus, W. A., Zimmerman, J. E., Wagner, D. P., Draper, E. A., & Lawrence, D. E. (1981) · Critical Care Medicine · doi:10.1097/00003246-198108000-00008 · Cited by 2,190 · see citing works
DeepDTA: deep drug–target binding affinity prediction
Öztürk, H., Özgür, A., & Özkırımlı, E. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty593 · Cited by 1,728 · see citing works
Convolutional Neural Networks for Sentence ClassificationPreprint
Kim, Y. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5882 · Cited by 1,112 · see citing works
Publicly Available Clinical BERT EmbeddingsPreprint
Alsentzer, E., Murphy, J. R., Boag, W., Weng, W., Jin, D., Naumann, T., & McDermott, M. B. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.03323 · Cited by 1,040 · see citing works
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · see citing works
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
Deep Fragment Embeddings for Bidirectional Image Sentence MappingPreprint
Karpathy, A., Joulin, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.5679 · Cited by 802 · see citing works
An Improved Non-monotonic Transition System for Dependency Parsing
Honnibal, M., & Johnson, M. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1162 · Cited by 618 · see citing works
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · see citing works
Explainable Prediction of Medical Codes from Clinical Text
Mullenbach, J., Wiegreffe, S., Duke, J., Sun, J., & Eisenstein, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1100 · Cited by 597 · see citing works
Deep learning in clinical natural language processing: a methodical review
Wu, S., Roberts, K. E., Datta, S., Du, J., Ji, Z., Si, Y., Soni, S., Wang, Q., Wei, Q., Xiang, Y., et al. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz200 · Cited by 495 · see citing works
Attend and Diagnose: Clinical Time Series Analysis Using Attention Models
Song, H., Rajan, D., Thiagarajan, J., & Spanias, A. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11635 · Cited by 468 · see citing works
Consistently dated Atlantic sediment cores over the last 40 thousand years
Waelbroeck, C., Lougheed, B. C., Riveiros, N. V., Missiaen, L., Pedro, J. B., Dokken, T., Hajdas, I., Wacker, L., Abbott, P. M., Dumoulin, J., et al. (2019) · Scientific Data · doi:10.1038/s41597-019-0173-8 · Cited by 440 · see citing works
A Convolutional Neural Network for Modelling SentencesPreprint
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2188 · Cited by 433 · see citing works
Benchmarking deep learning models on large healthcare datasets
Purushotham, S., Meng, C., Che, Z., & Liu, Y. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.04.007 · Cited by 406 · see citing works
Dybowski, R., Gant, V, Weller, P., & Chang, R. (1996) · The Lancet · doi:10.1016/s0140-6736(96)90609-1 · Cited by 240 · see citing works
Awad, A., Bader-El-Den, M., McNicholas, J., & Briggs, J. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.10.002 · Cited by 208 · see citing works
Barton, C., Chettipally, U., Zhou, Y., Jiang, Z., Lynn-Palevsky, A., Le, S., Calvert, J., & Das, R. (2019) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2019.04.027 · Cited by 175 · see citing works
Wang, S., McDermott, M. B. A., Chauhan, G., Ghassemi, M., Hughes, M. C., & Naumann, T. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384469 · Cited by 149 · see citing works
Kim, S., Kim, W., & Park, R. W. (2011) · Healthcare Informatics Research · doi:10.4258/hir.2011.17.4.232 · Cited by 132 · see citing works
Patient length of stay and mortality prediction: A survey
Awad, A., Bader–El–Den, M., & McNicholas, J. (2017) · Health Services Management Research · doi:10.1177/0951484817696212 · Cited by 128 · see citing works
Dependency-based Convolutional Neural Networks for Sentence Embedding
Ma, M.-B., Huang, L., Zhou, B., & Xiang, B. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-2029 · Cited by 122 · see citing works
Efficient Vector Representation for Documents through CorruptionPreprint
Chen, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.02377 · Cited by 88 · see citing works
Dynamically Modeling Patient's Health State from Electronic Medical Records
Barajas, K. L. C., & Akella, R. (2015) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2783258.2783289 · Cited by 84 · see citing works
Dependency-based Convolutional Neural Networks for Sentence EmbeddingPreprint
Ma, M., Huang, L., Xiang, B., & Zhou, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1507.01839 · Cited by 66 · see citing works
Clinical Concept Extraction with Contextual Word EmbeddingPreprint
Zhu, H., Paschalidis, I. C., & Tahmasebi, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.10566 · Cited by 66 · see citing works
Using Clinical Notes with Time Series Data for ICU Management
Khadanga, S., Aggarwal, K., Joty, S., & Srivastava, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1678 · Cited by 64 · see citing works
Early hospital mortality prediction using vital signals
Sadeghi, R., Banerjee, T., & Romine, W. (2018) · Smart Health · doi:10.1016/j.smhl.2018.07.001 · Cited by 63 · see citing works
A Database-driven Decision Support System: Customized Mortality Prediction
Celi, L. A., Galvin, S., Davidzon, G., Lee, J., Scott, D., & Mark, R. G. (2012) · Journal of Personalized Medicine · doi:10.3390/jpm2040138 · Cited by 62 · see citing works
Early Hospital Mortality Prediction Using Vital Signals
Sadeghi, R., Banerjee, T., & Romine, W. L. (2019) · Journal of Bioresource Management · Cited by 62 · see citing works
Improving Hospital Mortality Prediction with Medical Named Entities and Multimodal LearningPreprint
Jin, M., Bahadori, M. T., Colak, A., Bhatia, P., Celikkaya, B., Bhakta, R., Senthivel, S., Khalilia, M. A., Navarro, D., Zhang, B., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12276 · Cited by 58 · see citing works
What's in a Note? Unpacking Predictive Value in Clinical Note Representations.
Boag, W., Doss, D., Naumann, T., & Peter Szolovits, B. (2018) · Europe PMC (PubMed Central) · Cited by 55 · see citing works
Comprehend Medical: A Named Entity Recognition and Relationship Extraction Web Service
Bhatia, P., Celikkaya, B., Khalilia, M. A., & Senthivel, S. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00297 · Cited by 54 · see citing works
Learning Tasks for Multitask Learning: Heterogenous Patient Populations in the ICU
Suresh, H., Gong, J. J., & Guttag, J. V. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219930 · Cited by 54 · see citing works
Predicting intervention onset in the ICU with switching state space models.
Ghassemi, M., Wu, M., Hughes, M. C., Peter Szolovits, B., & Doshi‐Velez, F. (2017) · PubMed Central · Cited by 48 · see citing works
Deep EHR: Chronic Disease Prediction Using Medical NotesPreprint
Liu, J., Zhang, Z., & Razavian, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.04928 · Cited by 48 · see citing works
LSTM Recurrent Neural Networks for Cybersecurity Named Entity RecognitionPreprint
Gasmi, H., Laval, J., & Bouras, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.10521 · Cited by 48 · see citing works
MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-IIIPreprint
Wang, S., McDermott, M. B. A., Chauhan, G., Hughes, M. C., Naumann, T., & Ghassemi, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.08322 · Cited by 47 · see citing works
Multimodal Learning and Reasoning for Visual Question Answering
Ilievski, I., & Feng, J. (2017) · neural information processing systems · Cited by 46 · see citing works
Deep Patient Representation of Clinical Notes via Multi-Task Learning for Mortality Prediction.
Si, Y., & Roberts, K. E. (2019) · PubMed Central · Cited by 40 · see citing works
Semi-Supervised Biomedical Translation With Cycle Wasserstein Regression GANs
McDermott, M. B. A., Yan, T., Naumann, T., et al. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11890 · Cited by 32 · see citing works
Forecasting Mortality Risk for Patients Admitted to Intensive Care Units Using Machine Learning
Darabi, H. R., Tsinis, D., Zecchini, K., et al. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.10.313 · Cited by 32 · see citing works
Deep Multimodal Learning for Audio-Visual Speech RecognitionPreprint
Mroueh, Y., Marcheret, E., & Goel, V. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.05396 · Cited by 25 · see citing works
Med7: A transferable clinical natural language processing model for electronic health records
Kormilitzin, A., Vaci, N., Liu, Q., & Nevado‐Holgado, A. J. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102086 · Cited by 23 · see citing works
Yakovlev, A. N., Metsker, O., Kovalchuk, S. V., et al. (2018) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(18)30783-6 · Cited by 20 · see citing works
Extracting UMLS Concepts from Medical Text Using General and Domain-Specific Deep Learning ModelsPreprint
Fraser, K., Nejadgholi, I., de Bruijn, B., Li, M., LaPlante, A., & Abidine, K. Z. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01274 · Cited by 18 · see citing works
MIMIC-ExtractPreprint
Wang, S., McDermott, M. B. A., Chauhan, G., Hughes, M. C., Naumann, T., & Ghassemi, M. (2020) · arXiv (Cornell University) · Cited by 12 · see citing works
Shukla, S. N., & Marlin, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11059 · Cited by 8 · see citing works
Nallabasannagari, A. R., Reddiboina, M., Seltzer, R. G. N., Zeffiro, T., Sharma, A. K., & Bhandari, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01366 · Cited by 7 · see citing works
TAC SRIE 2018: Extracting Systematic Review Information with MedaCy.
Mulyar, A., Mahendran, D., Maffey, L., et al. (2018) · Theory and applications of categories · Cited by 5 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 2
- [5][5] Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference , pages 301-318, 2016. -
- [54][54] Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. Scispacy: Fast and robust models for biomedical natural language processing. arXiv preprint arXiv:1902.07669 , 2019. - · arXiv
Cited by
Sorted by citations
Multimodal machine learning in precision health: A scoping review
Kline, A., Wang, H., Li, Y., Dennis, S., Hutch, M., Xu, Z., Wang, F., Cheng, F., & Luo, Y. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00712-8 · Cited by 528 · see citing works
Integrated multimodal artificial intelligence framework for healthcare applications
Soenksen, L. R., Ma, Y., Zeng, C., Boussioux, L., Villalobos Carballo, K., Na, L., Wiberg, H. M., Li, M. L., Fuentes, I., & Bertsimas, D. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00689-4 · Cited by 380 · see citing works
“…11 , gynecology 12 , hematology 13 , immunology 14 , nephrology 15 , neurology 16,17 , oncology 18 – 20 , ophthalmology 21 , psychiatry 22 , radiology 23 – 25 , public health 26 and healthcare operational analytics (i.e., mortality, length-of-stay, and discharge predictions) 27 – 30 .”
Dang, M., Liu, B., Chen, Y., & Zhang, Z. (2026) · Psychoradiology · doi:10.1093/psyrad/kkag007 · Cited by 7 · see citing works
- Authors
- Bardak, B., & Tan, M.
- Affiliations
- TOBB University of Economics and Technology (TR)
- Corresponding author
- Tan, Mehmet
- Venue
- Artificial Intelligence in Medicine
- Publisher
- Elsevier BV
- Published
- 2021-07
- Volume
- 117
- Pages
- 102112
- ISSN
- 0933-3657 (ISSN-L)
- OpenAlex
- W3160359732
- Cited by
- 48
- References
- 72
- Citations by year
- 2026: 6 · 2025: 13 · 2024: 16 · 2023: 7 · 2022: 5 · 2021: 1
- FWCI
- 2.55
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Artificial Intelligence in Medicine (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Scientific and Technological Research Council of Turkey (120E173)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Epidemiology › Sepsis Diagnosis and Treatment
- Keywords
- mortality prediction, length of stay prediction, clinical notes, medical entity extraction, multimodal learning, convolutional neural networks, electronic health records, word embeddings, Word2Vec, FastText, clinical outcome prediction
- MeSH terms
- Machine Learning, Humans, Intensive Care Units, Prognosis, ROC Curve, Area Under Curve, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
