RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data

Xu, Yanbo; Biswal, Siddharth; Deshpande, Shriprasad R.; Maher, Kevin O.; Sun, Jimeng

doi:10.1145/3219819.3220051

Cited by 144Machine Learning in HealthcareHealthcare Technology and Patient MonitoringECG Monitoring and Analysis

RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data

Xu, Yanbo, Biswal, Siddharth, Deshpande, Shriprasad R., Maher, Kevin O., Sun, Jimeng·2018·Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

DOI 10.1145/3219819.3220051

Open this paper in CoChat — free account

Abstract

With the improvement of medical data capturing, vast amount of continuous patient monitoring data, e.g., electrocardiogram (ECG), real-time vital signs and medications, become available for clinical decision support at intensive care units (ICUs). However, it becomes increasingly challenging to model such data, due to high density of the monitoring data, heterogeneous data types and the requirement for interpretable models.

Integrity signals

Registry

Crossref · 10.1145/3219819.3220051

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Prior studies have shown that multimodal models outperform single-modality models in predictive performance.
“All of these studies have shown that multimodal models outperform single-modality models in predictive performance.”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [18]
Intermediate fusion methods modeled each modality separately over time before integrating learned representations using concatenation or attention, but most relied on retrospective alignment within relatively coarse time windows.
“Intermediate fusion methods modeled each modality separately over time before integrating learned representations using concatenation [57,58] or attention [18,59], but most relied on retrospective alignment within relatively coarse time windows.”
From Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review (2026) · cites this paper as [18]
An attention mechanism was adopted in 13 of the 49 included studies to improve performance and provide a degree of interpretability.
“To improve performance and provide a degree of interpretability to the model, an attention mechanism is adopted in 13 studies [6,14,27,31,43,57,64,72,76–78,80,88].”
From Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review (2021) · cites this paper as [14]
For outcomes that happen in hospital, such as inpatient mortality, length-of-stay, and intervention prediction, observation windows at hour level are developed.
“On the contrary, for those outcomes that happen in hospital, such as inpatient mortality [14,57,61,65], length-of-stay [14,83], and intervention prediction [32,55,65], observation windows at hour level are developed.”
From Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review (2021) · cites this paper as [14]

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
Background
Long Short-Term Memory
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · cited 1× · see citing works
“For the multiclass prediction task of forecasting length of stay, we evaluate the models in terms of Cohen’s Kappa [10] and Accuracy.”
Method
Untitled work
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 3× · see citing works
“Intervention-based guidance Gt[1] For predicting output yt”
Background
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftPreprint
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 15,273 · see citing works
PhysioBank, PhysioToolkit, and PhysioNet: Components of a New Research Resource for Complex Physiologic Signals
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809 · see citing works
Untitled work
(2017) · Cited by 11,625 · see citing works
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
Background
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248 · cited 1× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
Background
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · cited 1× · see citing works
“We demonstrate our model on the MIMIC-III Waveform Database Matched Subset 2 [22], a publicly available multimodal EHR data released last year on PhysioNet [15].”
Method
Untitled work
(2012) · Cited by 8,009 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · cited 1× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
Background
Recurrent Neural Networks for Multivariate Time Series with Missing Values
Che, Z., Purushotham, S., Cho, K., Sontag, D., & Liu, Y. (2018) · Scientific Reports · doi:10.1038/s41598-018-24271-9 · Cited by 2,370 · cited 1× · see citing works
“…data integrations could be solved by a combination of refining high-density inputs into low-density representations [40] and filling sparse inputs with smoothed missing values [3], advanced methods are still in need for extracting different levels of information from the integrated multimodel data.”
Background
Deep patient: An unsupervised representation to predict the future of patients from the electronic health records
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710 · cited 1× · see citing works
“diagnoses, procedures, medications) to learn vector representations of patients or medical concepts [5, 8, 9, 34]; recurrent neural networks (RNNs) like LSTM and GRU are used to capture the sequential manner of EHR data and predict future disease diagnosis or intervention codes [28, 39], onset of heart disease [4] or…”
Method
Building high-level features using large scale unsupervised learning
Le, Q. V. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6639343 · Cited by 1,695 · cited 6× · see citing works
“P where W βh , w βa , and b β are the parameters learnt in MLP. 3.2.2 Guidance-based Attention.”
MethodInfluential
Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · cited 1× · see citing works
“A more detailed survey of recent deep learning approaches for EHR analysis can be found here [43].”
Background
Look Closer to See Better: Recurrent Attention Convolutional Neural Network for Fine-Grained Image Recognition
Fu, J., Zheng, H., & Mei, T. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.476 · Cited by 1,459 · cited 1× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
Background
Temporal Relational Reasoning in Videos
Zhou, B., Andonian, A., Oliva, A., & Torralba, A. B. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_49 · Cited by 1,104 · see citing works
RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism
Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1608.05745 · Cited by 1,029 · cited 1× · see citing works
“A reverse time attention mechanism was proposed for identifying influential patient hospital visits and medical codes in predictions of heart failure [7].”
BackgroundSelf-citation
Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · cited 1× · see citing works
“org/physiobank/database/mimic3wdb/matched/ 3The two prediction tasks were originally defined in [16], a recently public multitask benchmark on MIMIC-III Clinical Database; but the data were not matched with the MIMIC-III Waveform Database.”
Method
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928 · cited 2× · see citing works
“There are many recent deep learning successes in modeling such event datasets.[4, 5, 8, 9, 12, 28].”
BackgroundMethodSelf-citation
Doctor AI: Predicting Clinical Events via Recurrent Neural Networks
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2015) · Europe PMC (PubMed Central) · doi:10.48550/arxiv.1511.05942 · Cited by 793 · cited 2× · see citing works
“There are many recent deep learning successes in modeling such event datasets.[4, 5, 8, 9, 12, 28].”
BackgroundMethodSelf-citation
Random search for hyper-parameter optimization
BergstraJames, & BengioYoshua (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2188395 · Cited by 691 · see citing works
GRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637 · see citing works
Self-citation
Cardiologist-Level Arrhythmia Detection with Convolutional Neural NetworksPreprint
Rajpurkar, P., Hannun, A., Haghpanahi, M., Bourn, C., & Ng, A. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.01836 · Cited by 611 · cited 3× · see citing works
“Prior work [40] has shown CNN can perform well in classifying 30-second ECG signals; for embedding the 1-hour ECG signals in our experiment, we initially trained CNN on 30-second segmentations and train another CNN or RNN on top of the 1-hour concatenated outputs, but these didn’t outperform a…”
BackgroundMethodInfluential
Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks
Ma, F., Chitta, R., Zhou, J., You, Q., Sun, T., & Gao, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098088 · Cited by 576 · see citing works
Guiding the Long-Short Term Memory Model for Image Caption Generation
Jia, X., Gavves, E., Fernando, B., & Tuytelaars, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.277 · Cited by 469 · cited 2× · see citing works
“Extensive attention-based deep models have been developed for images [14, 21, 36, 48], audios and videos [23, 49], machine translations [1, 30, 46].”
BackgroundMethod
Bidirectional RNN for Medical Event Detection in Electronic Health Records
Jagannatha, A. N., & Yu, H. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1056 · Cited by 314 · cited 1× · see citing works
“In addition to use discrete structural EHR data, other works target on unstructured data like clinical notes, the main purpose of which are to extract structured medical concepts [13, 19, 20, 29, 31].”
Method
Computational Phenotype Discovery Using Unsupervised Feature Learning over Noisy, Sparse, and Irregular Clinical Data
Lasko, T. A., Denny, J. C., & Levy, M. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0066341 · Cited by 313 · cited 1× · see citing works
“diagnoses, procedures, medications) to learn vector representations of patients or medical concepts [5, 8, 9, 34]; recurrent neural networks (RNNs) like LSTM and GRU are used to capture the sequential manner of EHR data and predict future disease diagnosis or intervention codes [28, 39], onset of heart disease [4] or…”
Method
DeepCare: A Deep Dynamic Memory Model for Predictive Medicine
Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-31750-2_3 · Cited by 269 · see citing works
Recurrent Neural Networks for Multivariate Time Series with Missing ValuesPreprint
Che, Z., Purushotham, S., Cho, K., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01865 · Cited by 230 · cited 1× · see citing works
“…data integrations could be solved by a combination of refining high-density inputs into low-density representations [40] and filling sparse inputs with smoothed missing values [3], advanced methods are still in need for extracting different levels of information from the integrated multimodel data.”
Background
Structured prediction models for RNN based sequence labeling in clinical text
Jagannatha, A., & yu, H. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1082 · Cited by 215 · cited 1× · see citing works
“ne learning methods. In addition to use discrete structural EHR data, other works target on unstructured data like clinical notes, the main purpose of which are to extract structured medical concepts [13, 19, 20, 29, 31]. Deep learning on patient monitoring data. Traditional signal processing algorithms were widely us…”
Method
Characteristic wave detection in ECG signal using morphological transform
Sun, Y., Chan, K. L., & Krishnan, S. M. (2005) · BMC Cardiovascular Disorders · doi:10.1186/1471-2261-5-28 · Cited by 175 · cited 1× · see citing works
“Traditional signal processing algorithms were widely used for analyzing patient monitoringdata,suchasP-QRS-Tdetection[44]orRRinterval/heart ratevariability(HRV)extraction 1 for ECG signal processing.”
Predicting Clinical Events by Combining Static and Dynamic Information Using Recurrent Neural Networks
Esteban, C., Staeck, O., Baier, S., Yang, Y., & Tresp, V. (2016) · IEEE International Conference on Healthcare Informatics · doi:10.1109/ichi.2016.16 · Cited by 172 · cited 2× · see citing works
“There are many recent deep learning successes in modeling such event datasets.[4, 5, 8, 9, 12, 28].”
BackgroundMethod
Medical Concept Representation Learning from Electronic Health Records and its Application on Heart Failure PredictionPreprint
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03686 · Cited by 110 · cited 2× · see citing works
“There are many recent deep learning successes in modeling such event datasets.[4, 5, 8, 9, 12, 28].”
BackgroundMethodSelf-citation
Interpretable Predictions of Clinical Outcomes with An Attention-based Recurrent Neural Network
Sha, Y., & Wang, M. D. (2017) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/3107411.3107445 · Cited by 102 · cited 1× · see citing works
“Later on, graph based attention [6] and hierarchical attention [42] were adapted for incorporating external information like medical ontologies into the deep EHR models; different deep architectures like bidirectional RNN [32], GRU [24] were also applied for improving predictive accuracy.”
Method
Beat by Beat: Classifying Cardiac Arrhythmias with Recurrent Neural Networks
Schwab, P., Scebba, G., Zhang, J., Delai, M., & Karlen, W. (2017) · Computing in cardiology · doi:10.22489/cinc.2017.363-223 · Cited by 102 · see citing works
DeepCare: A Deep Dynamic Memory Model for Predictive MedicinePreprint
Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.00357 · Cited by 85 · cited 1× · see citing works
“diagnoses, procedures, medications) to learn vector representations of patients or medical concepts [5, 8, 9, 34]; recurrent neural networks (RNNs) like LSTM and GRU are used to capture the sequential manner of EHR data and predict future disease diagnosis or intervention codes [28, 39], onset of heart disease [4] or…”
Method
Clinical Relation Extraction with Deep Learning
Lv, X., Guan, Y., Yang, J., & Wu, J. (2016) · International Journal of Hybrid Information Technology · doi:10.14257/ijhit.2016.9.7.22 · Cited by 58 · cited 1× · see citing works
“In addition to use discrete structural EHR data, other works target on unstructured data like clinical notes, the main purpose of which are to extract structured medical concepts [13, 19, 20, 29, 31].”
Method
Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
Liu, Y., Ge, T., Mathews, K., Ji, H., & McGuinness, D. L. (2015) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/w15-3810 · Cited by 52 · cited 1× · see citing works
“In addition to use discrete structural EHR data, other works target on unstructured data like clinical notes, the main purpose of which are to extract structured medical concepts [13, 19, 20, 29, 31].”
Method
Brundlefly at SemEval-2016 Task 12: Recurrent Neural Networks vs. Joint Inference for Clinical Temporal Information Extraction
Fries, J. (2016) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s16-1198 · Cited by 47 · cited 1× · see citing works
“In addition to use discrete structural EHR data, other works target on unstructured data like clinical notes, the main purpose of which are to extract structured medical concepts [13, 19, 20, 29, 31].”
Method
Temporal Pattern and Association Discovery of Diagnosis Codes Using Deep Learning
Mehrabi, S., Sohn, S., Li, D., Pankratz, J. J., Therneau, T. M., Sauver, J. L. S., Liu, H., & Palakal, M. J. (2015) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi.2015.58 · Cited by 44 · cited 1× · see citing works
“diagnoses, procedures, medications) to learn vector representations of patients or medical concepts [5, 8, 9, 34]; recurrent neural networks (RNNs) like LSTM and GRU are used to capture the sequential manner of EHR data and predict future disease diagnosis or intervention codes [28, 39], onset of heart disease [4] or…”
Method
Highrisk Prediction from Electronic Medical Records via Deep Attention NetworksPreprint
Kim, Y. J., Lee, Y.-G., Kim, J. W., Park, J. J., Ryu, B., & Ha, J.-W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00010 · Cited by 21 · cited 2× · see citing works
“For example, variations of convolutional neural network (CNN) and restricted boltzmann machine (RBM) are applied on structural clinical data (e.g. diagnoses, procedures, medications) to learn vector representations of patients or medical concepts [5, 8, 9, 34]; recurrent neural networks (RNNs) like LSTM and GRU are us…”
Method

Flagged first, then by citations

Sorted by citations

Multitask learning and benchmarking with clinical time series data
Harutyunyan, H., Khachatrian, H., Kale, D. C., Ver Steeg, G., & Galstyan, A. (2019) · Scientific Data · doi:10.1038/s41597-019-0103-9 · Cited by 958 · see citing works
“nchmark codebase, several teams used our dataset generation pipeline (fully or partially). In particular, the pipeline was used for in-hospital mortality prediction [38–43], decompensation prediction [44], length-of-stay prediction [42, 44, 45], phenotyping [38, 39, 46] and readmission prediction [47]. Additionally, a…”
Method
Remote patient monitoring using artificial intelligence: Current state, applications, and challenges
Shaik, T., Tao, X., Higgins, N., Li, L., Gururajan, R., Zhou, X., & Acharya, U. R. (2023) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1485 · Cited by 580 · see citing works
Opportunities and challenges of deep learning methods for electrocardiogram data: A systematic review
Hong, S., Zhou, Y., Shang, J., Xiao, C., & Sun, J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103801 · Cited by 488 · see citing works
“siﬁcation (e.g.,patients with paroxysmal atrial ﬁbrillation or Healthy [55]), multi-class classiﬁcation (e.g., detection of acute cognitive stress detection [31], decompensation of patients detection [131]) and multi-task classiﬁcation (e.g., detection of prevalent hypertension, sleep apnea, and diabetes [89]). 2) Ann…”
BackgroundSelf-citation
Deep learning in ECG diagnosis: A review
Liu, X., Wang, H., Li, Z., & Qin, L. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107187 · Cited by 443 · see citing works
Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects
Zhang, K., Wen, Q., Zhang, C., Cai, R., Jin, M., Liu, Y., Zhang, J. Y., Liang, Y., Pang, G., Song, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3387317 · Cited by 295 · see citing works
“T IME series data abound in many real-world scenarios [1], [2], including human activity recognition [3], industrial fault diagnosis [4], smart building management [5], and healthcare [6].”
Background
Deep representation learning of patient data from Electronic Health Records (EHR): A systematic review
Si, Y., Du, J., Li, Z., Jiang, X., Miller, T., Wang, F., Jim Zheng, W., & Roberts, K. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103671 · Cited by 235 · see citing works
Doctor XAI
Panigutti, C., Perotti, A., & Pedreschi, D. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372855 · Cited by 162 · see citing works
“Many interpretability approaches related to sequential data modeling focus on adding an attention mechanism [2, 44] to a sequential model and use the attention weights as a form of explanation [3, 11, 33, 47], however recent works have highlighted how this kind of explanation might lack consistency [6, 22, 41] and tha…”
Background
Exploring Interpretable LSTM Neural Networks over Multi-Variable DataPreprint
Guo, T., Lin, T., & Antulov-Fantulin, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12034 · Cited by 143 · see citing works
M3Care: Learning with Missing Modalities in Multimodal Healthcare Data
Zhang, C., Chu, X., Ma, L., Zhu, Y., Wang, Y., Wang, J., & Zhao, J. (2022) · Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3534678.3539388 · Cited by 102 · see citing works
Analyzing the role of model uncertainty for electronic health records
Dusenberry, M. W., Tran, D., Choi, E., Kemp, J., Nixon, J., Jerfel, G., Heller, K., & Dai, A. M. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384457 · Cited by 102 · see citing works
HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units
Hong, S., Xu, Y., Khare, A., Priambada, S., Maher, K., Aljiffry, A., Sun, J., & Tumanov, A. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3403212 · Cited by 89 · see citing works
“ectrocardiogram (ECG) and electroencephalogram (EEG) are being observed and collected especially in intensive care units (ICUs), which provides invaluable input for training deep learning (DL) models [21, 29, 37]. In fact deep learning models have achieved state-of-the-art performance in many healthcare and medical ap…”
BackgroundInfluentialSelf-citation
A Scalable and Transferable Federated Learning System for Classifying Healthcare Sensor Data
Sun, L., & Wu, J. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3171402 · Cited by 86 · see citing works
“Transformer [11] and recurrent attention models [12] are also used in intensive patient monitoring.”
Method
Deep learning for heterogeneous medical data analysis
Yue, L., Tian, D., Chen, W., Han, X., & Yin, M. (2020) · World Wide Web · doi:10.1007/s11280-019-00764-z · Cited by 85 · see citing works
“Xu et al. [71] proposed recurrent attentive and intensive model called RAIM for jointly analyzing continuous monitoring data, which generated a guidance matrix from lab measurements and interventions, used it to guide the multi-channel attention mechanism on processed multimodal input streams, and…”
Method
Benchmarking machine learning models on multi-centre eICU critical care dataset
Sheikhalishahi, S., Balaraman, V., & Osmani, V. (2020) · PLOS ONE · doi:10.1371/journal.pone.0235424 · Cited by 89 · see citing works
“Differently, Xu et al [35] proposed a deep learning model to predict the decompensation event.”
Background
Using Clinical Notes with Time Series Data for ICU Management
Khadanga, S., Aggarwal, K., Joty, S., & Srivastava, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1678 · Cited by 64 · see citing works
“Xu et al. (2018) accommodated supplemental information like diagnosis, medications, and lab events to improve model performance.”
Background
Explainable Multivariate Time Series Classification
Hsieh, T.-Y., Wang, S., Sun, Y., & Honavar, V. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441815 · Cited by 62 · see citing works
“RAIM [59] introduced a multi-channel attention mechanism guided by discrete clinical events to jointly analyze continuous monitoring data and discrete clinical events.”
Background
Anomaly Detection for Internet of Things Time Series Data Using Generative Adversarial Networks With Attention Mechanism in Smart Agriculture
Cheng, W., Ma, T., Wang, X., & Wang, G. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.890563 · Cited by 60 · see citing works
EHR-KnowGen: Knowledge-enhanced multimodal learning for disease diagnosis generation
Niu, S., Ma, J., Bai, L., Wang, Z., Guo, L., & Yang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102069 · Cited by 59 · see citing works
Predicting the Survival of Cancer Patients With Multimodal Graph Neural Network
Gao, J., Lyu, T., Xiong, F., Wang, J., Ke, W., & Li, Z. (2021) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2021.3083566 · Cited by 54 · see citing works
“…representations from multiple aspects, multimodal representation has been successfully applied to many tasks, such as action or expression recognition [25], [26], [27], image or video classiﬁcation [28], [29], medical diagnosis [30], [31], [32], disease diagnosis [33] and clinical prediction [34].”
Background
Optimizing artificial intelligence in sepsis management: Opportunities in the present and looking closely to the future
O'Reilly, D., McGrath, J., & Martin-Loeches, I. (2024) · Journal of Intensive Medicine · doi:10.1016/j.jointm.2023.10.001 · Cited by 55 · see citing works
“[42] This is a type of model that is easily interpretable for clinicians.”
Background
Predicting patient decompensation from continuous physiologic monitoring in the emergency department
Sundrani, S. C., Chen, J., Jin, B. T., Abad, Z. S. H., Rajpurkar, P., & Kim, D. A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00803-0 · Cited by 53 · see citing works
MNN: Multimodal Attentional Neural Networks for Diagnosis Prediction
Qiao, Z., Wu, X., Ge, S., & Fan, W. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/823 · Cited by 52 · see citing works
GRASP: Generic Framework for Health Status Representation Learning Based on Incorporating Knowledge from Similar Patients
Zhang, C., Gao, X., Ma, L., Wang, Y., Wang, J., & Tang, W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16152 · Cited by 53 · see citing works
“However, EMR data are usually sparse (Xu et al. 2018), and while dealing with a patient record with low quality, such methods will learn inadequate representations and lead to inferior performance.”
Background
Explainable Uncertainty-Aware Convolutional Recurrent Neural Network for Irregular Medical Time Series
Tan, Q., Ye, M., Ma, A. J., Yang, B., Yip, T. C., Wong, G. L., & Yuen, P. C. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3025813 · Cited by 52 · see citing works
Asthma Exacerbation Prediction and Risk Factor Analysis Based on a Time-Sensitive, Attentive Neural Network: Retrospective Cohort Study
Xiang, Y., Ji, H., Zhou, Y., Li, F., Du, J., Rasmy, L., Wu, S., Zheng, W. J., Xu, H., Zhi, D., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/16981 · Cited by 51 · see citing works
“Typical prediction tasks include the prediction of diabetes mellitus [23], Parkinson disease [29,33], chronic heart failure [26], sepsis [34], mortality, and readmission [25].”
BackgroundMethod
A novel artificial intelligence based intensive care unit monitoring system: using physiological waveforms to identify sepsis
Mollura, M., Lehman, L.-W. H., Mark, R. G., & Barbieri, R. (2021) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2020.0252 · Cited by 49 · see citing works
“[28] where the authors propose an attention-based deep learning algorithm which predicts patients’ length of stay and 24-h mortality risk combining high-resolution data (ECG sampled at 125Hz), low-resolution data (e.g. mean arterial blood pressure sampled at 2Hz) and uneven and sparse information…”
Background
Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series
Zhang, J., Zheng, S., Cao, W., Bian, J., & Li, J. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599543 · Cited by 49 · see citing works
“Early methods [24, 51, 52, 56] used either hourly aggregation or forward imputation to obtain uniformly spaced intervals, overlooking meaningful temporal patterns in irregular sampling.”
Method
Enhancing healthcare decision support through explainable AI models for risk prediction
Niu, S., Yin, Q., Ma, J., Song, Y., Xu, R. Y. D., Bai, L., Pan, W., & Yang, X. (2024) · Decision Support Systems · doi:10.1016/j.dss.2024.114228 · Cited by 45 · see citing works
Application of artificial intelligence methods in vital signs analysis of hospitalized patients: A systematic literature review
Kaieski, N., da Costa, C. A., Righi, R. D. R., Lora, P. S., & Eskofier, B. M. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106612 · Cited by 45 · see citing works
INSMA: An integrated system for multimodal data acquisition and analysis in the intensive care unit
Sun, Y., Guo, F., Kaffashi, F., Jacono, F. J., DeGeorgia, M. A., & Loparo, K. A. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103434 · Cited by 44 · see citing works
A Drug Recommendation Model Based on Message Propagation and DDI Gating Mechanism
Ren, Y., Shi, Y., Zhang, K., Wang, X., Chen, Z., & Li, H. (2022) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2022.3153342 · Cited by 42 · see citing works
“In addition, in health care research, many neural network structures (such as Dipole [14], RETAIN [10], HiTANet [15], and RAIM [16]) adopt attention mechanisms.”
Background
A Review of Deep Learning Methods for Irregularly Sampled Medical Time Series Data
Sun, C., Hong, S., Song, M., & Li, H. (2020) · PubMed · doi:10.48550/arxiv.2010.12493 · Cited by 38 · see citing works
“They [60, 61, 62] discretize the time axis into non-overlapping intervals with hand-designed intervals.”
Using neural attention networks to detect adverse medical events from electronic health records
Chu, J., Dong, W., He, K., Duan, H., & Huang, Z. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.10.002 · Cited by 35 · see citing works
Knowing your FATE
Tang, X., Liu, Y., Shah, N., Shi, X., Mitra, P., & Wang, S. (2020) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3394486.3403276 · Cited by 35 · see citing works
“The second group leverages attention methods to generate explanations on-the-fly, and gained tremendous popularity due to their efficiency [6, 9, 25, 31, 37].”
Background
Explaining Time Series Predictions with Dynamic MasksPreprint
Crabbé, J., & van der Schaar, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.05303 · Cited by 34 · see citing works
“A popular example, building on the success attention mechanisms (Vaswani et al., 2017), is the usage of attention layers to produce importance scores (Choi et al., 2016; Song et al., 2017; Xu et al., 2018; Kwon et al., 2018).”
Background
Predicting Optimal Hypertension Treatment Pathways Using Recurrent Neural Networks
Ye, X., Zeng, Q. T., Facelli, J. C., Brixner, D. I., Conway, M., & Bray, B. E. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104122 · Cited by 33 · see citing works
What went wrong and when? Instance-wise feature importance for time-series black-box models
Tonekaboni, S., Joshi, S. D., Campbell, K. R., Duvenaud, D., & Goldenberg, A. (2020) · neural information processing systems · Cited by 31 · see citing works
Harnessing the potential of multimodal EHR data: A comprehensive survey of clinical predictive modeling for intelligent healthcare
Wu, J., He, K., Mao, R., Shang, X., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103283 · Cited by 27 · see citing works
Knowledge-shot learning: An interpretable deep model for classifying imbalanced electrocardiography data
Chou, Y.-H., Hong, S., Zhou, Y., Shang, J., Song, M., & Li, H. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.07.045 · Cited by 26 · see citing works
Fairness and Inclusion in Biomedical Artificial Intelligence Research and Clinical Use: Technical and Social Perspectives
Griffin, A. C., Wang, K. H., Leung, T. I., & Facelli, J. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104693 · Cited by 19 · see citing works
A deep multimodal model for bug localization
Zhu, Z., Li, Y., Wang, Y., Wang, Y., & Tong, H. (2021) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-021-00755-7 · Cited by 24 · see citing works
“To process and correlate information from multiple modalities, multimodal machine learning has received a great deal of attention (Poria et al. 2016; Xu et al. 2018).”
Method
UA-CRNN: Uncertainty-Aware Convolutional Recurrent Neural Network for Mortality Risk Prediction
Tan, Q., Ma, A. J., Ye, M., Yang, B., Deng, H., Wong, V. W., Tse, Y., Yip, T. C., Wong, G. L., Ching, J. Y., et al. (2019) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3357384.3357884 · Cited by 23 · see citing works
Opportunities and Challenges of Deep Learning Methods for Electrocardiogram Data: A Systematic Review
Hong, S., Zhou, Y., Shang, J., Xiao, C., & Sun, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01550 · Cited by 23 · see citing works
“siﬁcation (e.g.,patients with paroxysmal atrial ﬁbrillation or Healthy [55]), multi-class classiﬁcation (e.g., detection of acute cognitive stress detection [31], decompensation of patients detection [131]) and multi-task classiﬁcation (e.g., detection of prevalent hypertension, sleep apnea, and diabetes [89]). 2) Ann…”
BackgroundSelf-citation
TOP-Net Prediction Model Using Bidirectional Long Short-term Memory and Medical-Grade Wearable Multisensor System for Tachycardia Onset: Algorithm Development Study
Liu, X., Liu, T., Zhang, Z., Kuo, P., Xu, H., Yang, Z., Lan, K., Li, P., Ouyang, Z., Ng, Y. L., et al. (2021) · JMIR Medical Informatics · doi:10.2196/18803 · Cited by 23 · see citing works
“Recently, several researchers have tried to combine the analysis of 2 kinds of materials to represent comprehensive information and improve the performance of the models: Xu et al [56] proposed a model to predict physiological decompensation and length of ICU stay by analyzing ECG and medical records data, and Nemati…”
Background
Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records
Niu, S., Yin, Q., Song, Y., Guo, Y., & Yang, X. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00056 · Cited by 22 · see citing works
“For example, [33] proposed RAIM by using the self-attention and guided-attention mechanisms, which used clinical events as the prior knowledge to regulate the continuous health status data.”
Influential
A novel hybrid GRU-CNN and residual bias (RB) based RB-GRU-CNN models for prediction of PTB Diagnostic ECG time series data
Khan, S., & Kumar, V. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106262 · Cited by 21 · see citing works
CardioLearn: A Cloud Deep Learning Service for Cardiac Disease Detection from Electrocardiogram
Hong, S., Fu, Z., Zhou, R., Jie, Y., Li, Y., Wang, K., & Cheng, G. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366424.3383529 · Cited by 21 · see citing works
Medication adherence prediction through temporal modelling in cardiovascular disease management
Hsu, W. W. Y., Warren, J., & Riddle, P. (2022) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-022-02052-9 · Cited by 23 · see citing works
“These studies include [56] where LSTM is used to model multivariate pediatric intensive care time series to predict diagnoses and [57] where LSTM is used to jointly analyse episodic clinical events and continuous monitoring data in the ICU settings to predict deterioration of patient conditions and their length of sta…”
Background
Modelling Patient Longitudinal Data for Clinical Decision Support: A Case Study on Emerging AI Healthcare Technologies
Niu, S., Ma, J., Yin, Q., Wang, Z., Bai, L., & Yang, X. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10513-x · Cited by 20 · see citing works
“In addition, the majority of current CDSS models (Xu et al., 2018; Choi et al., 2016; Ma et al., 2017) mainly process structured sequential EHRs, neglecting the abundant clinical information embedded in unstructured medical notes.”
Influential
Integrating Static and Time-Series Data in Deep Recurrent Models for Oncology Early Warning Systems
Li, D., Lyons, P. G., Klaus, J., Gage, B. F., Kollef, M. H., & Lu, C. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482441 · Cited by 19 · see citing works
“Existing models for integrating heterogeneous inputs are to concatenate the learned representations from each input [6, 9, 12, 13, 39] or to feed time-series inputs together with the learned representation of static inputs into RNN at each time step [20, 34].”
Background
Learning Saliency Maps to Explain Deep Time Series Classifiers
Parvatharaju, P. S., Doddaiah, R., Hartvigsen, T., & Rundensteiner, E. A. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482446 · Cited by 19 · see citing works
“As machine learning models have become more complex, the need for robust explanations of their predictions has become essential to garner the trust required for experts to react [6, 8, 9, 17, 37, 39, 40, 42, 44, 45, 47].”
Background
Interpreting patient-Specific risk prediction using contextual decomposition of BiLSTMs: application to children with asthma
AlSaad, R., Malluhi, Q. M., Janahi, I. A., & Boughorbel, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0951-4 · Cited by 19 · see citing works
“Attention-mechanism-based learning was used in [14– 20] for explaining what part of historical information weighs more in predicting future clinical events.”
Method
Prediction on critically ill patients: The role of “big data”
Bulgarelli, L., Deliberato, R. O., & Johnson, A. E. W. (2020) · Journal of Critical Care · doi:10.1016/j.jcrc.2020.07.017 · Cited by 18 · see citing works
Anomaly detection for sensor data of semiconductor manufacturing equipment using a GAN
Hashimoto, M., Ide, Y., & Aritsugi, M. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.08.090 · Cited by 18 · see citing works
Rare Disease Prediction by Generating Quality-Assured Electronic Health Records
Ma, F., Wang, Y., Gao, J., Xiao, H., & Zhou, J. L. (2020) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976236.58 · Cited by 17 · see citing works
Artificial Intelligence and Big Data Science in Neurocritical Care
Mainali, S., & Park, S. (2022) · Critical Care Clinics · doi:10.1016/j.ccc.2022.07.008 · Cited by 16 · see citing works
BaT: Beat-aligned Transformer for Electrocardiogram Classification
Li, X., Li, C., Wei, Y., Sun, Y., Wei, J., Li, X., & Qian, B. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00043 · Cited by 15 · see citing works
Predicting Risk of Mortality in Pediatric ICU Based on Ensemble Step-Wise Feature Selection
Hong, S., Hou, X., Jing, J., Ge, W., & Zhang, L. (2021) · Health Data Science · doi:10.34133/2021/9365125 · Cited by 14 · see citing works
“For general ICU patients, some methods mixed the whole population without age stratification [4, 6, 8], others removed patients who are younger than an age threshold [5, 7, 13].”
Method
DyHealth
Zheng, K., Cai, S., Chua, H., Herschel, M., Zhang, M., & Ooi, B. C. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3554821.3554835 · Cited by 14 · see citing works
“Further, with a guided multi-channel attention mechanism, RAIM provides interpretability for continuous monitoring data while leaving out the modality of time-series clinical data.”
BackgroundMethodInfluential
PyHealth: A Python Library for Health Predictive ModelsPreprint
Zhao, Y., Qiao, Z., Xiao, C., Glass, L. M., & Sun, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.04209 · Cited by 14 · see citing works
Self-citation
EBBE-Text: Explaining Neural Networks by Exploring Text Classification Decision Boundaries
Delaforge, A., Azé, J., Bringay, S., Mollévi, C., Sallaberry, A., & Servajean, M. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3184247 · Cited by 13 · see citing works
Temporal-Spatial Correlation Attention Network for Clinical Data Analysis in Intensive Care Unit
Nie, W., Yu, Y., Zhang, C., Song, D., Zhao, L., & Bai, Y. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3309956 · Cited by 12 · see citing works
“The calculation in the j-th Fusion-Encoder is as follows: The core is a multi-head cross-attention mechanism in which Q is derived from self-attentive output of the j-th time series X ij , K from the previous Fusion-Encoder’s output, and V from a mapping that combines Q and K.”
BackgroundMethodInfluential
Sequential Interpretability: Methods, Applications, and Future Direction for Understanding Deep Learning Models in the Context of Sequential DataPreprint
Shickel, B., & Rashidi, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12524 · Cited by 19 · see citing works
“Similar to RETAIN 71 and RAIM 73 , Qin et al. 74 implemented dual attention mechanisms along with LSTMs for forecasting real-valued financial time series that focused on both the timestep and the feature dimension of input multivariate sequences.”
MethodResult
DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data
Feng, Y., Xu, Z., Gan, L., Chen, N., Yu, B., Chen, T., & Wang, F. (2019) · 2019 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2019.00030 · Cited by 15 · see citing works
“values have been explored in many research to overcome the problems of multi-resolution and sampling irregularities of multimodal data, [3], [7].”
BackgroundMethod
Advances in Mining Heterogeneous Healthcare Data
Ma, F., Ye, M., Luo, J., Xiao, C., & Sun, J. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3470789 · Cited by 11 · see citing works
Self-citation
Strategies of Predictive Schemes and Clinical Diagnosis for Prognosis Using MIMIC-III: A Systematic Review
Khope, S. R., & Elias, S. (2023) · Healthcare · doi:10.3390/healthcare11050710 · Cited by 11 · see citing works
Stage-Aware Hierarchical Attentive Relational Network for Diagnosis Prediction
Wang, L., Liu, Q., Zhang, M., Hu, Y., Wu, S., & Wang, L. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310478 · Cited by 10 · see citing works
“To validate the effectiveness of the proposed HAR framework, without loss of generality, we choose four baseline models: LSTM, RETAIN [30], Dipole, [26], RAIM [59], StageNet [32], HiTANet [60].”
Method
Research on Multimodal Fusion of Temporal Electronic Medical Records
Ma, M., Wang, M., Gao, B., Li, Y., Huang, J., & Chen, H. (2024) · Bioengineering · doi:10.3390/bioengineering11010094 · Cited by 10 · see citing works
“In our future work, we plan to explore the applicability of the model in scenarios involving representations of other modalities of medical data, such as integrating clinical imaging or adapting the model for English-language datasets.”
Method
Addressing Noise and Skewness in Interpretable Health-Condition Assessment by Learning Model Confidence
Zhou, Y., Hong, S., Shang, J., Wu, M., Wang, Q., Li, H., & Xie, J. (2020) · Sensors · doi:10.3390/s20247307 · Cited by 10 · see citing works
“To explain the results of Deep Neural Network (DNN) models, most methods use the attention mechanism to highlight segments of an input record which are strongly associated with the model prediction and consider them as an explanation of prediction results [37,38], while others measure the role of each feature in the p…”
Method
Generalized and transferable patient language representation for phenotyping with limited data
Si, Y., Bernstam, E. V., & Roberts, K. E. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103726 · Cited by 10 · see citing works
Length of stay prediction for hospital management using domain adaptation
Momo, L. N. W., Moorosi, N., Nsoesie, E. O., Rademakers, F. E., & Moor, B. D. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108088 · Cited by 9 · see citing works
A transformer-based framework for temporal health event prediction with graph-enhanced representations
Liu, T., Liang, L., Che, C., Liu, Y., & Jin, B. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104826 · Cited by 10 · see citing works
DynEHR: Dynamic adaptation of models with data heterogeneity in electronic health records
Zhang, L., Chen, X., Chen, T., Wang, Z., & Mortazavi, B. J. (2021) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi50953.2021.9508558 · Cited by 10 · see citing works
“This heterogeneity is a challenge in machine learning and deep learning applications and is rarely addressed in existing work [7], [13], [15], [17].”
Background
DMRNet: Effective Network for Accurate Discharge Medication Recommendation
Shi, J., Wang, Y., Zhang, C., Luo, Z., Chai, C., & Zhang, M. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00262 · Cited by 8 · see citing works
“Speciﬁcally, in hospitals, Electronic Health Records (EHRs) [13]–[18], which contain different sources of structured data for the patients [19]–[22], have been widely adopted.”
Background
AID: Active Distillation Machine to Leverage Pre-Trained Black-Box Models in Private Data Settings
Hoang, T., Qiao, L., Xiao, C., Low, B., & Sun, J.-M. (2021) · The Web Conference · doi:10.1145/3442381.3449944 · Cited by 13 · see citing works
“First, attention-based methods often exploit the attention weights [5, 6, 27] to distil a local model for each subject (e.”
MethodSelf-citation
What went wrong and when? Instance-wise Feature Importance for Time-series ModelsPreprint
Tonekaboni, S., Joshi, S. D., Campbell, K., Duvenaud, D., & Goldenberg, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02821 · Cited by 8 · see citing works
“Attention models are suitable for sequential data and boost performance by learning to reliably learn from long range dependencies (Song et al., 2018; Kaji et al., 2019; Xu et al., 2018; Choi et al., 2016).”
Background
Integrating Physiological Time Series and Clinical Notes with Deep Learning for Improved ICU Mortality PredictionPreprint
Shukla, S. N., & Marlin, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11059 · Cited by 8 · see citing works
“Xu et al. (2018) developed clinical predictive models by integrating continuous monitoring data with discrete clinical event sequences.”
Method
CURENet: combining unified representations for efficient chronic disease prediction
Dao, C.-T., Phan, T. T., Ding, J.-E., Wu, C., Restrepo, D. S., Luo, D., Zhao, F., Liao, C.-C., Peng, W.-C., Wang, C., et al. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00396-w · Cited by 10 · see citing works
“By aligning LLM-derived semantic features with transformer-based temporal modeling, CURENet addresses a key limitation of prior single-modality or shallow fusion methods, which often fail to capture cross-modal dependencies [6, 7, 9, 44].”
Forecasting Treatment Outcomes Over Time Using Alternating Deep Sequential Models
Wu, F. C., Zhao, G., Zhou, Y., Qian, X., Elias, B.-K., & Lehman, L.-W. H. (2023) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2023.3331298 · Cited by 7 · see citing works
A deep clustering-based state-space model for improved disease risk prediction in personalized healthcare
Niu, S., Ma, J., Yin, Q., Bai, L., Chen, L., & Yang, X. (2024) · Annals of Operations Research · doi:10.1007/s10479-023-05817-1 · Cited by 7 · see citing works
How to Leverage Multimodal EHR Data for Better Medical Predictions?Preprint
Yang, B., & Wu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15763 · Cited by 7 · see citing works
“Xu et al. (2018) propose a recurrent attentive model to fuse continuous patient monitoring data, such as electrocardiogram (ECG), and discrete clinical events for predicting length of ICU stay.”
Background
ELDA: Learning Explicit Dual-Interactions for Healthcare Analytics
Cai, Q., Zheng, K., Ooi, B. C., Wang, W., & Yao, C. (2022) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde53745.2022.00034 · Cited by 6 · see citing works
Gated temporal convolutional neural network and expert features for diagnosing and explaining physiological time series: A case study on heart rates
Hong, S., Wang, C., & Fu, Z. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105847 · Cited by 6 · see citing works
Synthesizing Multimodal Electronic Health Records via Predictive Diffusion Models
Zhong, Y., Wang, X., Wang, J., Zhang, X., Wang, Y., Huai, M., Xiao, C., & Ma, F. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671836 · Cited by 5 · see citing works
Label-dependent and event-guided interpretable disease risk prediction using EHRs
Niu, S., Song, Y., Yin, Q., Guo, Y., & Yang, X. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669675 · Cited by 5 · see citing works
“[13] treated clinical events as interventions and also adopted the attention mechanism to generate the weighted embedding of electrocardiogram (ECG) for patients’ mortality prediction.”
BackgroundMethod
An intelligent adverse delivery outcomes prediction model based on the fusion of multiple obstetric clinical data
Zou, C., Zhang, Y., & Yuan, Z. (2023) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2023.2262663 · Cited by 5 · see citing works
“Most clinical prediction methods convert irregular time-series data into regular sequences (Xu et al. 2018; Tan et al. 2019).”
Attention-based Multimodal Fusion with Adversarial Network for In-Hospital Mortality Prediction
An, Y., Qiu, R., Lin, G., & Chen, X. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821815 · Cited by 5 · see citing works
Time-Guided High-Order Attention Model of Longitudinal Heterogeneous Healthcare Data
Huang, Y., Yang, X., & Xu, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-29908-8_5 · Cited by 5 · see citing works
“ep Learning on Heterogeneous EHRs data. [12] designed a heterogeneous LSTM structure to explore multiple inter-correlations of dierent medical sequences with dierent lengths and record frequencies. [25] proposed an e- cient multi-channel attention model of multimodal EHRs time series. However, these models only focus…”
BackgroundMethodInfluential
Explainable Multivariate Time Series Classification: A Deep Neural Network Which Learns To Attend To Important Variables As Well As Informative Time IntervalsPreprint
Hsieh, T.-Y., Wang, S., Sun, Y., & Honavar, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.11631 · Cited by 5 · see citing works
“RAIM [61] introduced a multi-channel attention mechanism guided by discrete clinical events to jointly analyze continuous monitoring data and discrete clinical events.”
PRIME: Pretraining for Patient Condition Representation with Irregular Multimodal Electronic Health Records
Li, B., Du, B., & Ye, J. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3744251 · Cited by 4 · see citing works
A Review of Deep Learning Methods for Irregularly Sampled Medical Time Series Data
Sun, C., Song, M., Cai, D., Zhang, B., Li, H., & Hong, S. (2026) · Health Data Science · doi:10.34133/hds.0456 · Cited by 55 · see citing works
“They [60, 61, 62] discretize the time axis into non-overlapping intervals with hand-designed intervals.”
Enabling Timely Medical Intervention by Exploring Health-Related Multivariate Time Series with a Hybrid Attentive Model
Xie, J., Wang, Z., Yu, Z., & Guo, B. (2022) · Sensors · doi:10.3390/s22166104 · Cited by 4 · see citing works
“Next, 1-D convolution is applied to the obtained segments to extract features P = conv(s), where s stands for g k or l (cl) k , P∈R U×J , U is the number of filters, and J is the length of the segment after convolution (a hyperparameter of CNN [29,34]).”
BackgroundMethodResultInfluential
A Disease Progression Prediction Model Based on EHR data
Zhao, F., Yu, X., Zhang, J., Li, X., & Li, R. (2022) · IEEE International Conference on High Performance Computing and Communications; IEEE International Conference on Smart City; IEEE International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc-dss-smartcity-dependsys57074.2022.00247 · Cited by 4 · see citing works
“Currently, EHR data play a major role in mortality prediction [2,3], disease diagnoses [4], length-of-stay forecasting [2,4,5] and admission prediction [6,7].”
Background
EvolveFNN: An Interpretable Framework for Early Detection Using Longitudinal Electronic Health Record Data
Zhang, Y., Wittrup, E. M., Hodgman, M. W., Mathis, M. R., & Najarian, K. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3551312 · Cited by 3 · see citing works
Explainable AI for the Operating Theater
Rudzicz, F., & Joshi, S. D. (2020) · Springer eBooks · doi:10.1007/978-3-030-49100-0_25 · Cited by 3 · see citing works
Machine learning for early prediction of circulatory failure in the intensive care unitPreprint
Hyland, S. L., Faltys, M., Hüser, M., Lyu, X., Gumbsch, T., Esteban, C., Bock, C., Horn, M., Moor, M., Rieck, B., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07990 · Cited by 3 · see citing works
Explainable Health Risk Predictor with Transformer-based Medicare Claim EncoderPreprint
Lahlou, C., Crayton, A., Trier, C., & Willett, E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09428 · Cited by 5 · see citing works
“A number of studies have leveraged dimensional reduction techniques, such as t-distributed stochastic neighbor embeddings (t-SNE), to construct clusters that may lend itself to an interpretation of learned representations [13, 25, 31, 41, 43, 50, 54].”
Background
Multimodal Artificial Intelligence for Precision Critical Care: A Scoping Review
Yang, M., Shi, N., Chen, H., Po, E. Y. P., Ng, P. Y., Lai, P. C. K., Wu, Y.-C., & Fong, D. Y. T. (2026) · Health Data Science · doi:10.34133/hds.0356 · Cited by 3 · see citing works
“…specific clinical tasks using a single data modality, such as using structured EHR data for sepsis prediction [ 9 ], patient subphenotyping [ 10 , 11 ], or medication dosing optimization [ 12 , 13 ], and utilizing medical imaging, such as MRI for neurological outcome assessment [ 14 , 15 ].”
GRU-TV: Time- and Velocity-aware Gated Recurrent Unit for patient representation
Liu, N., Gou, S., Gao, R., Su, B., Liu, W., Park, C., Xing, S., Yuan, J. Y., & Fenster, A. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104855 · Cited by 2 · see citing works
Time Series Classification Based on Supervised Contrastive Learning and Homoscedastic Uncertainty
Zhang, T., Li, K., & Wang, S. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3607901 · Cited by 2 · see citing works
“T IME series data exist in many real-world scenarios, including wind forecasting [1], stock forecasting [2], human activity recognition (HAR) [3], industrial fault diagnosis [4], [5], and healthcare [6].”

RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring DataPreprint
Xu, Y., Biswal, S., Deshpande, S. R., Maher, K. O., & Sun, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.08820 · Cited by 34 · see citing works
Self-citation
Integration of real-time and off-line clinical data in the MIMIC database
Moody, G. B., & Mark, R. G. (2002) · Computers in cardiology · doi:10.1109/cic.1997.648019 · Cited by 5 · see citing works
Information modeling for real-time decision support in intensive medicine
Santos, M. F., Portela, F., Vilas-Boas, M., Machado, J. M., Abelha, A., Neves, J., Silva, Á., & Rua, F. (2009) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) · Cited by 8 · see citing works
Integrating Data, Models, and Reasoning in Critical Care
Heldt, T., Long, B., Verghese, G. C., Peter Szolovits, B., & Mark, R. G. (2006) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/iembs.2006.259734 · Cited by 48 · see citing works
An integrated data mining approach to real-time clinical monitoring and deterioration warning
Mao, Y., Chen, W., Chen, Y., Lu, C., Kollef, M., & Bailey, T. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339709 · Cited by 76 · see citing works

Authors: Xu, Y., Biswal, S., Deshpande, S. R., Maher, K. O., & Sun, J.
Affiliations: Georgia Institute of Technology (US)
Emory University (US)
ORCID: Xu, Yanbo
Deshpande, Shriprasad R.
Maher, Kevin O.
Sun, Jimeng
Venue: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
Publisher: ACM
Published: 2018-07-19
Pages: 2565-2573
DOI: 10.1145/3219819.3220051
OpenAlex: W2884555939
Cited by: 144
References: 43
Citations by year: 2026: 13 · 2025: 18 · 2024: 15 · 2023: 11 · 2022: 15 · 2021: 26 · 2020: 31 · 2019: 14 · 2018: 1
FWCI: 7.16
Type: conference paper
Language: en
Open access: closed
Licence: www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations: ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
Funders: National Institutes of Health (1R01MD011682-01, R56HL138415)
National Science Foundation (IIS-#1418511, CCF-#1533768)
Topics: Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › ECG Monitoring and Analysis
Keywords: clinical decision support, intensive care unit, continuous patient monitoring, electrocardiogram, vital signs, drug information, interpretable models, heterogeneous data types

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Related

Record

About these checks