RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records
Open accessThe DOI matches its Crossref record; 103 of the 107 references it cites were found in a registry or an index.
Abstract
We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Even when interactive visual explanation tools were introduced (e.g., RetainVis [68]), these highlighted the need for more dynamic user evaluations beyond static metrics.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In our survey of the literature, only a handful of papers included formal user evaluations of their XAI components, and just one study explicitly investigated whether providing explanations actually helped users trust the system or use the model's output in their decisions [68].”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Making a complex algorithm understandable to end users can require simplification, which often comes at the cost of reduced accuracy of explanation [46,47,53,54,73].”
From Progressive Disclosure: When, Why, and How Do Users Want Algorithmic Transparency Information? (2020) · cites this paper as [46]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This critique builds on limitations that were often implicit in early XAI studies [38,45,46,66,68], where evaluations prioritized technical feasibility or proof-of-concept over robust behavioral assessment.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [68]
The cited source says what the claim attributes to it.
In the citing paper
“In addition, users can compare outcomes between patients and conduct what-if analyses by modifying patient attributes and seeing how predicted risk changes.”
From Progressive disclosure (2019) · cites this paper as [45]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 2×
“We trained our model on our training set using Adam [43] on learning rates of 0.01, 0.001, 0.0001 and hidden state sizes of 64, 128, 256, and tested them on the validation set to obtain the best performing hyperparameters and avoid overfitting.”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“Deep learning models provide interpretable results by showing attention [3, 58].”
BackgroundVisualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
(2017) · Cited by 11,625
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382 · cited 1×
“Deep learning models provide interpretable results by showing attention [3, 58].”
Background17. A Value for n-Person Games
Shapley, L. S. (1953) · Contributions to the Theory of Games (AM-28), Volume II · doi:10.1515/9781400881970-018 · Cited by 6,950
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“In case of deep learning models, it is difficult to describe the relationship between the input and the output [54].”
BackgroundDeep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
Shapley, L. S. (1988) · The Shapley Value · doi:10.1017/cbo9780511528446.003 · Cited by 4,954 · cited 1×
“An alternative form of interpretation is model-agnostic approaches such as partial dependence plot [27], Shapley values [71] and LIME [67], where they provide some form of explanation as to how a set of features affect the model output.”
MethodEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 1×
“Though the most prevalent use of deep learning techniques in medical domains is to predict diagnosis of a disease, such as breast cancer [1,28] and brain tumor [29,37] by training models on medical images, there has also been an increase in deep learning applications for longitudinal EMR data.”
BackgroundBrain tumor segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.05.004 · Cited by 3,278
Brain Tumor Segmentation with Deep Neural Networks
Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.-M., & Larochelle, H. (2015) · arXiv · doi:10.48550/arxiv.1505.03540 · Cited by 3,278
Obesity and the risk of heart failure
Kenchaiah, S., Evans, J. C., Levy, D., Wilson, P. W. F., Benjamin, E. J., Larson, M. G., Kannel, W. B., & Vasan, R. S. (2002) · New England Journal of Medicine · doi:10.1056/nejmoa020245 · Cited by 3,093
The progression from hypertension to congestive heart failure
Levy, D. (1996) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1996.03530440037034 · Cited by 2,155 · cited 1×
“The top-5 S1 scores in diagnosis support the premise that hypertensive disease are associated with heart failure, as well as being a major cause of other diseases and comorbidities [38, 52].”
BackgroundWhat's In A Name? Malay Seals As Onomastic Sources
Library, T. B., & Gallop, A. T. (2018) · Malay Literature · doi:10.37052/ml.31(1)no1 · Cited by 1,483
Klein, I., & Danzi, S. (2007) · Circulation · doi:10.1161/circulationaha.106.678326 · Cited by 1,324
RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism
Choi, E., Bahadori, M. T., Kulas, J. A., Schuetz, A., Stewart, W. F., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1608.05745 · Cited by 1,029
Self-citationClassification of breast cancer histology images using Convolutional Neural Networks
Araújo, T., Aresta, G., Castro, E., Rouco, J., Aguiar, P., Eloy, C., Polónia, A., & Campilho, A. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177544 · Cited by 1,013 · cited 1×
“Though the most prevalent use of deep learning techniques in medical domains is to predict diagnosis of a disease, such as breast cancer [1, 28] and brain tumor [29, 37] by training models on medical images, there has also been an increase in deep learning applications for longitudinal EMR data.”
BackgroundBreast cancer histopathological image classification using Convolutional Neural Networks
Spanhol, F. A., Oliveira, L. S., Petitjean, C., & Heutte, L. (2016) · 2016 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727519 · Cited by 942
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928
Self-citationLearning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · cited 2×
“In this paper, we focus on using such sequential data on a prediction task, ‘learning to diagnose(L2D)’ [55], where a model observes the visits of a patient and returns a prediction score indicating the probability of the patient being diagnosed with a target disease in near future.”
BackgroundVisualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845 · cited 1×
“Though it is possible to visualize the activations of hidden state cells [39, 61, 75], they do not propose the level of interpretability as in CNNs.”
BackgroundDesign Study Methodology: Reflections from the Trenches and the Stacks
Sedlmair, M., Meyer, M., & Munzner, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.213 · Cited by 826 · cited 1×
“We initially generated research questions of our target users’ potential interest (led by medical experts), derived visual analytics tasks (led by deep learning and visual analytics experts), and then further evaluated them by closely following a design study methodology [70].”
MethodAsk, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
Xu, H., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_28 · Cited by 727 · cited 1×
“Attention has proven to improve the performance of many sequence-tosequence based deep learning tasks such as machine translation [23], question answering [83], and speech generation [80].”
BackgroundBreast Cancer Multi-classification from Histopathological Images with Structured Deep Learning Model
Han, Z., Wei, B., Zheng, Y., Yin, Y., Li, K., & Li, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-04075-z · Cited by 683 · cited 1×
“Though the most prevalent use of deep learning techniques in medical domains is to predict diagnosis of a disease, such as breast cancer [1,28] and brain tumor [29,37] by training models on medical images, there has also been an increase in deep learning applications for longitudinal EMR data.”
BackgroundAutomatic Detection of Cerebral Microbleeds From MR Images via 3D Convolutional Neural Networks
Dou, Q., Chen, H., Yu, L., Zhao, L., Qin, J., Wang, D., Mok, V. C., Shi, L., & Heng, P.-A. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528129 · Cited by 679
GRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65,76], risk prediction [12,16,34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSelf-citationKim, L., Kim, J.-A., & Kim, S. (2014) · Epidemiology and Health · doi:10.4178/epih/e2014008 · Cited by 618
Self-citationPatient Subtyping via Time-Aware LSTM Networks
Baytas, I. M., Xiao, C., Zhang, X., Wang, F., Jain, A. K., & Zhou, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3097997 · Cited by 613 · cited 1×
“The first representation was introduced in [15] where time interval information was incorporated to RETAIN, and the latter two were proposed by [4].”
BackgroundRole of Blood Pressure in the Development of Congestive Heart Failure: The Framingham Study
Kannel, W. B., Castelli, W. P., McNamara, P. M., McKee, P. A., & Feinleib, M. (1972) · New England Journal of Medicine · doi:10.1056/nejm197210192871601 · Cited by 603
Ma, F., Chitta, R., Zhou, J., You, Q., Sun, T., & Gao, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098088 · Cited by 576 · cited 2×
“While there exist other models for EMR data that suggest interpretability such as Dipole [59], the level of inter-pretability is limited to each visit, thus failing to provide a complete decomposition of the prediction as RETAIN.”
Background(2016) · PubMed · Cited by 576
Self-citationRace, sex and occlusive cerebrovascular disease: a review.
Caplan, L. R., Gorelick, P. B., & Hier, D. B. (1986) · Stroke · doi:10.1161/01.str.17.4.648 · Cited by 553 · cited 1×
“as hypertension is a common characteristic of both diseases [5, 72].”
BackgroundTowards Better Analysis of Deep Convolutional Neural Networks
Liu, M., Shi, J., Li, Z., Li, C., Zhu, J., & Liu, S. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598831 · Cited by 498
Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
Zintgraf, L. M., Cohen, T. S., Adel, T., & Welling, M. (2017) · arXiv · doi:10.48550/arxiv.1702.04595 · Cited by 487 · cited 1×
“Especially for vision applications where convolutional neural networks (CNNs) have enjoyed a great success, many visualization methods such as heatmaps [85], blurring [79], dimensionality reduction [57, 66] of the filters and activation maps obtained during computation and backpropagation, and visualizing the model st…”
MethodMulti-layer Representation Learning for Medical Concepts
Choi, E., Bahadori, M. T., Searles, E., Coffey, C., Thompson, M., Bost, J., Tejedor-Sojo, J., & Sun, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939823 · Cited by 469 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65, 76], risk prediction [12, 16, 34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSelf-citationDoughty, R. N., Whalley, G. A., Gamble, G., MacMahon, S., & Sharpe, N. (1997) · Journal of the American College of Cardiology · doi:10.1016/s0735-1097(97)00012-0 · Cited by 422
Cleland, J. G. F., Findlay, I., Jafri, S., Sutton, G., Falk, R., Bulpitt, C., Prentice, C., Ford, I., Trainer, A., & Poole-Wilson, P. A. (2004) · American Heart Journal · doi:10.1016/j.ahj.2004.03.010 · Cited by 407 · cited 1×
“Bisoprolol, a medicine ingredient for which a relatively high S 1 score was recorded, is frequently prescribed for heart disease [22], while aspirin and atorvastatin are commonly used to prevent it [20].”
BackgroundSilent cerebrovascular disease in the elderly. Correlation with ambulatory pressure.
Shimada, K., Kawamoto, A., Matsubayashi, K., & Ozawa, T. (1990) · Hypertension · doi:10.1161/01.hyp.16.6.692 · Cited by 380 · cited 1×
“as hypertension is a common characteristic of both diseases [5, 72].”
BackgroundLSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 371 · cited 1×
“Though it is possible to visualize the activations of hidden state cells [39, 61, 75], they do not propose the level of interpretability as in CNNs.”
Background$\mathtt {Deepr}$: A Convolutional Net for Medical Records
Nguyen, P., Tran, T., Wickramasinghe, N., & Venkatesh, S. (2017) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2633963 · Cited by 352
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P. Y., Kalro, A., & Chau, D. H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 337 · cited 1×
“This led to a large number of studies dedicated to developing visualization frameworks that help users better understand their networks [19, 33, 35, 64, 74].”
BackgroundInteractive deep learning method for segmenting moving objects
Wang, Y., Luo, Z., & Jodoin, P.-M. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2016.09.014 · Cited by 334 · cited 1×
“However, there are only a small number of studies that apply user interaction to tasks requiring deep learning models, such as object segmentation [81].”
BackgroundVisualizing Dataflow Graphs of Deep Learning Models in TensorFlow
Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mane, D., Fritz, D., Krishnan, D., Viegas, F. B., & Wattenberg, M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744878 · Cited by 331 · cited 1×
“Especially for vision applications where convolutional neural networks (CNNs) have enjoyed a great success, many visualization methods such as heatmaps [85], blurring [79], dimensionality reduction [57, 66] of the filters and activation maps obtained during computation and backpropagation, and visualizing the model st…”
MethodYusuf, S. (1988) · JAMA · doi:10.1001/jama.260.15.2259 · Cited by 297
Visualizing the Hidden Activity of Artificial Neural Networks
Rauber, P. E., Fadel, S. G., Falcao, A. X., & Telea, A. C. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598838 · Cited by 280 · cited 1×
“…neural networks (CNNs) have enjoyed a great success, many visualization methods such as heatmaps [85], blurring [79], dimensionality reduction [57, 66] of the filters and activation maps obtained during computation and backpropagation, and visualizing the model structure itself [82] were used.”
MethodThe Role of Uncertainty, Awareness, and Trust in Visual Analytics
Sacha, D., Senaratne, H., Kwon, B. C., Ellis, G., & Keim, D. A. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467591 · Cited by 267 · cited 1×
“Domain experts often hesitate to accept what they see as facts since they are unable to tell various uncertainties propagated through the pipeline [69].”
BackgroundSelf-citationUTOPIAN: User-Driven Topic Modeling Based on Interactive Nonnegative Matrix Factorization
Choo, J., Lee, C., Reddy, C. K., & Park, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.212 · Cited by 262 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundSelf-citationDoctor AI: Predicting Clinical Events via Recurrent Neural Networks
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F., & Sun, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05942 · Cited by 244 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65, 76], risk prediction [12, 16, 34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSelf-citationDeep Computational Phenotyping
Che, Z., Kale, D., Li, W., Bahadori, M. T., & Liu, Y. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783365 · Cited by 239 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65, 76], risk prediction [12, 16, 34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSafety and efficacy of carvedilol in severe heart failure
Cohn, J. N., Fowler, M. B., Bristow, M. R., Colucci, W. S., Gilbert, E. M., Kinhal, V., Krueger, S. K., Lejemtel, T., Narahara, K. A., Packer, M., et al. (1997) · Journal of Cardiac Failure · doi:10.1016/s1071-9164(97)90013-0 · Cited by 221
Understanding Hidden Memories of Recurrent Neural Networks
Ming, Y., Cao, S., Zhang, R., Li, Z., Chen, Y., Song, Y., & Qu, H. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585721 · Cited by 207
Obesity and the risk of heart failure
Kenchaiah, S. (2003) · Biomedecine & Pharmacotherapy · doi:10.1016/s0753-3322(03)00066-0 · Cited by 178
Visual Classifier Training for Text Document Retrieval
Heimerl, F., Koch, S., Bosch, H., & Ertl, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.277 · Cited by 177 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18,45,49].”
BackgroundVisualizing and Understanding Neural Machine Translation
Ding, Y., Liu, Y., Luan, H., & Sun, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1106 · Cited by 168 · cited 1×
“Attention has proven to improve the performance of many sequence-tosequence based deep learning tasks such as machine translation [23], question answering [83], and speech generation [80].”
BackgroundDeepEyes: Progressive Visual Analytics for Designing Deep Neural Networks
Pezzotti, N., Hollt, T., Van Gemert, J., Lelieveldt, B. P. F., Eisemann, E., & Vilanova, A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744358 · Cited by 165 · cited 1×
“This led to a large number of studies dedicated to developing visualization frameworks that help users better understand their networks [19, 33, 35, 64, 74].”
BackgroundWhat you see is what you can change: Human-centered machine learning by interactive visualization
Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Peltonen, J., Weiskopf, D., North, S. C., & Keim, D. A. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.105 · Cited by 164 · cited 1×
“A topic of emerging importance in the visual analytics field is to integrate machine learning models with various user interactions [68].”
BackgroundChe, Z., Cheng, Y., Zhai, S., Sun, Z., & Liu, Y. (2017) · 2017 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2017.93 · Cited by 162
iVisClustering: An Interactive Visual Document Clustering via Topic Modeling
Lee, H., Kihm, J., Choo, J., Stasko, J., & Park, H. (2012) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2012.03108.x · Cited by 160 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundSelf-citationPredicting the Risk of Heart Failure With EHR Sequential Data Modeling
Jin, B., Che, C., Liu, Z., Zhang, S., Yin, X., & Wei, X. (2018) · IEEE Access · doi:10.1109/access.2017.2789324 · Cited by 160
CheckViz: Sanity Check and Topological Clues for Linear and Non‐Linear Mappings
Lespinats, S., & Aupetit, M. J. M. (2010) · Computer Graphics Forum · doi:10.1111/j.1467-8659.2010.01835.x · Cited by 134 · cited 2×
“, CheckViz [51]) and to resolve the issues by providing some guidance [2, 31].”
BackgroundClustervision: Visual Supervision of Unsupervised Clustering
Kwon, B. C., Eysenbach, B., Verma, J., Ng, K., De Filippi, C., Stewart, W. F., & Perer, A. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745085 · Cited by 132
Self-citationBlood Pressure and Cholesterol as Coronary Heart Disease Risk Factors
Tyroler, H. A., Heyen, S., Bartel, A. G., Cassel, J. C., Cornoni, J. C., Hames, C. G., & Kleinbaum, D. (1971) · Archives of Internal Medicine · doi:10.1001/archinte.1971.00310240061007 · Cited by 117
Bilirubin as a prognostic marker in patients with pulmonary arterial hypertension
Takeda, Y., Takeda, Y., Tomimoto, S., Tani, T., Narita, H., & Kimura, G. (2010) · BMC Pulmonary Medicine · doi:10.1186/1471-2466-10-22 · Cited by 93 · cited 1×
“The group also showed high contribution scores of bilirubin, suspected as a predictive marker of pulmonary arterial hypertension [77].”
BackgroundThe human touch: How non-expert users perceive, interpret, and fix topic models
Lee, T. Y., Smith, A., Seppi, K., Elmqvist, N., Boyd-Graber, J., & Findlater, L. (2017) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2017.03.007 · Cited by 92 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18,45,49].”
BackgroundProgressive Learning of Topic Modeling Parameters: A Visual Analytics Framework
El‐Assady, M., Sevastjanova, R., Sperrle, F., Keim, D. A., & Collins, C. M. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745080 · Cited by 91 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundDubach, P., MYERS, J. N., Bonetti, P. O., Schertler, T., Froelicher, V. F., Wagner, D., Scheidegger, M. B., Stuber, M., Luechinger, R., Schwitter, J., et al. (2002) · American Heart Journal · doi:10.1067/mhj.2002.121269 · Cited by 90
GRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07012 · Cited by 90 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65,76], risk prediction [12,16,34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSelf-citationÖzbaran, M., Omay, S. B., Nalbantgil, S., Kultursay, H., Kumanlioglu, K., Nart, D., & Pektok, E. (2004) · European Journal of Cardio-Thoracic Surgery · doi:10.1016/j.ejcts.2003.11.038 · Cited by 81
Direct-Manipulation Visualization of Deep Networks
Smilkov, D., Carter, S., Sculley, D., Viégas, F., & WATTENBERG, M. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03788 · Cited by 81 · cited 1×
“ion, and visualizing the model structure itself [82] were used. This led to a large number of studies dedicated to developing visualization frameworks that help users better understand their networks [19,33,35,64,74]. Compared to CNNs, RNNs have received less attention in visualization, mainly because of its intertwin…”
BackgroundCondensed Memory Networks for Clinical Diagnostic Inferencing
Prakash, A., Zhao, S., Hasan, S., Datla, V., Lee, K., Qadir, A., Liu, J., & Farri, O. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10964 · Cited by 76 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65,76], risk prediction [12,16,34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodSuo, Q., Ma, F., Canino, G., Gao, J., Zhang, A., Veltri, P., & Agostino, G. (2017) · PubMed Central · Cited by 73
AxiSketcher: Interactive Nonlinear Axis Mapping of Visualizations through User Drawings
Kwon, B. C., Kim, H., Wall, E., Choo, J., Park, H., & Endert, A. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598446 · Cited by 72
Self-citationVisualBackProp: visualizing CNNs for autonomous driving.Preprint
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Jackel, L., Müller, U., & Zieba, K. (2016) · arXiv (Cornell University) · Cited by 69
Chati, Z., Zannad, F. M., Jeandel, C., Robin-Lherbier, B., Escanyé, J., Robert, J., & Aliot, E. (1996) · American Heart Journal · doi:10.1016/s0002-8703(96)90537-2 · Cited by 65 · cited 1×
“Metabolic disorders, such as hemochromatosis, for which a relatively high S1 score was observed, were also shown to be likely to cause heart failure as a complication [8].”
BackgroundWang, W., Xu, S., & Xu, B. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-134 · Cited by 64 · cited 1×
“Attention has proven to improve the performance of many sequence-tosequence based deep learning tasks such as machine translation [23], question answering [83], and speech generation [80].”
BackgroundDeveloping the Inpatient Sample for the National Health Insurance Claims Data
Kim, L., Sakong, J., Kim, Y. H., Kim, S., Kim, S., Tchoe, B., Jeong, H., & Lee, T. R. (2013) · Health Policy and Management · doi:10.4332/kjhpa.2013.23.2.152 · Cited by 61 · cited 7×
“The representativeness and validity of this sample dataset have been confirmed by thorough evaluation against the entire patient population of Korea [42].”
BackgroundInfluentialSelf-citationVisualizing deep neural network by alternately image blurring and deblurring
Feng, W., Liu, H., & Cheng, J. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.09.007 · Cited by 54 · cited 1×
“Especially for vision applications where convolutional neural networks (CNNs) have enjoyed a great success, many visualization methods such as heatmaps [85], blurring [79], dimensionality reduction [57, 66] of the filters and activation maps obtained during computation and backpropagation, and visualizing the model st…”
MethodVisOHC: Designing Visual Analytics for Online Health Communities
Kwon, B. C., Kim, S.-H., Lee, S., Choo, J., Huh, J., & Yi, J. S. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467555 · Cited by 48
Self-citationRCLens: Interactive Rare Category Exploration and Identification
Lin, H., Gao, S., Gotz, D., Du, F., He, J., & Cao, N. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2711030 · Cited by 48 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18,45,49].”
BackgroundBridging the Gap of Domain and Visualization Experts with a Liaison
Simon, S., Mittelstädt, S., Keim, D. A., & Sedlmair, M. (2015) · Eurographics · doi:10.2312/eurovisshort.20151137 · Cited by 36 · cited 1×
“In particular, all leading authors, who were experts in two of the three domains of interest (i.e., deep learning, visual analytics, medical domains), each played the liaison role to fill the gap between domain and technical areas, as Simon et al. [73] suggests.”
BackgroundCho, N. H., Kim, J. Y., Kim, S. S., & Shin, C. (2013) · Diabetes & Metabolic Syndrome Clinical Research & Reviews · doi:10.1016/j.dsx.2013.08.003 · Cited by 34 · cited 1×
“Recently, there have been studies investigating the relationship between the Sasang types and prediction of cardiovascular disease [11].”
BackgroundSelf-citationWeakly supervised nonnegative matrix factorization for user-driven clustering
Choo, J., Lee, C., Reddy, C. K., & Park, H. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0384-8 · Cited by 32 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundSelf-citationProxiLens: Interactive Exploration of High-Dimensional Data using Projections
Heulot, N., Aupetit, M. J. M., & Fekete, J. (2013) · HAL · doi:10.2312/pe.vamp.vamp2013.011-015 · Cited by 32 · cited 1×
“, CheckViz [51]) and to resolve the issues by providing some guidance [2, 31].”
BackgroundPhenotyping of Clinical Time Series with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., & Wetzel, R. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.07641 · Cited by 29 · cited 1×
“RNN-based models have been extensively used for tasks such as patient diagnosis [65, 76], risk prediction [12, 16, 34], representation learning [13, 14] and patient phenotyping [9, 36, 56], outperforming rule-based and conventional machine learning baselines.”
MethodData programming with DDLite: putting humans in a different part of the loop
Ehrenberg, H. R., Shin, J., Ratner, A., Fries, J., & Ré, C. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939515 · Cited by 26 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundRole of blood pressure in the development of congestive heart failure: The framingham study
Kannel, W. B., Castelli, W. P., McNamara, P. M., McKee, P. A., & Feinleib, M. (1972) · The American Journal of Cardiology · doi:10.1016/0002-9149(72)90712-6 · Cited by 25
Causal Phenotype Discovery via Deep Networks.
Kale, D. C., Che, Z., Bahadori, M. T., Li, W., Liu, Y., & Wetzel, R. C. (2015) · PubMed Central · Cited by 24
Effects of bisoprolol in patients with stable congestive heart failure
de Groote, P., Delour, P., Lamblin, N., Dagorn, J., Verkindère, C., Tison, E., Millaire, A., & Bauters, C. (2004) · Annales de Cardiologie et d Angéiologie · doi:10.1016/j.ancard.2004.03.003 · Cited by 21 · cited 1×
“Bisoprolol, a medicine ingredient for which a relatively high S1 score was recorded, is frequently prescribed for heart disease [22], while aspirin and atorvastatin are commonly used to prevent it [20].”
BackgroundShapeShop: Towards Understanding Deep Learning Representations via Interactive Experimentation
Hohman, F., Hodas, N. O., & Chau, D. H. (2017) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3027063.3053103 · Cited by 17 · cited 1×
“This led to a large number of studies dedicated to developing visualization frameworks that help users better understand their networks [19, 33, 35, 64, 74].”
BackgroundRisk Factor Analysis Based on Deep Learning Models
Suo, Q., Xue, H., Gao, J., & Zhang, A. (2016) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/2975167.2975208 · Cited by 16
Personalized acupuncture treatment with Sasang typology
Chae, H., Lee, J., Jeon, E. S., & Kim, J. K. (2017) · Integrative Medicine Research · doi:10.1016/j.imr.2017.07.002 · Cited by 16 · cited 1×
“The Sasang typology is a traditional Korean personalized medicine, which aims to cluster patients into four groups based on patient’s phenotypic characteristics [7].”
BackgroundSelf-citationA multidimensional brush for scatterplot data analytics
Aupetit, M. J. M., Heulot, N., & Fekete, J. (2014) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2014.7042500 · Cited by 15 · cited 1×
“, CheckViz [51]) and to resolve the issues by providing some guidance [2, 31].”
BackgroundLearning Clinical Workflows to Identify Subgroups of Heart Failure Patients.
Yan, C., Chen, Y., Li, B., Liebovitz, D., & Malin, B. (2016) · PubMed Central · Cited by 14
Exploring Dimensionality Reductions with Forward and Backward ProjectionsPreprint
Cavallo, M., & Demiralp, Ç. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.04281 · Cited by 4 · cited 1×
“There have been a number of studies to develop tools where users can interact with the results of machine learning tasks such as classification [24,30,53], topic modeling [17,25,50], dimensionality reduction [6,46] and clustering [18, 45, 49].”
BackgroundIschemic heart disease, factor predisposing to Barrett’s adenocarcinoma: A case control study
Tsibouris, P. (2014) · World Journal of Gastrointestinal Pharmacology and Therapeutics · doi:10.4292/wjgpt.v5.i3.183 · Cited by 3
Effects of bisoprolol in patients with stable congestive heart failure
DEGROOTE, P. (2004) · Annales de Cardiologie et d Angéiologie · doi:10.1016/s0003-3928(04)00069-1 · Cited by 3
Research works printed but not linked: 5
- [74] D. Smilkov, S. Carter, D. Sculley, F. B. Vi´ egas, and M. Wattenberg. Directmanipulation visualization of deep networks. In International Conference on Machine Learning , 2016. -
- [26] T. Fawcett. Roc graphs: Notes and practical considerations for researchers. Machine learning , 31(1):1-38, 2004. -
- [54] Z. C. Lipton. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 , 2016. - · arXiv
- [19] S. Chung, C. Park, S. Suh, K. Kang, J. Choo, and B. C. Kwon. Re-V ACNN: Steering convolutional neural network via real-time visual analytics. In Future of Interactive Learning Machines Workshop at the 30th Annual Conference on Neural Information Processing Systems , 2016. -
- [48] B. C. Kwon, J. Verma, and A. Perer. Peekquence: Visual analytics for event sequence data. ACM SIGKDD Workshop on Interactive Data Exploration and Analytics , 2016. -
Cited by
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
“Kwon et al. [77] proposed a visually interpretable cardiac failure and cataract risk prediction model based on RETAIN (RetainVis).”
MethodVellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
Interpretability of machine learning-based prediction models in healthcare
Stiglic, G., Kocbek, P., Fijacko, N., Zitnik, M., Verbert, K., & Cilar, L. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1379 · Cited by 541
“RetainVis (Kwon et al., 2018) is a prominent example of a model-specific VA approach that visualizes a recurrent neural network to support interpretation and diagnosis of the model.”
BackgroundReal-world data: A brief review of the methods, applications, challenges and opportunities
Liu, F., & Panagiotakos, D. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01768-6 · Cited by 496
“…approaches to learn patterns, make new discoveries, assist preoperative planning, diagnostics, clinical prognostication, among others [18–27], improve predictions in selected outcomes especially if linked with administrative and claim data and usage of proper machine learning techniques…”
BackgroundŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
Metadata
- Authors
- Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J.
- Affiliations
- IBM (United States) (US)
- Korea University (KR)
- Georgia Institute of Technology (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2019-01
- Volume
- 25
- Issue
- 1
- Pages
- 299-309
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W2804604520
- Cited by
- 294
- References
- 107
- Citations by year
- 2026: 17 · 2025: 20 · 2024: 38 · 2023: 48 · 2022: 53 · 2021: 53 · 2020: 43 · 2019: 18
- FWCI
- 14.64
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Korea Electric Power Corporation (R18XA05)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- visual analytics, recurrent neural networks, electronic health records, interpretability, interactive machine learning, ReTain, black-box models, clinical risk prediction, heart failure, cataract, temporal information, clinical codes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.