The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Towards better analysis of machine learning models: A visual analytics perspective
Open accessAbstract
Interactive model analysis, the process of understanding, diagnosing, and refining a machine learning model with the help of interactive visualization, is very important for users to efficiently solve real-world artificial intelligence and data mining problems. Dramatic advances in big data analytics have led to a wide variety of interactive model analysis tasks. In this paper, we present a comprehensive analysis and interpretation of this rapidly developing area. Specifically, we classify the relevant work into three categories: understanding, diagnosis, and refinement. Each category is exemplified by recent influential work. Possible future research opportunities are also explored and discussed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
(2008) · Journal of Machine Learning Research · Cited by 33,051
Learning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,868
Wold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606
Human-level concept learning through probabilistic program induction
Lake, B. M., Salakhutdinov, R., & Tenenbaum, J. B. (2015) · Science · doi:10.1126/science.aab3050 · Cited by 2,817
Improving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591
Active Learning with Statistical Models
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 1,317
ImageNet classification with deep convolutional neural networks
Krizhevsky (2012) · Medical Entomology and Zoology · Cited by 799
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766
Employing EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689
Towards Better Analysis of Deep Convolutional Neural Networks
Liu, M., Shi, J., Li, Z., Li, C., Zhu, J., & Liu, S. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598831 · Cited by 498
Self-citationA survey on information visualization: recent advances and challenges
Liu, S., Cui, W., Wu, Y., & Liu, M. (2014) · The Visual Computer · doi:10.1007/s00371-013-0892-3 · Cited by 373
Self-citationTextFlow: Towards Better Understanding of Evolving Topics in Text
Cui, W., Liu, S., Tan, L., Shi, C., Song, Y., Gao, Z., Qu, H., & Tong, X. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.239 · Cited by 349
Self-citationApproximated and User Steerable tSNE for Progressive Visual Analytics
Pezzotti, N., Lelieveldt, B. P. F., Maaten, L. V. D., Hollt, T., Eisemann, E., & Vilanova, A. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2570755 · Cited by 308
Graying the black box: Understanding DQNsPreprint
Zahavy, T., Ben-Zrihem, N., & Mannor, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.02658 · Cited by 304
Visualizing the Hidden Activity of Artificial Neural Networks
Rauber, P. E., Fadel, S. G., Falcao, A. X., & Telea, A. C. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598838 · Cited by 280
The Role of Uncertainty, Awareness, and Trust in Visual Analytics
Sacha, D., Senaratne, H., Kwon, B. C., Ellis, G., & Keim, D. A. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467591 · Cited by 267
UTOPIAN: User-Driven Topic Modeling Based on Interactive Nonnegative Matrix Factorization
Choo, J., Lee, C., Reddy, C. K., & Park, H. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.212 · Cited by 262
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262
Progressive Visual Analytics: User-Driven Visual Exploration of In-Progress Analytics
Stolper, C. D., Perer, A., & Gotz, D. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346574 · Cited by 211
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200
#FluxFlow: Visual Analysis of Anomalous Information Spreading on Social Media
Zhao, J., Cao, N., Wen, Z., Song, Y., Lin, Y.-R., & Collins, C. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346922 · Cited by 178
OpinionSeer: Interactive Visualization of Hotel Customer Feedback
Wu, Y., Wei, F., Liu, S., Au, N., Cui, W., Zhou, H., & Qu, H. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.183 · Cited by 173
Self-citationHierarchicalTopics: Visually Exploring Large Text Collections Using Topic Hierarchies
Dou, W., Yu, L., Wang, X., Ma, Z., & Ribarsky, W. (2013) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.162 · Cited by 160
Self-citationAn Interactive Node-Link Visualization of Convolutional Neural Networks
Harley, A. W. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-27857-5_77 · Cited by 150
TargetVue: Visual Analysis of Anomalous User Behaviors in Online Communication Systems
Cao, N., Shi, C., Lin, S. C.-C., Lu, J., Lin, Y., & Lin, C. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467196 · Cited by 144
Visual Classifier Training for Text Document Retrieval
Heimerl, F., Koch, S., Bosch, H., & Ertl, T. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.277 · Cited by 141
Mühlbacher, T., Piringer, H., Gratzl, S., Sedlmair, M., & Streit, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346578 · Cited by 131
Visual Methods for Analyzing Probabilistic Classification Data
Alsallakh, B., Hanbury, A., Hauser, H., Miksch, S., & Rauber, A. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346660 · Cited by 119
TopicPanorama: A Full Picture of Relevant Topics
Wang, X., Liu, S., Liu, J., Chen, J., Zhu, J., & Guo, B. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2515592 · Cited by 109
Self-citationA framework for uncertainty-aware visual analytics
Correa, C. D., Chan, Y., & Ma, K. (2009) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2009.5332611 · Cited by 104
Opening the Black Box - Data Driven Visualization of Neural Networks
Tzeng, F.-Y., & Ma, K. (2006) · Visualization · doi:10.1109/visual.2005.1532820 · Cited by 104
Topic Model Diagnostics: Assessing Domain Relevance via Topical Alignment
Chuang, J., Gupta, S., Manning, C. D., & Heer, J. (2013) · Cited by 99
An Analysis of Machine- and Human-Analytics in Classification
Tam, G. K. L., Kothari, V., & Chen, M. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598829 · Cited by 92
Opening the Black Box - Data Driven Visualization of Neural Network
Tzeng, F.-Y., & Ma, K. (2006) · doi:10.1109/vis.2005.73 · Cited by 91
Turner, R. (2016) · Machine learning for signal processing ... · doi:10.1109/mlsp.2016.7738872 · Cited by 81
Online Visual Analytics of Text Streams
Liu, S., Yin, J., Wang, X., Cui, W., Cao, K., & Pei, J. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2509990 · Cited by 78
Self-citationTopicPanorama: A full picture of relevant topics
Liu, S., Wang, X., Chen, J., Zhu, J., & Guo, B. (2014) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2014.7042494 · Cited by 71
Self-citationAn Uncertainty-Aware Approach for Exploratory Microblog Retrieval
Liu, M., Liu, S., Zhu, X., Liao, Q., Wei, F., & Pan, S. (2015) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467554 · Cited by 69
Self-citationAn Approach to Supporting Incremental Visual Data Classification
Paiva, J. G. S., Schwartz, W. R., Pedrini, H., & Minghim, R. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2331979 · Cited by 68
VISUALIZING LEARNING AND COMPUTATION IN ARTIFICIAL NEURAL NETWORKS
Craven, M. W., & SHAVLIK, J. W. (1992) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213092000260 · Cited by 65
Improved Similarity Trees and their Application to Visual Data Classification
Paiva, J. G. S., Florian, L., Pedrini, H., Telles, G., & Minghim, R. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.212 · Cited by 52
Visualizing Flow of Uncertainty through Analytical Processes
Wu, Y., Yuan, G.-X., & Ma, K. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.285 · Cited by 48
NVIS: an interactive visualization tool for neural networks
Streeter, M. J. V., Ward, M. O., & Álvarez, S. A. (2001) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.424934 · Cited by 22
A Model Explanation System: Latest Updates and ExtensionsPreprint
Turner, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.09517 · Cited by 14
Research works printed but not linked: 8
- Chen, J., Zhu, J., Wang, Z., Zheng, X., Zhang, B. 2013. Scalable inference for logistic-normal topic models. In: NIPS
- Cui, 2014, How hierarchical topics evolve in large text corpora, IEEE TVCG, 20, 2281
- Dou, 2016, Topic-and time-oriented visual text analysis, IEEE Comput. Graph. Appl., 36, 8, 10.1109/MCG.2016.73 · doi:10.1109/mcg.2016.73
- Fekete, 2013, Visual analytics infrastructures: From data management to exploration, Computer, 46, 22, 10.1109/MC.2013.120 · doi:10.1109/mc.2013.120
- Liu, 2012, TIARA: Interactive, topic-based visual text summarization and analysis, ACM TIST, 3, 25:1
- Tzeng, F.Y., Ma, K.L. 2004. Acluster-space visual interface for arbitrary dimensional classification of volume data. In: Sixth Joint Eurographics - IEEE TCVG Conference on Visualization, pp. 17–24. http://dx.doi.org/10.2312/VisSym/VisSym04/017-024 · doi:10.2312/vissym/vissym04/017-024
- Wang, X., Liu, S., Song, Y., Guo, B. 2013. Mining evolutionary multi-branch trees from text streams. In: KDD, pp. 722–730 · doi:10.1145/2487575.2487603
- Wei, F., Li, W., Lu, Q., He, Y. 2008. Query-sensitive mutual reinforcement chain and its application in query-oriented multi-document summarization. In: SIGIR, pp. 283–290 · doi:10.1145/1390334.1390384
Cited by
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“Examples of lossless visualisation methods include dimensionality reduction techniques [135, 136] (such as PCA or t-SNE), scatter plots, and heat maps.”
MethodAli, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
The International Journal of Logistics Management
(2023) · The International Journal of Logistics Management · doi:10.1108/ijlm · Cited by 672
Vellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
Metadata
- Authors
- Liu, S., Wang, X., Liu, M., & Zhu, J.
- Venue
- Visual Informatics
- Publisher
- Elsevier BV
- Published
- 2017-03
- Volume
- 1
- Issue
- 1
- Pages
- 48-56
- ISSN
- 2468-502X (ISSN-L)
- OpenAlex
- W2587299461
- Cited by
- 397
- References
- 46
- FWCI
- 14.60
- Type
- journal article
- Language
- en
- Update notices
- erratum (2021-03-01)
- Open access
- diamond
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.