Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Understanding predictive models, in terms of interpreting and identifying actionable insights, is a challenging task. Often the importance of a feature in a model is only a rough estimate condensed into one number. However, our research goes beyond these naïve estimates through the design and implementation of an interactive visual analytics system, Prospector. By providing interactive partial dependence diagnostics, data scientists can understand how features affect the prediction overall. In addition, our support for localized inspection allows data scientists to understand how and why specific datapoints are predicted as they are, as well as support for tweaking feature values and seeing how the prediction responds. Our system is then evaluated using a case study involving a team of data scientists improving predictive models for detecting the onset of diabetes from electronic medical records.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 3×
“Partial dependence was proposed by Friedman [14] for analyzing gradient boosting models and has since been used for other models as well (e.”
MethodThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,370
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 2×
“Predictive modeling is defined as the process of developing a mathematical tool or model that generates an accurate prediction [24].”
MethodStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861
Research works printed but not linked: 8
- Cortez Paulo, IEEE Symposium on. 341-348
- Frank Eibe, Knowledge Discovery in Databases (PKDD), Nada Lavra, Dragan Gamberger, Ljupo Todorovski, and Hendrik Blockeel (Eds.)
- Kim Been, 2014, The Bayesian Case Model, In Advances in Neural Information Processing Systems., 1952
- Carson, Knowledge-Based and Intelligent Information & Engineering Systems 18th Annual Conference (KES), Proceedings of.
- 10.5555/2283516.2283682 · doi:10.5555/2283516.2283682
- Plate Tony, 1997, Progress in Connectionist-Based Information Systems: Proceedings of The Fourth International Conference on Neural Information Processing (ICONIP'97), 306
- Tzeng Fan-Yin, 2005, Proceedings of IEEE Visualization '05 Conference. IEEE, 383-390
- van den Elzen Stef, IEEE Conference on. 151-160
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
“This group of targeted users also includes researchers or workers who analyze data in specialized domains such as cybersecurity [36, 9], medical [19, 62], text analysis [76, 73], and image analysis [103].”
BackgroundAbdul, A., Vermeulen, J., Wang, D., Lim, B. Y., & Kankanhalli, M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174156 · Cited by 743
Model-Agnostic Interpretability of Machine Learning
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · arXiv · doi:10.48550/arxiv.1606.05386 · Cited by 699
“, 2010), perturbing inputs and seeing how the black box model reacts (Strumbelj & Kononenko, 2010; Krause et al., 2016), or both (Ribeiro et al.”
BackgroundMethodA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629
Notions of explainability and evaluation approaches for explainable artificial intelligence
Vilone, G., & Longo, L. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.009 · Cited by 598
The Building Blocks of Interpretability
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., & Mordvintsev, A. (2018) · Distill · doi:10.23915/distill.00010 · Cited by 591
The What-If Tool: Interactive Probing of Machine Learning Models
Wexler, J., Pushkarna, M., Bolukbasi, T., Wattenberg, M., Viegas, F., & Wilson, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934619 · Cited by 505
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 483
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 371
ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P. Y., Kalro, A., & Chau, D. H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 337
Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders
Cheng, H.-F., Wang, R., Zhang, Z., O'Connell, F., Gray, T., Harper, F. M., & Zhu, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300789 · Cited by 292
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3331846 · Cited by 278
Can we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, À. (2019) · IUI Workshops · Cited by 275
Explainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Arya, V., Bellamy, R. K. E., Chen, P.-Y., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2019) · arXiv · doi:10.48550/arxiv.1909.03012 · Cited by 268
“Lastly, several recent papers advocate for the vision of interactive explanation [32, 37, 42, 61], allowing users to drill down or ask for different types of explanation (e.g., through dialogues) until satisfied with their understanding.”
BackgroundExplainability fact sheets: a framework for systematic assessment of explainable approaches
Sokol, K., & Flach, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372870 · Cited by 264
explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
Spinner, T., Schlegel, U., Schafer, H., & El-Assady, M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934629 · Cited by 254
Seq2seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models
Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., & Rush, A. M. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865044 · Cited by 239
Self-citationMersha, M., Lam, K., Wood, J., AlShami, A. K., & Kalita, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128111 · Cited by 237
Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models
Hohman, F., Head, A., Caruana, R., DeLine, R., & Drucker, S. M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225
Explainability in human–agent systems
Rosenfeld, A., & Richardson, A. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09408-y · Cited by 224
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
Zhang, J., Wang, Y., Molino, P., Li, L., & Ebert, D. S. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864499 · Cited by 220
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
Meaningful Explanations of Black Box AI Decision Systems
Pedreschi, D., Giannotti, F., Guidotti, R., Monreale, A., Ruggieri, S., & Turini, F. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019780 · Cited by 213
Human-XAI Interaction: A Review and Design Principles for Explanation User Interfaces
Chromik, M., & Butz, A. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85616-8_36 · Cited by 210
Nasarian, E., Alizadehsani, R., Acharya, U. R., & Tsui, K.-L. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102412 · Cited by 207
iForest: Interpreting Random Forests via Visual Analytics
Zhao, X., Wu, Y., Lee, D. L., & Cui, W. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864475 · Cited by 203
“[30] recently introduces another heatmap-based design to represent PDPs, in which the color encodes prediction probabilities.”
BackgroundBig data hurdles in precision medicine and precision public health
Prosperi, M., Min, J. S., Bian, J., & Modave, F. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0719-2 · Cited by 200
“However, when integrated seamlessly in EHR as a clinical decisions support system and if they can identify clinically actionable features, they can be more acceptable [98].”
BackgroundHow do visual explanations foster end users' appropriate trust in machine learning?
Yang, F., Huang, Z., Scholtz, J., & Arendt, D. L. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377480 · Cited by 200
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200
Detecting Statistical Interactions from Neural Network WeightsPreprint
Tsang, M., Cheng, D., & Liu, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.04977 · Cited by 184
Sokol, K., & Flach, P. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00637-y · Cited by 186
Advancing Computational Toxicology by Interpretable Machine Learning
Jia, X., Wang, T., & Zhu, H. (2023) · Environmental Science & Technology · doi:10.1021/acs.est.3c00653 · Cited by 183
The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Rossi, F., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 182
Quantifying Climate and Catchment Control on Hydrological Drought in the Continental United States
Konapala, G., & Mishra, A. (2020) · Water Resources Research · doi:10.1029/2018wr024620 · Cited by 168
Impact of Lung Segmentation on the Diagnosis and Explanation of COVID-19 in Chest X-ray Images
Teixeira, L. O., Pereira, R. M., Bertolini, D. S., Oliveira, L. S., Nanni, L., Cavalcanti, G. D. C., & Costa, Y. M. G. (2021) · Sensors · doi:10.3390/s21217116 · Cited by 163
Survey on Visual Analysis of Event Sequence Data
Guo, Y., Guo, S., Jin, Z., Kaul, S., Gotz, D., & Cao, N. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3100413 · Cited by 157
Ethical machines: The human-centric use of artificial intelligence
Lepri, B., Oliver, N., & Pentland, A. (2021) · iScience · doi:10.1016/j.isci.2021.102249 · Cited by 149
Brik, B., Chergui, H., Zanzi, L., Devoti, F., Ksentini, A., Siddiqui, M. S., Costa‐Pérez, X., & Verikoukis, C. V. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3510543 · Cited by 151
Xie, Y., Chen, M., Kao, D., Gao, G., & Chen, X. '. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376807 · Cited by 141
Chowdhury, S., Joel-Edgar, S., Dey, P. K., Bhattacharya, S., & Kharlamov, A. (2023) · The International Journal of Human Resource Management · doi:10.1080/09585192.2022.2066981 · Cited by 140
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135
“Krause et al. [89], [104] proposed a technique called Prospector, which is an interactive visual analytics system.”
MethodRequirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132
Visual, textual or hybrid: the effect of user expertise on different explanations
Szymanski, M., Millecamp, M., & Verbert, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450662 · Cited by 131
Taylor, J. E. T., & Taylor, G. W. (2021) · Psychonomic Bulletin & Review · doi:10.3758/s13423-020-01825-5 · Cited by 130
FairSight: Visual Analytics for Fairness in Decision Making
Ahn, Y., & Lin, Y.-R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934262 · Cited by 126
Machine learning in cardiac CT: Basic concepts and contemporary data
Singh, G., Alaref, S. J., van Assen, M., Kim, T. S., van Rosendael, A., Kolli, K. K., Dwivedi, A., Maliakal, G., Pandey, M. K., Wang, J., et al. (2018) · Journal of cardiovascular computed tomography · doi:10.1016/j.jcct.2018.04.010 · Cited by 128
Naser, M. Z. (2021) · Fire Technology · doi:10.1007/s10694-020-01069-8 · Cited by 125
Obesity Prediction with EHR Data: A Deep Learning Approach with Interpretable Elements
Gupta, M., Phan, T.-L. T., Bunnell, H. T., & Beheshti, R. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3506719 · Cited by 127
DECE: Decision Explorer with Counterfactual Explanations for Machine Learning Models
Cheng, F., Ming, Y., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030342 · Cited by 122
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“The social nature of explanation also maps to an essential requirement for interactivity in XAI applications [57].”
BackgroundPereira, S., Meier, R., McKinley, R., Wiest, R., Alves, V., Silva, C. A., & Reyes, M. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.12.009 · Cited by 116
Research Trends in Artificial Intelligence Applications in Human Factors Health Care: Mapping Review
Asan, O., & Choudhury, A. (2021) · JMIR Human Factors · doi:10.2196/28236 · Cited by 114
Liu, H., Lai, V., & Tan, C. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479552 · Cited by 111
Visual Diagnosis of Tree Boosting Methods
Liu, S., Xiao, J., Liu, J., Wang, X., Wu, J., & Zhu, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744378 · Cited by 111
Supervised machine learning methods in psychology: A practical introduction with annotated R code
Rosenbusch, H., Soldner, F., Evans, A. M., & Zeelenberg, M. (2021) · Social and Personality Psychology Compass · doi:10.1111/spc3.12579 · Cited by 111
Designing a feature selection method based on explainable artificial intelligence
Zacharias, J., von Zahn, M., Chen, J., & Hinz, O. (2022) · Electronic Markets · doi:10.1007/s12525-022-00608-1 · Cited by 109
“The interactive visual analytics tool Prospector by Krause et al. (2016) focuses on explaining ML models by investigating the contributions of single features to the model’s output.”
MethodExplainable Active Learning (XAL): Toward AI Explanations as Interfaces for Machine Teachers
Ghai, B., Liao, Q. V., Zhang, Y., Bellamy, R., & Mueller, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3432934 · Cited by 106
“This idea underlies recent research into visual analytical tools that support debugging or feature engineering work [40, 47, 96].”
BackgroundHUMAN‐CENTERED DESIGN OF ARTIFICIAL INTELLIGENCE
Margetis, G., Ntoa, S., Antona, M., & Stephanidis, C. (2021) · HANDBOOK OF HUMAN FACTORS AND ERGONOMICS · doi:10.1002/9781119636113.ch42 · Cited by 106
Anik, A. I., & Bunt, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445736 · Cited by 103
Rationalization: A Neural Machine Translation Approach to Generating Natural Language Explanations
Ehsan, U., Harrison, B., Chan, L., & Riedl, M. O. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278736 · Cited by 99
Survey on Explainable AI: Techniques, challenges and open issues
Abusitta, A., Li, M. Q., & Fung, B. C. M. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124710 · Cited by 98
The State‐of‐the‐Art in Predictive Visual Analytics
Lu, Y., Garcia, R., Hansen, B., Gleicher, M., & Maciejewski, R. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13210 · Cited by 97
COGAM: Measuring and Moderating Cognitive Load in Machine Learning Model Explanations
Abdul, A., von der Weth, C., Kankanhalli, M., & Lim, B. Y. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376615 · Cited by 96
A Review of Guidance Approaches in Visual Data Analysis: A Multifocal Perspective
Ceneda, D., Gschwandtner, T., & Miksch, S. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13730 · Cited by 96
“Believe it or not: Designing a human-ai partnership for mixed-initiative fact-checking,”
Nguyen, A. T., Kharosekar, A., Krishnan, S., Krishnan, S., Tate, E., Wallace, B. C., & Lease, M. (2018) · Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3242587.3242666 · Cited by 95
ISeeU: Visually interpretable deep learning for mortality prediction inside the ICU
Caicedo-Torres, W., & Gutierrez, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103269 · Cited by 95
What’s up with Requirements Engineering for Artificial Intelligence Systems?
Ahmad, K., Bano, M., Abdelrazek, M., Arora, C., & Grundy, J. (2021) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re51729.2021.00008 · Cited by 95
What can AI do for me?: evaluating machine learning interpretations in cooperative play
Feng, S., & Boyd-Graber, J. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302265 · Cited by 92
Naser, M. Z., & Kodur, V. K. R. (2022) · Engineering Structures · doi:10.1016/j.engstruct.2021.113824 · Cited by 92
Explanation-Based Human Debugging of NLP Models: A Survey
Lertvittayakumjorn, P., & Toni, F. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00440 · Cited by 92
M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis
Wang, X., He, J., Jin, Z., Yang, M., Wang, Y., & Qu, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114794 · Cited by 91
Explaining Vulnerabilities to Adversarial Machine Learning through Visual Analytics
Ma, Y., Xie, T., Li, J., & Maciejewski, R. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934631 · Cited by 91
Ethically Responsible Machine Learning in Fintech
Rizinski, M., Peshov, H., Mishev, K., Chitkushev, L. T., Vodenska, I., & Trajanov, D. (2022) · IEEE Access · doi:10.1109/access.2022.3202889 · Cited by 90
A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations
Krause, J., Dasgupta, A., Swartz, J., Aphinyanaphongs, Y., & Bertini, E. (2017) · 2017 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585720 · Cited by 86
Self-citationRequirements engineering framework for human-centered artificial intelligence software systems
Ahmad, K., Abdelrazek, M., Arora, C., Baniya, A. A., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110455 · Cited by 85
Model extraction from counterfactual explanationsPreprint
Aïvodji, U., Bolot, A., & Gambs, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.01884 · Cited by 81
A Survey of Evaluation Methods and Measures for Interpretable Machine LearningPreprint
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv (Cornell University) · Cited by 80
Why these Explanations? Selecting Intelligibility Types for Explanation Goals.
Lim, B. Y., Yang, Q., Abdul, A., & Wang, D. (2019) · IUI Workshops · Cited by 80
Human-AI collaboration in route planning: An empirical efficiency-based analysis in retail logistics
Loske, D., & Klumpp, M. (2021) · International Journal of Production Economics · doi:10.1016/j.ijpe.2021.108236 · Cited by 80
Requirements practices and gaps when engineering human-centered Artificial Intelligence systems
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110421 · Cited by 80
Visual exploration of machine learning results using data cube analysis
Kahng, M., Fang, D., & Chau, D. H. (. (2016) · Proceedings of the Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/2939502.2939503 · Cited by 77
Risk assessment tools and data-driven approaches for predicting and preventing suicidal behavior
Velupillai, S., Hadlaczky, G., Baca-Garcia, E., Gorrell, G. M., Werbeloff, N., Nguyen, D., Patel, R., Leightley, D., Downs, J., Hotopf, M., et al. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00036 · Cited by 76
“Explanations should be tailored toward the specific task and the end users; employing and testing the output scores and explanations in a practical setting (60).”
BackgroundA Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning
Sperrle, F., El‐Assady, M., Guo, G., Borgo, R., Chau, D. H., Endert, A., & Keim, D. (2021) · Computer Graphics Forum · doi:10.1111/cgf.14329 · Cited by 75
Zając, H. D., Li, D., Dai, X., Carlsen, J. F., Kensing, F., & Andersen, T. O. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3582430 · Cited by 74
Human-centered explainability for life sciences, healthcare, and medical informatics
Dey, S., Chakraborty, P., Kwon, B. C., Dhurandhar, A., Ghalwash, M. F., Suárez, F., Ng, K., Sow, D., Varshney, K. R., & Meyer, P. (2022) · Patterns · doi:10.1016/j.patter.2022.100493 · Cited by 72
Self-citationInsideInsights: Integrating Data‐Driven Reporting in Collaborative Visual Analytics
Mathisen, A., Horak, T., Klokmose, C. N., Grønbæk, K., & Elmqvist, N. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13717 · Cited by 72
Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581314 · Cited by 71
Using Visual Analytics to Interpret Predictive Machine Learning ModelsPreprint
Krause, J., Perer, A., & Bertini, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05685 · Cited by 71
Self-citationCarePre: An Intelligent Clinical Decision Assistance System
Jin, Z., Cui, S., Guo, S., Gotz, D., Sun, J., & Cao, N. (2020) · ACM Transactions on Computing for Healthcare · doi:10.1145/3344258 · Cited by 70
“tion techniques to represent the internal structure of a deep learning model [31], [45], [30], and (2) instance-based analysis, which monitors changes to results in response to changes in model input [26], [36]. CarePre adopts the instance-based analysis approach via a set of interactive visualization designs that all…”
MethodVisualization for Recommendation Explainability: A Survey and New Perspectives
Chatti, M. A., Guesmi, M., & Muslim, A. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3672276 · Cited by 70
Interpreting Black-Box Classifiers Using Instance-Level Visual Explanations
Tamagnini, P., Krause, J., Dasgupta, A., & Bertini, E. (2017) · Proceedings of the 2nd Workshop on Human-In-the-Loop Data Analytics · doi:10.1145/3077257.3077260 · Cited by 69
Self-citationExplainExplore: Visual Exploration of Machine Learning Explanations
Collaris, D., & van Wijk, J. J. (2020) · IEEE Pacific Visualization Symposium · doi:10.1109/pacificvis48177.2020.7090 · Cited by 69
Metadata
- Authors
- Krause, J., Perer, A., & Ng, K.
- Affiliations
- New York University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5686-5697
- OpenAlex
- W2394669110
- Cited by
- 390
- References
- 50
- Citations by year
- 2026: 14 · 2025: 31 · 2024: 37 · 2023: 38 · 2022: 33 · 2021: 83 · 2020: 54 · 2019: 48 · 2018: 27 · 2017: 19 · 2016: 6
- FWCI
- 11.41
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- interactive visual analysis, partial dependence, model interpretability, feature importance, explainable machine learning, electronic health records, prospectors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.