Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Understanding predictive models, in terms of interpreting and identifying actionable insights, is a challenging task. Often the importance of a feature in a model is only a rough estimate condensed into one number. However, our research goes beyond these naïve estimates through the design and implementation of an interactive visual analytics system, Prospector. By providing interactive partial dependence diagnostics, data scientists can understand how features affect the prediction overall. In addition, our support for localized inspection allows data scientists to understand how and why specific datapoints are predicted as they are, as well as support for tweaking feature values and seeing how the prediction responds. Our system is then evaluated using a case study involving a team of data scientists improving predictive models for detecting the onset of diabetes from electronic medical records.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 3×
“Partial dependence was proposed by Friedman [14] for analyzing gradient boosting models and has since been used for other models as well (e.”
MethodThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,370
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 2×
“Predictive modeling is defined as the process of developing a mathematical tool or model that generates an accurate prediction [24].”
MethodStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861
Research works printed but not linked: 8
- Cortez Paulo, IEEE Symposium on. 341-348
- Frank Eibe, Knowledge Discovery in Databases (PKDD), Nada Lavra, Dragan Gamberger, Ljupo Todorovski, and Hendrik Blockeel (Eds.)
- Kim Been, 2014, The Bayesian Case Model, In Advances in Neural Information Processing Systems., 1952
- Carson, Knowledge-Based and Intelligent Information & Engineering Systems 18th Annual Conference (KES), Proceedings of.
- 10.5555/2283516.2283682 · doi:10.5555/2283516.2283682
- Plate Tony, 1997, Progress in Connectionist-Based Information Systems: Proceedings of The Fourth International Conference on Neural Information Processing (ICONIP'97), 306
- Tzeng Fan-Yin, 2005, Proceedings of IEEE Visualization '05 Conference. IEEE, 383-390
- van den Elzen Stef, IEEE Conference on. 151-160
Cited by
Items 501–513 of 513
Designing a Lightweight GenAI Interface for Visual Data AnalysisPreprint
Koonchanok, R., Kale, A., & Reda, K. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.02878
Inspection of Blackbox Models for Evaluating Vulnerability in Maternal, Newborn, and Child Health
Ogallo, W., Speakman, S., Akinwande, V., Varshney, K. R., Walcott-Bryant, A., Wayua, C., & Weldemariam, K. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/770
Overview, Trends and Future Applications of Artificial Intelligence in Manufacturing
Lenz, J. (2023) · International Conference on Industrial Engineering and Operations Management (IEOM) · doi:10.46254/eu6.20230468
Apple orchard visual inspection based on YOLOv5
Liu, J., Wu, R., & Zeng, J. (2024) · International Conference on Big Data, Artificial Intelligence and Risk Management (ICBAR) · doi:10.1145/3718751.3718816
Interactive Temporal Feature Construction: A User-Driven Approach to Predictive Model Development
Chaturvedi, R. (2017) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/qt88-d857
A Heuristic Approach for Dual Expert/End-User Evaluation of Guidance in Visual AnalyticsPreprint
Ceneda, D., Collins, C., El‐Assady, M., Miksch, S., Tominski, C., & Arleo, A. (2023) · reposiTUm (TU Wien) · doi:10.34726/5043
Making Predictive Process Monitoring useful insights from different perspectives
Rizzi, W. (2023) · Libera Università di Bolzano
Transforming Human Resources: Leveraging Human-Computer Interaction for the Future of Work
Venkata, R. V. V., Rani, C., & Kumar, P. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-1676-5.ch009
Explorando as interseções entre Visualização da Informação e Aprendizado de Máquina
Corrêa, I. B. (2018) · LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas)
Behavioral Drift Analytics for Preventive Fraud Risk Management in Digital Financial Platforms
Joyson, L., Seshan, N., Sivasankarapandi, S., & Bommuraj, S. (2026) · International Research Journal on Advanced Engineering Hub (IRJAEH) · doi:10.47392/irjaeh.2026.0255
HiVis: a web-based library for financial data visualizations
Han, D., Han, R., Zhou, C., Lin, Z., Hu, Y., Chai, Y., Ma, T., Zhu, H., & Chen, W. (2026) · Journal of Visualization · doi:10.1007/s12650-026-01135-3
Web Based Data Visualization and Data Preprocessing Tool
Chachra, S., Kumar, R. S., Kolur, S. D., Rohit, K., & Bharti, P. (2023) · Journal Of Advanced Zoology
Metadata
- Authors
- Krause, J., Perer, A., & Ng, K.
- Affiliations
- New York University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5686-5697
- OpenAlex
- W2394669110
- Cited by
- 390
- References
- 50
- Citations by year
- 2026: 14 · 2025: 31 · 2024: 37 · 2023: 38 · 2022: 33 · 2021: 83 · 2020: 54 · 2019: 48 · 2018: 27 · 2017: 19 · 2016: 6
- FWCI
- 11.41
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- interactive visual analysis, partial dependence, model interpretability, feature importance, explainable machine learning, electronic health records, prospectors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.