Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index.
Abstract
Understanding predictive models, in terms of interpreting and identifying actionable insights, is a challenging task. Often the importance of a feature in a model is only a rough estimate condensed into one number. However, our research goes beyond these naïve estimates through the design and implementation of an interactive visual analytics system, Prospector. By providing interactive partial dependence diagnostics, data scientists can understand how features affect the prediction overall. In addition, our support for localized inspection allows data scientists to understand how and why specific datapoints are predicted as they are, as well as support for tweaking feature values and seeing how the prediction responds. Our system is then evaluated using a case study involving a team of data scientists improving predictive models for detecting the onset of diabetes from electronic medical records.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Lecture Notes in Computer Science 1205
(1999) · Industrial Robot: An International Journal · doi:10.1108/ir.1999.04926fae.001 · Cited by 38,662
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 3×
“Partial dependence was proposed by Friedman [14] for analyzing gradient boosting models and has since been used for other models as well (e.”
MethodThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 25,410
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
The Elements of Statistical Learning: Data Mining, Inference, and Prediction
John Lu, Z. Q. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2010.00646_6.x · Cited by 12,390
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Kuhn, M., & Johnson, K. (2013) · doi:10.1007/978-1-4614-6849-3 · Cited by 6,788 · cited 2×
“Predictive modeling is defined as the process of developing a mathematical tool or model that generates an accurate prediction [24].”
MethodStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
Goldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Statistical modeling: The two cultures
Breiman, L. (2001) · Cited by 1,314
Olden, J. D., & Jackson, D. A. (2002) · Ecological Modelling · doi:10.1016/s0304-3800(02)00064-9 · Cited by 1,223
The challenge of information visualization evaluation
Plaisant, C. (2004) · Proceedings of the working conference on Advanced visual interfaces · doi:10.1145/989863.989880 · Cited by 590
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576
Information visualization in data mining and knowledge discovery
Fayyad, U. M. (2002) · Cited by 565
Using sensitivity analysis and visualization techniques to open black box data mining models
Cortez, P., & Embrechts, M. J. (2013) · Information Sciences · doi:10.1016/j.ins.2012.10.039 · Cited by 443
Shneiderman, B., & Plaisant, C. (2006) · Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization · doi:10.1145/1168149.1168158 · Cited by 398
The Bayesian Case Model: A Generative Approach for Case-Based Reasoning and Prototype Classification
Kim, B., Rudin, C. D., & Shah, J. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 340
ModelTracker: Redesigning Performance Analysis Tools for Machine Learning
Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015) · Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702509 · Cited by 262
EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers
Talbot, J., Lee, B., Kapoor, A., & Tan, D. S. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518895 · Cited by 226
BaobabView: Interactive construction and analysis of decision trees
van den Elzen, S., & van Wijk, J. J. (2011) · 2011 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2011.6102453 · Cited by 172
INFUSE: Interactive Feature Selection for Predictive Modeling of High Dimensional Data
Krause, J., Perer, A., & Bertini, E. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346482 · Cited by 169
Self-citationPerer, A., & Shneiderman, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357101 · Cited by 164
Self-citationRegroup: interactive machine learning for on-demand group creation in social networks
Amershi, S., Fogarty, J., & Weld, D. (2012) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2207680 · Cited by 144
Interactive optimization for steering machine classification
Kapoor, A., Lee, B., Tan, D., & Horvitz, E. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753529 · Cited by 133
Mind the Gap: a generative approach to interpretable feature selection and extraction
Kim, B., Shah, J., & Doshi‐Velez, F. (2015) · neural information processing systems · Cited by 122
Gestalt: integrated support for implementation and analysis in machine learning
Patel, K., Bancroft, N., Drucker, S. M., Fogarty, J., Ko, A. J., & Landay, J. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866038 · Cited by 112 · cited 1×
“Some recent interactive machine learning tools [1, 2, 3, 4, 19, 21, 26, 29, 32]) are more algorithm agnostic, but depend on general performance measures like confusion matrices, area under ROC curve (AUC) measures, result distribution of data points, and feature weights according to model independent statistics.”
BackgroundDiscovering Structure in High-Dimensional Data Through Correlation ExplanationPreprint
Steeg, G. V., & Galstyan, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.1222 · Cited by 111
Ng, K., Ghoting, A., Steinhubl, S. R., Stewart, W. F., Malin, B., & Sun, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.12.012 · Cited by 110
Self-citationOpening black box Data Mining models using Sensitivity Analysis
Cortez, P., & Embrechts, M. J. (2011) · 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM) · doi:10.1109/cidm.2011.5949423 · Cited by 107
Sports Data Mining: Predicting Results for the College Football Games
Leung, C. K.-S., & Joseph, K. W. (2014) · Procedia Computer Science · doi:10.1016/j.procs.2014.08.153 · Cited by 99
Opening the Black Box - Data Driven Visualization of Neural Network
Tzeng, F.-Y., & Ma, K.-L. (2006) · IEEE Visualization 2005 - (VIS'05) · doi:10.1109/vis.2005.73 · Cited by 87
Nomograms for visualizing support vector machines
Jakulin, A., Možina, M., Demšar, J., Bratko, I., & Zupan, B. (2005) · Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining · doi:10.1145/1081870.1081886 · Cited by 81
Gaining insights into support vector machine pattern classifiers using projection-based tour methods
Caragea, D., Cook, D., & Honavar, V. G. (2001) · Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/502512.502547 · Cited by 51
Designing for effective end-user interaction with machine learning
Amershi, S. (2011) · Proceedings of the 24th annual ACM symposium adjunct on User interface software and technology · doi:10.1145/2046396.2046416 · Cited by 46
CueT: human-guided fast and accurate network alarm triage
Amershi, S., Lee, B., Kapoor, A., Mahajan, R., & Christian, B. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1978966 · Cited by 46
Maximally Informative Hierarchical Representations of High-Dimensional DataPreprint
Steeg, G. V., & Galstyan, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.7404 · Cited by 41
Visualizing high-dimensional predictive model quality
Rheingans, P. L., & desJardins, M. (2000) · Visualization · doi:10.5555/375213.375313 · Cited by 37
Steed, C. A., Swan, J. E., Jankun-Kelly, T. J., & Fitzpatrick, P. J. (2009) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2009.5332586 · Cited by 37
Using Multiple Models to Understand Data
Patel, K., Drucker, S. M., Fogarty, J. A., Kapoor, A., & Tan, D. S. (2012) · Cited by 31
Improving Understanding and Trust with Intelligibility in Context-Aware Applications
Lim, B. Y. (2012) · KiltHub Repository · doi:10.1184/r1/6720179 · Cited by 29
Sentiment Analysis with Incremental Human-in-the-Loop Learning and Lexical Resource Customization
Mishra, S., Diesner, J., Byrne, J., & Surbeck, E. J. (2015) · ACM Conference on Hypertext and Social Media (HT) · doi:10.1145/2700171.2791022 · Cited by 22
Visualizing Class Probability Estimators
Frank, E., & Hall, M. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39804-2_17 · Cited by 20
Goldstein, A., Kapelner, A. D., Bleich, J., et al. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.6392 · Cited by 19
Tropical Cyclone Trend Analysis Using Enhanced Parallel Coordinates and Statistical Analytics
Steed, C. A., Fitzpatrick, P. J., Swan, J. E., & Jankun-Kelly, T. J. (2009) · Cartography and Geographic Information Science · doi:10.1559/152304009788988314 · Cited by 17
Scalable and Interpretable Data Representation for High-Dimensional, Complex Data
Kim, B., Patel, K., Rostamizadeh, A., & Shah, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9474 · Cited by 16
Martins, D., de Oliveira, E. A., Braga-Neto, U., Hashimoto, R. F., & Junior, R. M. C. (2012) · Information Sciences · doi:10.1016/j.ins.2012.10.027 · Cited by 9
Visualizing the Function Computed by a Feedforward Neural Network.
Plate, T. A., Band, P. R., Bert, J. L., & Grace, J. (1997) · International Conference on Neural Information Processing · Cited by 8
Summaries: A special issue on Evaluation for Information Visualization
Bertini, E., Lam, H., & Perer, A. (2011) · Information Visualization · doi:10.1177/1473871611421693 · Cited by 7
Self-citation
Research works printed but not linked: 8
- Cortez Paulo, IEEE Symposium on. 341-348
- Frank Eibe, Knowledge Discovery in Databases (PKDD), Nada Lavra, Dragan Gamberger, Ljupo Todorovski, and Hendrik Blockeel (Eds.)
- Kim Been, 2014, The Bayesian Case Model, In Advances in Neural Information Processing Systems., 1952
- Carson, Knowledge-Based and Intelligent Information & Engineering Systems 18th Annual Conference (KES), Proceedings of.
- 10.5555/2283516.2283682 · doi:10.5555/2283516.2283682
- Plate Tony, 1997, Progress in Connectionist-Based Information Systems: Proceedings of The Fourth International Conference on Neural Information Processing (ICONIP'97), 306
- Tzeng Fan-Yin, 2005, Proceedings of IEEE Visualization '05 Conference. IEEE, 383-390
- van den Elzen Stef, IEEE Conference on. 151-160
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
A Survey on the Explainability of Supervised Machine Learning
Burkart, N., & Huber, M. F. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12228 · Cited by 1,008
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Metadata
- Authors
- Krause, J., Perer, A., & Ng, K.
- Affiliations
- New York University (US)
- IBM (United States) (US)
- Venue
- Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2016-05-07
- Pages
- 5686-5697
- OpenAlex
- W2394669110
- Cited by
- 390
- References
- 50
- Citations by year
- 2026: 14 · 2025: 31 · 2024: 37 · 2023: 38 · 2022: 33 · 2021: 83 · 2020: 54 · 2019: 48 · 2018: 27 · 2017: 19 · 2016: 6
- FWCI
- 11.41
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- interactive visual analysis, partial dependence, model interpretability, feature importance, explainable machine learning, electronic health records, prospectors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.