The DOI matches its Crossref record; 76 of the 76 references it cites were found in a registry or an index.
Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
Open accessAbstract
As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interpretable models provide a common language for these experts by clearly explaining the inner workings of the detection system [80].”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Note that we chose to provide SHAP explanations for one black-box model because it emulates how black-box models are most commonly implemented in-practice [10,36].”
From It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy (2022) · cites this paper as [36]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Thus, it remains unclear if existing XAI methods meet the end-users' needs when trying to assess trustworthiness of an AI [24,32].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [24]
In the citing paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“scribe its behavior. Examples include methods that infer feature relevance Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW1, Article 68. Publication date: May 2020. 68:4 Hong et al. from the model [18] as well as methodsthat build whole surrogates that try to mimic the model’s behavior by building a more transparen…”
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,826 · cited 1×
“esent the perspectives of our participants systematically curated in an interpretive framework which was developed through discussions among the authors. We used aniterative qualitative coding process[66] characterizedby alternate phases ofcoding (to tag specific text segments with codes), analytic memo writing (to col…”
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We describe the methodology we followed to conduct the interviews and to analyze the data we collected.”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“In addition to proposing new algorithms and tools [6, 42, 64, 68, 72], the growing literature on interpretability includes attempts to define interpretability with scientific rigor [14, 15, 34, 67].”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim see model interpretability as a proxy for many “auxiliary criteria” such as safety, fairness, non-discrimination, avoiding technical debt, privacy, reliability, providing the right to explanation, trust, and more [14].”
BackgroundMethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“As several studies have shown, ML models can learn spurious associations [8, 19] and make decisions based on correlations that do not result from causal relationships.”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Anchors aims at providing instance explanations using a rule structure [63].”
MethodIntelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685 · cited 3×
“This aspect contrasts somewhat with definitions of interpretability that frame it as determinable from the interaction between an individual mental model and ML model.”
Uncertain Judgements: Eliciting Experts' Probabilities
O'Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E., & Rakow, T. (2006) · doi:10.1002/0470033312 · Cited by 1,550
Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2017)
Amendola, G., Ricca, F., & Truszczyński, M. (2017) · International Joint Conference on Artificial Intelligence · Cited by 1,179
Practical Lessons from Predicting Clicks on Ads at Facebook
He, X., Pan, J., Jin, O., Xu, T., Liu, B., Xu, T., Shi, Y., Atallah, A., Herbrich, R., Bowers, S., et al. (2014) · Proceedings of the Eighth International Workshop on Data Mining for Online Advertising · doi:10.1145/2648584.2648589 · Cited by 869 · cited 1×
“…have witnessed a rapid increase in the deployment of machine learning (ML) in a large variety of practical application areas, such as finance [54], healthcare [2, 16, 38], governance [50], 68:2 Sungsoo Ray Hong et al. education [25], space exploration [53], and digital product development [23, 49].”
Ad click prediction: a view from the trenches
McMahan, H. B., Holt, G., Sculley, D., Young, M., Ebner, D., Grady, J., Nie, L., Phillips, T., Davydov, E., Golovin, D., et al. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2488200 · Cited by 867
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · cited 1×
“…attempts at providing a window to the world of practitioners in ML and Data Science, such as Holstein et al.’s interview study on ML fairness [26]; Yang et al.’s, survey on how laypeople use ML [74]; and Kandel et al.’s interview study on how enterprise analysts use data analysis and…”
Letham, B., Rudin, C., McCormick, T. H., & Madigan, D. (2015) · The Annals of Applied Statistics · doi:10.1214/15-aoas848 · Cited by 766 · cited 1×
“Finally, we describe relevant empirical studies that motivated our work.”
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728 · cited 3×
“rehensibility, mental fit, explainability, acceptability, and justifiability, to refer to similar concepts[4] makesit hard to understand interpretability’s scope and to define metrics for its evaluation [61]. Such critiques have prompted reflection and at times more formal definitions. Doshi-Velez and Kim see model interpr…”
InfluentialSociological Practice: Linking Theory and Social Research
Bulmer, M., & Layder, D. (2001) · Canadian Journal of Sociology / Cahiers canadiens de sociologie · doi:10.2307/3341515 · Cited by 713
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 2×
“Model interpretability techniques are often categorized into two classes: global and local [15, 21, 55].”
Extracting Tree-Structured Representations of Trained Networks
Craven, M. W., & SHAVLIK, J. W. (1995) · neural information processing systems · Cited by 602
Principles of Explanatory Debugging to Personalize Interactive Machine Learning
Kulesza, T., Burnett, M., Wong, W.-K., & Stumpf, S. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701399 · Cited by 576 · cited 2×
“Finally, we describe relevant empirical studies that motivated our work.”
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564 · cited 4×
“, found that data scientists often “misuse” interpretability tools and over-trust the results [32].”
BackgroundInfluential(2016) · neural information processing systems · Cited by 556
The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems
Bussone, A., Stumpf, S., & O'Sullivan, D. (2015) · 2015 International Conference on Healthcare Informatics · doi:10.1109/ichi.2015.26 · Cited by 441 · cited 1×
“els [39], some studies paint a less positive picture. For example, Bussone et al., found users of clinical decision-support models tend to over-rely on a model recommendation over their own expertise [7]. Aligning with their findings, Kaur et al., found that data scientists often “misuse” interpretability tools and ove…”
Enterprise Data Analysis and Visualization: An Interview Study
Kandel, S., Paepcke, A., Hellerstein, J. M., & Heer, J. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.219 · Cited by 439 · cited 1×
“’s interview study on how enterprise analysts use data analysis and visualization tools [31].”
Workplace Studies: Recovering Work Practice and Informing System Design
(2000) · Cambridge University Press eBooks · doi:10.1017/cbo9780511628122 · Cited by 423
Classifying without discriminating
Kamiran, F., & Calders, T. (2009) · 2009 2nd International Conference on Computer, Control and Communication · doi:10.1109/ic4.2009.4909197 · Cited by 409 · cited 1×
“can learn spurious associations [8, 19] and makedecisions based on correlations that do not result from causal relationships. Modelscan alsolearnrelationshipsindatathatexemplifyundesirableformsofbias[30]oraremislabeled[57]. Hence in practice, model accuracy cannot be fully trusted as a reliable metric for automation a…”
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 2×
“…(HCI) have devoted effort to understanding which ML models can better conform to human understanding of prediction tasks [77], how people’s interaction with ML models affects the quality of decisions [20], and how data scientists use particular tools in conducting their interpretability tasks [32].”
Interacting with Predictions: Visual Inspection of Black-box Machine Learning Models
Krause, J., Perer, A., & Ng, K. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858529 · Cited by 390
Predictive policing: Review of benefits and drawbacks
Meijer, A., & Wessels, M. (2019) · International Journal of Public Administration · doi:10.1080/01900692.2019.1575664 · Cited by 383
Machine Learning for Science: State of the Art and Future Prospects
Mjolsness, E., & DeCoste, D. (2001) · Science · doi:10.1126/science.293.5537.2051 · Cited by 379
Characterizing Mental Models of Hazardous Processes: A Methodology and an Application to Radon
Bostrom, A., Fischhoff, B., & Morgan, M. G. (1992) · Journal of Social Issues · doi:10.1111/j.1540-4560.1992.tb01946.x · Cited by 358 · cited 1×
“Our identification of unaddressed technological needs among practitioners is intended to help researchers direct their attention to unaddressed, yet prevalent, challenges in industry.”
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342 · cited 2×
“One possibly fruitful research direction is research which focuses on hiding the complexity behind building anMLmodel and making model-building more accessible to non-specialists [27, 71].”
BackgroundActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models
Kahng, M., Andrews, P. Y., Kalro, A., & Chau, D. H. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744718 · Cited by 337 · cited 1×
“Some other approaches present visual analytics systems specialized in a specific model type, such as regression models [13], neural networks [29, 69, 73], and surrogate modes [52].”
BackgroundVisualizing Dataflow Graphs of Deep Learning Models in TensorFlow
Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mane, D., Fritz, D., Krishnan, D., Viegas, F. B., & Wattenberg, M. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744878 · Cited by 331 · cited 2×
“Some other approaches present visual analytics systems specialized in a specific model type, such as regression models [13], neural networks [29, 69, 73], and surrogate modes [52].”
Expert knowledge elicitation to improve formal and mental models
Ford, D. N., & Sterman, J. D. (1998) · System Dynamics Review · doi:10.1002/(sici)1099-1727(199824)14:4<309::aid-sdr154>3.0.co;2-5 · Cited by 301 · cited 1×
“Looking to research in belief elicitation and mental model discovery within psychology, economics, and related disciplines in designing such approaches may be informative (e.g., [ 5, 17, 35, 36, 58]).”
Expert knowledge elicitation to improve formal and mental models
Ford, D. N., & Sterman, J. D. (1998) · System Dynamics Review · doi:10.1002/(sici)1099-1727(199824)14:4<285::aid-sdr153>3.0.co;2-f · Cited by 299
Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff
Bansal, G., Nushi, B., Kamar, E., Weld, D. S., Lasecki, W. S., & Horvitz, E. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012429 · Cited by 284 · cited 2×
“Achieving this alignment can lead to measurable benefits such as a better performance of a human-model team on a task [3] or the user’s improved ability to debug the model [39].”
BackgroundHow do Data Science Workers Collaborate? Roles, Workflows, and Tools
Zhang, A. X., Muller, M., & Wang, D. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392826 · Cited by 268
Explaining Data-Driven Document Classifications1
Martens, D., & Provost, F. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.1.04 · Cited by 264 · cited 1×
“For instance, Miller argues that the majority of studies discuss model interpretability based on the unit of an individual, but few studies explain how humans describe how models work to one another [51].”
Rulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243
Self-citationKay, M., Kola, T., Hullman, J. R., & Munson, S. A. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858558 · Cited by 239
Self-citationStudent Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms
Holstein, K., McLaren, B. M., & Aleven, V. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-93843-1_12 · Cited by 235 · cited 1×
“We discuss the motivations behind model interpretability research and present key approaches to understanding and realizing interpretability in HCI, CSCW, and ML communities.”
EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers
Talbot, J., Lee, B., Kapoor, A., & Tan, D. S. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518895 · Cited by 226 · cited 2×
“Finally, we describe relevant empirical studies that motivated our work.”
Gamut: A Design Probe to Understand How Data Scientists Understand Machine Learning Models
Hohman, F., Head, A., Caruana, R., DeLine, R., & Drucker, S. M. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300809 · Cited by 225
Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
Zhang, J., Wang, Y., Molino, P., Li, L., & Ebert, D. S. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864499 · Cited by 220
Squares: Supporting Interactive Performance Analysis for Multiclass Classifiers
Ren, D., Amershi, S., Lee, B., Suh, J., & Williams, J. D. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598828 · Cited by 200 · cited 2×
“…building ML models [71, 74] or ML experts use specific data science packages in determining insights related to model interpretability [32], little research to date focuses on investigating holistic model interpretability processes or communication strategies among different roles in organizations.”
Explainable and Interpretable Models in Computer Vision and Machine Learning
(2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4 · Cited by 199
Hullman, J., Resnick, P., & Adar, E. (2015) · PLOS ONE · doi:10.1371/journal.pone.0142444 · Cited by 194 · cited 1×
“telligible visualizations that can effectively present the behavior of a model to the general public. For instance, presenting uncertainty in model predictions using more intuitive visual abstractions [28, 33] may be beneficial. 5.3 Scalable and Integratable Interpretability Tools Many participants shared their concerns…”
InfluentialSelf-citationConsiderations for Evaluation and Generalization in Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_1 · Cited by 176
Classification and Regression Trees
(2019) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789811201967_0002 · Cited by 173
Quantifying Interpretability and Trust in Machine Learning SystemsPreprint
Schmidt, P. S., & Bießmann, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08558 · Cited by 161 · cited 2×
“In addition to proposing new algorithms and tools [ 6, 42, 64, 68, 72], the growing literature on interpretability includes attempts to define interpretability with scientific rigor [14, 15, 34, 67].”
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · cited 2×
“’s, survey on how laypeople use ML [74]; and Kandel et al.”
BackgroundNarayanan, M., Chen, E., He, J., Kim, B., Gershman, S., & Doshi-Velez, F. (2018) · arXiv · doi:10.48550/arxiv.1802.00682 · Cited by 153 · cited 1×
“We describe the methodology we followed to conduct the interviews and to analyze the data we collected.”
Direct-Manipulation Visualization of Deep NetworksPreprint
Smilkov, D., Carter, S., Sculley, D., Viégas, F. B., & WATTENBERG, M. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.03788 · Cited by 152 · cited 1×
“tiple modelsinasingleview, enablingfastmodelcomparison[62]. Some other approaches present visual analytics systems specialized in a specific model type, such as regression models [13], neural networks [29, 69, 73], and surrogate modes [52]. 2.2 Limitations and Criticism of Model Interpretability As interpretability res…”
InfluentialMonotonic Calibrated Interpolated Look-Up TablesPreprint
Gupta, M. R., Cotter, A., Pfeifer, J., Voevodski, K., Canini, K. R., Mangylov, A., Moczydłowski, W., & Esbroeck, A. V. (2015) · arXiv (Cornell University) · doi:10.5555/2946645.3007062 · Cited by 128
Explaining the Gap: Visualizing One's Predictions Improves Recall and Comprehension of Data
Kim, Y., Reinecke, K., & Hullman, J. R. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025592 · Cited by 128 · cited 1×
“Our identification of unaddressed technological needs among practitioners is intended to help researchers direct their attention to unaddressed, yet prevalent, challenges in industry.”
Self-citationHuman-in-the-Loop Interpretability Prior
Lage, I., Ross, A. S., Kim, B., Gershman, S. J., & Doshi‐Velez, F. (2018) · PubMed · doi:10.57702/j7gpfk6v · Cited by 125
Luo, G. (2016) · Health Information Science and Systems · doi:10.1186/s13755-016-0015-4 · Cited by 85
Dissonance Between Human and Machine Understanding
Zhang, Z., Singh, J., Gadiraju, U., & Anand, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359158 · Cited by 80 · cited 3×
“…Human-Computer Interaction (HCI) have devoted effort to understanding which ML models can better conform to human understanding of prediction tasks [77], how people’s interaction with ML models affects the quality of decisions [20], and how data scientists use particular tools in conducting their…”
Interpretability of machine learning models and representations: an introduction.
Bibal, A., & Frénay, B. (2016) · Repository of the University of Namur · Cited by 77
Efficient Elicitation Approaches to Estimate Collective Crowd Answers
Chung, J. J. Y., Song, J. Y., Kutty, S., Hong, S.-S., Kim, J., & Lasecki, W. S. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359164 · Cited by 74
Self-citationRegressionExplorer: Interactive Exploration of Logistic Regression Models with Subgroup Analysis
Dingen, D., Veer, M. V. ’., Houthuizen, P., Mestrom, E. H. J., Korsten, E. H. H. M., Bouwman, R. A., & van Wijk, J. J. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865043 · Cited by 62 · cited 1×
“Some other approaches present visual analytics systems specialized in a specific model type, such as regression models [13], neural networks [29, 69, 73], and surrogate modes [52].”
BackgroundKim, Y.-S., Reinecke, K., & Hullman, J. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745240 · Cited by 60 · cited 1×
“Our identification of unaddressed technological needs among practitioners is intended to help researchers direct their attention to unaddressed, yet prevalent, challenges in industry.”
Self-citationChoi, M., Park, C., Yang, S., Kim, Y., Choo, J., & Hong, S.-S. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300460 · Cited by 59
Self-citationBeat the machine: Challenging workers to find the unknown unknowns
Attenberg, J., Ipeirotis, P. G., & Provost, F. J. (2011) · Cited by 57
Bayesian Rule Sets for Interpretable Classification
Wang, T., Rudin, C. D., Doshi‐Velez, F., Liu, Y., Klampfl, E., & MacNeille, P. (2016) · Proceedings · doi:10.1109/icdm.2016.0171 · Cited by 56
BayesDB: A probabilistic programming system for querying the probable implications of dataPreprint
Mansinghka, V. K., Tibbetts, R., Baxter, J., Shafto, P., & Eaves, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.05006 · Cited by 38 · cited 1×
“Bayesian approaches to providing answers to users’ queries based on conditional probabilities may be another promising approach [47, 65].”
Using Multiple Models to Understand Data
Patel, K., Drucker, S. M., Fogarty, J. A., Kapoor, A., & Tan, D. S. (2012) · Cited by 31
Towards Explainable Deep Learning for Credit Lending: A Case StudyPreprint
Modarres, C., Ibrahim, M., Louie, M., & Paisley, J. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.06471 · Cited by 23 · cited 2×
“20), 26 pages. https://doi.org/10.1145/3392878 1 INTRODUCTION Recentyearshavewitnessedarapidincreaseinthedeploymentofmachinelearning(ML)inalarge variety of practical application areas, such as finance [54], healthcare [2, 16, 38], governance [50], Authors’ addresses: Sungsoo Ray Hong, rayhong@nyu.edu, New York Universi…”
InfluentialImproving Human-Machine Cooperative Visual Search With Soft Highlighting
Kneusel, R. T., & Mozer, M. C. (2017) · ACM Transactions on Applied Perception · doi:10.1145/3129669 · Cited by 22
The Effect of Heterogeneous Data for Alzheimer's Disease Detection from SpeechPreprint
Balagopalan, A., Novikova, J., Rudzicz, F., & Ghassemi, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.12254 · Cited by 22 · cited 1×
“…have witnessed a rapid increase in the deployment of machine learning (ML) in a large variety of practical application areas, such as finance [54], healthcare [2, 16, 38], governance [50], 68:2 Sungsoo Ray Hong et al. education [25], space exploration [53], and digital product development [23, 49].”
A Bayesian framework for learning rule sets for interpretable classification
WangTong, RudinCynthia, Doshi-VelezFinale, LiuYimin, KlampflErica, & MacNeillePerry (2017) · Journal of Machine Learning Research · doi:10.5555/3122009.3176814 · Cited by 10
Saad, F. A., & Mansinghka, V. K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01708 · Cited by 5
Improving Human-Machine Cooperative Visual Search With Soft HighlightingPreprint
Kneusel, R. T., & Mozer, M. C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08117 · Cited by 1
Research works printed but not linked: 4
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115.
- Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.
- Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 ( 2016 ). http://arxiv.org/abs/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 (2016). http://arxiv.org/abs/1606.03490 · arXiv
- Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774.
Cited by
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
“An interview study with 22 machine learning professionals documented the value of explainability for developers, testers, managers, and users [52].”
BackgroundExpanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
“Their findings suggest that the process of interpreting or making sense of an AI system frequently involves cooperation and mental model comparison between people in different roles, aimed at building trust not only between people and the AI system, but also between people within the organization [43].”
BackgroundInterpretable Machine Learning: Moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · Queue · doi:10.1145/3511299 · Cited by 516
“Model debugging is not only a common consumer use case, 7,13 but also a well-grounded one.”
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312
Relation between prognostics predictor evaluation metrics and local interpretability SHAP values
Baptista, M. L., Goebel, K., & Henriques, E. M. P. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103667 · Cited by 270
“In other words, the goal is to build a surrogate model that is more transparent [60].”
Background
Metadata
- Authors
- Hong, S. R., Hullman, J., & Bertini, E.
- Affiliations
- New York University, New York, NY, USA
- Northwestern University (US)
- New York University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-05-28
- Volume
- 4
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3019489177
- Cited by
- 219
- References
- 74
- Citations by year
- 2026: 20 · 2025: 44 · 2024: 42 · 2023: 41 · 2022: 31 · 2021: 36 · 2020: 5
- FWCI
- 11.01
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model interpretability, human factors, industry professionals, machine learning adoption, mental models, organizational trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.