Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
Open accessThe DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Abstract
As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interpretable models provide a common language for these experts by clearly explaining the inner workings of the detection system [80].”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Note that we chose to provide SHAP explanations for one black-box model because it emulates how black-box models are most commonly implemented in-practice [10,36].”
From It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy (2022) · cites this paper as [36]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Thus, it remains unclear if existing XAI methods meet the end-users' needs when trying to assess trustworthiness of an AI [24,32].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [24]
In the citing paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“scribe its behavior. Examples include methods that infer feature relevance Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW1, Article 68. Publication date: May 2020. 68:4 Hong et al. from the model [18] as well as methodsthat build whole surrogates that try to mimic the model’s behavior by building a more transparen…”
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,826 · cited 1×
“esent the perspectives of our participants systematically curated in an interpretive framework which was developed through discussions among the authors. We used aniterative qualitative coding process[66] characterizedby alternate phases ofcoding (to tag specific text segments with codes), analytic memo writing (to col…”
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We describe the methodology we followed to conduct the interviews and to analyze the data we collected.”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“In addition to proposing new algorithms and tools [6, 42, 64, 68, 72], the growing literature on interpretability includes attempts to define interpretability with scientific rigor [14, 15, 34, 67].”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim see model interpretability as a proxy for many “auxiliary criteria” such as safety, fairness, non-discrimination, avoiding technical debt, privacy, reliability, providing the right to explanation, trust, and more [14].”
BackgroundMethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“As several studies have shown, ML models can learn spurious associations [8, 19] and make decisions based on correlations that do not result from causal relationships.”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Anchors aims at providing instance explanations using a rule structure [63].”
Method
Research works printed but not linked: 4
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115.
- Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.
- Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 ( 2016 ). http://arxiv.org/abs/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 (2016). http://arxiv.org/abs/1606.03490 · arXiv
- Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774.
Cited by
Items 201–217 of 217
Li, C., Zhou, S., Pan, X., Ding, S. Z., & Li, Z. (2026) · International Journal of Industrial Ergonomics · doi:10.1016/j.ergon.2026.103918
Alarcon, K. A. A. (2024) · theses.fr (ABES) · doi:10.70675/912cd6efz8e4cz48f2zafebz16e53a2570ca
Vereschak, O. (2022) · theses.fr (ABES) · doi:10.70675/7ecd53bfza7ccz4edfz8f1azdb8052ba1233 · Cited by 2
A survey on health care system with explainable AI
Babu, K. M., Gurusudha, C., Sreya, G., Saisresta, K., & Anusha, G. (2026) · AIP conference proceedings · doi:10.1063/5.0320169
CoNode: Visualizing Workflows for Knowledge Reuse and Recombination in Team–AI Collaborative Design
Zhou, Y., Xiao, Y., Wei, G., Ling, S., Ma, R., & Leung, C.-S. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791216 · Cited by 2
“AI involvement can further impose coordination overhead, as collaborators negotiate when to defer to AI, when to override it, and how to reconcile divergent human–AI contributions—blurring agency boundaries and slowing collaborative momentum [1, 21, 39, 72].”
Feng, H., Ara, Z., Hundt, A., Vučetić, S., Chung, J. J. Y., & Hong, S. R. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791853 · Cited by 2
“…in the challenges autistic job seekers identified in our study, and building on our prior research in designing human–AI collaboration systems for specific user groups [18, 28, 37, 85, 86], these directions reflect our own view on how future AI-driven systems can be more effectively leveraged.”
Self-citationHuman-in-the-Loop and Participatory Approaches to Trustworthy AI
Azeroual, O. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-6761-3.ch007
Human-Centered Recommender Systems
Hao, S., Ji, L., & Zhang, D. (2026) · Springer eBooks · doi:10.1007/978-981-95-3658-0_119
Language-Based Interpretability in Agentic Systems
Oliveira, P., Pereira, J. D. C., & Nováis, P. (2026) · SpringerBriefs in intelligent systems · doi:10.1007/978-3-032-24781-0_4
Popowski, L., Vasconcelos, H., Heinsen, I. J. F., Mgbahurike, C., Herbrich, R., Hancock, J. T., & Bernstein, M. S. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788067 · Cited by 1
Yaghoubi, E., Sianaki, O. A., Ghasemi, F., Kamali, Z., Rajabipour, A., & Sandanayake, M. S. (2026) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-032-23341-7_31
From Curiosity to Caution: How Expertise Shapes the Use of Interpretable Machine Learning
Naqvi, S., Balaji, G., Laju, A., & Kaur, H. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812376
“Users’ cognitive constraints and biases shape how they interpret and rely on explanations [1, 16, 70, 91, 140, 156], and interpretability practices vary by organizational role [83].”
On the Interplay of Explainability and Fairness in AI: A Survey
Fragkathoulas, C., Papanikou, V., Karidi, D. P., & Pitoura, E. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3706635
“Prior work characterizes stakeholders by expertise levels [64], [65], [66] or functional roles such as model builders, breakers, end users, or examiners [67], [68], [69].”
An artificial intelligence framework for transforming unstructured text into structured survey data
Manzi, G., Guo, Q., Russo, L., & Grané, A. (2026) · Decision Analytics Journal · doi:10.1016/j.dajour.2026.100730
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.010
Balayn, A., Rancourt, F., Casati, F., & Gadiraju, U. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816909
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2021) · arXiv · doi:10.48550/arxiv.2110.10790 · Cited by 120
“For model development or debugging, research suggests that users often need a range of explanations for different levels of the model behaviors to perform a comprehensive diagnosis [51, 52, 77].”
Background
Metadata
- Authors
- Hong, S. R., Hullman, J., & Bertini, E.
- Affiliations
- New York University, New York, NY, USA
- Northwestern University (US)
- New York University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-05-28
- Volume
- 4
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3019489177
- Cited by
- 219
- References
- 74
- Citations by year
- 2026: 20 · 2025: 44 · 2024: 42 · 2023: 41 · 2022: 31 · 2021: 36 · 2020: 5
- FWCI
- 11.01
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model interpretability, human factors, industry professionals, machine learning adoption, mental models, organizational trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.