CoChat

Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs

Open access
Hong, Sungsoo Ray, Hullman, Jessica, Bertini, Enrico · 2020 · Proceedings of the ACM on Human-Computer Interaction

The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.

Abstract

As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3392878

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 77 references

Research works printed but not linked: 4

  1. Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115.
  2. Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.
  3. Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 ( 2016 ). http://arxiv.org/abs/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 (2016). http://arxiv.org/abs/1606.03490 · arXiv
  4. Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774.

Cited by

Items 201–217 of 217

Metadata

Authors
Hong, S. R., Hullman, J., & Bertini, E.
Affiliations
  • New York University, New York, NY, USA
  • Northwestern University (US)
  • New York University (US)
Venue
Proceedings of the ACM on Human-Computer Interaction
Published
2020-05-28
Volume
4
Issue
CSCW1
Pages
1-26
ISSN
2573-0142 (ISSN-L)
OpenAlex
W3019489177
Cited by
219
References
74
Citations by year
2026: 20 · 2025: 44 · 2024: 42 · 2023: 41 · 2022: 31 · 2021: 36 · 2020: 5
FWCI
11.01
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
Keywords
model interpretability, human factors, industry professionals, machine learning adoption, mental models, organizational trust

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.