Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
Open accessThe DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Abstract
As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interpretable models provide a common language for these experts by clearly explaining the inner workings of the detection system [80].”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Note that we chose to provide SHAP explanations for one black-box model because it emulates how black-box models are most commonly implemented in-practice [10,36].”
From It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy (2022) · cites this paper as [36]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Thus, it remains unclear if existing XAI methods meet the end-users' needs when trying to assess trustworthiness of an AI [24,32].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [24]
In the citing paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“scribe its behavior. Examples include methods that infer feature relevance Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW1, Article 68. Publication date: May 2020. 68:4 Hong et al. from the model [18] as well as methodsthat build whole surrogates that try to mimic the model’s behavior by building a more transparen…”
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,826 · cited 1×
“esent the perspectives of our participants systematically curated in an interpretive framework which was developed through discussions among the authors. We used aniterative qualitative coding process[66] characterizedby alternate phases ofcoding (to tag specific text segments with codes), analytic memo writing (to col…”
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We describe the methodology we followed to conduct the interviews and to analyze the data we collected.”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“In addition to proposing new algorithms and tools [6, 42, 64, 68, 72], the growing literature on interpretability includes attempts to define interpretability with scientific rigor [14, 15, 34, 67].”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim see model interpretability as a proxy for many “auxiliary criteria” such as safety, fairness, non-discrimination, avoiding technical debt, privacy, reliability, providing the right to explanation, trust, and more [14].”
BackgroundMethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“As several studies have shown, ML models can learn spurious associations [8, 19] and make decisions based on correlations that do not result from causal relationships.”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Anchors aims at providing instance explanations using a rule structure [63].”
Method
Research works printed but not linked: 4
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115.
- Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.
- Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 ( 2016 ). http://arxiv.org/abs/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 (2016). http://arxiv.org/abs/1606.03490 · arXiv
- Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774.
Cited by
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
“An interview study with 22 machine learning professionals documented the value of explainability for developers, testers, managers, and users [52].”
BackgroundExpanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
“Their findings suggest that the process of interpreting or making sense of an AI system frequently involves cooperation and mental model comparison between people in different roles, aimed at building trust not only between people and the AI system, but also between people within the organization [43].”
BackgroundInterpretable Machine Learning: Moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · Queue · doi:10.1145/3511299 · Cited by 516
“Model debugging is not only a common consumer use case, 7,13 but also a well-grounded one.”
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312
Relation between prognostics predictor evaluation metrics and local interpretability SHAP values
Baptista, M. L., Goebel, K., & Henriques, E. M. P. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103667 · Cited by 270
“In other words, the goal is to build a surrogate model that is more transparent [60].”
BackgroundHaque, A. B., Islam, A. K. M. N., & Mikalef, P. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122120 · Cited by 240
“Therefore, they should design and develop an interactive system care-fully considering all the requirements of users, device diversity, and regulatory issues to promote the completeness of XAI (Brennen, 2020; Danry et al., 2020; Hong et al., 2020).”
BackgroundInfluentialHelp Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
Kim, S. S. Y., Watkins, E. A., Russakovsky, O., Fong, R., & Monroy-Hernández, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581001 · Cited by 233
“Although human-centered XAI is an actively growing area of research, much of the work still focuses on developers rather than end-users of AI systems [15, 52, 66].”
BackgroundHow to Evaluate Trust in AI-Assisted Decision Making? A Survey of Empirical Methodologies
Vereschak, O., Bailly, G., & Caramiaux, B. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476068 · Cited by 229
“…stated - ✗ ✓ ✓ [193] No source stated - ✓ ✗ ✓ [235] No definition: [1, 4, 8, 22, 24, 26, 27, 33, 34, 38, 42, 50, 56, 60, 67, 71, 75, 81, 84, 87, 89, 91, 97, 107, 112, 118, 122, 123, 126, 136, 150, 154, 156, 165, 166, 170, 176, 177, 180, 185, 196–198, 201, 207, 210, 214–216, 219–221, 225, 227,…”
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
Krishna, S., Han, T., Gu, A., Jabbari, S., Wu, Z. S., & Lakkaraju, H. (2023) · Research Square · doi:10.21203/rs.3.rs-2963888/v1 · Cited by 201
Mienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Interpretable machine learning: moving from mythos to diagnostics
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2022) · Communications of the ACM · doi:10.1145/3546036 · Cited by 203
“Model debugging is not only a common (Hong et al., 2020; Bhatt et al., 2020), but also well-grounded consumer use case.”
Applications, Challenges, and Future Directions of Human-in-the-Loop Learning
Kumar, S., Datta, S., Singh, V., Datta, D., Kumar Singh, S., & Sharma, R. (2024) · IEEE Access · doi:10.1109/access.2024.3401547 · Cited by 168
“Model interpretability refers to the ability to understand how a machine-learning model arrives at a decision [156], [157].”
BackgroundDegas, A., Islam, M. R., Hurter, C., Barua, S., Rahman, H., Poudel, M., Ruscio, D., Ahmed, M. U., Begum, S., Rahman, M. A., et al. (2022) · Applied Sciences · doi:10.3390/app12031295 · Cited by 168
Sivaraman, V., Bukowski, L. A., Levin, J., Kahn, J. M., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581075 · Cited by 141
“concert with human-centered design methods, these technical approaches can be integrated into algorithmic systems with the intent of improving trust and human-AI team performance [5, 37, 67].”
BackgroundSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
“…degree to which it may be biased—we see explicit evidence for this for machine learning experts [57] and data experts [87], model creators and breakers [58], model operators [27, 114], as well as for decision-and data-subjects who may wish to contest a decision or otherwise seek recourse [6, 69].”
BackgroundMethodInfluentialDhanorkar, S., Wolf, C. T., Qian, K., Xu, A., Popa, L., & Li, Y. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462131 · Cited by 119
“[23], noting how interpretability involves cooperation and trust-building among stakeholders.”
BackgroundObaido, G., Mienye, I. D., Egbelowo, O. F., Emmanuel, I. D., Ogunleye, A., Ogbuokiri, B., Mienye, P., & Aruleba, K. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100576 · Cited by 154
Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows
Xin, D., Wu, E. Y., Lee, D. J.-L., Salehi, N., & Parameswaran, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445306 · Cited by 110
“only between humans and models, but also between humans [23].”
BackgroundBell, A., Solano-Kamaiko, I., Nov, O., & Stoyanovich, J. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533090 · Cited by 103
“Other studies note that while post-hoc explain-ability techniques such as SHAP and LIME are useful, they also contain issues with regards to model volatility and have outlined how LIME has given unexpected results [10, 36].”
BackgroundMethodResultSuresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 103
Deepfake Media Forensics: Status and Future Challenges
Amerini, I., Barni, M., Battiato, S., Bestagini, P., Boato, G., Bruni, V., Caldelli, R., De Natale, F., De Nicola, R., Guarnera, L., et al. (2025) · Journal of Imaging · doi:10.3390/jimaging11030073 · Cited by 113
“Interpretable models provide a common language for these experts by clearly explaining the inner workings of the detection system [80].”
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders’ needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structure…”
BackgroundResultModel interpretability of financial fraud detection by group SHAP
Lin, K., & Gao, Y. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118354 · Cited by 108
Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
Kaur, H., Adar, E., Gilbert, E., & Lampe, C. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533135 · Cited by 92
“For example, [42, 45, 70, 113, 126] all highlight the challenges of operating within an organization that either develops or employs an AI-based system.”
BackgroundBrasse, J., Broder, H. R., Förster, M., Klier, M., & Sigler, I. (2023) · Electronic Markets · doi:10.1007/s12525-023-00644-5 · Cited by 148
“Insights from this research area pave the way for future research: First, when developing novel XAI methods, researchers should consider technical aspects, first and foremost, scalability (Hong et al., 2020).”
InfluentialLiao, Q. V., Subramonyam, H., Wang, J., & Wortman Vaughan, J. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 86
“Recent studies report that XAI techniques are increasingly used in industry practice as end user-facing features [66], by data scientists to debug models [52, 61, 68], and shared with stakeholders to verify the models [18, 57].”
BackgroundInfluentialExtending the Nested Model for User-Centric XAI: A Design Study on GNN-based Drug Repurposing
Wang, Q., Huang, K., Chandak, P., Zitnik, M., & Gehlenborg, N. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3209435 · Cited by 82
“, time sensitivity) XAI Goal domain-related problems that the users aim to solve using AI explanations Domain Explanation how a human expert would reason about a phenomenon in the applied domain user mental model in [17, 24, 42]”
BackgroundMethodInfluentialGoing Beyond XAI: A Systematic Survey for Explanation-Guided Learning
Gao, Y., Gu, S., Jiang, J., Hong, S.-S., Yu, D., & Zhao, L. (2024) · ACM Computing Surveys · doi:10.1145/3644073 · Cited by 121
“Lastly, the existing survey also provides a pressing field needs explaining why we need to improve the ways to steer models in the case of witnessing the evidence of bias [63].”
BackgroundSelf-citationExplainable machine learning for public policy: Use cases, gaps, and research directions
Amarasinghe, K., Rodolfa, K. T., Lamba, H., & Ghani, R. (2023) · Data & Policy · doi:10.1017/dap.2023.2 · Cited by 73
“[2013], Hu et al. [2019] may not correlate with results in actual deployment with real users.”
ResultLai, T. (2024) · BioMedInformatics · doi:10.3390/biomedinformatics4010008 · Cited by 87
Liao, Q. V., Zhang, Y., Luss, R., Doshi-Velez, F., & Dhurandhar, A. (2022) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v10i1.21995 · Cited by 69
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
Dai, J. C., Upadhyay, S., Aïvodji, U., Bach, S. H., & Lakkaraju, H. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534159 · Cited by 127
“[33] conduct a similar survey of practitioners, again identifying a variety of stakeholders across the model lifecycle, and highlight three core goals: model improvement, making decisions, and building trust and confidence in the model.”
BackgroundMaking It Possible for the Auditing of AI: A Systematic Review of AI Audits and AI Auditability
Li, Y., & Goel, S. (2024) · Information Systems Frontiers · doi:10.1007/s10796-024-10508-8 · Cited by 103
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55
“The key challenges when designing efective warning interfaces include the presentation of comprehensible messages and supporting evidence [15, 54], engaging users [50, 202], and preventing alert fatigue and habituation [5, 17].”
BackgroundFits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop
Crisan, A., & Fiore-Gartland, B. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445775 · Cited by 72
“As an example, Hong et al. [22] interviewed 20 individuals across different domains (the majority of whom identified as data scientists), and found that collaboration among different organizational roles was of chief importance for operationalizing machine learning models into organizational practices.”
BackgroundResultInfluentialExploring the effects of human-centered AI explanations on trust and reliance
Scharowski, N., Perrig, S. A. C., Svab, M., Opwis, K., & Brühlmann, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1151150 · Cited by 57
Banovic, N., Yang, Z., Ramesh, A., & Liu, A. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579460 · Cited by 55
“The trustworthy coach was everything that the existing XAI literature says is needed to build trust in AI.”
BackgroundMethodInfluentialLiu, Y., Liu, C., Zheng, J., Xu, C., & Wang, D. (2025) · Journal of Medical Internet Research · doi:10.2196/73374 · Cited by 80
“HCPs need to not only understand the normal decision-making logic of the model but also recognize the conditions under which the model may fail or generate incorrect decisions [13,39,42].”
InfluentialA.I. Robustness: a Human-Centered Perspective on Technological Challenges and Opportunities
Tocchetti, A., Corti, L., Balayn, A., Yurrita, M., Lippmann, P., Brambilla, M., & Yang, J. (2024) · ACM Computing Surveys · doi:10.1145/3665926 · Cited by 98
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 62
“[40] termed the people who find and test these hypotheses “model breakers,” roles who interact with customers and may have more direct knowledge of the ways in which a model may behave.”
BackgroundSeamful XAI: Operationalizing Seamful Design in Explainable AI
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637396 · Cited by 105
“However, recent work studying how practitioners deal with RAI issues on the ground, including AI fairness [17, 38, 55, 63], transparency [35, 40, 50], accountability [70], and overall harms mitigation [4, 71], report that practitioners grapple with tremendous challenges [67].”
BackgroundHopkins, A., & Booth, S. (2021) · Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3461702.3462527 · Cited by 40
“[25] suggested this style of proding-and-probing experimentation did not meet practitioners’ needs for transparency, we instead discovered that interviewees found this process of experimentation and rigorous documentation and versioning mitigated their concerns about using black box models.”
BackgroundInfluentialAligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment
Gao, Y., Sun, T., Zhao, L., & Hong, S.-S. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555590 · Cited by 58
“…networks (DNNs) are becoming the powerhouse of innovation in our society; they drive vehicles on behalf of humans [Tesla 2020], replace repetitive tasks in banking and finance industry [Hong et al. 2020], and provide powerful support for border control agencies to boost security [Kendall 2020].”
BackgroundInfluentialSelf-citationLi, H., Appleby, G., Brumar, C. D., Chang, R., & Suh, A. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326904 · Cited by 34
Epistemic values in feature importance methods: Lessons from feminist epistemology
Hancox-Li, L., & Kumar, I. E. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445943 · Cited by 64
“User studies back up the idea that explanation methods are used in very different contexts, for different purposes [35, 36].”
BackgroundInfluentialGNES: Learning to Explain Graph Neural Networks
Gao, Y., Sun, T., Bhatt, R., Yu, D., Hong, S.-S., & Zhao, L. (2021) · Proceedings · doi:10.1109/icdm51629.2021.00023 · Cited by 48
“As Deep Neural Networks (DNNs) are widely deployed in sensitive application areas, recent years have seen an explosion of research in understanding how DNNs work under the hood (e.g., explainable AI, or XAI) [1], [2] and more importantly, how to improve DNNs using human knowledge [3].”
BackgroundSelf-citationGuo, R., Wei, J., Sun, L., Yu, B., Chang, G., Liu, D., Zhang, S., Yao, Z., Xu, M., & Bu, L. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108709 · Cited by 69
Gebreyesus, Y., Dalton, D., Chiara, D. D., Chinnici, M., & Chinnici, A. (2024) · Electronics · doi:10.3390/electronics13091628 · Cited by 48
Designing Interactive Transfer Learning Tools for ML Non-Experts
Mishra, S., & Rzeszotarski, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445096 · Cited by 32
“One approach towards striking a balance between too few and too many afordances is to adopt a humancentered design methodology in designing iML tools [25, 56, 57].”
RES: A Robust Framework for Guiding Visual Explanation
Gao, Y., Sun, T., Bai, G., Gu, S., Hong, S.-S., & Liang, Z. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539419 · Cited by 34
“However, the research on supervising explanations on image data—where the explanation is represented through saliency maps—is still under-explored [19].”
BackgroundSelf-citationChatzimparmpas, A., Martins, R. M., Kucher, K., & Kerren, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3141040 · Cited by 38
“Another important reason why feature engineering is essential in real-world problems is that it increases the transparency and trustworthiness of the data and, in consequence, the ML process in general [4].”
BackgroundThe Need for Interpretable Features: Motivation and Taxonomy
Zytek, A., Arnaldo, I., Liu, D., Berti‐Équille, L., & Veeramachaneni, K. K. (2022) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3544903.3544905 · Cited by 57
“Importance of IFs: Discusses or claims the general importance of interpretable features 47 [3] [7] [16]”
BackgroundSubramonyam, H., Thakkar, D., Ku, A., Dieber, J., & Sinha, A. K. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713166 · Cited by 80
“In recent years, work across academia and industry has formulated design guidelines for task-specific ML models including designing interactions [2, 20, 27], explainability [75], privacy [26, 36], trans-parency [7, 29, 43], etc.”
Where Do Stories Come From? Examining the Exploration Process in Investigative Data Journalism
Showkat, D., & Baumer, E. P. S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479534 · Cited by 36
“Such efforts would require further user studies with various groups of users including journalists to understand its usefulness, acceptance, and the intricacies that might be involved [43, 54, 92].”
BackgroundWinecoff, A., & Watkins, E. A. (2022) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3514094.3534138 · Cited by 48
“Studies of AI ethics in organizational contexts often focus on interventions such as model fairness [22, 31, 32] and model interpretability techniques [4, 23, 29].”
BackgroundHow Do We Measure Trust in Visual Data Communication?
Elhamdadi, H., Gaba, A., Kim, Y., & Xiong, C. (2022) · IEEE Evaluation and Beyond - Methodological Approaches for Visualization (BELIV) · doi:10.1109/beliv57783.2022.00014 · Cited by 33
“For example, Hong et al. measured trust via the qualitative coding of existing research papers in machine learning that mention trust [32].”
BackgroundPfeuffer, N., Baum, L., Stammer, W., Abdel-Karim, B. M., Schramowski, P., Bucher, A. M., Hügel, C., Rohde, G. G. U., Kersting, K., & Hinz, O. F. (2023) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00806-x · Cited by 41
“As Hong et al. (2020) contend, it is crucial to design XAI features according to the different users’ needs, such that these users can effectively engage in the model inspection.”
BackgroundInfluentialRequirements Engineering in Machine Learning Projects
Gjorgjevikj, A., Mishev, K., Antovski, L., & Trajanov, D. (2023) · IEEE Access · doi:10.1109/access.2023.3294840 · Cited by 38
“The implementation of the requirements is described in [11]. local explanations, explanations by example, and etc., further discussed in [60]), it is essential that they are aligned with the user mental models, needs, and use cases [62].”
BackgroundMIPD: An Adaptive Gradient Sparsification Framework for Distributed DNNs Training
Zhang, Z., & Wang, C. (2022) · IEEE Transactions on Parallel and Distributed Systems · doi:10.1109/tpds.2022.3154387 · Cited by 46
“The growing recognition of the need to understand DL models has shifted from a unidirectional focus on model accuracy to a larger perspective including a strong focus on DL interpretability [10].”
BackgroundToward Human-centered XAI in Practice: A survey
Kong, X., Liu, S., & Zhu, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-022-1407-3 · Cited by 50
Hohman, F., Kery, M. B., Ren, D., & Moritz, D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642109 · Cited by 50
“Many other aspects of AI fairness, accountability, transparency, and ethics have been found to be practitioner-driven, where design decisions and process have enormous impact [21, 43, 46, 66, 67, 72, 86, 87].”
BackgroundA Survey of Domain Knowledge Elicitation in Applied Machine Learning
Kerrigan, D., Hullman, J. R., & Bertini, E. (2021) · Multimodal Technologies and Interaction · doi:10.3390/mti5120073 · Cited by 47
Self-citationAn Interpretable Time Series Data Prediction Framework for Severe Accidents in Nuclear Power Plants
Fu, Y., Zhang, D., Xiao, Y., Wang, Z., & Zhou, H. (2023) · Entropy · doi:10.3390/e25081160 · Cited by 27
“Therefore, in terms of practical engineering deployment, pursuing model interpretability represents a critical challenge for artificial intelligence models at this stage [11].”
BackgroundAdViCE: Aggregated Visual Counterfactual Explanations for Machine Learning Model Validation
Gómez, Ó., Holter, S., Yuan, J., & Bertini, E. (2021) · Visualization · doi:10.1109/vis49827.2021.9623271 · Cited by 38
Self-citationChildren's Mental Models of AI Reasoning: Implications for AI Literacy Education
Dangol, A., Wolfe, R., Zhao, R., Kim, J., Ramanan, T., Davis, K., & Kientz, J. A. (2025) · Proceedings of the 24th Interaction Design and Children · doi:10.1145/3713043.3728856 · Cited by 25
“One promising approach is to integrate AI explainability into these interactions, addressing common misconceptions about how AI makes decisions [41, 74].”
Farnoosh, R., & Abnoosian, K. (2024) · International Journal of Systems Assurance Engineering and Management · doi:10.1007/s13198-024-02354-3 · Cited by 32
“Machine learning is useful in healthcare because it can produce large datasets beyond what is possible for humans to do and then depend-ably modify the analysis of that data (Pattnayak and Panda 2021; Hong et al. 2020).”
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
Alvarez-Melis, D., Kaur, H., Daumé, H., Wallach, H., & Vaughan, J. W. (2021) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v9i1.18938 · Cited by 44
“Interpretability can enable model debugging and lead to more robust ML systems, support knowledge discovery, and boost trust (Hong, Hullman, and Bertini 2020).”
BackgroundKaate, I., Salminen, J. O., Jung, S., Xuan, T. T. T., Häyhänen, E., Azem, J. Y., & Jansen, B. J. J. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712160 · Cited by 53
“Trust Issues [14, 23, 36]: If users perceive AI-generated personas as unreliable or biased, it could undermine trust and reduce the effectiveness of personas in design processes.”
Suresh, H., Lewis, K. M., Guttag, J. V., & Satyanarayan, A. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511160 · Cited by 32
Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
Wang, Z. J., Kale, A. M., Nori, H., Stella, P., Nunnally, M., Chau, D. H., Vorvoreanu, M., Vaughan, J. W., & Caruana, R. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539074 · Cited by 44
“In practice, data scientists and domain experts often compare model interpretations with their knowledge [15].”
BackgroundA Systematic Literature Review on Machine Learning Algorithms for Human Status Detection
Sardar, S. K., Kumar, N., & Lee, S. C. (2022) · IEEE Access · doi:10.1109/access.2022.3190967 · Cited by 25
Suh, A., Appleby, G., Anderson, E. W., Finelli, L. A., Chang, R., & Cashman, D. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3259341 · Cited by 26
“In prior literature, studies have been done to understand AI/ML industry practices across a multitude of domains [7], in addition to sectors outside of ‘Big Tech’ [29].”
BackgroundKorica-Pehserl, P., Gayar, N. E., & Pang, W. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-91608-4_33 · Cited by 24
“• ML Ops: Integration and automation of XAI methods into ML model life cycle and deployment model, see [12, 20, 27, 28].”
MethodRahman, S., & Kandogan, E. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502068 · Cited by 28
“…engagement ( e.g. , collaborative [34, 82], single user [46, 70]), modalities ( e.g. , data [46, 50]), tools ( e.g. , note-book [35]), roles ( e.g. , with domain experts [43]), themes ( e.g. , interpretability [29], trust [51], explanation [60]), and goals ( e.g. , wrangling [26]), among others.”
Benjamin, J. J., Kinkeldey, C., Müller-Birn, C., Korjakow, T., & Herbst, E.-M. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492858 · Cited by 22
A Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mc762 · Cited by 26
Goodarzi, P., Schütze, A., & Schneider, T. (2025) · Journal of sensors and sensor systems · doi:10.5194/jsss-14-119-2025 · Cited by 25
Explainable Machine Learning in Risk Management: Balancing Accuracy and Interpretability
Wang, M., Zhang, X., Yang, Y., & Wang, J. (2025) · Journal of Financial Risk Management · doi:10.4236/jfrm.2025.143011 · Cited by 22
Explainable AI in Finance: Addressing the Needs of Diverse Stakeholders
Wilson, C. H. (2025) · doi:10.56227/25.1.25 · Cited by 39
Yin, X., Xu, B. L., Hu, L., Li, H., & He, W. (2023) · Scientific Reports · doi:10.1038/s41598-023-41305-z · Cited by 15
Kaur, H., Conrad, M. R., Rule, D., Lampe, C. A. C., & Gilbert, É. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637354 · Cited by 33
“…[16, 95, 128, 137]; how to present explanations to lay users, novices, and experts [100, 133]; which types of uncertainty information are helpful [48, 103, 139]; what causes information overload [13, 119]; whether explanations should presented as questions, answers, or a dialogue [23, 69, 70];…”
BackgroundDangol, A., Zhao, R., Wolfe, R., Ramanan, T., Kientz, J. A., & Yip, J. C. (2025) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3713043.3728836 · Cited by 22
Martinović, B., Bijanić, M., Danilović, D., Petrović, A., & Delibašić, B. (2023) · Mathematics · doi:10.3390/math11234782 · Cited by 18
3DPFIX: Improving Remote Novices' 3D Printing Troubleshooting through Human-AI Collaboration Design
Kwon, N., Sun, T., Gao, Y., Zhao, L., Wang, X., Kim, J., & Hong, S.-S. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637288 · Cited by 13
“While social sensemaking and annotation have inspired numerous applications in CSCW, there have been relatively a scarce of approaches that leverage AI in facilitating novices’ sensemaking process, e.g., [30, 34, 35, 58, 76] in troubleshooting.”
Self-citationAI and Financial Risk Management
Faisal, S. M., Khan, W., & Ishrat, M. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-1200-2.ch014 · Cited by 12
Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank
Sipos, L., Schäfer, U., Glinka, K., & Müller-Birn, C. (2023) · Mensch und Computer (MuC) · doi:10.1145/3603555.3608551 · Cited by 27
Challenges in Evaluating Interactive Visual Machine Learning Systems
Boukhelifa, N., Bezerianos, A., Chang, R., Collins, C., Drucker, S. M., Endert, A., Hullman, J. R., North, C. L., Sedlmair, M., & Rhyne, T.-M. (2020) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2020.3017064 · Cited by 12
“an interpretable IVML may be aimed at optimizing for analysts’ insight [20].”
BackgroundSelf-citationHopkins, A., & Booth, S. (2021) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 12
Personalized emotion analysis based on fuzzy multi-modal transformer model
Liu, J., Ang, M. C., Chaw, J. K., Ng, K. W., & Kor, A.-L. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05954-5 · Cited by 18
“Human emotions are intricate and elusive, constantly shifting in the real world, which makes them challenging to decipher [38].”
Almahmoud, J., DeLine, R. A., & Drucker, S. M. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3463934 · Cited by 18
“Work on bias mitigation looked into the interpretability and transparency of these models [27, 59, 60], what industry practitioners need to improve the fairness in ML systems [29, 30], and the perceived fairness of”
BackgroundGentile, D., Donmez, B., & Jamieson, G. A. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103945 · Cited by 15
Turri, V., Morrison, K., Robinson, K.-M., Abidi, C., Perer, A., Forlizzi, J., & Dzombak, R. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658985 · Cited by 10
On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
Amarasinghe, K., Rodolfa, K. T., Jesus, S., Chen, V., Balayan, V., Saleiro, P., Bizarro, P., Talwalkar, A., & Ghani, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30082 · Cited by 13
Tracing and Visualizing Human-ML/AI Collaborative Processes through Artifacts of Data Work
Rogers, J., & Crisan, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580819 · Cited by 25
“This gap is significant for the development of AutoML systems, as it does not account for the diversity of teams involved in human-ML/AI collaboration [4, 14, 37, 40, 55, 72, 102, 109].”
BackgroundSun, T., Gao, Y., Khaladkar, S., Liu, S., Zhao, L., Kim, Y., & Hong, S.-S. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610187 · Cited by 16
“The directions include understanding how they use XAI tools [43], how ML beginners learn XAI tools to work on their model building [99, 100], how ML experts view the automated AI [89], how ML experts collaborate in using XAI tools, and beyond [53, 102].”
BackgroundInfluentialSelf-citationYu, S. (2023) · International Academic Mindtrek Conference (Mindtrek) · doi:10.1145/3616961.3616965 · Cited by 17
“To build an explainable AI system, it is necessary for human-human collaboration to distinguish roles, processes, goals, and strategies in different organizations and AI environments [39].”
BackgroundTowards Visual Analytics for Explainable AI in Industrial Applications
Kucher, K., Zohrevandi, E., & Westin, C. (2025) · Analytics · doi:10.3390/analytics4010007 · Cited by 17
“…concerns about relying on black-box models have been voiced from a more general perspective [22], but also with respect to studies of human factors in model interpretability within industrial settings [83], including the identified needs to communicate and summarize model behavior, for instance.”
InfluentialExplainable AI for online disinformation detection: Insights from a design science research project
Bezzaoui, I., Stein, C., Weinhardt, C., & Fegert, J. (2025) · Electronic Markets · doi:10.1007/s12525-025-00799-3 · Cited by 21
“To promote trust in the system, it is recommended to reduce the knowledge gap between the user and the system by collaborating with users during the XAI development lifecycle (Chromik, 2021; Hong et al., 2020; Park et al., 2021).”
Balayn, A., Rikalo, N., Yang, J., & Bozzon, A. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581555 · Cited by 18
“Our results identify additional communication needs from the developer to non-developers, especially for defning when a model is suitable for deployment, whether specifc failures on single samples are acceptable, and which features one should expect [47, 88, 118].”
BackgroundMethodResultInfluentialReview on Artificial Intelligence and Human Computer Interaction
Kushwaha, S. (2023) · OPJU International Technology Conference on Smart Computing for Innovation and Advancement in Industry (OTCON) · doi:10.1109/otcon56053.2023.10114002 · Cited by 15
Metadata
- Authors
- Hong, S. R., Hullman, J., & Bertini, E.
- Affiliations
- New York University, New York, NY, USA
- Northwestern University (US)
- New York University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-05-28
- Volume
- 4
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3019489177
- Cited by
- 219
- References
- 74
- Citations by year
- 2026: 20 · 2025: 44 · 2024: 42 · 2023: 41 · 2022: 31 · 2021: 36 · 2020: 5
- FWCI
- 11.01
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model interpretability, human factors, industry professionals, machine learning adoption, mental models, organizational trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.