Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs
Open accessThe DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Abstract
As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Interpretable models provide a common language for these experts by clearly explaining the inner workings of the detection system [80].”
From Deepfake Media Forensics: Status and Future Challenges (2025) · cites this paper as [80]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Note that we chose to provide SHAP explanations for one black-box model because it emulates how black-box models are most commonly implemented in-practice [10,36].”
From It’s Just Not That Simple: An Empirical Study of the Accuracy-Explainability Trade-off in Machine Learning for Public Policy (2022) · cites this paper as [36]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Thus, it remains unclear if existing XAI methods meet the end-users' needs when trying to assess trustworthiness of an AI [24,32].”
From Being Trustworthy is Not Enough: How Untrustworthy Artificial Intelligence (AI) Can Deceive the End-Users and Gain Their Trust (2023) · cites this paper as [24]
In the citing paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“scribe its behavior. Examples include methods that infer feature relevance Proc. ACM Hum.-Comput. Interact., Vol. 4, No. CSCW1, Article 68. Publication date: May 2020. 68:4 Hong et al. from the model [18] as well as methodsthat build whole surrogates that try to mimic the model’s behavior by building a more transparen…”
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
Lewis, S. (2015) · Health Promotion Practice · doi:10.1177/1524839915580941 · Cited by 22,105
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
The Coding Manual for Qualitative Researchers
Lungu, M. (2022) · American Journal of Qualitative Research · doi:10.29333/ajqr/12085 · Cited by 10,826 · cited 1×
“esent the perspectives of our participants systematically curated in an interpretive framework which was developed through discussions among the authors. We used aniterative qualitative coding process[66] characterizedby alternate phases ofcoding (to tag specific text segments with codes), analytic memo writing (to col…”
Lipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427 · cited 1×
“We describe the methodology we followed to conduct the interviews and to analyze the data we collected.”
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 1×
“In addition to proposing new algorithms and tools [6, 42, 64, 68, 72], the growing literature on interpretability includes attempts to define interpretability with scientific rigor [14, 15, 34, 67].”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“Doshi-Velez and Kim see model interpretability as a proxy for many “auxiliary criteria” such as safety, fairness, non-discrimination, avoiding technical debt, privacy, reliability, providing the right to explanation, trust, and more [14].”
BackgroundMethodExplaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 1×
“As several studies have shown, ML models can learn spurious associations [8, 19] and make decisions based on correlations that do not result from causal relationships.”
BackgroundAnchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 1×
“Anchors aims at providing instance explanations using a rule structure [63].”
Method
Research works printed but not linked: 4
- Andre Esteva , Brett Kuprel , Roberto A Novoa , Justin Ko , Susan M Swetter , Helen M Blau , and Sebastian Thrun . 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 ( 2017 ), 115. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. 2017. Dermatologist-level Classification of Skin Cancer with Deep Neural Networks. Nature 542, 7639 (2017), 115.
- Sungsoo Ray Hong , Jorge Piazentin Ono , Juliana Freire , and Enrico Bertini . 2019 . Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process . In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM. Sungsoo Ray Hong, Jorge Piazentin Ono, Juliana Freire, and Enrico Bertini. 2019. Disseminating Machine Learning to domain experts: Understanding challenges and opportunities in supporting a model building process. In CHI 2019 Workshop, Emerging Perspectives in Human-Centered Machine Learning. ACM.
- Zachary Chase Lipton . 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 ( 2016 ). http://arxiv.org/abs/1606.03490 Zachary Chase Lipton. 2016. The Mythos of Model Interpretability. CoRR abs/1606.03490 (2016). http://arxiv.org/abs/1606.03490 · arXiv
- Scott M Lundberg and Su-In Lee . 2017. A Unified Approach to Interpreting Model Predictions . In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates , Inc ., 4765--4774. Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30 (NIPS '17. Curran Associates, Inc., 4765--4774.
Cited by
Items 101–200 of 217
Are We Closing the Loop Yet? Gaps in the Generalizability of VIS4ML Research
Subramonyam, H., & Hullman, J. R. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326591 · Cited by 16
Self-citationZhou, Y., Fukuda, T., & Yabuki, N. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106582 · Cited by 12
What can crowd computing do for the next generation of AI systems
Gadiraju, U., & Yang, J. (2020) · Research Repository (Delft University of Technology) · Cited by 22
Rolwes, A., & Böhm, K. (2021) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xlvi-4-w1-2021-97-2021 · Cited by 9
“They tend to put personal assessments on geospatial factors, even though they are not in the position to evaluate them neutrally (Hong et al., 2020).”
BackgroundAra, Z., Salemi, H., Hong, S.-S., Senarath, Y., Peterson, S., Hughes, A., & Purohit, H. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645214 · Cited by 11
“…sets directly contributes to the quality of Machine Learning (ML) models, research in data annotation focusing on producing high-quality labels has attracted attention from several communities, including HCI [19, 22, 28], Natural Language Processing (NLP) [8], Computer Vision [40], and beyond.”
BackgroundSelf-citationShadowMagic: Designing Human-AI Collaborative Support for Comic Professionals’ Shadowing
Ganguly, A., Yan, C., Chung, J. J. Y., Sun, T., Kiheon, Y., Gingold, Y. I., & Hong, S.-S. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676332 · Cited by 14
“They followed an iterative qualitative coding process [24, 39] that starts from coding and then connecting the repeating or re-lated insights in developing analytic memos.”
Self-citationExplainable Federated Learning: A Lifecycle Dashboard for Industrial Settings
Ungersböck, M., Hießl, T., Schall, D., & Michahelles, F. (2023) · IEEE Pervasive Computing · doi:10.1109/mprv.2022.3229166 · Cited by 8
“11 For example, if service engineers know about industrial pumps that work in similar environments and need to be monitored regarding their temperature, they can suggest using FL to improve anomaly detection models.”
BackgroundMethodInfluentialSeamful XAI: Operationalizing Seamful Design in Explainable AIPreprint
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2022) · arXiv (Cornell University) · doi:10.13016/at4q-63xb · Cited by 8
Sampaio, I. G. B., Fontes, S., Andrade, E. D. O., Bernardini, F., & Viterbo, J. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657274 · Cited by 9
“On the other hand, interpretability refers to the degree to which a human can understand the inner workings of a model and predict its output systematically [10].”
BackgroundAn Emerging Design Space of How Tools Support Collaborations in AI Design and Development
Muralikumar, M. D., & McDonald, D. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701181 · Cited by 11
“In fact, interpretability and explainability of AI models act as enablers of, and sometimes are the object of, collaborations to support sense-making in organizations [51].”
InfluentialEvaluating Non-AI Experts' Interaction with AI: A Case Study In Library Context
Zheng, Q.-X., Chen, M.-R., Park, H., Xu, Z., & Huang, Y. (2025) · International Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714219 · Cited by 9
“Effective interaction with AI depends on an interpretable and explainable understanding of how it works [45], particularly for non-AI experts.”
"Is It My Turn?": Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
Benk, M., Weibel, R. P., Feuerriegel, S., & Ferrario, A. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555580 · Cited by 14
“, legal and ethics experts), among others [43, 44].”
BackgroundTowards a Non-Ideal Methodological Framework for Responsible ML
Mothilal, R. K., Guha, S., & Ahmed, S. I. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642501 · Cited by 16
“Further, applying fairness metrics is not the only approach practitioners employ for RML.”
BackgroundLopez, M. G., & Halford, S. (2024) · Information Communication & Society · doi:10.1080/1369118x.2024.2400130 · Cited by 12
“Moreover, case studies of XAI systems in use show that these are not only interpreted at an individual cognitive level but can become the object of collective sense-making (Borch & Hee Min, 2022; Hong et al., 2020).”
A Prescriptive Machine Learning Approach for Assessing Goodwill in the Automotive Domain
Haas, S., & Hüllermeier, E. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26422-1_11 · Cited by 7
Graells-Garrido, E., Peña-Araya, V., & Bravo, L. (2020) · Sustainability · doi:10.3390/su12156001 · Cited by 7
“Furthermore, different roles also have different interpretations [24].”
BackgroundJin, W., Fan, J., Gromala, D., Pasquier, P., & Hamarneh, G. (2021) · arXiv (Cornell University) · Cited by 13
Evolutionary Reinforcement Learning for Interpretable Decision-Making in Supply Chain Management
Genetti, S., Longobardi, A., & Iacca, G. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-90062-4_12 · Cited by 11
“While these models can effectively support decision-making, they are often “black-box” [13], meaning their internal workings are not visible or comprehensible, which makes it difficult to trace how decisions are made.”
From Discovery to Adoption: Understanding the ML Practitioners’ Interpretability Journey
Ashtari, N., Mullins, R. S., Qian, C., Wexler, J., Tenney, I., & Pushkarna, M. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596046 · Cited by 13
“Method selection is most often determined by popularity of the methods [53], familiarity with the model building process [93, 104], and organizational practices of downstream stakeholders [49].”
BackgroundHaas, S., & Hüllermeier, E. (2024) · International Journal of Data Science and Analytics · doi:10.1007/s41060-024-00573-2 · Cited by 10
Khan, A. Q., Muhammad, S. G., Raza, A., & Pimanmas, A. (2026) · Journal of Road Engineering · doi:10.1016/j.jreng.2025.07.002 · Cited by 5
Personalized Guidelines for Design, Implementation and Evaluation of Anti-Phishing Interventions
Sarker, O., Haggag, S., Jayatilaka, A., & Liu, C. (2023) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem56168.2023.10304861 · Cited by 5
Thomas, I., Oh, S. Y., & Szafir, D. A. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645207 · Cited by 8
“These visualizations can help people build better mental models of an algorithm, leading to more efficient human-AI teaming in a range of classification tasks [4, 36, 83].”
BackgroundMental Models and Interpretability in AI Fairness Tools and Code Environments
Thompson, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-90963-5_43 · Cited by 8
Keelawat, P. (2023) · ACM Conference on Computer Supported Cooperative Work (CSCW) · doi:10.1145/3584931.3607020 · Cited by 10
“A lack of common ground is a serious problem that can lead to skepticism and trustworthiness of other members’ output [21, 43].”
BackgroundWang, D., Liu, K., Mohaisen, A., Wang, P., Lu, C., & Fu, Y. (2021) · Frontiers in Big Data · doi:10.3389/fdata.2021.762899 · Cited by 9
“Thus how to enhance the model interpretability attracts much attention of researchers (Elshawi et al., 2019; Hong et al., 2020; Stiglic et al., 2020; Poursabzi-Sangdeh et al., 2021).”
BackgroundAbnoosian, K., & Farnoosh, R. (2023) · Research Square · doi:10.21203/rs.3.rs-3310199/v1 · Cited by 5
ARTIFICIAL INTELLIGENCE IN CYBERSECURITY: THREATS, DEFENSES, AND FUTURE DIRECTIONS
Bakhronkulova, L., Ali, M., Khabirova, Z., Azimjon, A., Zebo, A., & Muslima, A. (2025) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2025vol2.8592 · Cited by 8
“A primary concern is ensuring the reliability and trustworthiness of the data used to train AI models, and the interpretability of AI-driven decisions poses a significant issue, as opaque "black-box" models can lead to false positives or missed threats [21][15].”
Explainable AI for Daily Scenarios from End-Users’ Perspective: Non-Use, Concerns, and Ideal Design
Wang, L., Anyi, C. L., Xu, K., Liu, Y., Arriaga, R. I., & Goel, A. K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735796 · Cited by 9
“However,empiricalstudieshavesofarfo-cusedonthemodeldevelopmentstage[13,58,76]oronhigh-stakes deploymentscenarios(e.g. medical[21,22,114]andjurisdictional [55,99]decisionmaking).”
AI-Supported EUD for Data Visualization: An Exploratory Case Study
Beschi, S., Fogli, D., Gargioni, L., & Locoro, A. (2025) · Future Internet · doi:10.3390/fi17080349 · Cited by 10
Dangol, A., Wolfe, R., Dewitt, A., Chickadel, B., Kientz, J. A., & Dasgupta, S. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3762807 · Cited by 8
Interpretability taxonomy: an approach for artificial intelligence developers technique selection
Cousineau, C., & Rozita, D. (2025) · Discover Data · doi:10.1007/s44248-025-00096-6 · Cited by 5
“Interpretability is intended to provide alignment for the AI end user’s mental framing of how the AI system operates [24].”
InfluentialIs this a violation? Learning and understanding norm violations in online communities
dos Santos, T. F., Osman, N., & Schorlemmer, M. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.104058 · Cited by 9
Understanding Human-side Impact of Sampling Image Batches in Subjective Attribute Labeling
Chung, C., Lee, J., Park, K., Lee, J., Kim, M., Song, M., Kim, Y., Choo, J., & Hong, S.-S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476037 · Cited by 6
“In general, the data collection acts as a barrier for both researchers and practitioners in building a machine learning (ML) models that can be applied in advanced and contextualized scenarios [33, 40].”
BackgroundSelf-citationCommunicating Performance of Regression Models Using Visualization in Pharmacovigilance
Suh, A., Appleby, G., Anderson, E. W., Finelli, L. A., & Cashman, D. (2021) · Workshop on Visual Analytics in Healthcare (VAHC) · doi:10.1109/vahc53616.2021.00006 · Cited by 5
“Stakeholders and SMEs often rely on model builders themselves to translate the reliability and limitations of predictive models, under the assumption that these translations will be accessible by the audience [14].”
BackgroundInfluentialHaas, S., Hegestweiler, K., Rapp, M. T., Muschalik, M., & Hüllermeier, E. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1471208 · Cited by 6
Governance of AI-Based Algorithms
Okolo, C. T. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_84 · Cited by 4
Zhou, H., Zhu, Z., Oh, K. M., & Hong, S.-S. (2025) · JMIR Formative Research · doi:10.2196/79975 · Cited by 5
“Future systems should incorporate mechanisms for continuously updating their knowledge base with validated sources, ensuring that recommendations [85-89] remain aligned with current standards of care.”
Self-citationConcept-Oriented Self-Explaining Neural Networks
Park, M. S., & Hwang, H. J. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11355-5 · Cited by 3
“However, as surrogate models must be trained for each instance separately, it has been reported that explanations by these surrogate models are usually unstable and volatile for every run [4].”
BackgroundUnderstanding Interpretability in Modeling Biotechnological Processes: A Critical Review
Aguirre-Zapata, E., Álvarez, H., Lema-Pérez, L., di Sciascio, F. A., & Amicarelli, A. (2023) · Workshop on Information Processing and Control · doi:10.1109/rpic59053.2023.10530792 · Cited by 3
“It explains the relationship between variables and the label (or dependent variable to be predicted) for a particular case or subset of cases [22], [23].”
BackgroundInfluentialGardasevic, S., & Lamba, M. (2024) · Digital Library Perspectives · doi:10.1108/dlp-02-2024-0025 · Cited by 6
Global explanation supervision for Graph Neural Networks
Etemadyrad, N., Gao, Y., Dinakarrao, S. M. P., & Zhao, L. (2024) · Frontiers in Big Data · doi:10.3389/fdata.2024.1410424 · Cited by 6
“However, similar to other DNNs’ architectures, GNNs also offer only limited transparency, imposing significant challenges in observing when GNNs make successful/unsuccessful predictions (Hong et al., 2020; Wu et al., 2021).”
BackgroundHybrid Models in Diabetes Prediction: A Review of Techniques, Performance, and Potential
Zaid, M. M. A., & Mohammed, A. A. (2024) · Journal of Al-Qadisiyah for Computer Science and Mathematics · doi:10.29304/jqcsm.2024.16.41805 · Cited by 7
Recourse, Repair, Reparation, & Prevention: A Stakeholder Analysis of AI Supply Chains
Hopkins, A., Struckman, I., Klyman, K., & Silbey, S. S. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732017 · Cited by 18
Towards Human-Centered Early Prediction Models for Academic Performance in Real-World Contexts
Zhang, H., Ren, Y., Nurius, P. S., Mankoff, J. C., & Dey, A. K. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757433 · Cited by 10
“Researchers have increasingly advocated for integrating technical innovations in ML/AI with human needs and social values [9, 14, 78, 94, 109, 158, 171].”
Characterizing Data Scientists’ Mental Models of Local Feature Importance
Collaris, D., Weerts, H., Miedema, D., van Wijk, J. J., & Pechenizkiy, M. (2022) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3546155.3546670 · Cited by 10
“Instability may cause users to be reluctant to use explanation methods [17].”
BackgroundTRIVEA: Transparent Ranking Interpretation using Visual Explanation of black-box Algorithmic rankers
Yuan, J., Bhattacharjee, K., Islam, A., & Dasgupta, A. (2023) · The Visual Computer · doi:10.1007/s00371-023-03055-x · Cited by 4
“We identify the following questions that ranking users might ask to motivate our proposed analytical abstraction: Q1: Which attributes have a strong influence on the ranking, and why?”
Patil, D. T. (2023) · IEEE/ACIS International Conference on Big Data, Cloud Computing, and Data Science (BCD) · doi:10.1109/bcd57833.2023.10466351 · Cited by 3
“Looking ahead, Explainable AI (XAI) [13][14] holds potential in enhancing transparency and interpretability of models.”
BackgroundPredicting Future Cyber Threats: Analysing Trends and Predicting Future Cybersecurity Challenges
-, R. A., -, E. A., -, B. A. A., -, N. A. K. D., & -, K. O. A. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i06.33755 · Cited by 5
“It is essential to understand that the collection of ‘anonymised,’ strongly aggregated data can still carry comparative or predictive information for marginalised and underserved individuals (Hong et al., 2020).”
Modeling Disclosive Transparency in NLP Application Descriptions
Saxon, M., Levy, S., Wang, X., Albalak, A., & Wang, W. Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.153 · Cited by 8
“studies have directly asked users of experimental XAI systems to respond to (Bhatt et al., 2019; Kaur et al., 2020; Hong et al., 2020) the quality and effects of the explanations.”
BackgroundMichael, B., Ise, A., Kawabata, K., & Matsubara, T. (2022) · IEEE Access · doi:10.1109/access.2022.3140758 · Cited by 3
“Learning interpretable models is of paramount importance [27], not only for furnace safety and integration with existing tools [28], but also to necessitate the desired aim of extracting frequency components corresponding to the longterm dynamics, while ignoring short-term anomalies.”
BackgroundAssessing Credibility in Bayesian Networks Structure Learning
Barth, V. B. D. O., Serrão, F. V., & Maciel, C. D. (2024) · Entropy · doi:10.3390/e26100829 · Cited by 2
“…for this narrow user base of ML models in the medical area are diverse, such as the strict regulations of medical protocols [37,44–46], the difficulty in updating models with new data or expert knowledge [37,44–47], and the lack of transparency of the models when making predictions [44,46,47].”
BackgroundResearch, Ethics and Artificial Intelligence Challenges and Opportunities
Cano, C. A. G. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-88304-0_68 · Cited by 3
Behavior Matters: An Alternative Perspective on Promoting Responsible Data Science
Zhang, D., Patil, A., Shoda, Y., Battle, L., & Wall, E. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710932 · Cited by 7
Felice, F. D., Cannito, A. R., Monte, D., & Vitulano, F. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85607-6_22 · Cited by 2
The Atlas of AI Incidents in Mobile Computing: Visualizing the Risks and Benefits of AI Gone Mobile
Bogucka, E. P., Constantinides, M., Velázquez, J. D. M., Šćepanović, S., Quercia, D., & Gvirtz, A. (2024) · International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI) · doi:10.1145/3640471.3680447 · Cited by 4
“These tools, including impact assessment reports [30, 46], harm description templates [5], and databases such as the IBM Risk Atlas [23] and AI Risk Database [41], focus on detailing technical risks for AI practitioners [8, 22], often neglecting the broader societal implications.”
VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level
Yuan, J., Miao, K., Oh, H., Walker, I., Xue, Z., Katolikyan, T., & Cavallo, M. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712147 · Cited by 5
“With the advancement of XAI techniques, numerous empirical studies have explored human-AI decision-making [31] and conducted interviews with domain experts to examine industry practices, challenges, and needs [21, 35].”
Toward faithful and human-aligned self-explanation of deep models
Lee, S., Wang, X., Zhang, A., Han, S., Yao, J., Yi, X., Xie, X., & Cha, M. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00023-9 · Cited by 2
“However, practitioners report unstableexplanationswiththisapproach 11 ,implyingalackoffaithfulness — the degree to which explanations truly represent the model ’ s decisions.”
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
Liao, Q. V., & Varshney, K. R. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.011 · Cited by 480
Interpretable Machine Learning: Moving From Mythos to DiagnosticsPreprint
Chen, V., Li, J., Kim, J. S., Plumb, G., & Talwalkar, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06254 · Cited by 15
“Model debugging is not only a common (Hong et al., 2020; Bhatt et al., 2020), but also well-grounded consumer use case.”
Group Person Identification and Communication Using Deep Learning
Verma, V., Acharya, A., Shahi, S., Kumar, V., & Patil, S. (2024) · International Conference on Emerging Technologies in Computer Science for Interdisciplinary Applications (ICETCS) · doi:10.1109/icetcs61022.2024.10543632 · Cited by 1
“Hong et al.[8] investigated human factors in model interpretability, focusing on industry practices, challenges, and needs related to enhancing the interpretability of machine learning models.”
BackgroundZhang, T., Yang, F., & Zhang, P. (2024) · Digital Health · doi:10.1177/20552076241293498 · Cited by 5
“109 Interpretability frameworks: Developing dedicated interpretability frameworks, such as local interpretable model-agnostic explanations 110 and Shapley additive explanations , 111 to explain model predictions.”
BackgroundGenerative Pattern Learning for Missing Sensor Data Imputation in Industrial IoT
Amuri, B., Zahra, F. T., & Soytürk, M. (2025) · Smart Systems Integration Conference and Exhibition (SSI) · doi:10.1109/ssi65953.2025.11107192 · Cited by 1
“While recent research has explored machine learning-based approaches, existing methods struggle with large missing gaps, domain-specific patterns, and the lack of interpretability in industrial decision-making [6].”
Fu, J., Ding, H., Jiang, Z., Zhang, L., Wang, Y., Chen, Y., Song, Z., Sun, S., & Huang, X. (2025) · Electric Power Systems Research · doi:10.1016/j.epsr.2025.112585 · Cited by 1
Designing a human-centered AI solution with radiologists to improve prostate cancer diagnosis
Saßmannshausen, S. M., Ontika, N. N., Tran, K., Ukleja, K., de Carvalho, A. F. P., & Rohde, M. (2025) · i-com · doi:10.1515/icom-2025-0043 · Cited by 1
“128,129 On the other hand, Berkel et al., found that the participants were focused on relying solely on their skills and judgment, and they disregarded any visual indicators that could potentially disrupt their capability to identify abnormalities.”
A Human-Centered Interpretability Framework Based on Weight of Evidence.
Alvarez-Melis, D., Kaur, H., Daumé, H., Wallach, H., & Vaughan, J. W. (2021) · Cited by 1
SHAI 2023: Workshop on Designing for Safety in Human-AI Interactions
Goyal, N., Hong, S.-S., Mandryk, R. L., Li, T. J.-J., Luther, K., & Wang, D. (2023) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3581754.3584169 · Cited by 1
“Amongst many facets, this will require iterative sociotechnical understanding of human behavior that motivates such outcomes [6]; theoretical models and frameworks that define such behaviors [24]; sociological observation of impact of such human and algorithmic behaviors [16]; computational advances in pursuing resear…”
BackgroundSelf-citationRolwes, A., & Böhm, K. (2023) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi12080305 · Cited by 1
A Domain Adaptive Feature Relevance Analysis for Human Activity Recognition using Transfer Learning
Jemima, P. P., Singh, G., Madhurika, K., & Pandey, A. K. (2023) · International Conference on Research Methodologies in Knowledge Management, Artificial Intelligence and Telecommunication Engineering (RMKMATE) · doi:10.1109/rmkmate59243.2023.10368816 · Cited by 1
Towards a Framework for Interdisciplinary Studies in Explainable Artificial Intelligence
Ziethmann, P., Stieler, F., Pfrommer, R., Schlögl-Flierl, K., & Bauer, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60606-9_18 · Cited by 1
“Inspired by Hong et al., this compass delineates three distinct stakeholder groups: Model Consumers, Model Builders, and Model Breakers [17].”
Balloccu, S., Podda, A. S., Pompianu, L., Saia, R., & Salatino, A. A. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3658530 · Cited by 1
“In an era where digital literacy is becoming synonymous with social empowerment, human-centred modelling and adaptation method can represent innovative solutions [13] that not only speed up the digital transformation [6] but actively contribute to the growth of disadvantaged territories [5].”
BackgroundMURDOC: Transforming Pixels into Perception for Camouflage Detection
Hogue, D., Kastl, Z., Karch, J., Nguyen, D., Schley, L., Lewis, J., Connelly, S., & Weaver, C. (2024) · IEEE/AIAA Digital Avionics Systems Conference (DASC) · doi:10.1109/dasc62030.2024.10748781 · Cited by 1
“Tools like Model Performance Dashboards [20] and Local Interpretable Model-Agnostic Explanations (LIME) [21] enable continuous monitoring of model performance and individual predictions.”
Deepfake Threats Detection and Countermeasures in Cybersecurity
Alauthman, M., Al–Qerem, A., Alateef, S., Hadi, W. M., Aldweesh, A. Y., & Almomani, A. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-0832-6.ch007 · Cited by 1
The Double-Edged Sword of Behavioral Responses in Strategic Classification: Theory and User Studies
Ebrahimi, R., Vaccaro, K., & Naghizadeh, P. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732056 · Cited by 11
Tinkering: A Way Towards Designing Transparent Algorithmic User Interfaces.
Showkat, D. (2021) · IUI Workshops · Cited by 2
Towards Evaluating Exploratory Model Building Process with AutoML SystemsPreprint
Hong, S.-S., Castelo, S., D'Orazio, V. J., Benthune, C., Santos, A. S. R., Langevin, S., Jonker, D., Bertini, E., & Freire, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.00449 · Cited by 6
“Meanwhile, another line of research stresses the importance of gaining an empirical understanding from end-users in designing AutoML systems [30, 31].”
BackgroundSelf-citationUm método Bayesiano orientado a dados para o aprendizado estrutural de Redes Bayesianas
Barth, V. B. D. O. (2023) · doi:10.11606/d.18.2023.tde-10042023-154512 · Cited by 1
Towards automated dermatology triage: deep learning and knowledge-driven approachesPreprint
Wang, M., Kloczko, E., Altayeb, A., Farrugia, M., Gupta, G., Wu, H., & Hirani, N. (2023) · Research Square · doi:10.21203/rs.3.rs-2889033/v1 · Cited by 2
Developing Explainable AI Methods for Predictive Analytics and Prognostics
Alomari, Y. (2025) · Eötvös Loránd Tudományegyetem · doi:10.15476/elte.2024.388 · Cited by 1
Modeling Discolsive Transparency in NLP Application DescriptionsPreprint
Saxon, M., Levy, S., Wang, X., Albalak, A., & Wang, W. Y. (2021) · arXiv (Cornell University)
Krishnan, G., & Hofmann, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.01131 · Cited by 2
“Hong et al[20] explain this in their work by pointing towards the increasing scholarly interest in model interpretability in research communities like ML and HCI, but lack of any knowledge about what practitioners perceive and understand from the presented interpretability.”
BackgroundCognitive Pairwise Comparisons for Human Centric Design in Artificial Intelligence and Data SciencePreprint
Yuen, K. K. F. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4048692
Jin, S., Choi, H.-Y., & Han, K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4200133
Designing a Glyph-Based Polar Chart to Interpret the Results of Machine Learning Models
Nguyen, T. Q., Gentile, D., Jamieson, G. A., Gosine, R. G., & Purmehdi, H. (2023) · Ergonomics in Design The Quarterly of Human Factors Applications · doi:10.1177/10648046231166047
“Previous studies attempted to deploy multiple XAI tools to provide DSs with a more comprehensive picture of the relationships between data inputs and ML model outcomes, usually in terms of feature importance (Gashi et al., 2022; Hong et al., 2020; Kuzlu et al., 2020).”
Model-Agnostic Interpretation via Feature Perturbation Visualization
Marcílio-Jr, W. E., Eler, D. M., & Breve, F. (2023) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi59091.2023.10347141
“Besides defining the interpretability of machine learning with scientific rigor [2] and identifying the human factors in model interpretability, such as practices, challenges, and needs [14], the literature is focused on proposing novel techniques to interpret machine learning models.”
MethodQuality control of plastic sack products using the DMAIC method at PT. XYZ
Ismail, I., & Marwan, M. (2024) · International Journal of Mechanical Computational and Manufacturing Research · doi:10.35335/computational.v13i4.136
“The factors that influence the cause of high product defects are human factors, machines, materials and how they work (Hong et al., 2020; Nugraha et al., 2022; Sgarbossa et al., 2020).”
BackgroundDemonstration of MaskSearch: Efficiently Querying Image Masks for Machine Learning Workflows
Wei, L. L., Yeung, C. Y. E., Yu, H., Zhou, J., He, D., & Bałazińska, M. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3685800.3685859
“However, efficient execution of these queries suffers from insufficient systems support [8].”
BackgroundDesirability vs. feasibility: a research through design inquiry of explainable AI
Corti, L., & Yang, J. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-340553-2.00010-1
Decoding MRI Data Machine Learning for Structural Biomarker Discovery in Modern Radiodiagnosis
Sharma, M., Kumar, P., & Verma, P. (2025) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-5801-7.ch002
Zhou, H., Zhu, Z., Hong, S.-S., & Oh, K. M. (2025) · doi:10.2196/preprints.79975
Self-citationDigitally Enabled AI-Interpreted Salivary Ferning–Based Ovulation Prediction: Feasibility Study
Peebles, E., Finlay, W., Nguyen, T.-M., Barrett, S., Thirumalaraju, P., Kanakasabapathy, M. K., Kandula, H., Sarcione, C., James, K. E., Shafiee, H., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/73028
“In addition to architectural considerations, interpretability will be essential and should remain a central focus throughout model development and implementation [35].”
MaskSearch: Querying Image Masks at Scale
He, D., Zhang, J., Daum, M., Ratner, A., & Bałazińska, M. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00184 · Cited by 5
“Unfortunately, there is a lack of system support to efficiently execute these queries [28].”
Graph-based visual analysis for HVAC system anomaly detection
Cao, T., Wang, Y., Fu, Q., Lu, Y., Chen, J., Li, Y., & Ji, C. (2025) · Energy and Buildings · doi:10.1016/j.enbuild.2025.116751 · Cited by 1
Suvorova, A. V. (2025) · Высшая школа: научные исследования · doi:10.15622/rcai.2025.028
Comprehensive Evaluation of Explainable AI for Multivariate Time-Series Regression Tasks
More, V., Ramesh, K., & Ahmed, N. (2025) · Arabian Journal for Science and Engineering · doi:10.1007/s13369-025-10968-3
Gilal, R., Omar, M., Rejab, M. M., & Gilal, A. R. (2025) · Journal of Software Evolution and Process · doi:10.1002/smr.70071
Machine Learning-Based Management of Climatic Factors in Hydroponic Greenhouses
Selvan, S. A., & Pal, T. (2026) · Smart agriculture · doi:10.1007/978-981-95-3963-5_13 · Cited by 1
Human-Centered Recommender Systems
Hao, S., Ji, L., & Zhang, D. (2026) · Springer eBooks · doi:10.1007/978-981-97-8440-0_119-1 · Cited by 1
Sun, T., Feng, H., Ye, J., Yun, S., Kim, Y., & Hong, S. R. (2026) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3797953
“This requires effort for data annotation and reasoning— typically through outsourcing—which keeps engineers waiting and causes workflow inefficiencies [21, 22, 40].”
Self-citationLiang, X., Chuah, S. H., & Wee, V. E. L. (2026) · Journal of Hospitality and Tourism Technology · doi:10.1108/jhtt-01-2025-0067
Metadata
- Authors
- Hong, S. R., Hullman, J., & Bertini, E.
- Affiliations
- New York University, New York, NY, USA
- Northwestern University (US)
- New York University (US)
- Venue
- Proceedings of the ACM on Human-Computer Interaction
- Published
- 2020-05-28
- Volume
- 4
- Issue
- CSCW1
- Pages
- 1-26
- ISSN
- 2573-0142 (ISSN-L)
- OpenAlex
- W3019489177
- Cited by
- 219
- References
- 74
- Citations by year
- 2026: 20 · 2025: 44 · 2024: 42 · 2023: 41 · 2022: 31 · 2021: 36 · 2020: 5
- FWCI
- 11.01
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings of the ACM on Human-Computer Interaction (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model interpretability, human factors, industry professionals, machine learning adoption, mental models, organizational trust
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.