Transparency in the Wild: Navigating Transparency in a Deployed AI System to Broaden Need-Finding Approaches
Open accessThe DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Abstract
Transparency is a critical component when building artificial intelligence (AI) decision-support tools, especially for contexts in which AI outputs impact people or policy. Effectively identifying and addressing user transparency needs in practice remains a challenge. While a number of guidelines and processes for identifying transparency needs have emerged, existing methods tend to approach need-finding with a limited focus that centers around a narrow set of stakeholders and transparency techniques. To broaden this perspective, we employ numerous need-finding methods to investigate transparency mechanisms in a widely deployed AI-decision support tool developed by a wildlife conservation non-profit. Throughout our 5-month case study, we conducted need-finding through semi-structured interviews with end-users, analysis of the tool’s community forum, experiments with their ML model, and analysis of training documents created by end-users. We also held regular meetings with the tool’s product and machine learning teams. By approaching transparency need-finding from a broad lens, we uncover insights into end-users’ transparency needs as well as unexpected uses and challenges with current transparency mechanisms. Our study is one of the first to incorporate such diverse perspectives to reveal an unbiased and rich view of transparency needs. Lastly, we offer the FAccT community recommendations on broadening transparency need-finding approaches, contributing to the evolving field of transparency research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Turri et al. [57] found that achieving meaningful transparency in a deployed AI system required addressing nuanced user needs and practical limitations far beyond implementing a basic XAI technique.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [57]
In the citing paper
“For this, we recommend a reasoning-driven evaluation of XAI approaches, similar to the framework proposed by Wang et al. [57], which emphasizes human cognitive principles, to systematically determine which XAI methods best align with users' reasoning goals.”
From Explainable AI for Clinical Decision Support Systems: Literature Review, Key Gaps, and Research Synthesis (2025) · cites this paper as [57]
Citations
Cites
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
The Use of Triangulation in Qualitative Research
Carter, N., Bryant-Lukosius, D., DiCenso, A., Blythe, J., & Neville, A. J. (2014) · Oncology Nursing Forum · doi:10.1188/14.onf.545-547 · Cited by 3,915
Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks
Chattopadhay, A., Sarkar, A., Howlader, P., & Balasubramanian, V. N. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00097 · Cited by 3,639
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785
A Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
Cai, C. J., Winter, S., Steiner, D., Wilcox, L., & Terry, M. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359206 · Cited by 594
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405
Research works printed but not linked: 10
- Chirag Agarwal Satyapriya Krishna Eshika Saxena Martin Pawelczyk Nari Johnson Isha Puri Marinka Zitnik and Himabindu Lakkaraju. 2022. OpenXAI: Towards a Transparent Evaluation of Model Explanations. https://arxiv.org/abs/2206.11104v3 · arXiv
- Tanya Y Berger-Wolf, Daniel I Rubenstein, Charles V Stewart, Jason A Holmberg, Jason Parham, Sreejith Menon, Jonathan Crall, Jon Van Oast, Emre Kiciman, and Lucas Joppa. 2017. Wildbook: Crowdsourcing, computer vision, and data science for conservation. arXiv preprint arXiv:1710.08880 (2017). https://arxiv.org/abs/1710.08880 · arXiv
- Hugh Beyer and Karen Holtzblatt. 1999. Contextual design. interactions 6, 1 (1999), 32–42. https://dl.acm.org/doi/fullHtml/10.1145/291224.291229 · DOI · Paper page
- Stephen Casper, Tong Bu, Yuxiao Li, Jiawei Li, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. 2023. Red Teaming Deep Neural Networks with Feature Synthesis Tools. In Thirty-seventh Conference on Neural Information Processing Systems. https://arxiv.org/abs/2302.10894 · arXiv
- Hussein Mozannar, Jimin J Lee, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. 2023. Effective Human-AI Teams via Learned Natural Language Rules and Onboarding. arXiv preprint arXiv:2311.01007 (2023). https://proceedings.neurips.cc/paper_files/paper/2023/file/61355b9c218505505d1bedede9da56b2-Paper-Conference.pdf · arXiv
- Sahil Singla and Soheil Feizi. 2022. Salient ImageNet: How to discover spurious features in Deep Learning?. In International Conference on Learning Representations. https://openreview.net/forum?id=XVPqLyNxSyh
- Simone Stumpf Adrian Bussone and Dympna O’sullivan. 2016. Explanations considered harmful? user interactions with machine learning systems. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). https://www.yumpu.com/cs/document/view/55465112/explanations-considered-harmful-user-interactions-with-machine-learning-systems
- Sana Tonekaboni Shalmali Joshi Melissa D McCradden and Anna Goldenberg. 2019. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference. PMLR 359–380. http://proceedings.mlr.press/v106/tonekaboni19a/tonekaboni19a.pdf
- Michael Williams and Tami Moser. 2019. The art of coding and thematic exploration in qualitative research. International management review 15, 1 (2019), 45–55. https://www.academia.edu/download/82465402/imr-v15n1art4.pdf
- 10.1145/3287560.3287596 · DOI · Paper page
Cited by
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39
“Turri et al. [57] found that achieving meaningful transparency in a deployed AI system required addressing nuanced user needs and practical limitations far beyond implementing a basic XAI technique.”
Dhanorkar, S., Passi, S., & Vorvoreanu, M. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812402 · Cited by 32
“Information provided by AI systems can be difficult to understand for some users and mislead others [133], exacerbating automation bias [127] and overreliance [73, 109].”
Towards semantic versioning of open pre-trained language model releases on hugging face
Ajibode, A., Bangash, A. A., Cogo, F. R., Adams, B., & Hassan, A. E. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10631-3 · Cited by 24
“(Turri et al., 2024) explore transparency needs in AI decision-support tools through a comprehensive case study, highlighting the challenges and diverse requirements of end-users.”
Towards certifiable AI in aviation: landscape, challenges, and opportunitiesPreprint
Bello, H., Geißler, D., Ray, L. S. S., Müller-Divéky, S., Müller, P., Kittrell, S., Liu, M., Zhou, B., & Lukowicz, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.08666 · Cited by 23
“Hence, identifying and addressing users’ transparency needs becomes a challenge and a critical element of the Human-AI teaming [144].”
BackgroundAI Agents Push Humans Out of the LoopPreprint
Mitchell, M., Ghosh, A., & Passi, S. (2026) · arXiv.org · doi:10.48550/arxiv.2608.23642 · Cited by 12
“But the mechanisms users would need to do this effectively are virtually absent [104, 33, 79].”
Who Controls the Conversation? User Perspectives on Generative AI (LLM) System Prompts
Neumann, A., Pi, Y., & Singh, J. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791726 · Cited by 15
Human Resource Management and AI: Building A Collaborative Contextual Transparency Database
Simpson, E., Ermovick, R., & Sloane, M. (2026) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3846167.3848594 · Cited by 7
Qian, A., Shaw, R., Dabbish, L., Suh, J., & Shen, H. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3788065 · Cited by 5
Alharbi, R., Cheong, A. D., Herskovitz, J., Brewer, R. N., & Schoenebeck, S. (2025) · Proceedings of the 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746376 · Cited by 3
“The vast majority of prior work on AI transparency has focused on engineers and data scientists (e.g., [41, 49, 66], with comparatively little attention given to user experiences [22, 104] and even less to accessibility contexts [1, 7, 74].”
Mathur, N., Zubatiy, T., Rozga, A., Forlizzi, J., & D Mynatt, E. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790812 · Cited by 3
“Yet , most XAI approaches today continue to prioritize transparency for AI engineers and developers, rather than for everyday end-users interacting with these systems [97, 141].”
Self-citationPoirier, L., Huang, J., & MacGibbon, C. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732114 · Cited by 3
AI Safety Evaluations Need to Consider Cascading Effects
Neumann, A., & Singh, J. (2026) · Proceedings of IASEAI Conference · doi:10.1609/iaseai.v2i1.43045 · Cited by 3
Hein, I., Ullrich, D., & Diefenbach, S. (2026) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2026.101025 · Cited by 2
Gyamfi, S., Kondoro, A. M., Öztürk, Y., Schreiber, R., & Borisov, V. (2026) · Workshop on African Natural Language Processing (AfricaNLP) · doi:10.18653/v1/2026.africanlp-main.12 · Cited by 2
“However, producing high-quality, diverse, and culturally grounded synthetic text for low-resource languages requires careful control to ensure linguistic fidelity (Kirk et al., 2024; Li et al., 2023).”
Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support
Sivaraman, V., Morrison, K., Epperson, W., & Perer, A. (2025) · arXiv · doi:10.48550/arxiv.2504.07423 · Cited by 1
Self-citationYi, H., Xian, L., Zhang, Y., Zeng, J., & Zhang, Z. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812239 · Cited by 1
“By making aspects of algorithmic decision-making processes visible (e.g., data flows [34], failure modes [7], and for what purpose they are created [45]) and by providing disclosures or labels [81], technical systems become inspectable by stakeholders when they are deployed in practice [93].”
He, J., & Liu, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.14611 · Cited by 1
Now You See Me: Designing Responsible AI Dashboards for Early-Stage Health InnovationPreprint
Surodina, S., Görücü, S., Golmohammadi, L., Delaney, E., & Borgo, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.23378 · Cited by 1
“Recent methodological work has increasingly problematized the sustainability of design studies in governance-heavy contexts: information needs and other factors change over time and across roles, limiting the effectiveness of one-off design and evaluation cycles [65, 73].”
From Preventive to Reactive: How AI Coding Assistants Transform Developers' Security AwarenessPreprint
Bappy, F. H., Hossain, T., Meheraj, S. M., Akhand, A. S., Tabassum, T., Zaman, T. S., Hasan, R., & Islam, T. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.23130 · Cited by 1
“Turri et al. [44] demonstrated through a case study that current methods for identifying user transparency needs are too narrow, requiring multi-method approaches to capture diverse stakeholder needs.”
Mammo-CLIP Dissect: A Framework for Analysing Mammography Concepts in Vision-Language ModelsPreprint
Salahuddin, S. A., Dorszewski, T., Martiniussen, M. A., Hovda, T., Portaluri, A., Thrun, S., Kampffmeyer, M., Wetzer, E., Wickstrøm, K., & Jenssen, R. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.21102
“One of the most pressing issues is the lack of transparency and interpretability in many AI systems 16,17 , which becomes especially problematic in safety-critical applications like healthcare and breast cancer detection.”
Barradas, A., Sanchéz-García, Á. J., Alonso-Ramirez, O., & Limón, X. (2025) · Mexican International Conference on Computer Science · doi:10.1109/enc68268.2025.11311899
“…data, as misinterpretation can lead to clusters that lack relevance or practical utility [21] Another critical issue concerns the validity of the results, since the generated clusters must accurately capture patterns that support software enhancement and maintenance [27] [28] [30] [40] [41] [46].”
When Explanations Deceive: Understanding Unintentional and Intentional Deception in XAI
Tekkesinoglu, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3812351
“…[4, 41, 42, 55, 93, 100, 105] False Causality: [15, 18, 30, 42, 72, 108] Inconsistencies: [9, 15, 15, 17, 26, 39] Over-confidence: [23, 48, 81, 85, 88] Human Factors Misinterpretation: [8, 18, 24, 30, 35, 51, 73, 91, 102, 107] False Sense of Security: [12, 25, 40, 52, 61, 64, 71, 87, 89, 90] Fig.”
Balayn, A., Rancourt, F., Casati, F., & Gadiraju, U. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3816909
Kang, E. J., Lin, F., & Hwang, A. H. (2026) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v9i2.43238
Metadata
- Authors
- Turri, V., Morrison, K., Robinson, K.-M., Abidi, C., Perer, A., Forlizzi, J., & Dzombak, R.
- Affiliations
- Software Engineering Institute (US)
- Carnegie Mellon University (US)
- ORCID
- Venue
- The 2024 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2024-06-03
- Pages
- 1494-1514
- OpenAlex
- W4399363537
- Cited by
- 10
- References
- 58
- Citations by year
- 2026: 5 · 2025: 5
- FWCI
- 2.96
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Political Science and International Relations › E-Government and Public Services
- Keywords
- AI transparency, AI decision support, wildlife conservation, end-user perspective, transparency mechanisms, stakeholder perspectives, community forums, training data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.