CoChat
Sign Up
Cited by 925Explainable Artificial Intelligence (XAI)Usability and User Interface DesignInnovative Human-Technology Interaction

Questioning the AI: Informing Design Practices for Explainable AI User Experiences

Open access
Liao, Q. Vera, Gruen, Daniel, Miller, Sarah·2020·Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3313831.3376590
Open this paper in CoChat — free accountOpen PDF

Abstract

A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic. While many recognize the necessity to incorporate explainability features in AI systems, how to address real-world user needs for understanding AI remains an open question. By interviewing 20 UX and design practitioners working on various AI products, we seek to identify gaps between the current XAI algorithmic work and practices to create explainable AI products. To do so, we develop an algorithm-informed XAI question bank in which user needs for explainability are represented as prototypical questions users might ask about the AI, and use it as a study probe. Our work contributes insights into the design space of XAI, informs efforts to support design practices in this space, and identifies opportunities for future XAI work. We also provide an extended XAI question bank and discuss how it can be used for creating user-centered XAI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3313831.3376590

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Users' explainability needs emerge in a usage context, guided by their goals and shaped by their backgrounds, expectations, and socio-organizational and cultural contexts.

    “users' explainability needs emerge in a usage context, guided by their goals and shaped by their backgrounds, expectations, as well as socio, organizational and cultural contexts [20,48,52].”

    From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [48]

    In “Understanding user needs for explainability” (page unavailable), the paper identifies usage points, decision contexts, and user characteristics as factors relevant to explainability needs. It also links motivation for explanation to users’ system goals and discusses backgrounds and beliefs in “In quest for human-like explanations.” Our copy does not establish that cultural contexts shape those needs, and the paper explicitly says it does not conclude how the identified factors vary them. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • CAs offering technical explanations or counterfactual options, even if not fully comprehended, signal transparency.

    “CAs offering technical explanations or counterfactual options, even if not fully comprehended, signal transparency (Liao et al., 2020;Rong et al., 2024).”

    From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Liao et al., 2020

    The “What if, How to be—inspecting counterfactual” section describes counterfactual options that let users test scenarios and gain insight. The discussion of explanation goals also links explanations to transparency. Neither passage says that technical explanations or counterfactual options signal transparency when users do not fully comprehend them, and the paper does not establish this effect specifically for CAs. (LLM, conf=0.80)

    Experimental — accuracy not yet measured

  • Through measures such as ethical evaluation of AI applications used in learning, regulating access to generative models, establishing protocols for the use of AI in research, or promoting decision-making transparency in automated systems, institutions create a concrete framework that facilitates legal and ethical literacy.

    “Through measures such as ethical evaluation of AI applications used in learning, regulating access to generative models, establishing protocols for the use of AI in research, or promoting decision-making transparency in automated systems, institutions create a concrete framework that facilitates legal and ethical literacy [51-53].”

    From Artificial Intelligence Governance in Higher Education: The Role of Knowledge-Based Strategies in Fostering Legal Awareness and Ethical Artificial Intelligence Literacy (2025) · cites this paper as [53]

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Liao et al.'s framework shows users are interested in a broad range of explanatory information about an AI model, including its overall logic, how it reasons to produce a particular output, the training data, and its performance and range of output.

    “This framework demonstrates that users are interested in a broad range of explanatory information about an AI model, including its overall logic, how it reasons to produce a particular output, the training data, and its performance and range of output.”

    From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [48]

  • Liao et al. proposed to define users' explainability needs by the questions they ask to understand the AI and developed a framework of common questions.

    “Liao et al. [48] proposed to define user's explainablity needs by what questions they ask to understand the AI [48] and developed a framework of common questions.”

    From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [48]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 65

  1. [3]3. Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli. 2018. Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda. In Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, 582.
  2. [6]6. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, and others. 2019. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 3.
  3. [12]12. Jacob Bien, Robert Tibshirani, and others. 2011. Prototype selection for interpretable classification. The Annals of Applied Statistics 5, 4 (2011), 2403-2424.
  4. [14]14. Nadia Boukhelifa, Marc-Emmanuel Perrin, Samuel Huron, and James Eagan. 2017. How data workers cope with uncertainty: A task characterisation study. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 3645-3656.
  5. [15]15. Glenn A Bowen. 2006. Grounded theory and sensitizing concepts. International journal of qualitative methods 5, 3 (2006), 12-23.
  6. [16]16. Sylvain Bromberger. 1992. On what we know we don't know: Explanation, theory, linguistics, and how questions shape them. University of Chicago Press.
  7. [18]18. Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. 2019. Machine Learning Interpretability: A Survey on Methods and Metrics. Electronics 8, 8 (2019), 832.
  8. [19]19. Bruce Chandrasekaran, Michael C Tanner, and John R Josephson. 1989. Explaining control strategies in problem solving. IEEE Intelligent Systems 1 (1989), 9-15.
  9. [21]21. William J Clancey. 1983. The epistemology of a rule-based expert system-a framework for explanation. Artificial intelligence 20, 3 (1983), 215-251.
  10. [22]22. Herbert H Clark, Susan E Brennan, and others. 1991. Grounding in communication. Perspectives on socially shared cognition 13, 1991 (1991), 127-149.
  11. [25]25. Sanjeeb Dash, Oktay Gunluk, and Dennis Wei. 2018. Boolean decision rules via column generation. In Advances in Neural Information Processing Systems. 4655-4665.
  12. [26]26. Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems. 592-603.
  13. [27]27. Nicholas Diakopoulos. 2015. Algorithmic accountability: Journalistic investigation of computational power structures. Digital journalism 3, 3 (2015), 398-415.
  14. [30]30. Malin Eiband, Hanna Schneider, Mark Bilandzic, Julian Fazekas-Con, Mareike Haug, and Heinrich Hussmann. 2018. Bringing transparency design into practice. In 23rd International Conference on Intelligent User Interfaces. ACM, 211-223.
  15. [31]31. Thomas Erickson, Catalina M Danis, Wendy A Kellogg, and Mary E Helander. 2008. Assistance: the work practices of human administrative assistants and their implications for it and organizations. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. ACM, 609-618.
  16. [32]32. Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189-1232.
  17. [33]33. Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. 2018. Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA). IEEE, 80-89.
  18. [34]34. Alyssa Glass, Deborah L McGuinness, and Michael Wolverton. 2008. Toward establishing trust in adaptive agents. In Proceedings of the 13th international conference on Intelligent user interfaces. ACM, 227-236.
  19. [35]35. Alex Goldstein, Adam Kapelner, Justin Bleich, and Emil Pitkin. 2015. Peeking inside the black box: Visualizing statistical learning with plots of individual conditional expectation. Journal of Computational and Graphical Statistics 24, 1 (2015), 44-65.
  20. [37]37. Shirley Gregor and Izak Benbasat. 1999. Explanations from intelligent systems: Theoretical foundations and implications for practice. MIS quarterly (1999), 497-530.
  21. [39]39. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2019. A survey of methods for explaining black box models. ACM computing surveys (CSUR) 51, 5 (2019), 93.
  22. [40]40. Andreas Henelius, Kai Puolamäki, Henrik Boström, Lars Asker, and Panagiotis Papapetrou. 2014. A peek into the black box: exploring classifiers by randomization. Data mining and knowledge discovery 28, 5-6 (2014), 1503-1529.
  23. [41]41. Jonathan L Herlocker, Joseph A Konstan, and John Riedl. 2000. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 241-250.
  24. [42]42. Michael Hind. 2019. Explaining explainable AI. XRDS: Crossroads, The ACM Magazine for Students 25, 3 (2019), 16-19.
  25. [44]44. Fred Hohman, Andrew Head, Rich Caruana, Robert DeLine, and Steven M Drucker. 2019. Gamut: A design probe to understand how data scientists understand machine learning models. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 579.
  26. [45]45. Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 600.
  27. [46]46. Ulf Johansson and Lars Niklasson. 2009. Evolving decision trees using oracle guides. In 2009 IEEE Symposium on Computational Intelligence and Data Mining. IEEE, 238-244.
  28. [47]47. Been Kim, Cynthia Rudin, and Julie A Shah. 2014. The bayesian case model: A generative approach for case-based reasoning and prototype classification. In Advances in Neural Information Processing Systems. 1952-1960.
  29. [48]48. Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. 2019. Will You Accept an Imperfect AI?: Exploring Designs for Adjusting End-user Expectations of AI Systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 411.
  30. [49]49. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 1885-1894.
  31. [50]50. Josua Krause, Adam Perer, and Kenney Ng. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 5686-5697.
  32. [51]51. R Krishnan, G Sivakumar, and P Bhattacharya. 1999. Extracting decision trees from trained neural networks. Pattern recognition 32, 12 (1999).
  33. [52]52. Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th international conference on intelligent user interfaces. ACM, 126-137.
  34. [53]53. Vivian Lai and Chenhao Tan. 2018. On Human Predictions with Explanations and Predictions of Machine Learning Models: A Case Study on Deception Detection. arXiv preprint arXiv:1811.07901 (2018). · arXiv
  35. [58]58. Zachary C Lipton. 2016. The mythos of model interpretability. arXiv preprint arXiv:1606.03490 (2016). · arXiv
  36. [61]61. Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere. 2019. A Grounded Interaction Protocol for Explainable Artificial Intelligence. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 1033-1041.
  37. [63]63. Sina Mohseni, Niloofar Zarei, and Eric D Ragan. 2018. A survey of evaluation methods and measures for interpretable machine learning. arXiv preprint arXiv:1811.11839 (2018). · arXiv
  38. [65]65. Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. 2020. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 607-617.
  39. [66]66. Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q Vera Liao, Casey Dugan, and Thomas Erickson. 2019. How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 126.
  40. [69]69. Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. 2018. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.07810 (2018). · arXiv
  41. [72]72. Gabriëlle Ras, Marcel van Gerven, and Pim Haselager. 2018. Explanation methods in deep learning: Users, values, concerns and challenges. In Explainable and Interpretable Models in Computer Vision and Machine Learning. Springer, 19-36.
  42. [73]73. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 1135-1144.
  43. [75]75. David Ribes. 2017. Notes on the concept of data interoperability: Cases from an ecology of AIDS research infrastructures. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1514-1526.
  44. [76]76. Marko Robnik-Šikonja and Marko Bohanec. 2018. Perturbation-Based Explanations of Prediction Models. In Human and Machine Learning. Springer, 159-175.
  45. [77]77. Adam Rule, Aurélien Tabard, and James D Hollan. 2018. Exploration and explanation in computational notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 32.
  46. [78]78. Wojciech Samek and Klaus-Robert Müller. 2019. Towards explainable artificial intelligence. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer, 5-22.
  47. [81]81. Milene Selbach Silveira, Clarisse Sieckenius de Souza, and Simone DJ Barbosa. 2001. Semiotic engineering contributions for designing online help systems. In Proceedings of the 19th annual international conference on Computer documentation. ACM, 31-38.
  48. [84]84. Erik Štrumbelj and Igor Kononenko. 2014. Explaining prediction models and individual predictions with feature contributions. Knowledge and information systems 41, 3 (2014), 647-665.
  49. [85]85. Simone Stumpf, Vidya Rajaram, Lida Li, Margaret Burnett, Thomas Dietterich, Erin Sullivan, Russell Drummond, and Jonathan Herlocker. 2007. Toward harnessing user feedback for machine learning. In Proceedings of the 12th international conference on Intelligent user interfaces. ACM, 82-91.
  50. [86]86. William R Swartout. 1983. XPLAIN: A system for creating and explaining expert consulting programs. Artificial intelligence 21, 3 (1983), 285-325.
  51. [87]87. William R Swartout and Stephen W Smoliar. 1987. On making expert systems more like experts. Expert Systems 4, 3 (1987), 196-208.
  52. [88]88. Andrea L Thomaz and Cynthia Breazeal. 2006. Transparency and socially guided machine learning. In 5th Intl. Conf. on Development and Learning (ICDL).
  53. [89]89. Gabriele Tolomei, Fabrizio Silvestri, Andrew Haines, and Mounia Lalmas. 2017. Interpretable predictions of tree-based ensembles via actionable feature tweaking. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. ACM, 465-474.
  54. [90]90. Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2017. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GPDR. Harv. JL & Tech. 31 (2017), 841.
  55. [91]91. Danding Wang, Qian Yang, Ashraf Abdul, and Brian Y Lim. 2019. Designing Theory-Driven User-Centric Explainable AI. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 601.
  56. [92]92. Dennis Wei, Sanjeeb Dash, Tian Gao, and Oktay Gunluk. 2019. Generalized Linear Rule Models. In International Conference on Machine Learning. 6687-6696.
  57. [93]93. Daniel S Weld and Gagan Bansal. 2018. The challenge of crafting intelligible intelligence. arXiv preprint arXiv:1803.04263 (2018). · arXiv
  58. [94]94. Adrian Weller. 2017. Challenges for transparency. arXiv preprint arXiv:1708.01870 (2017). · arXiv
  59. [95]95. Christine T Wolf. 2019. Explainability scenarios: towards scenario-based XAI design. In Proceedings of the 24th International Conference on Intelligent User Interfaces. ACM, 252-257.
  60. [96]96. Qian Yang. 2018. Machine Learning as a UX Design Material: How Can We Imagine Beyond Automation, Recommenders, and Reminders?. In 2018 AAAI Spring Symposium Series.
  61. [98]98. Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. 2019. Interpreting cnns via decision trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6261-6270.
  62. [99]99. Xin Zhang, Armando Solar-Lezama, and Rishabh Singh. 2018. Interpreting neural network judgments via minimal, stable, and symbolic corrections. In Advances in Neural Information Processing Systems. 4874-4885.
  63. [100]100. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2921-2929.
  64. [101]101. Zhi-Hua Zhou, Yuan Jiang, and Shi-Fu Chen. 2003. Extracting symbolic rules from trained neural network ensembles. Ai Communications 16, 1 (2003), 3-15.
  65. [102]102. Jichen Zhu, Antonios Liapis, Sebastian Risi, Rafael Bidarra, and G Michael Youngblood. 2018. Explainable AI for designers: A human-centered perspective on mixed-initiative co-creation. In 2018 IEEE Conference on Computational Intelligence and Games (CIG). IEEE, 1-8.

Cited by

Sorted by citations

Authors
Liao, Q. V., Gruen, D., & Miller, S.
Affiliations
  • IBM (United States) (US)
  • IBM Research - Cambridge (United States) (US)
ORCID
Liao, Q. Vera
Venue
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2020-04-21
Pages
1-15
DOI
10.1145/3313831.3376590
OpenAlex
W2999765337
Cited by
925
References
29
Citations by year
2026: 192 · 2025: 153 · 2024: 178 · 2023: 144 · 2022: 129 · 2021: 107 · 2020: 22
FWCI
38.38
Type
conference paper
Language
en
Open access
Licence
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
Keywords
explainable AI, user experience design, design practice, user-centered design, AI products, design space, UX practitioners

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review