The DOI matches its Crossref record; 212 of the 212 references it cites were found in a registry or an index.
Towards Human-Centered Explainable AI: A Survey of User Studies for Model Explanations
Open accessAbstract
Explainable AI (XAI) is widely viewed as a sine qua non for ever-expanding AI research. A better understanding of the needs of XAI users, as well as human-centered evaluations of explainable models are both a necessity and a challenge. In this paper, we explore how human-computer interaction (HCI) and AI researchers conduct user studies in XAI applications based on a systematic literature review. After identifying and thoroughly analyzing 97 core papers with human-based XAI evaluations over the past five years, we categorize them along the measured characteristics of explanatory methods, namely trust, understanding, usability, and human-AI collaboration performance. Our research shows that XAI is spreading more rapidly in certain application domains, such as recommender systems than in others, but that user evaluations are still rather sparse and incorporate hardly any insights from cognitive or social sciences. Based on a comprehensive discussion of best practices, i.e., common models, design choices, and measures in user studies, we propose practical guidelines on designing and conducting user studies for XAI researchers and practitioners. Lastly, this survey also highlights several open research directions, particularly linking psychological science and human-centered XAI.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CAs offering technical explanations or counterfactual options, even if not fully comprehended, signal transparency (Liao et al., 2020;Rong et al., 2024).”
From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Rong et al., 2024
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–212 of 212
Li, T., Convertino, G., Tayi, R. K., et al. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302294 · Cited by 10
Inherently Explainable Reinforcement Learning in Natural LanguagePreprint
Peng, X., Riedl, M., & Ammanabrolu, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.08907 · Cited by 8
Visual correspondence-based explanations improve AI robustness and human-AI team accuracyPreprint
Nguyen, G., Taesiri, M. R., & Nguyen, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.00780 · Cited by 8
How Do People Rank Multiple Mutant Agents?
Dodge, J. E., Anderson, A. A., Olson, M., Dikkala, R., & Burnett, M. (2022) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3490099.3511115 · Cited by 7
Fel, T., Colin, J., Cadène, R., et al. (2021) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 6
A Psychological Theory of ExplainabilityPreprint
Yang, S. C., Folke, T., & Shafto, P. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08452 · Cited by 6
Smith, A., Kleanthous, S., Lim, B. Y., Kuflik, T., Stumpf, S., Otterbacher, J., Sarkar, A., Dugan, C., & Tal, A. S. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3379336.3379361 · Cited by 5
Do Users Benefit From Interpretable Vision? A User Study, Baseline, And DatasetPreprint
Sixt, L., Schuessler, M., Popescu, O.-I., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.11642 · Cited by 4
Sanity Checks for Saliency Metrics
Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., & Preece, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6064 · Cited by 3
Quantifying Learnability and Describability of Visual Concepts Emerging in Representation LearningPreprint
Laina, I., Fong, R., & Vedaldi, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14551 · Cited by 3
Kuhl, U., Artelt, A., & Hammer, B. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.03398 · Cited by 2
How Much Information Is Enough?
Jacobson, J. A. (2008) · The AMA Journal of Ethic · doi:10.1001/virtualmentor.2008.10.8.ccas4-0808 · Cited by 1
Flagged first, then by citations
Research works printed but not linked: 9
- Paleja, The utility of explainable AI in ad hoc human-machine teaming, 34, 610
- Borowski, Exemplary natural images explain {CNN} activations better than state-of-the-art feature visualization
- Yeh, On completeness-aware concept-based explanations in deep neural networks, 20554
- Ghorbani, Towards automatic concept-based explanations, 9277
- Leemann, Coherence evaluation of visual concepts with objects and language
- Lai, “why is’ Chicago’deceptive?, 1
- Iso, 1998, 9241–11: 1998, Ergonomic requirements for work with visual display terminals (VDTs)-Part 11: Guidance on usability,, Int. Org. Standardization, 45
- Qian, Evaluating the role of interactivity on improving transparency in autonomous agents, 1083
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1
Cited by
Sorted by citations
What large language models know and what people think they know
Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W., & Smyth, P. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00976-7 · Cited by 209
On opportunities and challenges of large multimodal foundation models in education
Küchemann, S., Avila, K. E., Dinc, Y., Hortmann, C., Revenga, N., Ruf, V., Stausberg, N., Steinert, S., Fischer, F., Fischer, M., et al. (2025) · npj Science of Learning · doi:10.1038/s41539-025-00301-w · Cited by 47
Self-citationSalimzadeh, S., He, G., & Gadiraju, U. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641905 · Cited by 41
“In addition to these measures of reliance, we also evaluated participants’ decision-making accuracy, demonstrating the human-AI team performance [11, 108].”
Building Appropriate Mental Models: What Users Know and Want to Know about an Agentic AI Chatbot
Brachman, M., Kunde, S., Miller, S., Fucs, A., Dempsey, S., Jabbour, J., & Geyer, W. (2025) · Proceedings of the 30th International Conference on Intelligent User Interfaces · doi:10.1145/3708359.3712071 · Cited by 18
“Our work builds upon and contributes to the fields of human-centered AI transparency and explainable AI [46, 47, 59, 80].”
Survey of Computerized Adaptive Testing: A Machine Learning Perspective
Zhuang, Y., Liu, Q., Bi, H., Huang, Z., Huang, W., Li, J., Yu, J., Liu, Z., Hu, Z., Hong, Y., et al. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3672850 · Cited by 9
“The assessment of intelligent agents, whether human or AI systems, is essential for ensuring that individuals are well-prepared to meet the demands of their respective roles [1], [2].”
Metadata
- Authors
- Rong, Y., Leemann, T., Nguyen, T.-T., Fiedler, L., Qian, P., Unhelkar, V., Seidel, T., Kasneci, G., & Kasneci, E.
- Affiliations
- Technical University of Munich (DE)
- University of Tübingen (DE)
- Rice University (US)
- ORCID
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2024-04
- Volume
- 46
- Issue
- 4
- Pages
- 2104-2122
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W4388624604
- Cited by
- 278
- References
- 212
- Citations by year
- 2026: 120 · 2025: 98 · 2024: 57
- FWCI
- 5.77
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
- Keywords
- explainable AI, model explanation, user study, human-centered AI, human-computer interaction, trust in AI, usability, human-AI collaboration, explanation evaluation, recommender systems
- MeSH terms
- Algorithms, Humans
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
