Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 121 of the 121 references it cites were found in a registry or an index.

Human-centered evaluation of explainable AI applications: a systematic review

Open access
Kim, Jenia, Maathuis, Henry, Sent, Danielle · 2024 · Frontiers in Artificial Intelligence

Abstract

Explainable Artificial Intelligence (XAI) aims to provide insights into the inner workings and the outputs of AI systems. Recently, there's been growing recognition that explainability is inherently human-centric, tied to how people perceive explanations. Despite this, there is no consensus in the research community on whether user evaluation is crucial in XAI, and if so, what exactly needs to be evaluated and how. This systematic literature review addresses this gap by providing a detailed overview of the current state of affairs in human-centered XAI evaluation. We reviewed 73 papers across various domains where XAI was evaluated with users. These studies assessed what makes an explanation “good” from a user's perspective, i.e., what makes an explanation meaningful to a user of an AI system. We identified 30 components of meaningful explanations that were evaluated in the reviewed papers and categorized them into a taxonomy of human-centered XAI evaluation, based on: (a) the contextualized quality of the explanation, (b) the contribution of the explanation to human-AI interaction, and (c) the contribution of the explanation to human-AI performance. Our analysis also revealed a lack of standardization in the methodologies applied in XAI user studies, with only 19 of the 73 papers applying an evaluation framework used by at least one other study in the sample. These inconsistencies hinder cross-study comparisons and broader insights. Our findings contribute to understanding what makes explanations meaningful to users and how to measure this, guiding the XAI community toward a more unified approach in human-centered explainability.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/frai.2024.1456486

Cites

121 of 121 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The high-level categories in existing taxonomies for XAI evaluation. Doshi-Velez and Kim (2018), Hoffman et al. (2018, 2023), and Lopes et al. (2022) discuss both evaluation with users (green) and without users (orange); Nauta et al. (2023) focus on evaluation without users (orange); Zhou et al. (2021), Vilone and Longo (2021), and Mohseni et al. (2021) focus on evaluation with users (green).
Figure 1 The high-level categories in existing taxonomies for XAI evaluation. Doshi-Velez and Kim (2018), Hoffman et al. (2018, 2023), and Lopes et al. (2022) discuss both evaluation with users (green) and without users (orange); Nauta et al. (2023) focus on evaluation without users (orange); Zhou et al. (2021), Vilone and Longo (2021), and Mohseni et al. (2021) focus on evaluation with users (green).CC BY · PubMed Central
Figure 2 Search query for the systematic literature review.
Figure 2 Search query for the systematic literature review.CC BY · PubMed Central
Figure 3 PRISMA flow diagram.
Figure 3 PRISMA flow diagram.CC BY · PubMed Central
Figure 4 The labeling scheme used for data extraction. (A) Labeling scheme of AI systems. (B) Labeling scheme of XAI methods. (C) Labeling scheme of user studies.
Figure 4 The labeling scheme used for data extraction. (A) Labeling scheme of AI systems. (B) Labeling scheme of XAI methods. (C) Labeling scheme of user studies.CC BY · PubMed Central
Figure 5 Distribution of papers by application domain and type user study.
Figure 5 Distribution of papers by application domain and type user study.CC BY · PubMed Central
Figure 6 The proposed taxonomy of human-centered evaluation of XAI. The blue, orange and red boxes contain the 30 evaluation measures identified in the reviewed papers. They are grouped based on the aspect of human-centered explanation quality that they evaluate: (A) in-context quality of the explanation, (B) contribution of the explanation to human-AI interaction, and (C) contribution of the explanation to human-AI performance. An additional aspect, the a priori explanation quality, is not covered by our review since it is not evaluated with users (expl = explanation; sys = system).
Figure 6 The proposed taxonomy of human-centered evaluation of XAI. The blue, orange and red boxes contain the 30 evaluation measures identified in the reviewed papers. They are grouped based on the aspect of human-centered explanation quality that they evaluate: (A) in-context quality of the explanation, (B) contribution of the explanation to human-AI interaction, and (C) contribution of the explanation to human-AI performance. An additional aspect, the a priori explanation quality, is not covered by our review since it is not evaluated with users (expl = explanation; sys = system).CC BY · PubMed Central
Figure 7 The number of studies in which each meaningfulness component is evaluated (total: 77 studies in 73 papers).
Figure 7 The number of studies in which each meaningfulness component is evaluated (total: 77 studies in 73 papers).CC BY · PubMed Central
Figure 8 Our taxonomy, compared to other XAI evaluation frameworks. The main difference is in the level of detail of the evaluated properties, and in the novel categorization into the three dimensions of human-centered evaluation.
Figure 8 Our taxonomy, compared to other XAI evaluation frameworks. The main difference is in the level of detail of the evaluated properties, and in the novel categorization into the three dimensions of human-centered evaluation.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 118 references

Research works printed but not linked: 4

  1. “Capturing users' reality: a novel approach to generate coherent counterfactual explanations,” Förster M. Hühn P. Klier M. Kluge K. Hawaii International Conference on System Sciences 2021
  2. Ibrahim, 2023, “Do explanations improve the quality of AI-assisted human decisions? An algorithm-in-the-loop analysis of factual and counterfactual explanations,”, 326
  3. Kim, “Identifying XAI user needs: gaps between literature use cases in the financial sector,”, Proceedings of the 2nd Workshop on Responsible Applied Artificial Intelligence (RAAIT), at HHAI 2024
  4. Schellingerhout, 2022, Explainable career path predictions using neural models

Cited by

See all 153

Metadata

Authors
Kim, J., Maathuis, H., & Sent, D.
Affiliations
University of Applied Sciences Utrecht (NL)
Corresponding author
  • Kim, Jenia
  • Maathuis, Henry
Venue
Frontiers in Artificial Intelligence
Published
2024-10-17
Volume
7
Pages
1456486-1456486
ISSN
2624-8212 (ISSN-L)
OpenAlex
W4403501023
Cited by
130
References
118
Citations by year
2026: 105 · 2025: 25
FWCI
4.45
Type
journal article
Language
en
Open access
Locations
  • Frontiers in Artificial Intelligence (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
  • Social Sciences › Human Factors and Ergonomics › Human-Automation Interaction and Safety
Keywords
explainable AI, human-centered evaluation, human-AI interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Human-centered evaluation of explainable AI applications: a systematic review. https://app.cochat.ai/papers/10.3389/frai.2024.1456486
The paper
Kim, J., Maathuis, H., & Sent, D. (2024). Human-centered evaluation of explainable AI applications: a systematic review. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2024.1456486

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.