CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Making Sense of the Unsensible: Reflection, Survey, and Challenges for XAI in Large Language Models Toward Human-Centered AI

Herrera, Francisco·2025·arXiv
DOI 10.48550/arxiv.2505.20305arXiv 2505.20305
Open this paper in CoChat — free account

Abstract

As large language models (LLMs) are increasingly deployed in sensitive domains such as healthcare, law, and education, the demand for transparent, interpretable, and accountable AI systems becomes more urgent. Explainable AI (XAI) acts as a crucial interface between the opaque reasoning of LLMs and the diverse stakeholders who rely on their outputs in high-risk decisions. This paper presents a comprehensive reflection and survey of XAI for LLMs, framed around three guiding questions: Why is explainability essential? What technical and ethical dimensions does it entail? And how can it fulfill its role in real-world deployment? We highlight four core dimensions central to explainability in LLMs, faithfulness, truthfulness, plausibility, and contrastivity, which together expose key design tensions and guide the development of explanation strategies that are both technically sound and contextually appropriate. The paper discusses how XAI can support epistemic clarity, regulatory compliance, and audience-specific intelligibility across stakeholder roles and decision settings. We further examine how explainability is evaluated, alongside emerging developments in audience-sensitive XAI, mechanistic interpretability, causal reasoning, and adaptive explanation systems. Emphasizing the shift from surface-level transparency to governance-ready design, we identify critical challenges and future research directions for ensuring the responsible use of LLMs in complex societal contexts. We argue that explainability must evolve into a civic infrastructure fostering trust, enabling contestability, and aligning AI systems with institutional accountability and human-centered decision-making.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.20305

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Explainable AI solutions' implementation within legal frameworks remains theoretically insufficiently developed.

    “Explainable AI solutions present possible remedies; however, their implementation within legal frameworks remains theoretically insufficiently developed (Herrera, 2025).”

    From Causation in AI Tort Litigation: Legal Dilemmas of Algorithmic Black Boxes and Burden of Proof Allocation (2025) · cites this paper as Herrera, 2025

    Section 7.2 (p. 24) calls for future work to translate XAI principles into deployable systems aligned with legal expectations, supporting an implementation gap. It does not say that the theoretical basis for implementing XAI within legal frameworks is insufficiently developed; its stated gap is between theory and practice. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 39

  1. Bertrand, A., Viard, T., Belloum, R., Eagan, J. R., & Maxwell, W. (2023). On selective, mutable and dialogic xai: A review of what users say about different types of interactive explanations. In Proceedings of the 2023 chi conference on human factors in computing systems (pp. 1-21).
  2. Biecek, P., & Samek, W. (2024). Explain to question not to justify. arXiv preprint arXiv:2402.13914 . · arXiv
  3. Bilal, A., Ebert, D., & Lin, B. (2025). Llms for explainable ai: A comprehensive survey. arXiv preprint arXiv:2504.00125 . · arXiv
  4. Binkyte, R., Sheth, I., Jin, Z., Havaei, M., Schölkopf, B., & Fritz, M. (2025). Causality is key to understand and balance multiple goals in trustworthy ml and foundation models. arXiv preprint arXiv:2502.21123 . · arXiv
  5. Boge, F. J. (2022). Two dimensions of opacity and the deep learning predicament. Minds and Machines , 32 (1), 43-75.
  6. Burden, J., Cebrian, M., & Hernandez-Orallo, J. (2024). Conversational complexity for assessing risk in large language models. arXiv preprint arXiv:2409.01247 . · arXiv
  7. Cambria, E., Malandri, L., Mercorio, F., Nobani, N., & Seveso, A. (2024). Xai meets llms: A survey of the relation between explainable ai and large language models. arXiv preprint arXiv:2407.15248 . · arXiv
  8. Chen, Z., Sun, X., Jiao, X., Lian, F., Kang, Z., Wang, D., & Xu, C. (2024). Truth forest: Toward multi-scale truthfulness in large language models through intervention without tuning. In Proceedings of the aaai conference on artificial intelligence (Vol. 38, pp. 20967-20974).
  9. Fraser, H., Simcock, R., & Snoswell, A. J. (2022). Ai opacity and explainability in tort litigation. In Proceedings of the 2022 acm conference on fairness, accountability, and transparency (pp. 185-196).
  10. García-Carrasco, J., Maté, A., & Trujillo, J. (2024). How does gpt-2 predict acronyms? extracting and understanding a circuit via mechanistic interpretability. In International conference on artificial intelligence and statistics (pp. 3322-3330).
  11. García-Carrasco, J., Maté, A., & Trujillo, J. (2025). Extracting interpretable task-specific circuits from large language models for faster inference. In Proceedings of the aaai conference on artificial intelligence (Vol. 39, pp. 1677216780).
  12. Haresamudram, K., Larsson, S., & Heintz, F. (2023). Three levels of AI transparency. Computer , 56 (2), 93-100.
  13. He, G., Aishwarya, N., & Gadiraju, U. (2025). Is conversational xai all you need? human-ai decision making with a conversational xai assistant. In Proceedings of the 30th international conference on intelligent user interfaces (pp. 907-924).
  14. Hsieh, W., Bi, Z., Jiang, C., Liu, J., Peng, B., Zhang, S., . . . others (2024). A comprehensive guide to explainable ai: from classical models to llms. arXiv preprint arXiv:2412.00800 . · arXiv
  15. Jacovi, A., & Goldberg, Y. (2020). Towards faithfully interpretable nlp systems: How should we define and evaluate faithfulness? arXiv preprint arXiv:2004.03685 . · arXiv
  16. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., . . . Fung, P. (2023). Survey of hallucination in natural language generation. ACM computing surveys , 55 (12), 1-38.
  17. Li, M., Vitányi, P., et al. (2008). An introduction to kolmogorov complexity and its applications (Vol. 3). Springer.
  18. Lin, S., Hilton, J., & Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 . · arXiv
  19. Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024). Towards trustworthy llms: a review on debiasing and dehallucinating in large language models. Artificial Intelligence Review , 57 (9), 243.
  20. Liu, X., Xu, P., Wu, J., Yuan, J., Yang, Y ., Zhou, Y ., . . . others (2024). Large language models and causal inference in collaboration: A comprehensive survey. arxiv. 2024. arXiv preprint arXiv:2403.09606 . · arXiv
  21. Lu, Z., Li, X., Cai, D., Yi, R., Liu, F., Zhang, X., . . . Xu, M. (2024). Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790 . · arXiv
  22. Luo, H., & Specia, L. (2024). From understanding to utilization: A survey on explainability for large language models. arXiv preprint arXiv:2401.12874 . · arXiv
  23. Mansurov, J., Sakip, A., & Aji, A. F. (2024). Data laundering: Artificially boosting benchmark results through knowledge distillation. arXiv preprint arXiv:2412.15255 . · arXiv
  24. Mersha, M. A., Yigezu, M. G., Byun, S., Kalita, J., et al. (2025). A unified framework with novel metrics for evaluating the effectiveness of xai techniques in llms. arXiv preprint arXiv:2503.05050 . · arXiv
  25. Mumuni, F., & Mumuni, A. (2025). Explainable artificial intelligence (xai): from inherent explainability to large language models. arXiv preprint arXiv:2501.09967 . · arXiv
  26. Rai, D., Zhou, Y., Feng, S., Saparov, A., & Yao, Z. (2025). A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646 . · arXiv
  27. Rohekar, R. Y., Gurwicz, Y., & Nisimov, S. (2023). Causal interpretation of self-attention in pre-trained transformers. Advances in Neural Information Processing Systems , 36 , 31450-31465.
  28. Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., . . . Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems , 36 , 68539-68551.
  29. Schneider, J. (2025). Generative to agentic ai: Survey, conceptualization, and challenges. arXiv preprint arXiv:2504.18875 . · arXiv
  30. Subramanian, S., Elango, V., & Gungor, M. (2025). Small language models (slms) can still pack a punch: A survey. arXiv preprint arXiv:2501.05465 . · arXiv
  31. Sun, M., Chen, X., Kolter, J. Z., & Liu, Z. (2024). Massive activations in large language models. arXiv preprint arXiv:2402.17762 . · arXiv
  32. Taghanaki, S. A., & Monteiro, J. (2025). Explain-query-test: Self-evaluating llms via explanation and comprehension discrepancy. arXiv preprint arXiv:2501.11721 . · arXiv
  33. Tariq, S., Singh, R., Chhetri, M. B., Nepal, S., & Paris, C. (2025). Bridging expertise gaps: The role of llms in human-ai collaboration for cybersecurity. arXiv preprint arXiv:2505.03179 . · arXiv
  34. Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., . . . others (2024). A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness. arXiv preprint arXiv:2411.03350 . · arXiv
  35. Wu, X., Zhao, H., Zhu, Y., Shi, Y., Yang, F., Liu, T., . . . others (2024). Usable xai: 10 strategies towards exploiting explainability in the llm era. arXiv preprint arXiv:2403.08946 . · arXiv
  36. Xu, C., Guan, S., Greene, D., Kechadi, M., et al. (2024). Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244 . · arXiv
  37. Xu, Z., Song, T., & Lee, Y.-C. (2025). Confronting verbalized uncertainty: Understanding how llm's verbalized uncertainty influences users in ai-assisted decision-making. International Journal of Human-Computer Studies , 103455.
  38. Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., . . . Du, M. (2024). Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology , 15 (2), 1-38.
  39. Zhu, Z., Chen, H., Ye, X., Lyu, Q., Tan, C., Marasovi´ c, A., & Wiegreffe, S. (2024). Explanation in the era of large language models. In Proceedings of the 2024 conference of the north american chapter of the association for computational linguistics: Human language technologies (volume 5: Tutorial abstracts) (pp. 19-25).

Cited by

Sorted by citations

Metadata

Authors
Herrera, F.
Venue
arXiv
Publisher
arXiv
Published
2025
DOI
10.48550/arxiv.2505.20305
arXiv
2505.20305
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review