Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Open accessThe DOI matches its Crossref record; 180 of the 180 references it cites were found in a registry or an index; 20 of the 74 sources its claims cite were checked for support.
Abstract
This review surveys explainable AI, emphasizing transparency, interpretability, trust, and evaluation across high-stakes domains.
Overview
CoChat Summary
## TL;DR
This paper is a survey of explainable artificial intelligence (XAI) that organizes the field around why XAI is needed, how explanations are produced, and where the main challenges lie. It argues that black-box AI systems need explainability for trust, accountability, regulation, and safer deployment across domains such as healthcare, finance, and transportation.
## Key findings
- The review proposes a revised hierarchical taxonomy for XAI and says it is intended to serve as a building block for future researchers.
- It distinguishes two broad interpretability classes: perceptive interpretability and interpretability by mathematical structures; it also separates explanations by scope (global vs. local) and by stage (ante hoc vs. post hoc).
- The paper summarizes model-agnostic methods such as PDP, ICE, ALE, LIME, and counterfactuals, and reports their main trade-offs; for example, PDP is intuitive but valid for at most three features, ICE can become unreadable due to overcrowding, and ALE is described as unbiased for correlated features and faster to compute.
- In the paper’s escitalopram example, the interpretable version of the model achieved approximately 70% accuracy in identifying patients more likely to benefit from the treatment.
- It also notes a reported example where CrystalCandle explanations increased a LinkedIn software sales team’s subscription revenue by 8%.
## Methods
- Narrative review / survey of XAI literature; the authors state that they “careful[ly]” selected and analyzed current state-of-the-art XAI research.
- The paper synthesizes prior work into a taxonomy and organizes the discussion into conceptual background, need for XAI, evaluation frameworks, challenges, discussion/reflection, and future directions.
- The analysis is qualitative and comparative, using cited examples from multiple application domains and explaining the strengths and weaknesses of different explanation methods.
## Limitations
- The paper states that a “thorough and systematic survey is still lacking despite the large number of surveys” in the area, which indicates the field remains fragmented.
- It explicitly says XAI is still in its infancy and that a more formal approach is needed to address diverse user questions; it also notes that guidelines alone do not guarantee success.
- Evident from the text: no new datasets were generated or analyzed in this study, so the conclusions are based on literature synthesis rather than original empirical testing.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Valid · 25 Not valid · 49 Not checked · 59 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“With the aim of providing an explanation for legal judgment predictions generated by Legal Judgement Prediction (LJP), Zhong et al. [71] designed the QAjudge.”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Need For Reasoning
What the sources say
- Supportedcites [71] Zhong H, Wang Y, Tu C, Zhang T, Liu Z, Sun M. Iteratively questioning and answering for interpretable legal judgment prediction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34. 01 ed. 2020. p. 1250–7. https://ojs.aaai.org/index.php/AAAI/article/view/5479. source
The Abstract (p. 1) states QAjudge is 'a model based on reinforcement learning to visualize the prediction process and give interpretable judgments' for Legal Judgment Prediction, and the Conclusion (p. 7) reiterates that QAjudge is proposed 'to provide interpretable results for LJP' addressing the lack of interpretability in existing LJP methods. The Introduction (p. 2) likewise frames the goal as detecting elements so that professionals and non-specialists can understand the prediction results. Thus the attribution — that Zhong et al. designed QAjudge to explain/interpret LJP predictions — is directly stated in our copy. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Additionally, by exposing the system to contradictory examples where the same symptoms can indicate different diseases, the system can learn to evaluate multiple possibilities and make a more accurate diagnosis [91].”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Contradictions Provide Alternative Approaches
What the sources say
- Could not verifycites [91] Chou YL, Moreira C, Bruza P, Ouyang C, Jorge J. Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications. Inf Fusion. 2022;81:59–83. sourceExperimental — accuracy not yet measured
Section 7 (p. 39) discusses counterfactual explanations in medical decision support, and Section 8.1 (p. 42) discusses questions a clinician might ask about a patient's fever. Neither says that exposing a diagnostic system to examples in which the same symptoms indicate different diseases helps it consider alternatives or improve diagnostic accuracy. Searches for contradictory symptom examples and diagnostic accuracy found no passage addressing that claim. (LLM, conf=0.73)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“HCI skills such as philosophy, psychology, and cognitive science have proven their worth in the field of XAI by developing explanations that were able to stimulate the human explanation process [89].”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Collaborative Techniques Should Be Implemented More
What the sources say
- Partly supportedcites [89] Miller T. Explanation in artificial intelligence: Insights from the social sciences. Artif Intell. 2019;267:1–38. https://www.sciencedirect.com/science/article/pii/S0004370218305988. sourceExperimental — accuracy not yet measured
The Introduction says research in philosophy, psychology, and cognitive science offers valuable insights into how humans explain, and §5.3.2 describes an interactive XAI system that tailored explanations to users. But the paper does not say these fields have already proven their worth by developing explanations that stimulate the human explanation process: the Introduction says most XAI work has not built on social-science frameworks, and §5.3.6 says recent XAI has not truly adopted socially interactive explanation. (LLM, conf=0.84)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Distillation turns out to be a more effective method for obtaining interpretable models.”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Model Extraction
What the sources say
- Could not verifycites [102] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network. 2015. https://arxiv.org/abs/1503.02531. sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes distillation as transferring knowledge to a smaller model suited to deployment, and the Discussion (p. 8) says the resulting model is easier to deploy. Neither passage addresses whether distilled models are more interpretable; a search for interpretability and interpretable models found no passage making that comparison. (LLM, conf=0.74)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
In the paper
“The chart (Fig. 5) shows that IUD has the least proportional interaction effect with other variables compared to years of using hormonal contraception [107].”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Feature Interaction
What the sources say
- Not matchedcites [107] Molnar C. Interpretable Machine Learning. 2nd ed. 2022. https://christophm.github.io/interpretable-ml-book.
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
27 more claims
Create a free account to see all 148 claims and to verify the 21 references still open.
- Probably not a substantial claim. 108
- It may state two separate claims as one. 37
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For safety-critical systems, explainability is not enough for a system to be trustworthy.Various requirements such as robustness, reliability, traceability, data quality, and other factors must be demonstrated [25,43,62,63].”
From A requirements model for AI algorithms in functional safety-critical systems with an explainable self-enforcing network from a developer perspective (2024) · cites this paper as [43]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Furthermore, users' primary focus is usually on obtaining the desired outcome efficiently, prioritizing resolution over understanding of why a failure occurred (Collier et al., 2017;Hassija et al., 2024).”
From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Hassija et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“By revealing the conditions under which the CA succeeds or fails, XAI-based strategies transform an unpredictable black-box AI system into a more comprehensible one, helping users evaluate whether to continue interacting with the CA despite having experienced an error (Hassija et al., 2024;Zhao et al., 2024).”
From Explainability in AI: Comparing Human-Like and System-Like Trust Repair Strategies (2026) · cites this paper as Hassija et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unfortunately, AI does not provide explanations for its decisions, often termed a ‘Black-Box Model’; thus, adopting such models in sensitive decisions such as research retraction is difficult.50”
From Artificial Intelligence in Detecting Statistical Errors: Implications for Authors, Reviewers, and Editors (2025) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results from this study support prior studies demonstrating that organizations will lose trust in “black box” AI systems until they implement explainability mechanisms and auditability controls (Buhrmester et al. 2021; Hassija et al. 2024).”
From AI-driven financial fraud detection in Pakistan’s banking sector: bridging strategic intent and operational implementation (2026) · cites this paper as Hassija et al. 2024
Experimental — accuracy not yet measured
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Human-Machine Collaboration
“Figure 14 shows the benefit of AI in the workplace, according to the survey.”
§ Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence > Responsible AI
“To ensure that the design and development of responsible AI reflect the ethical concerns of humans, we propose to incorporate the above-discussed factors with the principles proposed by Dignum [185], i.e., accountability, responsibility, and transparency (ART) as depicted in Fig. 14.”
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“One of the noteworthy methods is the permutation feature importance (PFI) by Breiman [108] for random forests.”
MethodImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Human beings have been defeated for the first time in related open challenges [3] (e.g., ImageNet image-classification [4], COCO object-detection [5]) since the introduction of AlexNet [6].”
BackgroundMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 1×
“Some of the popularly implemented visualisation techniques are: Partial dependence plot (PDP) uses graph-based explanations for visualising the relationship between one or more features (at maximum three features) and the prediction generated by the black-box model [130].”
MethodGrad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513
A Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · cited 5×
“To develop local explanations for black-box models, Ribeiro et al. introduced the Local Interpretable Model-Agnostic explanation (LIME) [34].”
BackgroundMethodInfluentialMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 1×
“One of the significant milestones was witnessed in 2016 when an AI player AlphaGo [9, 10], was able to defeat the human world champion Sedol Lee in a game of Go.”
BackgroundDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Research works printed but not linked: 9
- Li XH, Cao CC, Shi Y, Bai W, Gao H, Qiu L, Wang C, Gao Y, Zhang S, Xue X, Chen L. A survey of data-driven and knowledge-aware explainable ai. IEEE Trans Knowl Data Eng. 2022;34(1):29–49.
- Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv. 2014.
- Sarkar S, Weyde T, d’Avila Garcez AS, Slabaugh GG, Dragicevic S, Percy C. Accuracy and interpretability trade-offs in machine learning applied to safer gambling. In: CoCo@NIPS. 2016.
- Green D, Kern H. Modeling heterogeneous treatment effects in large-scale experiments using Bayesian additive regression trees. Iowa City: The Annual Summer Meeting of the Society of Political Methodology; 2010.
- Su J, Vargas D, Sakurai K. One pixel attack for fooling deep neural networks. IEEE Trans Evolut Comput. 2017.
- Sadiku M, Shadare A, Musa S, Akujuobi C, Perry R. Data visualization. Int J Eng Res Adv Technol (IJERAT). 2016;12:2454–6135.
- Chang J, Boyd-Graber J, Gerrish S, Wang C, Blei D. Reading tea leaves: how humans interpret topic models. vol 32. 2009. p. 288–96.
- Ballav A, Ghosh M. Human factors of human machine interaction: Analyzing future trends through the past and the present. Int J Res. 2017;4:138–44.
- Xiong W, Fan H, Ma L, Wang C. Challenges of human—machine collaboration in risky decision-making. Front Eng Manag. 2022;9(1):89–103. · DOI · Paper page
Cited by
Vimbi, V., Shaffi, N., & Mahmud, M. (2024) · Brain Informatics · doi:10.1186/s40708-024-00222-1 · Cited by 433
“While ML and DL models have shown great promise in AD prediction, their black-box nature remains a significant hurdle to their adoption in real-world scenarios [81].”
BackgroundMethodSelf-citationMarques, L., Costa, B., Pereira, M., Silva, A., Santos, J., Saldanha, L., Silva, I., Magalhães, P., Schmidt, S., & Vale, N. (2024) · Pharmaceutics · doi:10.3390/pharmaceutics16030332 · Cited by 390
“Advanced models can play a pivotal role in verifying the accurate application of de-identification techniques, thereby promoting both data confidentiality and usability while aligning with ethical and regulatory standards [214].”
BackgroundExplainable artificial intelligence (XAI) in finance: a systematic literature review
Černevičienė, J., & Kabašinskas, A. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10854-8 · Cited by 289
Ethical and legal considerations in healthcare AI: innovation and policy for safe and fair use
Pham, T. (2025) · Royal Society Open Science · doi:10.1098/rsos.241873 · Cited by 261
Babatunde, S. O., Odejide, O. A., Edunjobi, T. E., & Ogundipe, D. O. (2024) · International Journal of Management & Entrepreneurship Research · doi:10.51594/ijmer.v6i3.964 · Cited by 247
Metadata
- Authors
- Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A.
- Affiliations
- KIIT University (IN)
- Birla Institute of Technology and Science, Pilani (IN)
- Jaypee Institute of Information Technology (IN)
- Duke Kunshan University (CN)
- Sapienza University of Rome (IT)
- Istituto Nazionale di Fisica Nucleare, Sezione di Roma I (IT)
- Nottingham Trent University (GB)
- Edinburgh Napier University (GB)
- Corresponding author
- Hassija, Vikas
- Chamola, Vinay
- Singal, Abhinandan
- Goel, Divyansh
- Huang, Kaizhu
- Scardapane, Simone
- Spinelli, Indro
- Mahmud, Mufti
- Hussain, Amir
- ORCID
- Venue
- Cognitive Computation
- Published
- 2023-08-24
- Volume
- 16
- Issue
- 1
- Pages
- 45-74
- ISSN
- 1866-9956 (ISSN-L)
- 1866-9964
- OpenAlex
- W4386142022
- Semantic Scholar
- 20c02c51420a69ae39b74d8d73148417819dacc1
- Cited by
- 2,495
- References
- 139
- Citations by year
- 2026: 1109 · 2025: 1030 · 2024: 338 · 2023: 18
- FWCI
- 177.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Cognitive Computation (publishedVersion, cc-by)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, other-oa)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion, other-oa)
- Funders
- European Commission (618615-EPP-1-2020-1-UKEPPKA2-CBHEJP, 2020-1-UK01-KA201-079167)
- Erasmus+ (618615-EPP-1-2020-1-UK-EPPKA2-CBHE-JP, 2020-1-UK01-KA201-079167)
- CHIST-ERA (CHIST-ERA-19-XAI-009)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- explainable AI, black-box models, model interpretability, model transparency, decision explanation, machine learning model evaluation, deep learning model interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence. https://app.cochat.ai/papers/10.1007/s12559-023-10179-8
- The paper
- Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024). Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence. Cognitive Computation. https://doi.org/10.1007/s12559-023-10179-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.