CoChat
Sign Up
Cited by 283Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationCOVID-19 diagnosis using AI

Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks

Open access
Nazir, Sajid, Dickson, Diane M., Akram, Muhammad Usman·2023·Computers in Biology and Medicine
DOI 10.1016/j.compbiomed.2023.106668
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial Intelligence (AI) techniques of deep learning have revolutionized the disease diagnosis with their outstanding image classification performance. In spite of the outstanding results, the widespread adoption of these techniques in clinical practice is still taking place at a moderate pace. One of the major hindrance is that a trained Deep Neural Networks (DNN) model provides a prediction, but questions about why and how that prediction was made remain unanswered. This linkage is of utmost importance for the regulated healthcare domain to increase the trust in the automated diagnosis system by the practitioners, patients and other stakeholders. The application of deep learning for medical imaging has to be interpreted with caution due to the health and safety concerns similar to blame attribution in the case of an accident involving autonomous cars. The consequences of both a false positive and false negative cases are far reaching for patients' welfare and cannot be ignored. This is exacerbated by the fact that the state-of-the-art deep learning algorithms comprise of complex interconnected structures, millions of parameters, and a 'black box' nature, offering little understanding of their inner working unlike the traditional machine learning algorithms. Explainable AI (XAI) techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements. This survey provides a comprehensive review of the promising field of XAI for biomedical imaging diagnostics. We also provide a categorization of the XAI techniques, discuss the open challenges, and provide future directions for XAI which would be of interest to clinicians, regulators and model developers.

Verified 2026-10-07 · 1 reference not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

This survey reviews explainable AI (XAI) techniques for deep neural network–based biomedical imaging, categorizing methods and mapping them to imaging modalities and medical specialties. It finds Grad-CAM is the most widely used XAI method, while evaluation standards, definitions, and clinical validation remain open challenges.

Key findings

  • The survey categorizes XAI techniques into visualization (perturbation-based and gradient/backpropagation-based) and explanation by example, further classified by scope (local/global), applicability (model-specific/model-agnostic), and usage (intrinsic/post hoc).
  • Across biomedical imaging, Grad-CAM is the most common and preferred XAI method, followed by LRP, LIME, and SHAP; use is increased in neurology, respiratory, and ophthalmic studies, with MRI dominant in neurology, CXR in respiratory, and fundus images in ophthalmology.
  • Reported modality-specific performance examples include COVID-19 CXR with EfficientNet-B7 at 95.01% accuracy and AUC 0.950; COVID-19 CT/CXR with VGG-19 at 95.61% accuracy; low-quality CT with ResNet-50 at 99.87% accuracy and 99.58% sensitivity; Parkinson’s disease 3D CNN at 95.29% accuracy and ROC-AUC 0.98; neonatal MRI brain dysmaturation at 98.5% accuracy; coronary CTA atherosclerosis AUC 0.91; and coronary artery calcium detection AUC 0.9627 with 0.9272 accuracy.
  • Evaluation of explainability remains unsettled: the paper states there is no agreement on measuring or evaluating explainability, and it reviews quantitative metrics such as deletion/insertion, CLEVR, and MSFI alongside qualitative human evaluation.

Methods

  • Literature survey; papers were selected by searching Web of Science, ScienceDirect, IEEE Explore, and Google Scholar for articles on explainable, interpretable, and trustworthy XAI techniques for biomedical applications.
  • No primary experiments or new data are reported; the paper synthesizes XAI techniques, tools/frameworks, evaluation methods, and applications grouped by imaging modality and medical specialty.
  • Techniques are categorized by scope, applicability, and usage/nature, with summaries provided in tables.

Limitations

  • Stated: there is no universally agreed definition of “explanation”; no agreement on evaluation metrics; lack of algorithm standardization; most explainability techniques are visualization-based and qualitative; current definitions lack consistency and may represent a false hope for patient-level decision support; model-highlighted features may differ from human-identified features.
  • Stated: the paper calls for inherently interpretable and accurate models, human-in-the-loop systems, regulatory benchmarks, and federated learning, implying these are not yet resolved.
  • Evident from the text: the survey does not report a systematic review protocol, search dates, inclusion/exclusion counts, or formal quality appraisal, and it does not independently validate the XAI methods it reviews.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.compbiomed.2023.106668

Cites

218 of 219 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 278 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The most used explainable technique for X-rays seems to be Grad-CAM, as evidenced by Table 4.

    “The most used explainable technique for X-rays seems to be Grad-CAM, as evidenced by Table 4.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Again, Grad-CAM seems to be the most favored approach for MRI as evidenced by Table 4.

    “Again, Grad-CAM seems to be the most favored approach as evidenced by Table 4.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Grad-CAM seems to be the most widely used XAI technique across different medical imaging modalities, as evidenced by Table 4.

    “There are different XAI techniques used for different medical imaging modalities but Grad-CAM seems to be the most widely used as evidenced by Table 4.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    There is not a predominant single XAI technique used across the various imaging modalities.

    “Although use of Grad-CAM and other XAI techniques have been reported in Table 4 but there is not a predominant single XAI technique used across the various imaging modalities.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The most common and preferred XAI method across various medical imaging modalities seems to be Grad-CAM, followed by LRP, LIME and SHAP and other methods.

    “However, the most common and preferred XAI method across the various medical imaging modalities seems to be Grad-CAM (Table 4), followed by LRP, LIME and SHAP and other methods.”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 102
  • May combine two claims. 81
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Increased use of XAI techniques is seen in neurology, respiratory and ophthalmic studies.

    “The increased use of XAI techniques is seen in neurology, respiratory and ophthalmic studies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    CXR images are most favored for respiratory studies due to ease of obtaining them, with Grad-CAM as the most prevalent XAI method.

    “CXR images are most favored for respiratory studies due to the ease with which these can be obtained, with Grad-CAM as the most prevalent XAI methods.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    In ophthalmic studies, fundus images seem to be the dominant modality.

    “In ophthalmic studies, fundus images seem to be the dominant modality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    In neurology, the most prevalent image modality is MRI.

    “In neurology, the most prevalent image modality is MRI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    Due to paucity of medical data, some research is focused on addressing associated challenges.

    “Due to the paucity of the medical data, some research is focused on addressing the associated challenges [140,227].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [140] Choi, 2020, Cognitive signature of brain FDG PET based on deep learning: domain transfer from Alzheimer's disease to Parkinson's disease, Eur. J. Nucl. Med. Mol. Imag. source

      Not checked in the free run.

    • cites [227] Rajpurkar, 2020, AppendiXNet: deep learning for diagnosis of appendicitis from a small dataset of CT exams using video pretraining, Sci. Rep. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The problem of bias can be addressed by an open sharing of the research process, methods, data and results, with such open sharing across institutions and funding agencies enabling open science.

    “The problem of bias can be addressed by an open sharing of the research process, methods, data and results, with such open sharing across institutions and funding agencies enabling open science [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Norori, 2021, Addressing bias in big data and AI for health care: a call for open science, Patterns source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    AI can potentially transform the image-based diagnosis in radiology.

    “AI can potentially transform the image-based diagnosis in radiology [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Folke, 2021, Explainable AI for medical imaging: explaining pneumothorax diagnoses with Bayesian teaching, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Neural networks require a lot of training data and time before they can produce acceptable results but are like a black-box providing little or no insight into the decision-making process.

    “Neural networks require a lot of training data and time before they can produce acceptable results but are like a 'black-box' providing little or no insight into the decision-making process [13].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [13] McCarthy, 2021, Enterprise imaging and big data: a review from a medical physics perspective, Phys. Med. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Encoder-decoder transformer architectures have also been proposed for image segmentation and classification.

    “Convolutional Neural Networks (CNN) are state-of-the-art models for image segmentation and classification [2], although encoder-decoder transformer architectures have also been proposed [3][4][5].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3] Ahamed, 2022, Joint learning with local and global consistency for improved medical image segmentation

      Not checked in the free run.

    • cites [4] Dosovitskiy, 2010

      Not checked in the free run.

    • cites [5] Strudel, 2021, Segmenter: transformer for semantic segmentation

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Open models can be trained with local data from hospitals across the globe to mitigate bias and boost sample size.

    “The open models can be trained with local data from, for example, hospitals across the globe to mitigate bias and boost sample size [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Norori, 2021, Addressing bias in big data and AI for health care: a call for open science, Patterns source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Algorithm or model bias can be a risk in AI as the models can generate a wrongful output based on an intentional or unintentional bias.

    “Algorithm or model bias can be a risk in AI as the models can generate a wrongful output based on an intentional or unintentional bias [40].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Meske, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Disease diagnosis through these techniques is inevitable to help overcome the shortage of, for example, 2000 radiologists in UK.

    “Disease diagnosis through these techniques is inevitable to help overcome the shortage of, for example, 2000 radiologists in UK [9].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    IBM Watson for health system had very successful performance and agreements with oncologists in some areas but poor in other areas, indicating adaptations to local practices and not extensible at a global scale.

    “The performance of IBM Watson system for oncology had very successful performance and agreements with oncologists in some areas but poor in other areas, indicating adaptations to local practices and not extensible at a global scale [38].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] Durán, 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Clinicians are the primary users of XAI systems and they may have differing views but all have a need for the explainability for the patient.

    “Clinicians are the primary users of XAI systems and they may have differing views but all have a need for the explainability for the patient [27].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27] Antoniadi, 2021, Current challenges and future opportunities for XAI in machine learning-based clinical decision support systems: a systematic review, Appl. Sci. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Most XAI work is by the AI experts which might help engineers to build better AI systems but may be limited for other stakeholders especially the end-user.

    “Most XAI work is by the AI experts which might help engineers to build better AI systems but may be limited for other stakeholders especially the end-user [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Folke, 2021, Explainable AI for medical imaging: explaining pneumothorax diagnoses with Bayesian teaching, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The regulators would most likely be interested in an overall performance and a combination of explanations of the model rather than specific cases of individual patients.

    “The regulators would most likely be interested in an overall performance and a combination of explanations of the model rather than specific cases of individual patients [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] W. Samek and K. R. Müller, “Towards explainable artificial intelligence,” in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science, vol. vol. 11700, Springer, Cham..

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The models with an intrinsic interpretability provide better explainability at lower predictive performance compared to post-hoc methods that in contrast have less explainability but a higher predictive performance.

    “The models with an intrinsic interpretability provide better explainability at lower predictive performance compared to post-hoc methods that in contrast have less explainability but a higher predictive performance [59].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [59] Bauer, 2021, Expl(AI)n it to me – explainable AI and information systems research, Business & Information Systems Engineering source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The proposed 2D capsule network system provided better visualizations and malignancy predictions compared to 3D CNN and achieved better diagnostic accuracy.

    “It was demonstrated that the proposed 2D capsule network system provided better visualizations and malignancy predictions compared to 3D CNN and achieved better diagnostic accuracy [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] R. LaLonde, D. Torigian and U. Bagci, “Encoding visual attributes in capsules for explainable medical diagnoses,” in Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    VGG-16 model requires 15.5 billion floating-point operations for a single image classification with approximately 138 million parameters.

    “For example, VGG-16 model requires 15.5 billion floating-point operations for a single image classification with approximately 138 million parameters [56].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [56] Yeom, 2021, Pruning by explaining: a novel criterion for deep neural network pruning, Pattern Recogn. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Failure to identify and address the concerns of the system users can lead to another AI Winter, where a small market segment could change the research and development focus from XAI.

    “Failure to identify and address the concerns of the system users can lead to another “AI Winter”, where a small market segment could change the research and development focus from XAI [61].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61] A. Preece, D. Harborne, D. Braines, R. Tomsett and S. Chakraborty, “Stakeholders in Explainable AI,” [Online]. Available: https://doi.org/10.48550/arXiv.1810.00184. [Accessed 15 September 2022].

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    NICE uses a mask over the entire image for each pixel that captures saliency based on its contribution to the prediction.

    “Natural Image Compression and Explanation (NICE) uses a mask over the entire image for each pixel in the image that captures the saliency based on its contribution to the prediction.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [75] Li, 2020, Neural image compression and explanation, IEEE Access source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    While trustworthiness should be a property of every explainable system, this does not mean that every trustworthy system should also be explainable.

    “While trustworthiness should be a property of every explainable system, this does not mean that every trustworthy system should also be explainable [2].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [2] Barredo Arrieta, 2020, Explainable Artificial Intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Self-explaining AI produces a decision and an explanation, provides a human understandable explanation and confidence level for each decision, and was deemed to motivate and lead towards trustworthy AI.

    “Self-explaining AI is proposed that produces a decision and an explanation.It provides a human understandable explanation and confidence level for each decision and which was deemed to motivate and lead towards trustworthy AI [55].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [55] Elton, 2020, Self-explaining AI as an alternative to interpretable AI

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Heatmaps are commonly used to explain AI models and the quality of heatmaps can be appreciated using perturbation analysis.

    “The heatmaps are commonly used to explain AI models and the quality of heatmaps can be appreciated using perturbation analysis [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] W. Samek and K. R. Müller, “Towards explainable artificial intelligence,” in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science, vol. vol. 11700, Springer, Cham..

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The introduction of a hierarchy excluded the regions with less important features.

    “The introduction of a hierarchy excluded the regions with less important features [74].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [74] Monroe, 2021, HihO: accelerating artificial intelligence interpretability for medical imaging in IoT applications using hierarchical occlusion, Neural Comput. Appl. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    There is a need for certifications of DNN models for clinical adoption.

    “Similar to ISO 13485:2016 Medical devices standard for Quality management systems that prescribes the medical devices regulatory requirements, there is a need for certifications of DNN models for clinical adoption [64].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [64] 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    LIME was proposed to explain any classifier's predictions by approximating it locally around the prediction.

    “Local Interpretable Modelagnostic Explanations (LIME) was proposed to explain any classifier's predictions by approximating it locally around the prediction [70].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [70] Ribeiro, 2016, Why should i trust you?': explaining the predictions of any classifier

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The patients require simplified explanations for the clinical decision as the DNN models are very complex and cannot be explained to a lay user.

    “The patients require simplified explanations for the clinical decision as the DNN models are very complex and cannot be explained to a lay user [49].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] Gryz, 2021, Black box algorithms and the rights of individuals: no easy solution to the ‘explainability’ problem, Internet Policy Review source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Formal mathematical notations and techniques can unambiguously provide explanations for domain experts but are not targeted to the end users.

    “Similarly, formal mathematical notations and techniques can unambiguously provide explanations for domain experts but are not targeted to the end users [63].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] Heuillet, 2021, Explainability in deep reinforcement learning, Knowl. Base Syst. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The application developers would like to use the XAI techniques to understand the inner working and model improvement.

    “The application developers would like to use the XAI techniques to understand the inner working and model improvement [62].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [62] Wang, 2020, CNN hyperparameter optimization based on CNN visualization and perception hash algorithm

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Certification frameworks for a formal validation of models could testify the model operation statistically and ensure that the model predictions meet the required regulatory standards.

    “Certification frameworks for a formal validation of models could testify the model operation statistically and ensure that the model predictions meet the required regulatory standards such as ISO/IEC CD 23053 (2020) for establishing a framework for use of AI systems [49].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] Gryz, 2021, Black box algorithms and the rights of individuals: no easy solution to the ‘explainability’ problem, Internet Policy Review source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    RISE generates an importance map based on the importance of each pixel to the prediction.

    “Randomized Input Sampling for Explanation of Black-box Models (RISE) generates an importance map based on the importance of each pixel to the prediction [72].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [72] V. Petsiuk, A. Das and K. Saenko, “RISE: Randomized Input Sampling for Explanation of Black-box Models,” [Online]. Available: https://doi.org/10.48550/arXiv.1806.07421. [Accessed 15 September 2022].

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Important features are localized in an image.

    “A novel hierarchical occlusion algorithm is proposed and is based on two observations; that the important features are localized in an image, and features collection in multiple scales can affect the prediction outcome [74].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] Monroe, 2021, HihO: accelerating artificial intelligence interpretability for medical imaging in IoT applications using hierarchical occlusion, Neural Comput. Appl. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    IASSA was proposed that used an iterative and adaptive sampling module to determine the saliency map of each pixel for the model prediction.

    “Iterative and Adaptive Sampling with Spatial Attention (IASSA) was proposed that used an iterative and adaptive sampling module to determine the saliency map of each pixel for the model prediction [73].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [73] Vasu, 2020, Iterative and adaptive sampling with spatial attention for black-box model explanations

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An interpretable model can obtain transparency by providing the reasons behind their output.

    “An interpretable model can obtain transparency by providing the reasons behind their output [52].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [52] Abeyrathna, 2021, Extending the tsetlin machine with integer-weighted clauses for increased interpretability, IEEE Access source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Feature collection in multiple scales can affect the prediction outcome.

    “A novel hierarchical occlusion algorithm is proposed and is based on two observations; that the important features are localized in an image, and features collection in multiple scales can affect the prediction outcome [74].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] Monroe, 2021, HihO: accelerating artificial intelligence interpretability for medical imaging in IoT applications using hierarchical occlusion, Neural Comput. Appl. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    There is little agreement on what is an interpretable ML system and how it should be evaluated.

    “There is little agreement on what is an interpretable ML system and how it should be evaluated [51].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] Doshi-Velez, 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    There is a need to have a more formalized definition of interpretability in the context of DNNs.

    “There is a need to have a more formalized definition of interpretability in the context of DNNs [51].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [51] Doshi-Velez, 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A lack of consensus in the meaning of explainability originates due to different stakeholders, each having their own understanding of the terms.

    “A lack of consensus in the meaning of explainability originates due to different stakeholders, each having their own understanding of the terms [61].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [61] A. Preece, D. Harborne, D. Braines, R. Tomsett and S. Chakraborty, “Stakeholders in Explainable AI,” [Online]. Available: https://doi.org/10.48550/arXiv.1810.00184. [Accessed 15 September 2022].

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    There is a trade-off between the accuracy and interpretability of the models, where non-linear models with better performance, such as DNNs are less explainable, and linear models which are least accurate, such as decision trees, are more explainable.

    “There is a trade-off between the accuracy and interpretability of the models, as shown in Fig. 2, where non-linear models with better performance, such as DNNs are less explainable, and linear models which are least accurate, such as decision trees, are more explainable [2,21,45].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [2] Barredo Arrieta, 2020, Explainable Artificial Intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI, Inf. Fusion source

      Not checked in the free run.

    • cites [21] Adadi, 2018, Peeking inside the black-box: a survey on explainable artificial intelligence (XAI), IEEE Access source

      Not checked in the free run.

    • cites [45] Gunning, 2019

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The model during training ideally should learn the discriminating image features of the various classes but instead can also learn extraneous features.

    “The model during training ideally should learn the discriminating image features of the various classes but instead can also learn extraneous features [42].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [42] Zech, 2018, Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study, PLoS Med. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    For domains with a lack of knowledge about the underlying causal process, such as healthcare, it is prudent not to prefer explainability over diagnostic or predictive accuracy as this may affect the health and well-being of patients.

    “For domains with a lack of knowledge about the underlying causal process, such as healthcare, it is prudent not to prefer explainability over diagnostic or predictive accuracy as this may affect the health and well-being of patients with some who get additional investigations and treatment and similarly those who are mjs-or undiagnosed [57].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [57] London, 2019, Artificial intelligence and black-box medical decisions: accuracy versus explainability, Hastings Cent. Rep. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The attention-based caption model exploited learned attention for better interpretability, well-aligned to human intuition on datasets with METEOR and BLEU metrics.

    “The proposed technique exploited learned attention for better interpretability, well-aligned to human intuition on the datasets with METEOR and BLEU metrics [92].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [92] Xu, 2015

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LRP has limitations such as heatmaps being noisy and non-discriminative.

    “The LRP has some limitations such as the heatmaps were noisy and could be non-discriminative [84].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [84] Jung, 2021, Explaining CNN and RNN using selective layer-wise relevance propagation, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Among the evaluated XAI methods on the evaluation task, Deconvnet and Grad-CAM were least accurate, whereas LRP, Integrated Gradients, and Guided Backpropagation were most accurate.

    “Among the evaluated XAI methods on the evaluation task, Deconvnet and Grad-CAM were found to be least accurate, whereas LRP, Integrated Gradients and Guided Backpropagation were found to be the most accurate methods [130].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [130] Arras, 2022, CLEVR-XAI: a benchmark dataset for the ground truth evaluation of neural network explanations, Inf. Fusion source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    SHAP evaluation was more consistent with human understanding and intuition with improved computational efficiency compared to other selected approaches.

    “The evaluation of proposed technique was found to be more consistent with human understanding and intuition with an improved computational efficiency compared to the other selected approaches [90].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [90] Lundberg, 2017, A unified approach to interpreting model predictions, Adv. Neural Inf. Process. Syst.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Grad-CAM produces visual explanations by using gradients in the final model layer to identify and highlight important image regions.

    “Gradient-weighted Class Activation Mapping (Grad-CAM) was proposed for producing visual explanations by using the gradients in the final model layer to identify and highlight the important image regions [79].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [79] Selvaraju, 2017, Visual explanations from deep networks via gradient-based localization

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Eigen-CAM could work in the presence of adversarial noise and irrespective of model accuracy.

    “It was demonstrated that the proposed method could work in the presence of adversarial noise and irrespective of the model accuracy [82].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [82] Muhammad, 2020, Eigen-CAM: class activation map using principal components

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    SIDU was motivated by Grad-CAM and RISE, but unlike Grad-CAM it was gradient-free, and unlike RISE it used a combined final mask instead of a random mask.

    “The method was motivated by Grad-CAM and RISE, however, unlike Grad-CAM it was gradient-free, and unlike RISE, it used a combined final mask instead of a random mask [76].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] S. M. Muddamsetty, N. S. J. Mohammad and T. B. Moeslund, “SIDU: similarity difference and uniqueness method for explainable AI,” in IEEE International Conference on Image Processing (ICIP).

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    SLRP provided less noise, better class discrimination, and held entire target objects compared to LRP, CLRP, and SGLRP.

    “The model was evaluated through a comparison with LRP, Contrastive LRP (CLRP), and Softmax Gradient LRP (SGLRP), and found to provide less noise, better class discrimination and held entire target objects [84].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [84] Jung, 2021, Explaining CNN and RNN using selective layer-wise relevance propagation, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The combined text-to-bounding-box and GAIN approach achieved better attention maps and higher classification accuracy.

    “It was shown that the combined approach achieved better attention maps and a higher classification accuracy [114].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [114] Kashyap, 2020, Looking in the right place for anomalies: explainable ai through automatic location learning

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The web-based interactive CNN visualization interface was evaluated using surveys for both experts and non-experts and showed the approach to be effective.

    “The interface was evaluated using surveys for both experts of DNN and non-experts and showed the approach to be effective [126].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [126] Vyas, 2020, An interactive graphical visualization approach to CNNs and RNNs

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Gradient-based methods are lacking because neurons' gradients do not relate to top-down saliency.

    “It was argued that the Gradient-Based methods are lacking as the neurons' gradients do not relate to the top-down saliency [85].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [85] Zhang, 2018, Top-down neural attention by excitation Backprop, Int. J. Comput. Vis. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Grad-CAM++ provided better visual explanations compared to Grad-CAM and was better in explaining multiple occurrences of the same class objects in a single image.

    “Grad-CAM++ was proposed as a method to build on Grad-CAM and provided better visual explanations compared to it and was better in explaining multiple occurrences of the same class objects in a single image [80].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [80] Chattopadhay, 2018, Grad-CAM++: generalized gradient-based visual explanations for deep convolutional networks

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    DeepCOVIDExplainer's best performing individual DNN architectures were VGG-19 and DenseNet-161, and heatmaps generated by Grad-CAM++ were the most reliable compared with Grad-CAM and LRP.

    “The best performing individual DNN architectures were VGG-19 and DenseNet-161 and the heatmaps generated by Grad-CAM++ were the most reliable in comparison with Grad-CAM and LRP [162].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [162] Karim, 2020, Deepcovidexplainer: explainable COVID-19 diagnosis from chest X-ray images

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    HFnet showed that low-level layer features are superior in comparison to low-level semantic questions.

    “A Hierarchical Feature Network (HFnet) was proposed that showed that the low-level layer features are superior in comparison to low-level semantic questions [115].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [115] Hong, 2019, Exploiting hierarchical visual features for visual question answering, Neurocomputing source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    In multiclass classification of NIH CXR images into COVID-19, pneumonia, and normal, any foreign substance in the lung cavity resulted in misclassification, and sometimes a correct classification was due to the wrong reasons.

    “It was found that any foreign substance in the lung cavity resulted in misclassification of images, and sometimes a correct classification was due to the wrong reasons [164].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [164] Lee, 2020, Evaluation of scalability and degree of fine-tuning of deep convolutional neural networks for COVID-19 screening on chest X-ray images using explainable deep-learning algorithm, J. Personalized Med. source

      Not checked in the free run.

  • The quote only partly states this claim.

    Premisecitation

    VGG-19 with transfer learning had an accuracy of 95.61% in COVID-19 detection.

    “The model performance was more accurate on CT compared to CXR images, and had an accuracy of 95.61% in COVID-19 detection.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [158] Panwar, 2020

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Gradient-based methods lack generalizability and robustness.

    “Thus, the gradient based methods lack in generalizability and robustness [85].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [85] Zhang, 2018, Top-down neural attention by excitation Backprop, Int. J. Comput. Vis. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Compared to qualitative approaches through visual observations and comparisons, quantitative approaches can provide more objectivity and reliability in the process.

    “Compared to qualitative approaches through visual observations and comparisons, quantitative approaches can provide more objectivity and reliability in the process.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    explAIner was qualitatively evaluated on the MNIST dataset with different types of human users and received very positive feedback.

    “The framework was qualitatively evaluated on MNIST dataset with different types of human users and received very positive feedback [124].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [124] Spinner, 2020, IEEE Trans. Visual. Comput. Graph.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Deconvnet visualizations showed features are not random or uninterpretable patterns but have class discrimination and increasing invariance between layers.

    “It was shown through visualizations that the features are not random or uninterpretable patterns but rather have class discrimination and increasing invariance between layers [87].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [87] Zeiler, 2014, Visualizing and understanding convolutional networks

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    SIDU estimates pixel saliency by extracting feature maps from the last CNN layer and creating a difference mask to form the visual explanation of the prediction.

    “A visual explainable system Similarity Difference and Uniqueness Method (SIDU) was proposed to estimate pixel saliency by extracting feature maps from the last CNN layer, and creating a difference mask to form the visual explanation of the prediction [76].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] S. M. Muddamsetty, N. S. J. Mohammad and T. B. Moeslund, “SIDU: similarity difference and uniqueness method for explainable AI,” in IEEE International Conference on Image Processing (ICIP).

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    SIDU showed better visual explanations with greater trust for the human expert on clinical and general datasets.

    “The proposed scheme was evaluated on clinical and general datasets and showed better visual explanations with a greater trust for the human expert [76].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [76] S. M. Muddamsetty, N. S. J. Mohammad and T. B. Moeslund, “SIDU: similarity difference and uniqueness method for explainable AI,” in IEEE International Conference on Image Processing (ICIP).

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    COVID-19 detection on CXR images with a fine-tuned EfficientNet-B7 achieved an accuracy of 95.01% and AUC scores of 0.950, which were better than many other DNN approaches for COVID-19 detection.

    “COVID-19 detection on CXR images with a fine tuned EfficientNet-B7 achieved an accuracy of 95.01% and AUC scores of 0.950, which were better in comparison to many other DNN approaches for COVID-19 detection [161].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [161] Chetoui, 2021, Deep efficient neural networks for explainable COVID-19 detection on CXR images

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There is no agreement on techniques for measuring or evaluating the explainability performance of an AI model.

    “Despite the large number of existing explainable models, there is no agreement on techniques for measuring or evaluating the explainability performance of an AI model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Saliency map and explanation-by-examples are complementary: saliency map highlights pixels with significant contribution to the model prediction, and explanation-by-examples selects training images likely for prediction in Bayesian Teaching.

    “The two explainability approaches are complementary, in that, saliency map highlights the pixels with significant contribution to the model prediction and explanation-by-examples selects the training images likely for prediction in Bayesian Teaching [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Folke, 2021, Explainable AI for medical imaging: explaining pneumothorax diagnoses with Bayesian teaching, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Automated DNN method was better than radiological image biomarkers for stroke outcome prediction and was helpful in treatment selection on MR CLEAN Registry dataset of CT angiography images.

    “An adapted ResNet architecture was used on MR CLEAN Registry dataset of CT angiography images and determined that the automated DNN method was better than radiological image biomarkers for stroke outcome prediction and was helpful in the treatment selection [136].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [136] Hilbert, 2019, Data-efficient deep learning of radiological image data for outcome prediction after endovascular treatment of patients with acute ischemic stroke, Comput. Biol. Med. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Feature map showed positive values for portions with significant influence on disease diagnosis for cardiomegaly using CNN and CXR images.

    “The feature map showed positive values for the portions with significant influence on the disease diagnosis [236].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [236] Yoo, 2021, Diagnosis support model of cardiomegaly based on CNN using ResNet and explainable feature map, IEEE Access source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    CT images from multiple sources were mixed to avoid overfitting and bias.

    “The CT images from multiple sources were mixed to avoid overfitting and bias [175].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [175] Wu, 2021, JCS: an explainable COVID-19 diagnosis system by Joint classification and segmentation, IEEE Trans. Image Process. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    XAI can help in identifying all factors which form the basis of a particular decision.

    “The XAI can help in identifying all such factors which form the basis of a particular decision.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Premisecitation

    Concept attribution method provided insights of high-level feature importance in trained CNN model for liver tumor segmentation on CT images LiTS database.

    “The method provided insights of the high-level feature importance in the trained CNN model [224].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [224] Couteaux, 2019, Towards interpretability of segmentation networks by analyzing deepdreams

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Stacked interpretable architecture provided better insights into model's decisions and achieved better performance compared to considered approaches for classification of abnormal lung nodules for lung cancer prediction on LIDC-IDRI dataset.

    “The proposed architecture provided better insights into the model's decisions and achieved better performance compared to the considered approaches.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [182] Kumar, 2019, SISC: end-to-end interpretable discovery radiomics-driven lung cancer prediction via stacked interpretable sequencing cells, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    CNN ensemble improved classification and achieved an AUC of 90.29% in predicting nodules that would probably be diagnosed as cancerous in two years.

    “The ensemble was found to improve the classification of the system and achieved an AUC of 90.29% in predicting nodules that would probably be diagnosed as cancerous in two years.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [179] Paul, 2020, Convolutional Neural Network ensembles for accurate lung nodule malignancy prediction 2 years in the future, Comput. Biol. Med. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductcitation

    The dataset for paediatric pneumonia diagnosis using ensemble of CNNs was small with low-resolution images.

    “However, the dataset was small with low-resolution images [167].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [167] Liz, 2021, Ensembles of Convolutional Neural Network models for pediatric pneumonia diagnosis, Future Generat. Comput. Syst. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Emergent Symbolic language are cornerstones of GOFAI systems which are explainable.

    “Emergent Symbolic language are the cornerstones of Good Old-Fashioned AI (GOFAI) systems which are explainable [252].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [252] Chowdhury, 2021, Emergent Symbolic language based deep medical image classification

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    X-Caps XAI technique generated malignancy scores with explanations at human-level that were better interpretable by radiologists for lung cancer prediction using CXR images.

    “CXR images were used for lung cancer prediction using the proposed X-Caps XAI technique generating malignancy scores with explanations at human-level that were better interpretable by the radiologists [15].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15] R. LaLonde, D. Torigian and U. Bagci, “Encoding visual attributes in capsules for explainable medical diagnoses,” in Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    JSC identified and segmented the opacification areas, a basic CT feature of COVID-19.

    “The JSC identified and segmented the opacification areas, a basic CT feature of COVID-19 [175].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [175] Wu, 2021, JCS: an explainable COVID-19 diagnosis system by Joint classification and segmentation, IEEE Trans. Image Process. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Emergent languages have been shown to enhance neural network capabilities.

    “Emergent languages involve multi-agent communication games with a sender/ receiver coordinate to complete a task and have been shown to enhance the neural network capabilities [252].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [252] Chowdhury, 2021, Emergent Symbolic language based deep medical image classification

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    AppendiXNet model performance was comparable to similar 3D CNN models and had high diagnostic performance for acute appendicitis detection.

    “The model performance was found comparable to similar 3D CNN models and had a high diagnostic performance for acute appendicitis detection [227].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [227] Rajpurkar, 2020, AppendiXNet: deep learning for diagnosis of appendicitis from a small dataset of CT exams using video pretraining, Sci. Rep. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Proposed radiomic sequencer achieved correct predictions that can potentially improve clinical adoption.

    “The proposed radiomic sequencer achieved correct predictions that can potentially improve clinical adoption [182].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [182] Kumar, 2019, SISC: end-to-end interpretable discovery radiomics-driven lung cancer prediction via stacked interpretable sequencing cells, IEEE Access source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    AID-Net achieved an AUC of 0.9627 and a detection accuracy of 0.9272 for Coronary Artery Calcium detection.

    “The proposed system achieved an AUC of 0.9627 and a detection accuracy of 0.9272 [240].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [240] Huo, 2019, Coronary calcium detection using 3D attention identical dual deep network based on weakly supervised learning, Med. Imaging Image Process.

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    XAI can help in overcoming challenges associated with limited information present in an X-ray and highlighting how density, shape, size and location of abnormalities affect final diagnosis.

    “This can also help in overcoming the challenges associated with the limited information present in an X-ray and actually highlighting how the density, shape, size and location of abnormalities affect the final diagnosis.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    RFGB found simpler rules for modelling features and could be easily compared to a large single highly accurate model.

    “RFGB found simpler rules for modelling the features and could be easily compared to a large single highly accurate model [219].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [219] Kunapuli, 2018, A decision-support tool for renal mass classification,” journal of digital imaging, J. Digit. Imag. source

      Not checked in the free run.

  • The quote only partly states this claim.

    Premisecitation

    Multi-scale attention network achieved an average classification accuracy of 94.78% for classification of seven types of pulmonary textures on high resolution CT images.

    “An average classification accuracy of 94.78% was achieved.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [177] Xu, 2019, Pulmonary textures classification via a multi-scale attention network, IEEE journal of biomedical and health informatics source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Hierarchical diagnosis performed better than other models for classification and identified relevant image regions with attention maps.

    “It was shown that the hierarchical diagnosis performed better than the other models for classification, and identified the relevant image regions with attention maps [217].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [217] Barata, 2021, Explainable skin lesion diagnosis using taxonomies, Pattern Recogn. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LRP was useful for explainability of Alzheimer's classification predictions and applicable to similar disease diagnosis using MRI data.

    “It was shown that LRP was useful for explainability of Alzheimer's classification predictions and applicable to similar disease diagnosis using MRI data [144].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [144] Böhle, 2019, Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer's disease classification, Front. Aging Neurosci. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ABCD-rule of dermoscopy is known to outperform other methods.

    “The ABCD-rule of dermoscopy is known to outperform other methods [212].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [212] Stieler, 2021, Towards domain-specific explainable AI:model interpretation of a skin image classifier using a human approach

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    Embedding model explainability in early model training can save time and help clinicians in using a model's prediction for clinical decisions.

    “The study concluded that by embedding model explainability in the early model training can save time and help clinicians in using a model's prediction for the clinical decisions [152].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [152] Windisch, 2020, Implementation of model explainability for a basic brain tumor detection using convolutional neural networks on MRI slices, Neuroradiology source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Four CNN models and heatmap visualizations determined important features which were compared against expert's judgement for diabetic retinopathy.

    “Four CNN models were used and visualizations by heatmaps determined the important features which were compared against the expert's judgement [187].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [187] Reguant, 2021, Understanding inherent image features in CNN-based assessment of diabetic retinopathy, Sci. Rep. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    From aggregated explanations it was concluded that cisterns and ventricles governed age predictions.

    “From the aggregated explanations it was concluded that cisterns and ventricles governed the age predictions [146].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [146] Levakov, 2020, From a deep learning model back to the brain—identifying regional predictors and their relation to aging, Hum. Brain Mapp. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    It can be difficult to control ultrasound image quality due to manual operation, operator dependency and acoustic shadows.

    “It can be difficult to control the image quality due to manual operation, operator dependency and acoustic shadows [253].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [253] Komatsu, 2021, Towards clinical application of artificial intelligence in ultrasound imaging, Biomedicines source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    CA-Net provided better explainability by visualizing attention weight maps, although it had around 15 times smaller model size.

    “CA-Net also provided better explainability by visualizing the attention weight maps, although it had around 15 times smaller model size [153].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [153] Lopatina, 2020, Investigation of deep-learning-driven identification of multiple sclerosis patients based on susceptibility-weighted images using relevance analysis, Front. Neurosci. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Significant features for prediction were identified by isolating neuron activation patterns for diabetic retinopathy detection.

    “The significant features for the prediction were identified by isolating the neuron activation patterns [192].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [192] Niu, 2021, Explainable diabetic retinopathy detection and retinal image generation, IEEE journal of biomedical and health informatics

      Not checked in the free run.

  • The quote only partly states this claim.

    Premisecitation

    Grad-CAM visualisations helped identify tumor location for multiclass classification on IXI dataset of MRI slices.

    “Grad-CAM was used for visualisations and helped to identify the tumor location.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [152] Windisch, 2020, Implementation of model explainability for a basic brain tumor detection using convolutional neural networks on MRI slices, Neuroradiology source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Video data converted to barcode type timeline aided explainability and made it easier to identify substructures for fetal ultrasound abnormality detection.

    “The video data was converted to a barcode type timeline to aid the explainability which made it easier to identify the substructures [243].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [243] Komatsu, 2021, Detection of cardiac structural abnormalities in fetal ultrasound videos using deep learning, Appl. Sci. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Different CNN models had consistent classification performance but image feature disagreement between different architectures was 70% as the highest value on EyePACS and DIARETDB1 datasets.

    “The datasets EyePACS and DIARETDB1 were used and although different CNN models had a consistent classification performance, the image feature disagreement between different architectures was 70% as the highest value [187].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [187] Reguant, 2021, Understanding inherent image features in CNN-based assessment of diabetic retinopathy, Sci. Rep. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Choice of MRI slice/image is important and selecting the best MRI slice improves model performance.

    “The choice of the MRI slice/image is important and the authors proposed selecting the best MRI slice for improved model performance [220].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [220] Hassan, 2022, Prostate cancer classification from ultrasound and MRI images using deep learning based Explainable Artificial Intelligence, Future Generat. Comput. Syst. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Breast ultrasound and MRI can provide additional information for mammographic diagnosis.

    “Breast ultrasound and MRI can provide additional information for the mammographic diagnosis [254].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [254] Le, 2019, Artificial intelligence in breast imaging, Clin. Radiol. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    3D CNN system achieved an accuracy of 95.29% and ROC-AUC of 0.98 for both classes for Parkinson's disease detection and classification of MRI scans from PPMI dataset.

    “The system achieved an accuracy of 95.29% and Receiver Operating Characteristics (ROC)-AUC of 0.98 for both classes.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [148] Chakraborty, 2020, Detection of Parkinson's disease from 3T T1 weighted MRI scans using 3D convolutional neural network, Diagnostics source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Postulates provided association between disease and pathogens for interpretability in diabetic retinopathy detection.

    “These postulates provided the association between the disease and the pathogens [192].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [192] Niu, 2021, Explainable diabetic retinopathy detection and retinal image generation, IEEE journal of biomedical and health informatics

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Results demonstrated high classification accuracy and explainability for multilabel classification of DR images.

    “The results demonstrated a high classification accuracy and explainability [191].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [191] Quellec, 2021, ExplAIn: explanatory artificial intelligence for diabetic retinopathy diagnosis, Med. Image Anal. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Use of frameworks combining automated segmentation and classification could use segmentation results to guide classification.

    “Similarly, the use of frameworks combining the automated segmentation and classification could for example, can use the segmentation results to guide the classification process [214].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [214] Xie, 2020, A mutual bootstrapping model for automated skin lesion segmentation and classification, IEEE Trans. Med. Imag. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Visualizations can provide a degree of confidence in the model, but without a standard way of evaluating different models and their outcomes, they cannot be trusted for medical imaging.

    “This can provide a degree of confidence in the model but without a standard way of evaluating the different models and their outcomes cannot be trusted for medical imaging.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Large and high-quality data is required by DNN for better results, without which the model may overfit.

    “Large and high-quality data is required by DNN for better results without which the model may overfit, that is, becomes closely aligned to the limited set of datapoints [265].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [265] Tayarani-N, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    XAI techniques can help in understanding the predictions of a deep learning model, yet there are challenges to be overcome before these techniques become mainstream for healthcare applications.

    “The XAI techniques can help in understanding the predictions of a deep learning model yet there are challenges to be overcome before these techniques become mainstream for healthcare applications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Most studies relating to cancer detection use a post-hoc explainability technique, whereas development of intrinsic explanation techniques would be advantageous for adoption and explainability.

    “Most of the studies relating to cancer detection use a post-hoc explainability technique whereas a development of intrinsic explanation techniques would be advantageous for adoption and explainability [258].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [258] Gulum, 2021, A review of explainable deep learning cancer detection models in medical imaging, Appl. Sci. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Diagnosis system using CNN for segmentation and classification of glaucoma detection with fundus images was considered useful for mass screening as it could be run without Internet connection on a mobile device.

    “The system was considered useful for mass screening as it could be run without an Internet connection on a mobile device [193].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [193] Martins, 2020, Offline computer-aided diagnosis for Glaucoma detection using fundus images targeted at mobile devices, Comput. Methods Progr. Biomed. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The adoption of AI in clinical practice would take more effort in making the XAI techniques more comprehensible.

    “The adoption of AI in clinical practice would thus take more effort in making the XAI techniques to be more comprehensible.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Creation of quantitative metrics for model comparison for explainability will give rise to successful application of relevant explainability techniques to different image modalities.

    “The creation of quantitative metrics for model comparison for explainability will give rise to the successful application of relevant explainability techniques to different image modalities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    XAI techniques using multimodal DNN models can help with hidden patterns related to each modality in segmentation and classification tasks.

    “XAI techniques using multimodal DNN models can help with the hidden patterns related to each modality in segmentation and classification tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Without sufficient data, AI models can suffer from overfitting and data from different hospitals differ due to variety of data collection policies and protocols.

    “Without sufficient data, the AI models can suffer from overfitting and the data from different hospitals differ due to the variety of data collection policies and protocols [265].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [265] Tayarani-N, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Ophthalmology is one of the most widely used applications of AI and efforts towards explainability in this area are still at a very basic level.

    “Ophthalmology is one of the most widely used application of AI and there are efforts towards explainability in this area but are still at very basic level.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Explanatory Interactive Learning (XIL), which adds the human in the model training loop, can help increase the adoption of XAI techniques.

    “Explanatory Interactive Learning (XIL) which adds the human in the model training loop [259] can also help to increase the adoption of the XAI techniques.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [259] Schramowski, 2020, Making deep neural networks right for the right scientic reasons by interacting with their explanations, Nat. Mach. Intell. source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The quality of images improves the performance of DNN algorithms in correctly predicting a disease and therefore can reduce the explainability gap.

    “The quality of images improves the performance of DNN algorithms in correctly predicting a disease and therefore can reduce the explainability gap.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The term 'explanation' as used in XAI is not very well defined.

    “The term 'explanation' as used in XAI is not very well defined [60,260,261].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [60] Páez, 2019, The pragmatic turn in explainable artificial intelligence (XAI), Minds Mach. source

      Not checked in the free run.

    • cites [260] Payrovnaziri, 2020, Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review, J. Am. Med. Inf. Assoc. source

      Not checked in the free run.

    • cites [261] Gillmann, 2021, Ten open challenges in medical visualization, Comput. Graphics Appl. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A universally agreed definition of explanations is lacking.

    “A universally agreed definition of explanations is lacking and there has been some efforts in this direction by developing mathematical methods [25].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25] W. Samek and K. R. Müller, “Towards explainable artificial intelligence,” in Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Lecture Notes in Computer Science, vol. vol. 11700, Springer, Cham..

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Current definitions lack consistency with explainable AI representing a false hope to provide patient-level decision support.

    “This differs for the different perspectives of a patient, clinician and engineer and the current definitions lack consistency with explainable AI representing a false hope to provide a patient level decision support [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] Ghassemi, 2021, The false hope of current approaches to explainable artificial intelligence in health care, The Lancet Digital Health source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    There is no agreement on what is meant by 'explanation'.

    “There is no agreement on what is meant by 'explanation' [262].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [262] Guidotti, 2018, A survey of methods for explaining black box models, ACM Comput. Surv. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The development of DNN and attempts to identify their internal workings and decision-making process has resulted in development of many useful techniques.

    “The development of DNN and then having attempts to identify their internal workings and decision-making process has resulted in development of many useful techniques.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Such metrics can also meet the needs for regulations and standards in assessing and approving XAI techniques for clinical practice.

    “Such metrics can also meet the needs for the regulations and standards in assessing and approving the XAI techniques for the clinical practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Various techniques can be employed to address the lack of data, such as data augmentation.

    “Various techniques can be employed to address the lack of data, such as, data augmentation [34,37].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [34] Karthik, 2021, AI for COVID-19 detection from radiographs: incisive analysis of state of the art techniques, IRBM source

      Not checked in the free run.

    • cites [37] Singh, 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The ultimate responsibility for image-based disease diagnosis would still be with human radiologists.

    “The ultimate responsibility would still be with the human radiologists [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Strudel, 2021, Segmenter: transformer for semantic segmentation

      Not checked in the free run.

    • Rests on an assumption: The final decision for image-based disease diagnosis would rest with the specialist, whereas XAI model insights should be used only to aid the decision.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    It is aimed at translating algorithmic model research to actual practice in domains such as healthcare.

    “It is aimed at translating the algorithmic model research to the actual practice in domains such as healthcare [275].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [275]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Such automated workflows are expected to make the process of incorporating XAI easier into clinical decisions by making it possible to select the best visualizations in response to a few simple questions.

    “Such automated workflows are expected to make the process of incorporating XAI easier into clinical decisions by making it possible to select the best visualizations in response to a few simple questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    DNN systems have sensitivity similar to radiologists but often detect different lesion characteristics.

    “DNN systems have sensitivity similar to radiologists but often times these DNN can detect different lesion characteristics [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] Chan, 2020, Deep learning in medical image analysis, Adv. Exp. Med. Biol. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Funders and governments are emphasizing open sharing of data for AI model training, but most data are not representative, interoperable, and sufficiently diverse.

    “The funders and governments are emphasizing an open sharing of data for AI model training but most data are not representative, interoperable, and sufficiently diverse [39].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Norori, 2021, Addressing bias in big data and AI for health care: a call for open science, Patterns source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The model architecture has many hyperparameters that need to be tried and re-tried for designing an architecture with optimum performance.

    “The model architecture has many hyperparameters that need to be tried and re-tried for designing an architecture with an optimum performance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Automation will not only make XAI techniques widely available but will enable medical practitioners to make better and informed decisions by leveraging insights through XAI.

    “The automation will not only make these XAI techniques widely available but will enable medical practitioners to make better and informed decisions by leveraging the insights through XAI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Use of sufficient, representative, and diverse type of data for DNN models can help reduce bias and enhance generalizability.

    “The use of sufficient, representative, and diverse type of data for DNN models can help to reduce bias and enhance generalizability.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A framework for trustworthy and XAI combined selected features of trusted oracles, blockchain and smart contracts to reduce adversarial attacks and bias.

    “A framework for trustworthy and XAI combined selected features of trusted oracles, blockchain and smart contracts to reduce adversarial attacks and bias [11].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] M. Nassar, K. Salah, M. ur Rehman and D. Svetinovic, “Blockchain for explainable and trustworthy artificial intelligence,” Wiley Interdisciplinary Reviews: Data Min. Knowl. Discov., vol. 10, no. 1. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    End-to-end automated XAI applications, such as combining segmentation and classification, will help XAI adoption in clinical practice.

    “End to end automated XAI applications, such as, combining segmentation and classification, will help XAI adoption in clinical practice.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Certification of DNN models by agencies external to the developer or user could help eliminate risk of unauthorized changes to how the system works and can prove beneficial in litigation to prove the correctness of the system.

    “Certification of DNN models by agencies external to the developer or user could help to eliminate risk of unauthorized changes to how the system works and can prove beneficial in case of litigation to prove the correctness of the system [49].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] Gryz, 2021, Black box algorithms and the rights of individuals: no easy solution to the ‘explainability’ problem, Internet Policy Review source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Successful use of AI requires a multi-disciplinary effort encompassing guarantees of ethical, legal and social aspects of AI and increasing awareness.

    “In order to use AI successfully requires a multi-disciplinary effort encompassing guarantees of ethical, legal and social aspects of AI and increasing awareness [5].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [5] Strudel, 2021, Segmenter: transformer for semantic segmentation

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Few-shot learning can work with very few training samples.

    “Few-shot learning can work with very few training samples [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13] McCarthy, 2021, Enterprise imaging and big data: a review from a medical physics perspective, Phys. Med. source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    The CNN fixation method was applied to detect an adversarial attack on sets of images with a clean and perturbed image, and even despite the labels being changed, the proposed method could still localize the regions correctly in both images.

    “The CNN fixation method was applied to detect an adversarial attack on sets of images with a clean and perturbed image, and even despite the labels being changed, the proposed method could still localize the regions correctly in both images [69].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69] Mopuri, 2019, An unraveling approach to visualize the discriminative image regions, IEEE Trans. Image Process. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationcitation

    This can in future help towards models' certifications and standardizations such as by FDA.

    “This can in future help towards models' certifications and standardizations such as by Food and Drug Administration (FDA) [274].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [274] Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) -Discussion Paper and Request for Feedback”.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Decisions affecting patients' lives cannot be entrusted fully to an algorithm.

    “The decisions affecting patients' lives cannot be entrusted fully to an algorithm”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [261] Gillmann, 2021, Ten open challenges in medical visualization, Comput. Graphics Appl. source

      Not checked in the free run.

    • Rests on an assumption: DNN predictions comparable to human experts are not sufficient to fully delegate medical diagnosis to an algorithm.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Adversarial inputs comprising a modified image can drive a model to misclassify and is termed an adversarial attack.

    “The adversarial inputs comprising a modified image can drive a model to misclassify and is termed as an adversarial attack.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Biomedical imaging datasets sometimes suffer from noise, whereas CT images are commonly affected by noise.

    “Biomedical imaging datasets sometime suffer from noise, whereas CT images are commonly affected by noise [265].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [265] Tayarani-N, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    A counterfactual approach, Global Explanations for Bias Identification (GEBI), was proposed for identification of bias in the data for image classification although it can be applied to other problems.

    “A counterfactual approach, Global Explanations for Bias Identification (GEBI) was proposed for identification of bias in the data for image classification although it can be applied to other problems [279].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [279] Mikołajczyk, 2020

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    AutoML is a step towards automatically creating an optimum architecture and making it possible for architectures to be reproducible and comparable.

    “AutoML is a step towards the direction of automatically creating an optimum architecture and making it possible for the architectures to be reproducible and comparable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Federated learning would provide opportunities for measuring the accuracy and explainability of DNN against the same dataset and mitigate the different biases in the model and data.

    “This would provide opportunities for measuring the accuracy and explainability of DNN against the same dataset and mitigate the different biases in the model and data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Undetectable adversarial attacks can be devised to change importance maps dramatically, and interpretations of breast ultrasound images were shown to be susceptible to adversarial attacks similar to other image modalities.

    “Undetectable adversarial attacks can be devised to change the importance maps dramatically and interpretations of breast ultrasound images were shown to be susceptible to adversarial attacks similar to other image modalities [203].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [203] Rasaee, 2021, Explainable AI and susceptibility to adversarial attacks: a case study in classification of breast ultrasound images

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Use of XAI techniques such as visualizations are helpful to identify model bias, and experienced clinicians can identify unsupported or bad DNN decisions.

    “The use of XAI techniques such as visualizations are helpful to identify the model bias and an experienced clinicians can identify any unsupported or bad DNN decisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Google's What-If Tool can be used for optimizing the model's AI fairness.

    “Similarly, Google's What-If Tool can be used for optimizing the model's AI fairness [276].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [276]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Most ML studies on medical images use trained models that are often based on a single unit or hospital, which brings in a selective bias and sometimes inclusion of low-quality scans.

    “Most ML studies on medical images use trained models that are often based on a single unit or hospital which brings in a selective bias and sometimes inclusion of low-quality scans [273].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [273] Hampe, 2019

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Additional realistic data could be generated through GANs using the latent data distributions.

    “Additional realistic data could be generated through GANs using the latent data distributions [13,34,37].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [13] McCarthy, 2021, Enterprise imaging and big data: a review from a medical physics perspective, Phys. Med. source

      Not checked in the free run.

    • cites [34] Karthik, 2021, AI for COVID-19 detection from radiographs: incisive analysis of state of the art techniques, IRBM source

      Not checked in the free run.

    • cites [37] Singh, 2021

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Explainable AI coupled with deep neural network models can help with both the detection and diagnosis of disease by providing additional insights to the clinician.

    “Explainable AI coupled with deep neural network models can help with both the detection and diagnosis of disease, by providing additional insights to the clinician.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Trained DNN models can provide predictions comparable to human experts in a fraction of the time.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The black box and opaque nature of deep neural networks precludes their usefulness despite excellent performance of multimodal methods.

    “The black box and opaque nature of deep neural networks precludes their usefulness despite excellent performance of multimodal methods [288].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [288] Joshi, 2021, A review on explainability in multimodal deep neural nets, IEEE Access source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Clinical acceptance of AI techniques would be aided by multi-modality and information fusion of both imaging and non-imaging data.

    “The clinical acceptance of AI techniques would be aided by multi-modality and information fusion of both imaging and non-imaging data [34,170].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [34] Karthik, 2021, AI for COVID-19 detection from radiographs: incisive analysis of state of the art techniques, IRBM source

      Not checked in the free run.

    • cites [170] Dunnmon, 2019, Assessment of convolutional neural networks for automated classification of chest radiographs, Radiology source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Two significant issues for multimodal imaging are class imbalances and lack of sufficient labelled data.

    “Two significant issues faced for multimodal imaging are the class imbalances and lack of sufficient labelled data [35].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] O'Sullivan, 2020, Developments in AI and machine learning for neuroimaging

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    There is a lack of algorithm standardization for biomedical imaging with diverse and independent efforts for improvements.

    “We see a lack of algorithm standardization for biomedical imaging with diverse and independent efforts for improvements.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Currently, image-based diagnosis and prognosis in biomedical imaging require expert opinion of practitioners who are already overburdened.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Overcoming challenges to clinical adoption of XAI will foster growth of the field and result in better health and safety measures for patients through timely interventions by building trust in the model's predictions.

    “The overcoming of challenges to clinical adoption of XAI will foster growth of the field, and result in better health and safety measures for the patients through timely interventions by building trust in the model's predictions.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Increasingly, there are health and legal regulations that mandate approval and compliance for deep learning techniques.

  • May combine two claims.

    Premisecitation

    Model visualizations can be used to determine model problems and can help improve the models.

    “It was shown with the proposed deconvnet that the model visualizations can be used to determine model problems and can help improve the models [87].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [87] Zeiler, 2014, Visualizing and understanding convolutional networks

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Graph Neural Networks can learn from graph structures and can represent information capturing different types of relationships rather than just an ordered grid structure.

    “Graph Neural Networks can learn from graph structures and can represent information capturing different types of relationships rather than just an ordered grid structure [100].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [100] Holzinger, 2021, Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AI, Inf. Fusion source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Mathematical approaches can improve the interpretability of models but may not be suitable as concrete examples for the end user.

    “The mathematical approaches can improve the interpretability of models, however, may not be suitable as concrete examples for the enduser [63].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [63] Heuillet, 2021, Explainability in deep reinforcement learning, Knowl. Base Syst. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Federated learning is promising for biomedical applications with different hospitals contributing their data to train and improve a deep learning model without violating data privacy.

    “Federated learning is also promising for biomedical applications with different hospitals contributing their data to train and improve a deep learning model without violating data privacy.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Influence functions are useful for generation of visually similar training set attacks, determining dataset errors, understanding, and debugging model behavior.

    “Influence functions have basis in statistics and were demonstrated to be useful for generation of visually similar training set attacks, determining dataset errors, understanding, and debugging the model behavior [104].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [104] Koh, 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    A human-in-the-loop is a pre-requisite for any XAI application based clinical decision.

    “We consider a human-in-the-loop as a pre-requisite for any XAI application based clinical decision.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Trained DNN models can provide predictions comparable to human experts in a fraction of the time.
    • Rests on an assumption: DNN predictions comparable to human experts are not sufficient to fully delegate medical diagnosis to an algorithm.
    • Rests on an assumption: Human expertise cannot be dispensed, and only simple and time-consuming tasks should be directed to AI.
    • Rests on an assumption: The final decision for image-based disease diagnosis would rest with the specialist, whereas XAI model insights should be used only to aid the decision.
160 of 160

  • The paper relies on this without support.

    XAI techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements.

    “Explainable AI (XAI) techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements.”

  • The paper relies on this without support.

    There is often an explainability gap between the model's prediction and its explanations being understandable to a human.

    “However, there is often an explainability gap between the model's prediction and its explanations being understandable to a human.”

  • The paper relies on this without support.

    Incorporating explainability into a model takes time and effort, and may affect its prediction accuracy.

    “Incorporating explainability into a model takes time and effort, and may affect its prediction accuracy.”

  • The paper relies on this without support.

    The use of XAI techniques can reduce the workload of the clinicians, help to make better diagnosis, and reduce the complications associated with a false positive and a false negative outcome.

    “The use of XAI techniques can reduce the workload of the clinicians, help to make better diagnosis, and reduce the complications associated with a false positive and a false negative outcome.”

  • The paper relies on this without support.

    Medical practitioners are more likely to employ and trust a system if they can understand the reasoning of a prediction by a model.

    “They are more likely to employ and trust a system if they can understand the reasoning of a prediction by a model.”

  • The paper relies on this without support.

    Patients require assurance that the XAI methods can be trusted, and are providing a valid and consistent prediction.

    “Patients require assurance that the XAI methods can be trusted, and are providing a valid and consistent prediction.”

  • The paper relies on this without support.

    DNN can provide very accurate predictive performance but are hard to interpret or explain.

    “DNN can provide very accurate predictive performance but owing to a large number of trainable parameters, layers, and the performance determined through the hyperparameters, these are hard to interpret or explain.”

  • The paper relies on this without support.

    CXR is the most commonly used imaging modality for detecting COVID-19 or lung infections, but it is less accurate than CT.

    “CXR is the most commonly used imaging modality for detecting COVID-19, or lung infections, however, in comparison to CT, it is less accurate.”

  • The paper relies on this without support.

    MRI is the most favored imaging modality for neurology and is particularly useful for neurodegenerative diseases.

    “MRI is the most favored imaging modality for neurology and particularly useful for the neurodegenerative diseases.”

  • The paper relies on this without support.

    Ophthalmologists are interested in finding the parameters responsible for a specific finding, rather than just knowing the final diagnosis.

    “The ophthalmologists are interested in finding out the parameters responsible for a specific finding, rather than just knowing the final diagnosis.”

  • The paper relies on this without support.

    In fundus images, the actual variations in landmarks (blood vessels, optic disc, fovea/macula) along with appearances of any new lesion are important.

    “In fundus images, the actual variations in landmarks i.e. blood vessels, optic disc and fovea/macula region along with appearances of any new lesion are important.”

  • The paper relies on this without support.

    For OCT images, it is important to look into layer variants and lesions appearing in those layers.

    “For OCT images, it is important to look into layer variants and lesions appearing in those layers.”

  • The paper relies on this without support.

    The labelled datasets used to train DNNs are often from multiple sources or maintained by different organisations owning the data.

    “The labelled datasets used to train DNNs are often from multiple sources or maintained by different organisations owning the data.”

  • The paper relies on this without support.

    One of the stated advantages of DNN is automated learning of image features and overcoming problems associated with hand-crafted features.

    “One of the stated advantages of DNN is that of an automated learning of the image features and overcoming the problems associated with the hand-crafted features.”

  • The paper relies on this without support.

    Trained DNN models can provide predictions comparable to human experts in a fraction of the time.

    “The trained DNN model can provide predictions comparable to human expert in a fraction of the time”

  • The paper relies on this without support.

    DNN predictions comparable to human experts are not sufficient to fully delegate medical diagnosis to an algorithm.

    “however, this is not sufficient to fully delegate the medical diagnosis to an algorithm”

  • The paper relies on this without support.

    Human expertise cannot be dispensed, and only simple and time-consuming tasks should be directed to AI.

    “However, human expertise cannot be dispensed and only simple, and time-consuming tasks should be directed to AI.”

  • The paper relies on this without support.

    The ultimate beneficiary of model explainability is a human user.

    “The ultimate beneficiary of the model explainability is a human user.”

  • The paper relies on this without support.

    The final decision for image-based disease diagnosis would rest with the specialist, whereas XAI model insights should be used only to aid the decision.

    “Similar to the role of an autonomous car driver, the final decision for image-based disease diagnosis would rest with the specialist whereas the XAI model insights should be used only to aid the decision.”

  • The paper relies on this without support.

    Currently, image-based diagnosis and prognosis in biomedical imaging require expert opinion of practitioners who are already overburdened.

    “Currently, the image-based diagnosis and prognosis in the different disciplines of biomedical imaging such as radiology and ophthalmology require expert opinion of practitioners who are already overburdened.”

  • The paper relies on this without support.

    Increasingly, there are health and legal regulations that mandate approval and compliance for deep learning techniques.

    “Increasingly, there are health and legal regulations that mandate approval and compliance for the deep learning techniques which can be helped using XAI techniques.”
21 of 21

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • SHAP and Guided Backpropagation are both called the best method in the same comparison

    Major

    For the same comparison of six interpretation methods and four DNN models, the text first says SHAP produced the better heatmaps, then says Guided Backpropagation performed best. It does not distinguish the criteria behind these conflicting rankings.

  • Echocardiography is labeled as ECG video data

    Minor

    The prose describes the input for study [246] as transthoracic echocardiography images, while Table 4 labels the same study's modality "ECG video data." ECG and echocardiography are different modalities.

2 of 2

Cites

Flagged first, then by citations

96 of the 315 references couldn’t be matched to a known paper.

Research works printed but not linked: 10

  1. Durán, 2021, vol. 297
  2. A. Preece, D. Harborne, D. Braines, R. Tomsett and S. Chakraborty, “Stakeholders in Explainable AI,” [Online]. Available: https://doi.org/10.48550/arXiv.1810.00184. [Accessed 15 September 2022]. · DOI · Paper page
  3. V. Petsiuk, A. Das and K. Saenko, “RISE: Randomized Input Sampling for Explanation of Black-box Models,” [Online]. Available: https://doi.org/10.48550/arXiv.1806.07421. [Accessed 15 September 2022]. · DOI
  4. Adebayo, 2018, Sanity checks for saliency maps, Adv. Neural Inf. Process. Syst., 31
  5. Alber, 2019, iNNvestigate neural networks, J. Mach. Learn. Res., 20, 1
  6. Spinner, 2020, IEEE Trans. Visual. Comput. Graph., 26, 1064
  7. Panwar, 2020, vol. 140
  8. Liz, 2021, Ensembles of Convolutional Neural Network models for pediatric pneumonia diagnosis, Future Generat. Comput. Syst., 122, 220, 10.1016/j.future.2021.04.007 · DOI
  9. Kim, 2018, ICADx: interpretable computer aided diagnosis of breast masses, Medical Imaging 2018: Computer-Aided Diagnosis, 10575, 450
  10. Hampe, 2019, vol. 6

Cited by

Sorted by citations

Related

Authors
Nazir, S., Dickson, D. M., & Akram, M. U.
Affiliations
  • Glasgow Caledonian University (GB)
  • National University of Sciences and Technology (PK)
Corresponding author
Nazir, Sajid
ORCID
Venue
Computers in Biology and Medicine
Publisher
Elsevier BV
Published
2023-04
Volume
156
Pages
106668
ISSN
0010-4825 (ISSN-L)
DOI
10.1016/j.compbiomed.2023.106668
OpenAlex
W4321350922
Semantic Scholar
77d942cfa2d4b9266080258943048825e7449ad2
Cited by
283
References
315
Citations by year
2026: 70 · 2025: 104 · 2024: 70 · 2023: 37 · 2022: 1 · 2016: 1
FWCI
5.83
Type
journal article
Language
en
Open access
Licence
Locations
  • Computers in Biology and Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ResearchOnline (acceptedVersion, cc-by)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • ResearchOnline (Glasgow Caledonian University) (submittedVersion, cc-by)
Funders
  • Glasgow Caledonian University (10386)
  • Defense Advanced Research Projects Agency
  • Global Challenges Research Fund
  • Office of Defense Nuclear Nonproliferation
  • Alzheimer's Disease Neuroimaging Initiative
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
Keywords
explainable AI, biomedical imaging, deep neural network, medical image classification, diagnosis, model interpretability, regulatory compliance, black-box models
MeSH terms
Machine Learning, Algorithms, Artificial Intelligence, Diagnostic Imaging, Humans, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review