CoChat
Sign Up
Cited by 368Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationArtificial Intelligence in Healthcare

Applications of Explainable Artificial Intelligence in Diagnosis and Surgery

Open access
Zhang, Yiming, Weng, Ying, Lund, Jonathan·2022·Diagnostics
DOI 10.3390/diagnostics12020237PMC PMC8870992
Open this paper in CoChat — free account

Abstract

In recent years, artificial intelligence (AI) has shown great promise in medicine. However, explainability issues make AI applications in clinical usages difficult. Some research has been conducted into explainable artificial intelligence (XAI) to overcome the limitation of the black-box nature of AI methods. Compared with AI techniques such as deep learning, XAI can provide both decision-making and explanations of the model. In this review, we conducted a survey of the recent trends in medical diagnosis and surgical applications using XAI. We have searched articles published between 2019 and 2021 from PubMed, IEEE Xplore, Association for Computing Machinery, and Google Scholar. We included articles which met the selection criteria in the review and then extracted and analyzed relevant information from the studies. Additionally, we provide an experimental showcase on breast cancer diagnosis, and illustrate how XAI can be applied in medical XAI applications. Finally, we summarize the XAI methods utilized in the medical XAI applications, the challenges that the researchers have met, and discuss the future research directions. The survey result indicates that medical XAI is a promising research direction, and this study aims to serve as a reference to medical experts and AI scientists when designing medical XAI applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/diagnostics12020237

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • SHAP or Grad-CAM representations let doctors identify which image regions or data points influence model predictions.

    “SHAP or Grad-CAM representations let doctors identify which image regions or data points influence model predictions [175,176].”

    From Biomarker-Guided Imaging and AI-Augmented Diagnosis of Degenerative Joint Disease (2025) · cites this paper as [176]

  • In joint imaging, attention mechanisms could emphasize radiographic features linked with osteoarthritis.

    “In joint imaging, for instance, attention mechanisms could emphasize radiographic features linked with osteoarthritis”

    From Biomarker-Guided Imaging and AI-Augmented Diagnosis of Degenerative Joint Disease (2025) · cites this paper as [176]

    Section 1.2 lists attention mechanisms as a category of explainability methods, but does not describe their use in joint imaging. Section 3.1 describes LIME emphasizing regions in Parkinson’s SPECT images, not attention mechanisms highlighting osteoarthritis features. Searches for osteoarthritis, joint radiographs, and attention mechanisms found no passage addressing the claimed example; page numbers were not available in our copy. (LLM, conf=0.74)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 54 · Cited by 368

Research works printed but not linked: 46

  1. [1]Alloghani M. Al-Jumeily D. Aljaaf A.J. Khalaf M. Mustafina J. Tan S.Y. The Application of Artificial Intelligence Technology in Healthcare: A Systematic Review Commun. Comput. Inf. Sci. 2020 1174 248 261 10.1007/978-3-030-38752-5_20 · DOI
  2. [2]Loh E. Medicine and the rise of the robots: A qualitative review of recent advances of artificial intelligence in health BMJ Lead. 2018 2 59 63 10.1136/leader-2018-000071 · DOI
  3. [3]Zhou X. Guo Y. Shen M. Yang G.-Z. Application of artificial intelligence in surgery Front. Med. 2020 14 417 430 10.1007/s11684-020-0770-0 32705406 · DOI · Paper page
  4. [5]Christopher B.M. Pattern Recognition and Machine Learning Springer Berlin/Heidelberg, Germany 2006 978-0-387-31073-2
  5. [6]Peterson L. K-nearest neighbor Scholarpedia 2009 4 1883 10.4249/scholarpedia.1883 · DOI
  6. [7]Vapnik V. The Support Vector Method of Function Estimation Nonlinear Modeling Springer Boston, MA, USA 1998 55 85
  7. [8]Safavian S.R. Landgrebe D. A Survey of Decision Tree Classifier Methodology IEEE Trans. Syst. Man Cybern. 1991 21 660 674 10.1109/21.97458 · DOI
  8. [9]Breiman L. Random forests Mach. Learn. 2001 45 5 32 10.1023/A:1010933404324 · DOI · Paper page
  9. [10]LeCun Y. Bengio Y. Hinton G. Deep learning Nature 2015 521 436 444 10.1038/nature14539 26017442 · DOI · Paper page
  10. [11]Mohammad H. Sulaiman M.N. A Review on Evaluation Metrics for Data Classification Evaluations Int. J. Data Min. Knowl. Manag. Process 2015 5 1 11 10.5121/ijdkp.2015.5201 · DOI · Paper page
  11. [12]Doshi-Velez F. Kim B. Towards A Rigorous Science of Interpretable Machine Learning arXiv 2017 1702.08608
  12. [14]Kim M.-Y. Atakishiyev S. Babiker H.K.B. Farruque N. Goebel R. Zaïane O.R. Motallebi M.-H. Rabelo J. Syed T. Yao H. A Multi-Component Framework for the Analysis and Design of Explainable Artificial Intelligence Mach. Learn. Knowl. Extr. 2021 3 45 10.3390/make3040045 · DOI
  13. [16]Kleinbaum D.G. Kleinbaum D.G. Logistic Regression Springer Berlin/Heidelberg, Germany 1994 9781441917416
  14. [18]Zhou B. Khosla A. Lapedriza A. Oliva A. Torralba A. Learning Deep Features for Discriminative Localization Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Las Vegas, NV, USA 27–30 June 2016 2921 2929
  15. [19]Pearson K. LIII. On lines and planes of closest fit to systems of points in space Lond. Edinb. Dublin Philos. Mag. J. Sci. 1901 2 559 572 10.1080/14786440109462720 · DOI
  16. [20]Cogswell M. Das A. Vedantam R. Parikh D. Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization Proceedings of the IEEE International Conference on Computer Vision Venice, Italy 22–29 October 2017 618 662
  17. [22]Yang G. Ye Q. Xia J. Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond Inf. Fusion 2022 77 29 52 10.1016/j.inffus.2021.07.016 34980946 PMC8459787 · DOI · Paper page
  18. [24]Kavya R. Christopher J. Panda S. Lazarus Y.B. Machine Learning and XAI approaches for Allergy Diagnosis Biomed. Signal Process. Control 2021 69 102681 10.1016/j.bspc.2021.102681 · DOI
  19. [25]Amoroso N. Pomarico D. Fanizzi A. Didonna V. Giotta F. La Forgia D. Latorre A. Monaco A. Pantaleo E. Petruzzellis N. A roadmap towards breast cancer therapies supported by explainable artificial intelligence Appl. Sci. 2021 11 4881 10.3390/app11114881 · DOI
  20. [26]Dindorf C. Konradi J. Wolf C. Taetz B. Bleser G. Huthwelker J. Werthmann F. Bartaguiz E. Kniepert J. Drees P. Classification and automated interpretation of spinal posture data using a pathology-independent classifier and explainable artificial intelligence (Xai) Sensors 2021 21 6323 10.3390/s21186323 34577530 PMC8470313 · DOI
  21. [28]Peng J. Zou K. Zhou M. Teng Y. Zhu X. Zhang F. Xu J. An Explainable Artificial Intelligence Framework for the Deterioration Risk Prediction of Hepatitis Patients J. Med. Syst. 2021 45 1 9 10.1007/s10916-021-01736-5 33847850 · DOI
  22. [30]Sarp S. Kuzlu M. Wilson E. Cali U. Guler O. The enlightening role of explainable artificial intelligence in chronic wound classification Electronics 2021 10 1406 10.3390/electronics10121406 · DOI
  23. [31]Tan W. Guan P. Wu L. Chen H. Li J. Ling Y. Fan T. Wang Y. Li J. Yan B. The use of explainable artificial intelligence to explore types of fenestral otosclerosis misdiagnosed when using temporal bone high-resolution computed tomography Ann. Transl. Med. 2021 9 969 10.21037/atm-21-1171 34277769 PMC8267294 · DOI
  24. [32]Wu H. Chen W. Xu S. Xu B. Counterfactual Supporting Facts Extraction for Explainable Medical Record Based Diagnosis with Graph Network Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online 6–11 June 2021 Association for Computational Linguistics Stroudsburg, PA, USA 2021 1942 1955
  25. [33]Chen J. Dai X. Yuan Q. Lu C. Huang H. Towards Interpretable Clinical Diagnosis with Bayesian Network Ensembles Stacked on Entity-Aware CNNs Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Association for Computational Linguistics Stroudsburg, PA, USA 2020 3143 3153
  26. [34]Rucco M. Viticchi G. Falsetti L. Towards personalized diagnosis of glioblastoma in fluid-attenuated inversion recovery (FLAIR) by topological interpretable machine learning Mathematics 2020 8 770 10.3390/math8050770 · DOI
  27. [35]Gu D. Li Y. Jiang F. Wen Z. Liu S. Shi W. Lu G. Zhou C. VINet: A Visually Interpretable Image Diagnosis Network IEEE Trans. Multimed. 2020 22 1720 1729 10.1109/TMM.2020.2971170 · DOI
  28. [36]Kroll J.P. Eickhoff S.B. Hoffstaedter F. Patil K.R. Evolving complex yet interpretable representations: Application to Alzheimer’s diagnosis and prognosis Proceedings of the 2020 IEEE Congress on Evolutionary Computation (CEC) Glasgow, UK 19–24 July 2020 10.1109/CEC48606.2020.9185843 · DOI
  29. [37]Meldo A. Utkin L. Kovalev M. Kasimov E. The natural language explanation algorithms for the lung cancer computer-aided diagnosis system Artif. Intell. Med. 2020 108 101952 10.1016/j.artmed.2020.101952 32972653 · DOI
  30. [38]Yeboah D. Steinmeister L. Hier D.B. Hadi B. Wunsch D.C. Olbricht G.R. Obafemi-Ajayi T. An Explainable and Statistically Validated Ensemble Clustering Model Applied to the Identification of Traumatic Brain Injury Subgroups IEEE Access 2020 8 180690 180705 10.1109/ACCESS.2020.3027453 · DOI
  31. [39]Wang L. Lin Z.Q. Wong A. COVID-Net: A tailored deep convolutional neural network design for detection of COVID-19 cases from chest X-ray images Sci. Rep. 2020 10 19549 10.1038/s41598-020-76550-z 33177550 PMC7658227 · DOI
  32. [40]Wong A. Shafiee M.J. Chwyl B. Li F. FermiNets: Learning generative machines to generate efficient neural networks via generative synthesis arXiv 2018 1809.05989
  33. [41]Sabol P. Sinčák P. Hartono P. Kočan P. Benetinová Z. Blichárová A. Verbóová Ľ. Štammová E. Sabolová-Fabianová A. Jašková A. Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images J. Biomed. Inform. 2020 109 103523 10.1016/j.jbi.2020.103523 32758538 · DOI
  34. [42]Wei X. Zhu J. Zhang H. Gao H. Yu R. Liu Z. Zheng X. Gao M. Zhang S. Visual Interpretability in Computer-Assisted Diagnosis of Thyroid Nodules Using Ultrasound Images Med. Sci. Monit. 2020 26 e927007 10.12659/MSM.927007 32798214 PMC7446277 · DOI
  35. [43]Chang Y.-W. Tsai S.-J. Wu Y.-F. Yang A.C. Development of an Al-Based Web Diagnostic System for Phenotyping Psychiatric Disorders Front. Psychiatry 2020 11 1 10 10.3389/fpsyt.2020.542394 33250789 PMC7674487 · DOI
  36. [45]Cho J. Alharin A. Hu Z. Fell N. Sartipi M. Predicting Post-stroke Hospital Discharge Disposition Using Interpretable Machine Learning Approaches Proceedings of the 2019 IEEE International Conference on Big Data (Big Data) Los Angeles, CA, USA 9–12 December 2019 IEEE New York, NY, USA 2019 4817 4822
  37. [46]Lamy J.B. Sekar B. Guezennec G. Bouaud J. Séroussi B. Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach Artif. Intell. Med. 2019 94 42 53 10.1016/j.artmed.2019.01.001 30871682 · DOI · Paper page
  38. [47]Das D. Ito J. Kadowaki T. Tsuda K. An interpretable machine learning model for diagnosis of Alzheimer’s disease PeerJ 2019 7 e6543 10.7717/peerj.6543 30842909 PMC6398390 · DOI
  39. [48]Yoo T.K. Ryu I.H. Choi H. Kim J.K. Lee I.S. Kim J.S. Lee G. Rim T.H. Explainable machine learning approach as a tool to understand factors used to select the refractive surgery technique on the expert level Transl. Vis. Sci. Technol. 2020 9 1 14 10.1167/tvst.9.2.8 PMC7346876 32704414 · DOI
  40. [49]Mirchi N. Bissonnette V. Yilmaz R. Ledwos N. Winkler-Schwartz A. Del Maestro R.F. The virtual operative assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine PLoS ONE 2020 15 1 15 10.1371/journal.pone.0229596 PMC7046231 32106247 · DOI · Paper page
  41. [50]Fawaz H.I. Forestier G. Weber J. Idoumghar L. Accurate and interpretable evaluation of surgical skills from kinematic data using fully convolutional neural networks Int. J. Comput. Assist. Radiol. Surg. 2019 14 1611 1617 10.1007/s11548-019-02039-4 31363983 · DOI
  42. [51]Kletz S. Schoeffmann K. Husslein H. Learning the representation of instrument images in laparoscopy videos Healthc. Technol. Lett. 2019 6 197 203 10.1049/htl.2019.0077 32038857 PMC6952264 · DOI
  43. [52]Chittajallu D.R. Dong B. Tunison P. Collins R. Wells K. Fleshman J. Sankaranarayanan G. Schwaitzberg S. Cavuoto L. Enquobahrie A. XAI-CBIR: Explainable ai system for content based retrieval of video frames from minimally invasive surgery videos Proc. Int. Symp. Biomed. Imaging 2019 2019 66 69 10.1109/ISBI.2019.8759428 · DOI
  44. [53]Shorten C. Khoshgoftaar T.M. A survey on Image Data Augmentation for Deep Learning J. Big Data 2019 6 60 10.1186/s40537-019-0197-0 PMC8287113 34306963 · DOI · Paper page
  45. [55]Chen W.-Y. Liu Y.-C. Kira Z. Wang Y.-C.F. Huang J.-B. A Closer Look at Few-shot Classification arXiv 2019 1904.04232
  46. [56]Holzinger A. Carrington A. Müller H. Measuring the Quality of Explanations: The System Causability Scale (SCS): Comparing Human and Machine Explanations KI Kunstl. Intell. 2020 34 193 198 10.1007/s13218-020-00636-z PMC7271052 32549653 · DOI · Paper page

Authors
Zhang, Y., Weng, Y., & Lund, J.
Affiliations
  • University of Nottingham Ningbo China (CN)
  • University of Nottingham (GB)
Corresponding author
Weng, Ying
ORCID
Venue
Diagnostics
Publisher
MDPI AG
Published
2022-01-19
Volume
12
Issue
2
Pages
237
ISSN
2075-4418 (ISSN-L)
DOI
10.3390/diagnostics12020237
PMC
PMC8870992
OpenAlex
W4206348828
Cited by
368
References
54
Citations by year
2026: 52 · 2025: 94 · 2024: 111 · 2023: 84 · 2022: 27
FWCI
5.50
Type
journal article
Language
en
Open access
Licence
Locations
  • Diagnostics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • NBCP (2019C50052)
  • NCHI (I01200100023)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
Keywords
explainable AI, diagnosis, surgical applications, black-box models, model interpretability, breast cancer diagnosis, clinical decision support, deep learning in healthcare

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review