CoChat
Sign Up
PreprintCited by 19Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationMachine Learning in Healthcare

Explainable AI applications in the Medical Domain: a systematic review

Open access
Prentzas, Nicoletta, Kakas, Antonis, Pattichis, Constantinos S.·2023·arXiv
DOI 10.48550/arxiv.2308.05411arXiv 2308.05411
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial Intelligence in Medicine has made significant progress with emerging applications in medical imaging, patient care, and other areas. While these applications have proven successful in retrospective studies, very few of them were applied in practice.The field of Medical AI faces various challenges, in terms of building user trust, complying with regulations, using data ethically.Explainable AI (XAI) aims to enable humans understand AI and trust its results. This paper presents a literature review on the recent developments of XAI solutions for medical decision support, based on a representative sample of 198 articles published in recent years. The systematic synthesis of the relevant articles resulted in several findings. (1) model-agnostic XAI techniques were mostly employed in these solutions, (2) deep learning models are utilized more than other types of machine learning models, (3) explainability was applied to promote trust, but very few works reported the physicians participation in the loop, (4) visual and interactive user interface is more useful in understanding the explanation and the recommendation of the system. More research is needed in collaboration between medical and AI experts, that could guide the development of suitable frameworks for the design, implementation, and evaluation of XAI solutions in medicine.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.05411

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Many machine learning and deep learning systems for predicting sepsis onset and patient outcomes function as black boxes, generating predictions without explanations.

    “many of these systems function as “black boxes,” generating predictions without providing explanations for their reasoning.”

    From Explainable AI for critical care: a systematic review of interpretable models for sepsis and ICU mortality prediction (2026) · cites this paper as [2]

    The discussion of explainability says that machine-learning models, including deep neural networks, are opaque and less explainable (Explainable AI applications in the Medical; page not provided). The review supports the general black-box characterization, but the passages found do not establish it specifically for systems predicting sepsis onset or patient outcomes. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • In critical care, lack of transparency in AI tools can reduce clinician confidence, delay necessary interventions, and limit practical adoption.

    “In critical care, where timely decisions can directly affect patient survival, this lack of transparency can reduce clinician confidence, delay necessary interventions, and limit the practical adoption of AI tools [1, 2].”

    From Explainable AI for critical care: a systematic review of interpretable models for sepsis and ICU mortality prediction (2026) · cites this paper as [2]

    The Introduction and “Unique challenges in the medical domain” link opaque medical AI to difficulties gaining physicians’ trust and deploying it in clinical practice. The paper does not establish the claim specifically for critical care or say that lack of transparency delays necessary interventions. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • The lack of standardized metrics for assessing explainability across studies complicates the comparison of different approaches and limits the evaluation of their reliability in real-world ICU settings.

    “Furthermore, while XAI methods offer valuable interpretability, the lack of standardized metrics for assessing explainability across studies complicates the comparison of different approaches and limits the evaluation of their reliability in real-world ICU settings ([2, 20]).”

    From Explainable AI for critical care: a systematic review of interpretable models for sepsis and ICU mortality prediction (2026) · cites this paper as [2]

    The 'Medical XAI studies, Explainability Evaluation' section says there is a lack of established methods for assessing XAI solutions and that the authors had to identify common metrics to analyze selected studies. This supports the concern about inconsistent assessment across studies, but the paper does not establish the claimed consequence for reliability evaluation in real-world ICU settings. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Much current research prioritizes improving prediction accuracy while giving less attention to whether explainable AI explanations are truly understandable and useful for clinicians.

    “In addition, much of the current research prioritizes improving prediction accuracy while giving less attention to whether the explanations produced by XAI models are truly understandable and useful for clinicians [1, 2, 12].”

    From Explainable AI for critical care: a systematic review of interpretable models for sepsis and ICU mortality prediction (2026) · cites this paper as [2]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 152

  1. Fogel AL, Kvedar JC. Artificial intelligence powers digital medicine. npj Digit Med; 1. Epub ahead of print 2018. DOI: 10.1038/s41746-017-0012-2. · DOI
  2. Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: Past, present and future. Stroke and Vascular Neurology 2017; 2: 230-243.
  3. Winter G. Machine learning in healthcare. 2019. Epub ahead of print 2019. DOI: 10.12968/bjhc.2019.25.2.100. · DOI
  4. Weng SF, Reps J, Kai J, et al. Can Machine-learning improve cardiovascular risk prediction using routine clinical data? PLoS One; 12. Epub ahead of print 2017. DOI: 10.1371/journal.pone.0174944. · DOI · Paper page
  5. Rajpurkar P, Irvin J, Zhu K, et al. CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. arXiv, http://arxiv.org/abs/1711.05225 (2017). · arXiv
  6. Ardila D, Kiraly AP, Bharadwaj S, et al. End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat Med; 25. Epub ahead of print 2019. DOI: 10.1038/s41591-019-0447-x. · DOI · Paper page
  7. Zhou D, Tian F, Tian X, et al. Diagnostic evaluation of a deep learning model for optical diagnosis of colorectal cancer. Nat Commun; 11. Epub ahead of print 2020. DOI: 10.1038/s41467-020- 16777-6. · DOI · Paper page
  8. Holzinger A, Biemann C, Pattichis CS, et al. What do we need to build explainable AI systems for the medical domain? arXiv, http://arxiv.org/abs/1712.09923 (2017). · arXiv
  9. Prentzas N, Nicolaides A, Kyriacou E, et al. Integrating machine learning with symbolic reasoning to build an explainable ai model for stroke prediction. In: Proceedings -2019 IEEE 19th International Conference on Bioinformatics and Bioengineering, BIBE 2019. Institute of Electrical and Electronics Engineers Inc., 2019, pp. 817-821.
  10. Shortliffe EH, Davis R, Axline SG, et al. Computer-based consultations in clinical therapeutics: Explanation and rule acquisition capabilities of the MYCIN system. Comput Biomed Res 1975; 8: 303-320.
  11. Clancey WJ, Letsinger R. Neomycin: Reconfiguring a Rule-Based Expert System for Application To Teaching. In: Department of Computer Science, Stanford University Stanford. 1981, pp. 829- 836.
  12. Weiss SM, Kulikowski CA, Amarel S, et al. A model-based method for computer-aided medical decision-making. Artif Intell 1978; 11: 145-172.
  13. Washbrook J, Keravnou E. Making deepness explicit. Artif Intell Med 1990; 2: 129-134.
  14. Holzinger A, Langs G, Denk H, et al. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery; 9. Epub ahead of print 2019. DOI: 10.1002/widm.1312. · DOI · Paper page
  15. Adadi A, Berrada M. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access 2018; 6: 52138-52160.
  16. Dosilovic FK, Brcic M, Hlupic N. Explainable artificial intelligence: A survey. In: 2018 41st International Convention on Information and Communication Technology, Electronics and Microelectronics, MIPRO 2018 -Proceedings. 2018, pp. 210-215.
  17. Guidotti R, Monreale A, Ruggieri S, et al. A survey of methods for explaining black box models. ACM Comput Surv; 51. Epub ahead of print 1 August 2018. DOI: 10.1145/3236009. · DOI · Paper page
  18. Gilpin LH, Bau D, Yuan BZ, et al. Explaining explanations: An overview of interpretability of machine learning. In: Proceedings - 2018 IEEE 5th International Conference on Data Science and Advanced Analytics, DSAA 2018. Institute of Electrical and Electronics Engineers Inc., 2019, pp. 80-89.
  19. Carvalho D V., Pereira EM, Cardoso JS. Machine learning interpretability: A survey on methods and metrics. Electronics (Switzerland); 8. Epub ahead of print 2019. DOI: 10.3390/electronics8080832. · DOI · Paper page
  20. Xu F, Uszkoreit H, Du Y, et al. Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019, pp. 563-574.
  21. Xie N, Ras G, van Gerven M, et al. Explainable deep learning: A field guide for the uninitiated. arXiv, https://arxiv.org/abs/2004.14545 (2020, accessed 17 November 2020). · arXiv
  22. Chakraborti T, Sreedharan S, Kambhampati S. The emerging landscape of explainable automated planning & decision making. In: IJCAI International Joint Conference on Artificial Intelligence. 2020, pp. 4803-4811.
  23. Barredo Arrieta A, Díaz-Rodríguez N, Del Ser J, et al. Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion 2020; 58: 82-115.
  24. Angelov P, Soares E. Towards explainable deep neural networks (xDNN). Neural Networks 2020; 130: 185-194.
  25. Montavon G, Samek W, Müller KR. Methods for interpreting and understanding deep neural networks. Digital Signal Processing: A Review Journal 2018; 73: 1-15.
  26. Molnar C, Casalicchio G, Bischl B. Interpretable Machine Learning -A Brief History, State-of-the-Art and Challenges. In: Communications in Computer and Information Science. 2020. Epub ahead of print 2020. DOI: 10.1007/978-3-030-65965-3_28. · DOI · Paper page
  27. Linardatos P, Papastefanopoulos V, Kotsiantis S. Explainable ai: A review of machine learning interpretability methods. Entropy; 23. Epub ahead of print 2021. DOI: 10.3390/e23010018. · DOI · Paper page
  28. Anjomshoae S, Calvaresi D, Najjar A, et al. Explainable agents and robots: Results from a systematic literature review. In: Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS. 2019, pp. 1078-1088.
  29. Payrovnaziri SN, Chen Z, Rengifo-Moreno P, et al. Explainable artificial intelligence models using real-world electronic health record data: A systematic scoping review. Journal of the American Medical Informatics Association; 27. Epub ahead of print 2020. DOI: 10.1093/jamia/ocaa053. · DOI
  30. Gade K, Geyik S, Kenthapadi K, et al. Explainable AI in Industry: Practical Challenges and Lessons Learned. In: The Web Conference 2020 -Companion of the World Wide Web Conference, WWW 2020. 2020. Epub ahead of print 2020. DOI: 10.1145/3366424.3383110. · DOI · Paper page
  31. Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine; 6. Epub ahead of print 2009. DOI: 10.1371/journal.pmed.1000097. · DOI · Paper page
  32. Van Lent M, Fisher W, Mancuso M. An explainable artificial intelligence system for small-unit tactical behavior. In: Proceedings of the National Conference on Artificial Intelligence. 2004, pp. 900- 907.
  33. Doran D, Schulz S, Besold TR. What does explainable ai really mean? A new conceptualization of perspectives. In: arXiv. CEUR- WS, 2017.
  34. Alonso JM, Castiello C, Mencar C. A bibliometric analysis of the explainable artificial intelligence research field. In: Communications in Computer and Information Science. 2018, pp. 3-15.
  35. Biran O, Cotton C. Explanation and Justification in Machine Learning: A Survey. IJCAI-17 Work Explain AI 2017; 8-13.
  36. Miller T, Howe P, Sonenberg L. Explainable AI: Beware of inmates running the asylum or: How i learnt to stop worrying and love the social and behavioural sciences. arXiv, http://arxiv.org/abs/1712.00547 (2017). · arXiv
  37. Nohara Y, Iihara K, Nakashima N. Interpretable Machine Learning Techniques for Causal Inference Using Balancing Scores as Meta- features. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS. 2018, pp. 4042-4045.
  38. Miller T. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence 2019; 267: 1-38.
  39. Ploug T, Holm S. The four dimensions of contestable AI diagnostics-A patient-centric approach to explainable AI. Artif Intell Med; 107. Epub ahead of print 2020. DOI: 10.1016/j.artmed.2020.101901. · DOI
  40. Amann J, Blasimme A, Vayena E, et al. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Med Inform Decis Mak; 20. Epub ahead of print 1 December 2020. DOI: 10.1186/s12911-020-01332-6. · DOI · Paper page
  41. Samek W, Wiegand T, Müller KR. Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv, http://arxiv.org/abs/1708.08296 (2017). · arXiv
  42. Loizos M. Machine Coaching. In: Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence, https://cognition.ouc.ac.cy/loizos/pubs.html (2019).
  43. Kim TW, Routledge BR. Informational Privacy, A Right to Explanation, and Interpretable AI. In: Proceedings -2018 2nd IEEE Symposium on Privacy-Aware Computing, PAC 2018. Institute of Electrical and Electronics Engineers Inc., 2018, pp. 64-74.
  44. Lipton ZC. The mythos of model interpretability. Commun ACM 2018; 61: 35-43.
  45. Ahmad MA, Teredesai A, Eckert C. Interpretable machine learning in healthcare. In: Proceedings -2018 IEEE International Conference on Healthcare Informatics, ICHI 2018. 2018, p. 447.
  46. Doshi-Velez F, Kortz M, Budish R, et al. Accountability of AI under the law: The role of explanation. arXiv. Epub ahead of print 2017. DOI: 10.2139/ssrn.3064761. · DOI · Paper page
  47. Lou Y, Caruana R, Gehrke J. Intelligible models for classification and regression. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2012, pp. 150-158.
  48. Letham B, Rudin C, McCormick TH, et al. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. Ann Appl Stat 2015; 9: 1350-1371.
  49. Che Z, Purushotham S, Khemani R, et al. Interpretable Deep Models for ICU Outcome Prediction. AMIA . Annu Symp proceedings AMIA Symp 2016; 2016: 371-380.
  50. Arnold M, Piorkowski D, Reimer D, et al. FactSheets: Increasing trust in AI services through supplier's declarations of conformity. IBM J Res Dev; 63. Epub ahead of print 2019. DOI: 10.1147/JRD.2019.2942288. · DOI · Paper page
  51. Robnik-Šikonja M, Bohanec M. Perturbation-Based Explanations of Prediction Models. In: Human and machine learning. 2018, pp. 159-175.
  52. Greenyer J, Lochau M, Vogel T. Explainable Software for Cyber- Physical Systems (ES4CPS) Report from the GI Dagstuhl Seminar 19023 January 06-11 2019, Schloss Dagstuhl. arXiv, http://arxiv.org/abs/1904.11851 (2019). · arXiv
  53. Sokol K, Flach P. Explainability fact sheets: A framework for systematic assessment of explainable approaches. In: FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 2020, pp. 56-67.
  54. Wang D, Yang Q, Abdul A, et al. Designing theory-driven user- centric explainable AI. In: Conference on Human Factors in Computing Systems -Proceedings. 2019. Epub ahead of print 2019. DOI: 10.1145/3290605.3300831. · DOI · Paper page
  55. Hoffman RR, Mueller ST, Klein G, et al. Metrics for Explainable AI: Challenges and Prospects. arXiv.
  56. Vilone G, Longo L. Notions of explainability and evaluation approaches for explainable artificial intelligence. Inf Fusion; 76. Epub ahead of print 2021. DOI: 10.1016/j.inffus.2021.05.009. · DOI · Paper page
  57. Zhou J, Gandomi AH, Chen F, et al. Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics (Switzerland) 2021; 10: 1-19.
  58. Singh A, Sengupta S, Lakshminarayanan V. Explainable deep learning models in medical image analysis. J Imaging; 6. Epub ahead of print 2020. DOI: 10.3390/JIMAGING6060052. · DOI · Paper page
  59. Durán JM, Jongsma KR. Who is afraid of black box algorithms? On the epistemological and ethical basis of trust in medical AI. J Med Ethics; 47. Epub ahead of print 2021. DOI: 10.1136/medethics- 2020-106820. · DOI · Paper page
  60. London AJ. Artificial Intelligence and Black-Box Medical Decisions: Accuracy versus Explainability. Hastings Cent Rep 2019; 49: 15-21.
  61. Sampson DK, Dwyer LM, Tseng EH, et al. FDA Proposes Regulatory Framework for Artificial Intelligence/Machine Learning Software as a Medical Device. Intellect Prop Technol Law J 2019; 31: 12-16.
  62. Kwon BC, Choi MJ, Kim JT, et al. RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records. IEEE Trans Vis Comput Graph; 25. Epub ahead of print 2019. DOI: 10.1109/TVCG.2018.2865027. · DOI · Paper page
  63. Bruckert S, Finzel B, Schmid U. The Next Generation of Medical Decision Support: A Roadmap Toward Transparent Expert Companions. Front Artif Intell; 3. Epub ahead of print 2020. DOI: 10.3389/frai.2020.507973. · DOI · Paper page
  64. Müller J, Stoehr M, Oeser A, et al. A visual approach to explainable computerized clinical decision support. Comput Graph; 91. Epub ahead of print 2020. DOI: 10.1016/j.cag.2020.06.004. · DOI
  65. Holzinger A. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Informatics 2016; 3: 119-131.
  66. Čyras K, Badrinath R, Mohalik SK, et al. Machine reasoning explainability. arXiv.
  67. Liao QV, Gruen D, Miller S. Questioning the AI: Informing Design Practices for Explainable AI User Experiences. In: Conference on Human Factors in Computing Systems -Proceedings. 2020. Epub ahead of print 2020. DOI: 10.1145/3313831.3376590. · DOI · Paper page
  68. Clark P, Niblett T. The CN2 Induction Algorithm. Mach Learn 1989; 3: 261-283.
  69. Zhang Q, Cao R, Shi F, et al. Interpreting CNN knowledge via an explanatory graph. In: arXiv. 2017.
  70. Futia G, Vetrò A. On the integration of knowledge graphs into deep learning models for a more comprehensible AI-Three challenges for future research. Information (Switzerland); 11. Epub ahead of print 2020. DOI: 10.3390/info11020122. · DOI · Paper page
  71. Chimatapu R, Hagras H, Kern M, et al. Hybrid deep learning type-2 fuzzy logic systems for explainable AI. In: IEEE International Conference on Fuzzy Systems. 2020. Epub ahead of print 2020. DOI: 10.1109/FUZZ48607.2020.9177817. · DOI
  72. Mao J, Gan C, Kohli P, et al. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In: arXiv. International Conference on Learning Representations, ICLR, 2019.
  73. Ribeiro MT, Singh S, Guestrin C. 'Why should i trust you?' Explaining the predictions of any classifier. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2016, pp. 1135-1144.
  74. Deng H. Interpreting tree ensembles with inTrees. Int J Data Sci Anal 2019; 7: 277-287.
  75. Petsiuk V, Das A, Saenko K. RISE: Randomized input sampling for explanation of black-box models. In: arXiv. 2018.
  76. Lundberg SM, Lee SI. A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems. 2017, pp. 4766-4775.
  77. Frosst N, Hinton G. Distilling a neural network into a soft decision tree. In: arXiv. 2017.
  78. Lakkaraju H, Bach SH, Leskovec J. Interpretable decision sets: A joint framework for description and prediction. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016, pp. 1675-1684.
  79. Zilke JR, Mencía EL, Janssen F. DeepRED -Rule extraction from deep neural networks. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2016, pp. 457-473.
  80. Zhou ZH, Jiang Y, Chen SF. Extracting symbolic rules from trained neural network ensembles. AI Communications 2003; 16: 3-15.
  81. Hayashi Y. Neural network rule extraction by a new ensemble concept and its theoretical and historical background: A review. In: International Journal of Computational Intelligence and Applications. 2013. Epub ahead of print 2013. DOI: 10.1142/S1469026813400063. · DOI
  82. Barakat NH, Bradley AP. Rule extraction from support vector machines: A sequential covering approach. IEEE Trans Knowl Data Eng 2007; 19: 729-741.
  83. Barakat N, Diederich J. Eclectic rule-extraction from support vector machines. Int J Comput Intell 2005; 2: 59-62.
  84. Valdes G, Luna JM, Eaton E, et al. MediBoost: A Patient Stratification Tool for Interpretable Decision Making in the Era of Precision Medicine. Sci Rep; 6. Epub ahead of print 2016. DOI: 10.1038/srep37854. · DOI
  85. Caruana R, Lou Y, Gehrke J, et al. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery, 2015, pp. 1721-1730.
  86. Ustun B, Rudin C. Supersparse linear integer models for optimized medical scoring systems. Mach Learn 2016; 102: 349-391.
  87. Tibshirani R. Regression shrinkage and selection via the lasso: A retrospective. J R Stat Soc Ser B Stat Methodol; 73. Epub ahead of print 2011. DOI: 10.1111/j.1467-9868.2011.00771.x. · DOI
  88. El-Sappagh S, Alonso JM, Ali F, et al. An ontology-based interpretable fuzzy decision support system for diabetes diagnosis. IEEE Access 2018; 6: 37371-37394.
  89. Gorzałczany MB, Rudziński F. Heart-disease diagnosis decision support employing fuzzy systems with genetically optimized accuracy-interpretability trade-off. In: 2017 IEEE Symposium Series on Computational Intelligence, SSCI 2017 -Proceedings. 2018, pp. 1-8.
  90. Xu Z, Zhang J, Zhang Q, et al. A Comorbidity Knowledge-Aware Model for Disease Prognostic Prediction. IEEE Trans Cybern. Epub ahead of print 2021. DOI: 10.1109/TCYB.2021.3070227. · DOI
  91. Messalas A, Kanellopoulos Y, Makris C. Model-Agnostic Interpretability with Shapley Values. In: 10th International Conference on Information, Intelligence, Systems and Applications, IISA 2019. 2019, pp. 1-7.
  92. Chen Z, Zhang L, Cao Z, et al. Distilling the Knowledge from Handcrafted Features for Human Activity Recognition. IEEE Trans Ind Informatics 2018; 14: 4334-4342.
  93. Fu LM. Rule Generation from Neural Networks. IEEE Trans Syst Man Cybern 1994; 24: 1114-1124.
  94. Taha IA, Ghosh J. Symbolic interpretation of artificial neural networks. IEEE Trans Knowl Data Eng 1999; 11: 448-463.
  95. Ziegel ER. The Elements of Statistical Learning. Technometrics 2003; 45: 267-268.
  96. Lou Y, Caruana R, Gehrke J, et al. Accurate intelligible models with pairwise interactions. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2013, pp. 623-631.
  97. Hara S, Hayashi K. Making tree ensembles interpretable: A Bayesian model selection approach. In: International Conference on Artificial Intelligence and Statistics, AISTATS 2018. 2018, pp. 77- 85.
  98. Friedman JH, Popescu BE. Predictive learning via rule ensembles. Ann Appl Stat 2008; 2: 916-954.
  99. Ferreira P, Dutra I, Salvini R, et al. Interpretable models to predict Breast Cancer. In: Proceedings -2016 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2016. Institute of Electrical and Electronics Engineers Inc., 2017, pp. 1507-1511.
  100. Nguyen A, Yosinski J, Clune J. Understanding Neural Networks via Feature Visualization: A Survey. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019, pp. 55-76.
  101. Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In: Advances in Neural Information Processing Systems. 2016, pp. 3395-3403.
  102. Selvaraju RR, Cogswell M, Das A, et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int J Comput Vis 2020; 128: 336-359.
  103. Odusami M, Maskeliūnas R, Damaševičius R. An Intelligent System for Early Recognition of Alzheimer's Disease Using Neuroimaging. Sensors; 22. Epub ahead of print 2022. DOI: 10.3390/s22030740. · DOI
  104. Jansen C, Penzel T, Hodel S, et al. Network physiology in insomnia patients: Assessment of relevant changes in network topology with interpretable machine learning models. Chaos; 29. Epub ahead of print 2019. DOI: 10.1063/1.5128003. · DOI
  105. Bach S, Binder A, Montavon G, et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One; 10. Epub ahead of print 10 July 2015. DOI: 10.1371/journal.pone.0130140. · DOI · Paper page
  106. Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2014, pp. 818-833.
  107. Baehrens D, Schroeter T, Harmeling S, et al. How to explain individual classification decisions. J Mach Learn Res 2010; 11: 1803-1831.
  108. Ieracitano C, Mammone N, Versaci M, et al. A fuzzy-enhanced deep learning approach for early detection of Covid-19 pneumonia from portable chest X-ray images. Neurocomputing; 481. Epub ahead of print 2022. DOI: 10.1016/j.neucom.2022.01.055. · DOI
  109. Cortez P, Embrechts MJ. Opening black box Data Mining models using Sensitivity Analysis. In: IEEE SSCI 2011: Symposium Series on Computational Intelligence -CIDM 2011: 2011 IEEE Symposium on Computational Intelligence and Data Mining. 2011, pp. 341-348.
  110. Castro J, Gómez D, Tejada J. Polynomial calculation of the Shapley value based on sampling. Comput Oper Res 2009; 36: 1726-1730.
  111. Kim B, Wattenberg M, Gilmer J, et al. Interpretability beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In: arXiv. 2017.
  112. Pisov M, Goncharov M, Kurochkina N, et al. Incorporating task- specific structural knowledge into CNNs for brain midline shift detection. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019, pp. 30-38.
  113. Choi E, Bahadori MT, Kulas JA, et al. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In: Advances in Neural Information Processing Systems. Neural information processing systems foundation, 2016, pp. 3512-3520.
  114. Karimi M, Wu D, Wang Z, et al. DeepAffinity: Interpretable deep learning of compound-protein affinity through unified recurrent and convolutional neural networks. Bioinformatics 2019; 35: 3329- 3338.
  115. Zellers R, Bisk Y, Farhadi A, et al. From recognition to cognition: Visual commonsense reasoning. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2019, pp. 6713-6724.
  116. Li O, Liu H, Chen C, et al. Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions. In: arXiv. 2017.
  117. Du M, Liu N, Hu X. Techniques for interpretable machine learning. Commun ACM 2020; 63: 68-77.
  118. Ras G, Van Gerven M, Haselager P. Explanation methods in deep learning: Users, values, concerns and challenges. In: arXiv. 2018. Epub ahead of print 2018. DOI: 10.1007/978-3-319-98131-4_2. · DOI · Paper page
  119. Alves MA, Castro GZ, Oliveira BAS, et al. Explaining machine learning based diagnosis of COVID-19 from routine blood tests with decision trees and criteria graphs. Comput Biol Med; 132. Epub ahead of print 2021. DOI: 10.1016/j.compbiomed.2021.104335. · DOI
  120. Barata C, Celebi ME, Marques JS. Explainable skin lesion diagnosis using taxonomies. Pattern Recognit; 110. Epub ahead of print 2021. DOI: 10.1016/j.patcog.2020.107413. · DOI
  121. Ren H, Wong AB, Lian W, et al. Interpretable Pneumonia Detection by Combining Deep Learning and Explainable Models with Multisource Data. IEEE Access; 9. Epub ahead of print 2021. DOI: 10.1109/ACCESS.2021.3090215. · DOI
  122. Saleem H, Shahid AR, Raza B. Visual interpretability in 3D brain tumor segmentation network. Comput Biol Med; 133. Epub ahead of print 2021. DOI: 10.1016/j.compbiomed.2021.104410. · DOI
  123. Habib S, Us Salam W, Butt MA, et al. A neutrosophic clinical decision-making system for cardiovascular diseases risk analysis. J Intell Fuzzy Syst; 39. Epub ahead of print 2020. DOI: 10.3233/JIFS- 201163. · DOI
  124. Simonyan K, Vedaldi A, Zisserman A. Deep inside convolutional networks: Visualising image classification models and saliency maps. In: 2nd International Conference on Learning Representations, ICLR 2014 -Workshop Track Proceedings. International Conference on Learning Representations, ICLR, 2014.
  125. Montavon G, Binder A, Lapuschkin S, et al. Layer-Wise Relevance Propagation: An Overview. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019, pp. 193-209.
  126. Lundberg SM, Erion G, Chen H, et al. Explainable AI for trees: From local explanations to global understanding. arXiv; 2. Epub ahead of print 2019. DOI: 10.1038/s42256-019-0138-9. · DOI · Paper page
  127. Shrikumar A, Greenside P, Shcherbina A, et al. Not Just a Black Box: Learning Important Features Through Propagating Activation Differences. arXiv; 1.
  128. Štrumbelj E, Kononenko I. Explaining prediction models and individual predictions with feature contributions. Knowl Inf Syst 2014; 41: 647-665.
  129. Chen P, Dong W, Wang J, et al. Interpretable clinical prediction via attention-based neural network. BMC Med Inform Decis Mak; 20. Epub ahead of print 2020. DOI: 10.1186/s12911-020-1110-7. · DOI
  130. Alsinglawi B, Alshari O, Alorjani M, et al. An explainable machine learning framework for lung cancer hospital length of stay prediction. Sci Rep; 12. Epub ahead of print 2022. DOI: 10.1038/s41598-021-04608-7. · DOI
  131. Kibria, Hafsa Binte; Nahiduzzaman, Md; Goni, Md Omaer Faruq; Ahsan, Mominul and Haider J. An Ensemble Approach for the Prediction of Diabetes Mellitus Using a Soft Voting Classifier with an Explainable AIle. Sensors (Basel) 2022; 22: 7268. doi: 10.3390/s22197268. · DOI
  132. Ntakolia C, Priftis D, Charakopoulou-Travlou M, et al. An Explainable Machine Learning Approach for COVID-19's Impact on Mood States of Children and Adolescents during the First Lockdown in Greece. Healthc; 10. Epub ahead of print 2022. DOI: 10.3390/healthcare10010149. · DOI
  133. Oh S, Park Y, Cho KJ, et al. Explainable machine learning model for glaucoma diagnosis and its interpretation. Diagnostics; 11. Epub ahead of print 2021. DOI: 10.3390/diagnostics11030510. · DOI
  134. Hatwell J, Gaber MM, Atif Azad RM. Ada-WHIPS: Explaining AdaBoost classification with applications in the health sciences. BMC Med Inform Decis Mak; 20. Epub ahead of print 2020. DOI: 10.1186/s12911-020-01201-2. · DOI
  135. Casiraghi E, Malchiodi D, Trucco G, et al. Explainable Machine Learning for Early Assessment of COVID-19 Risk Prediction in Emergency Departments. IEEE Access; 8. Epub ahead of print 2020. DOI: 10.1109/ACCESS.2020.3034032. · DOI
  136. Shi W, Tong L, Zhu Y, et al. COVID-19 Automatic Diagnosis with Radiographic Imaging: Explainable Attention Transfer Deep Neural Networks. IEEE J Biomed Heal Informatics; 25. Epub ahead of print 2021. DOI: 10.1109/JBHI.2021.3074893. · DOI
  137. Jansen T, Geleijnse G, van Maaren M, et al. Machine learning explainability in breast cancer survival. In: Studies in Health Technology and Informatics. 2020, pp. 307-311.
  138. Lombardi A, Diacono D, Amoroso N, et al. Explainable Deep Learning for Personalized Age Prediction With Brain Morphology. Front Neurosci; 15. Epub ahead of print 2021. DOI: 10.3389/fnins.2021.674055. · DOI
  139. Khodabandehloo E, Riboni D, Alimohammadi A. HealthXAI: Collaborative and explainable AI for supporting early diagnosis of cognitive decline. Futur Gener Comput Syst; 116. Epub ahead of print 2021. DOI: 10.1016/j.future.2020.10.030. · DOI
  140. Chary M, Boyer EW, Burns MM. Diagnosis of Acute Poisoning using explainable artificial intelligence. Comput Biol Med; 134. Epub ahead of print 2021. DOI: 10.1016/j.compbiomed.2021.104469. · DOI
  141. Lamy JB, Sedki K, Tsopra R. Explainable decision support through the learning and visualization of preferences from a formal ontology of antibiotic treatments. J Biomed Inform; 104. Epub ahead of print 2020. DOI: 10.1016/j.jbi.2020.103407. · DOI · Paper page
  142. Kokciyan N, Sassoon I, Sklar E, et al. Applying Metalevel Argumentation Frameworks to Support Medical Decision Making. IEEE Intell Syst. Epub ahead of print 2021. DOI: 10.1109/MIS.2021.3051420. · DOI
  143. Fang HSA, Tan NC, Tan WY, et al. Patient similarity analytics for explainable clinical risk prediction. BMC Med Inform Decis Mak; 21. Epub ahead of print 2021. DOI: 10.1186/s12911-021-01566-y. · DOI
  144. Sabol P, Sinčák P, Hartono P, et al. Explainable classifier for improving the accountability in decision-making for colorectal cancer diagnosis from histopathological images. J Biomed Inform; 109. Epub ahead of print 2020. DOI: 10.1016/j.jbi.2020.103523. · DOI
  145. de Souza LA, Mendel R, Strasser S, et al. Convolutional Neural Networks for the evaluation of cancer in Barrett's esophagus: Explainable AI to lighten up the black-box. Comput Biol Med; 135. Epub ahead of print 2021. DOI: 10.1016/j.compbiomed.2021.104578. · DOI
  146. Lamy JB, Sekar B, Guezennec G, et al. Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach. Artif Intell Med 2019; 94: 42-53.
  147. Kumar A, Manikandan R, Kose U, et al. Doctor's dilemma: Evaluating an explainable subtractive spatial lightweight convolutional neural network for brain tumor diagnosis. ACM Trans Multimed Comput Commun Appl; 17. Epub ahead of print 2021. DOI: 10.1145/3457187. · DOI
  148. Dahalan AJ, Razak TR, Ismail MH, et al. Heart rate events classification via explainable fuzzy logic systems. IAES Int J Artif Intell; 10. Epub ahead of print 2021. DOI: 10.11591/IJAI.V10.I4.PP1036-1047. · DOI
  149. Barda AJ, Horvat CM, Hochheiser H. A qualitative research framework for the design of user-centered displays of explanations for machine learning model predictions in healthcare. BMC Med Inform Decis Mak; 20. Epub ahead of print 2020. DOI: 10.1186/s12911-020-01276-x. · DOI · Paper page
  150. Yoo TK, Ryu IH, Choi H, et al. Explainable machine learning approach as a tool to understand factors used to select the refractive surgery technique on the expert level. Transl Vis Sci Technol; 9. Epub ahead of print 2020. DOI: 10.1167/tvst.9.2.8. · DOI
  151. Nguyen-Duc T, Mulligan N, Mannu GS, et al. Deep EHR Spotlight: a Framework and Mechanism to Highlight Events in Electronic Health Records for Explainable Predictions. AMIA . Annu Symp proceedings AMIA Symp; 2021.
  152. Cheng F, Liu D, Du F, et al. VBridge: Connecting the Dots between Features and Data to Explain Healthcare Models. IEEE Trans Vis Comput Graph; 28. Epub ahead of print 2022. DOI: 10.1109/TVCG.2021.3114836. i Selected articles list of references: https://refworks.proquest.com/public- share/ · DOI

Cited by

Sorted by citations

Authors
Prentzas, N., Kakas, A., & Pattichis, C. S.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.05411
arXiv
2308.05411
OpenAlex
W4385774886
Cited by
19
References
1
Citations by year
2026: 1 · 2025: 8 · 2024: 10
FWCI
0.40
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • University of Cyprus (8037P-22046)
  • European Commission (HORIZON2020, 739578)
  • Research and Innovation Foundation (739578)
  • Deputy Ministry of Research, Innovation and Digital Policy
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
explainable AI, clinical decision support, artificial intelligence in healthcare, model-agnostic explainability, deep learning, interactive user interface, diagnostic imaging, clinical adoption

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review