Designing an Interpretability-Based Model to Explain the Artificial Intelligence Algorithms in Healthcare
Open accessAbstract
The lack of interpretability in artificial intelligence models (i.e., deep learning, machine learning, and rules-based) is an obstacle to their widespread adoption in the healthcare domain. The absence of understandability and transparency frequently leads to (i) inadequate accountability and (ii) a consequent reduction in the quality of the predictive results of the models. On the other hand, the existence of interpretability in the predictions of AI models will facilitate the understanding and trust of the clinicians in these complex models. The data protection regulations worldwide emphasize the relevance of the plausibility and verifiability of AI models’ predictions. In response and to take a role in tackling this challenge, we designed the interpretability-based model with algorithms that achieve human-like reasoning abilities through statistical analysis of the datasets by calculating the relative weights of the variables of the features from the medical images and the patient symptoms. The relative weights represented the importance of the variables in predictive decision-making. In addition, the relative weights were used to find the positive and negative probabilities of having the disease, which indicated high fidelity explanations. Hence, the primary goal of our model is to shed light and give insights into the prediction process of the models, as well as to explain how the model predictions have resulted. Consequently, our model contributes by demonstrating accuracy. Furthermore, two experiments on COVID-19 datasets demonstrated the effectiveness and interpretability of the new model.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lack of interpretability and transparency in AI models can make it challenging to identify and address biases.
“Lack of interpretability and transparency in AI models can make it challenging to identify and address biases.”
From Edge AI for Early Detection of Chronic Diseases and the Spread of Infectious Diseases: Opportunities, Challenges, and Future Directions (2023) · cites this paper as [102]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Anchors: High-Precision Model-Agnostic Explanations
Ribeiro, M. T., Singh, S., & Guestrin, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11491 · Cited by 2,162 · cited 2× · see citing works
“Anchors Anchors is a model-independent, rule-based local explainer approach [23].”
BackgroundMethodExplainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718 · see citing works
A Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · see citing works
Interpretable Machine Learning in Healthcare
Ahmad, M. A., Eckert, C., & Teredesai, A. (2018) · Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/3233547.3233667 · Cited by 583 · see citing works
Loyola-Gonzalez, O. (2019) · IEEE Access · doi:10.1109/access.2019.2949286 · Cited by 559 · see citing works
Malware Detection with Deep Neural Network Using Process Behavior
Tobiyama, S., Yamaguchi, Y., Shimada, H., Ikuse, T., & Yagi, T. (2016) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2016.151 · Cited by 328 · see citing works
Chest CT findings of COVID-19 pneumonia by duration of symptoms
Ding, X., Jia, X., Zhou, J., & Long, Q. (2020) · European Journal of Radiology · doi:10.1016/j.ejrad.2020.109009 · Cited by 270 · see citing works
Interpreting tree ensembles with inTrees
Deng, H. (2018) · International Journal of Data Science and Analytics · doi:10.1007/s41060-018-0144-8 · Cited by 254 · see citing works
Attention-based interpretable neural network for building cooling load prediction
Li, A., Xiao, F., Zhang, C., & Fan, C. (2021) · Applied Energy · doi:10.1016/j.apenergy.2021.117238 · Cited by 231 · see citing works
Elshawi, R., Mohamed, Y., Al-Mallah, M., & Sakr, S. (2019) · 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS) · doi:10.1109/cbms.2019.00065 · Cited by 224 · see citing works
Perturbation-based methods for explaining deep neural networks: A survey
Ivanovs, M., Kadiķis, R., & Ozols, K. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.030 · Cited by 207 · see citing works
Global Explanations of Neural Networks
Ibrahim, M., Louie, M., Modarres, C., & Paisley, J. W. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314230 · Cited by 126 · see citing works
AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches
Cao, N., Yan, X., Shi, Y., & Chen, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33012564 · Cited by 55 · see citing works
A General Method for Visualizing and Explaining Black-Box Regression Models
Štrumbelj, E., & Kononenko, I. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20267-4_3 · Cited by 53 · see citing works
Lee, H., Kim, S. T., & Ro, Y. M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33850-3_3 · Cited by 47 · see citing works
Regional Tree Regularization for Interpretability in Deep Neural Networks
Wu, M., Parbhoo, S., Hughes, M., Kindle, R., Celi, L., Zazzi, M., Roth, V., & Doshi-Velez, F. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6112 · Cited by 38 · see citing works
Survey on Techniques, Applications and Security of Machine Learning Interpretability
Ji, S.-L., Li, J., Du, T., & Li, B. (2019) · Journal of Computer Research and Development · doi:10.7544/issn1000-1239.2019.20190540 · Cited by 35 · see citing works
Model-agnostic vs. Model-intrinsic Interpretability for Explainable Product Search
Ai, Q., & Narayanan.R, L. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482276 · Cited by 24 · see citing works
Explainable 3D convolutional neural network using GMM encoding
Stano, M., Benešová, W., & Marták, L. S. (2020) · International Conference on Machine Vision · doi:10.1117/12.2557314 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- [1]Sun Y. Liang D. Wang X. Tang X. DeepID3: Face Recognition with Very Deep Neural Networks arXiv 2015 1502.00873
- [12]Junjie W.U. Guannan L.I.U. Jingyuan W. Yuan Z.U.O. Hui B.U. Hao L.I.N. Data Intelligence: Trends and Challenges Syst. Eng. Theory Pract. 2020 40 2116 2149 10.12011/1000-6788-2020-0027-34 · DOI
- [14]Shrikumar A. Greenside P. Kundaje A. Learning Important Features Through Propagating Activation Differences arXiv 2017 1704.02685
- [15]Nurminen J. Rusanen A.-M. Chapter 4: Should We Know How AI Works Available online: https://ethics-of-ai.mooc.fi/chapter-4/2-what-is-transparency (accessed on 3 April 2022)
- [17]Doshi-Velez F. Kim B. Towards A Rigorous Science of Interpretable Machine Learning arXiv 2017 1702.08608
- [21]Puri N. Gupta P. Agarwal P. Verma S. Krishnamurthy B. MAGIX: Model Agnostic Globally Interpretable Explanations arXiv 2017 1706.07160
- [24]Shapley L.S. A Value for n-Person Games Contributions to the Theory of Games (AM-28) Kuhn H.W. Tucker A.W. Princeton University Press Princeton, NJ, USA 2016 Volume II 307 318
- [29]Kim B. Wattenberg M. Gilmer J. Cai C. Wexler J. Viegas F. Sayres R. Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV) arXiv 2017 1711.11279
- [33]Choi E. Bahadori M.T. Schuetz A. Stewart W.F. Sun J. RETAIN: Interpretable Predictive Model in Healthcare Using Reverse Time Attention Mechanism arXiv 2016 1608.05745
- [34]Molnar C. Interpretable Machine Learning Lulu Press Morrisville, NC, USA 2022
- [35]Modarres C. Ibrahim M. Louie M. Paisley J.W. Towards Explainable Deep Learning for Credit Lending: A Case Study arXiv 2018 1811.06471
Cited by
Sorted by citations
Mathew, D. E., Ebem, D. U., Ikegwu, A. C., Ukeoma, P. E., & Dibiaezue, N. F. (2025) · Neural Processing Letters · doi:10.1007/s11063-025-11732-2 · Cited by 168 · see citing works
Badidi, E. (2023) · Future Internet · doi:10.3390/fi15110370 · Cited by 113 · see citing works
- Authors
- Ennab, M., & Mcheick, H.
- Affiliations
- Université du Québec à Chicoutimi (CA)
- Corresponding author
- Ennab, Mohammad
- Venue
- Diagnostics
- Publisher
- MDPI AG
- Published
- 2022-06-26
- Volume
- 12
- Issue
- 7
- Pages
- 1557
- ISSN
- 2075-4418 (ISSN-L)
- PMC
- PMC9319389
- OpenAlex
- W4283521686
- Cited by
- 76
- References
- 21
- Citations by year
- 2026: 15 · 2025: 31 · 2024: 20 · 2023: 7 · 2022: 3
- FWCI
- 5.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Diagnostics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Constellation (Université du Québec à Chicoutimi) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Keywords
- explainable AI, interpretability, artificial intelligence in healthcare, feature importance, feature weights, medical image analysis, patient symptoms, COVID-19 prediction, clinical trust in AI, model transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
