XAI-Based Clinical Decision Support Systems: A Systematic Review
Open accessPublished version of a preprint
The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
With increasing electronic medical data and the development of artificial intelligence, clinical decision support systems (CDSSs) assist clinicians in diagnosis and prescription. Traditional knowledge-based CDSSs follow an accumulated medical knowledgebase and a predefined rule system, which clarifies the decision-making process; however, maintenance cost issues exist in the medical data quality control and standardization processes. Non-knowledge-based CDSSs utilize vast amounts of data and algorithms to effectively make decisions; however, the deep learning black-box problem causes unreliable results. EXplainable Artificial Intelligence (XAI)-based CDSSs provide valid rationales and explainable results. These systems ensure trustworthiness and transparency by showing the recommendation and prediction result process using explainable techniques. However, existing systems have limitations, such as the scope of data utilization and the lack of explanatory power of AI models. This study proposes a new XAI-based CDSS framework to address these issues; introduces resources, datasets, and models that can be utilized; and provides a foundation model to support decision-making in various disease domains. Finally, we propose future directions for CDSS technology and highlight societal issues that need to be addressed to emphasize the potential of CDSSs in the future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
DrugBank 5.0: a major update to the DrugBank database for 2018
Wishart, D. S., Feunang, Y. D., Guo, A. C., Lo, E. J., Marcu, A., Grant, J. R., Sajed, T., Johnson, D., Li, C., Sayeeda, Z., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gkx1037 · Cited by 8,404
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761
Clinically applicable deep learning for diagnosis and referral in retinal disease
De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al. (2018) · Nature Medicine · doi:10.1038/s41591-018-0107-6 · Cited by 2,639
Marcus, D. S., Wang, T. H., Parker, J., Csernansky, J. G., Morris, J. C., & Buckner, R. L. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.9.1498 · Cited by 2,163
Pollard, T. J., Johnson, A. E. W., Raffa, J. D., Celi, L. A., Mark, R. G., & Badawi, O. (2018) · Scientific Data · doi:10.1038/sdata.2018.178 · Cited by 1,993
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Ali, S., Abuhmed, T., El-Sappagh, S., Muhammad, K., Alonso-Moral, J. M., Confalonieri, R., Guidotti, R., Del Ser, J., Díaz-Rodríguez, N., & Herrera, F. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101805 · Cited by 1,762
The Alzheimer's Disease Neuroimaging Initiative
Mueller, S. G., Weiner, M. W., Thal, L. J., Petersen, R. C., Jack, C., Jagust, W., Trojanowski, J. Q., Toga, A. W., & Beckett, L. (2005) · Neuroimaging Clinics of North America · doi:10.1016/j.nic.2005.09.008 · Cited by 1,525
SmoothGrad: removing noise by adding noise
Smilkov, D., Thorat, N., Kim, B., Viégas, F., & Wattenberg, M. (2017) · arXiv · doi:10.48550/arxiv.1706.03825 · Cited by 1,467
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Preparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164
Chen, Z., Chen, J., Collins, R., Guo, Y., Peto, R., Wu, F., Li, L., & on behalf of the China Kadoorie Biobank (CKB) collaborative group (2011) · International Journal of Epidemiology · doi:10.1093/ije/dyr120 · Cited by 1,034
Disease Ontology: a backbone for disease semantic integration
Schriml, L. M., Arze, C., Nadendla, S., Chang, Y.-W. W., Mazaitis, M., Felix, V., Feng, G., & Kibbe, W. A. (2012) · Nucleic Acids Research · doi:10.1093/nar/gkr972 · Cited by 929
Using recurrent neural network models for early detection of heart failure onset
Choi, E., Schuetz, A., Stewart, W. F., & Sun, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw112 · Cited by 928
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734
Bien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734
Clinical decision support systems for the practice of evidence-based medicine
Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., & Tang, P. C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080527 · Cited by 732
Polynomial calculation of the Shapley value based on sampling
Castro, J., Gómez, D., & Tejada, J. (2009) · Computers & Operations Research · doi:10.1016/j.cor.2008.04.004 · Cited by 601
DARPA's explainable artificial intelligence (XAI) program
Gunning, D. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3308446 · Cited by 546
Building a knowledge graph to enable precision medicine
Chandak, P., Huang, K., & Zitnik, M. (2023) · Scientific Data · doi:10.1038/s41597-023-01960-3 · Cited by 522
El-Sappagh, S., Alonso, J. M., Islam, S. M. R., Sultan, A. M., & Kwak, K. S. (2021) · Scientific Reports · doi:10.1038/s41598-021-82098-3 · Cited by 363
Explainable Artificial Intelligence (XAI): Concepts and Challenges in Healthcare
Hulsen, T. (2023) · AI · doi:10.3390/ai4030034 · Cited by 305
Magesh, P. R., Myloth, R. D., & Tom, R. J. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.104041 · Cited by 295
DiPietro, R., & Hager, G. D. (2020) · Handbook of Medical Image Computing and Computer Assisted Intervention · doi:10.1016/b978-0-12-816176-0.00026-0 · Cited by 293
Himmelstein, D. S., & Baranzini, S. E. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004259 · Cited by 172
Design and Development of a Multimodal Biomedical Information Retrieval System
Demner-Fushman, D., Antani, S., Simpson, M., & Thoma, G. R. (2012) · Journal of Computing Science and Engineering · doi:10.5626/jcse.2012.6.2.168 · Cited by 161
Woodfield, R., Grant, I., UK Biobank Stroke Outcomes Group, UK Biobank Follow-Up and Outcomes Working Group, & Sudlow, C. L. M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0140533 · Cited by 145
Hu, C.-A., Chen, C.-M., Fang, Y.-C., Liang, S.-J., Wang, H.-C., Fang, W.-F., Sheu, C.-C., Perng, W.-C., Yang, K.-Y., Kao, K.-C., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033898 · Cited by 106
Yang, X., Bian, J., Fang, R., Bjarnadottir, R. I., Hogan, W. R., & Wu, Y. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz144 · Cited by 74
Lin, Y.-S., Lee, W.-C., & Celik, Z. B. (2021) · Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining · doi:10.1145/3447548.3467213 · Cited by 64
Artificial intelligence in glioma imaging: Challenges and advances
Jin, W., Fatehi, M., Abhishek, K., Mallya, M., Toyota, B., & Hamarneh, G. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab8131 · Cited by 55
Lee, K. H., Goo, J. M., Park, C. M., Lee, H. J., & Jin, K. N. (2012) · Korean Journal of Radiology · doi:10.3348/kjr.2012.13.5.564 · Cited by 37
Chmiel, F. P., Burns, D. K., Azor, M., Borca, F., Boniface, M. J., Zlatev, Z. D., White, N. M., Daniels, T. W. V., & Kiuber, M. (2020) · doi:10.1101/2020.12.02.20239194 · Cited by 30
Cited by
Shaping the future of healthcare: ethical clinical challenges and pathways to trustworthy AI
Goktas, P., & Grzybowski, A. (2025) · Journal of Clinical Medicine · doi:10.3390/jcm14051605 · Cited by 336
Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Noor, A. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W. (2025) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.70018 · Cited by 52
Salimparsa, M., Sedig, K., Lizotte, D. J., Abdullah, S. S., Chalabianloo, N., & Muanda, F. T. (2025) · Informatics · doi:10.3390/informatics12040119 · Cited by 39
Unveiling Explainable AI in Healthcare: Current Trends, Challenges, and Future Directions
Aziz, N. A., Manzoor, A., Mazhar Qureshi, M. D., Qureshi, M. A., & Rashwan, W. (2024) · doi:10.1101/2024.08.10.24311735 · Cited by 30
Jo, D.-M., Ko, S.-C., Kim, K. W., Yang, D., Kim, J.-Y., Oh, G.-W., Choi, G., Lee, D.-S., Tabassum, N., Kim, Y.-M., et al. (2025) · Trends in Food Science & Technology · doi:10.1016/j.tifs.2025.105309 · Cited by 20
Metadata
- Authors
- Kim, S. Y., Kim, D. H., Kim, M. J., Ko, H. J., & Jeong, O. R.
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-07-30
- Volume
- 14
- Issue
- 15
- Pages
- 6638
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4401117531
- Cited by
- 68
- References
- 40
- FWCI
- 3.05
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.