The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review
Open accessAbstract
Objective: To conduct a systematic scoping review of explainable artificial intelligence (XAI) models that use real-world electronic health record data, categorize these techniques according to different biomedical applications, identify gaps of current studies, and suggest future research directions. Materials and Methods: We searched MEDLINE, IEEE Xplore, and the Association for Computing Machinery (ACM) Digital Library to identify relevant papers published between January 1, 2009 and May 1, 2019. We summarized these studies based on the year of publication, prediction tasks, machine learning algorithm, dataset(s) used to build the models, the scope, category, and evaluation of the XAI methods. We further assessed the reproducibility of the studies in terms of the availability of data and code and discussed open issues and challenges. Results: Forty-two articles were included in this review. We reported the research trend and most-studied diseases. We grouped XAI methods into 5 categories: knowledge distillation and rule extraction (N = 13), intrinsically interpretable models (N = 9), data dimensionality reduction (N = 8), attention mechanism (N = 7), and feature interaction and importance (N = 5). Discussion: XAI evaluation is an open issue that requires a deeper focus in the case of medical applications. We also discuss the importance of reproducibility of research work in this field, as well as the challenges and opportunities of XAI from 2 medical professionals’ point of view. Conclusion: Based on our review, we found that XAI evaluation in medicine has not been adequately and formally practiced. Reproducibility remains a critical concern. Ample opportunities exist to advance XAI research in medicine.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Artificial intelligence: a modern approach
(1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 21,974
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller, H., Loh, M. L., Downing, J. R., Caligiuri, M. A., et al. (1999) · Science · doi:10.1126/science.286.5439.531 · Cited by 11,516
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,583
Rudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
Alon, U., Barkai, N., Notterman, D. A., Gish, K., Ybarra, S., Mack, D., & Levine, A. J. (1999) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.96.12.6745 · Cited by 4,167
Artificial intelligence in radiology
Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H., & Aerts, H. J. W. L. (2018) · Nature Reviews Cancer · doi:10.1038/s41568-018-0016-5 · Cited by 3,803
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Self-citationGene expression correlates of clinical prostate cancer behavior
Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo, P., Renshaw, A. A., D'Amico, A. V., Richie, J. P., et al. (2002) · Cancer Cell · doi:10.1016/s1535-6108(02)00030-2 · Cited by 2,512
The practical implementation of artificial intelligence technologies in medicine
He, J., Baxter, S. L., Xu, J., Xu, J., Zhou, X., & Zhang, K. (2019) · Nature Medicine · doi:10.1038/s41591-018-0307-0 · Cited by 2,246
Scalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Self-citationCausability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885 · cited 1×
“Thus, XAI for medicine is of vital importance to support the implementation of AI in clinical decision support systems.(16,17) The increasing capabilities of AIM married to the necessity of XAI demand a review of the state-of-the-art research in the field.”
BackgroundMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842
MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia
Armstrong, S. A., Staunton, J. E., Silverman, L. B., Pieters, R., den Boer, M. L., Minden, M. D., Sallan, S. E., Lander, E. S., Golub, T. R., & Korsmeyer, S. J. (2002) · Nature Genetics · doi:10.1038/ng765 · Cited by 1,723
Mining electronic health records: towards better research applications and clinical care
Jensen, P. B., Jensen, L. J., & Brunak, S. (2012) · Nature Reviews Genetics · doi:10.1038/nrg3208 · Cited by 1,680
DARPA's Explainable Artificial Intelligence Program
Gunning, D., & Aha, D. W. (2019) · AI Magazine · doi:10.1609/aimag.v40i2.2850 · Cited by 1,360
Techniques for interpretable machine learning
Du, M., Liu, N., & Hu, X. (2019) · Communications of the ACM · doi:10.1145/3359786 · Cited by 1,268
Artificial intelligence in cardiology
Johnson, K. W., Torres Soto, J., Glicksberg, B. S., Shameer, K., Miotto, R., Ali, M., Ashley, E., & Dudley, J. T. (2018) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2018.03.521 · Cited by 1,202
Artificial Intelligence in Precision Cardiovascular Medicine
Krittanawong, C., Zhang, H., Wang, Z., Aydar, M., & Kitai, T. (2017) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2017.03.571 · Cited by 1,120
Interpretation of Neural Networks Is Fragile
Ghorbani, A., Abid, A., & Zou, J. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013681 · Cited by 765
Interpretable Decision Sets: A Joint Framework for Description and Prediction
Lakkaraju, H., Bach, S. H., & Leskovec, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939874 · Cited by 687
Vellido, A. (2020) · Neural Computing and Applications · doi:10.1007/s00521-019-04051-w · Cited by 635
(2016) · PubMed · Cited by 576
Artificial intelligence faces reproducibility crisis
Hutson, M. (2018) · Science · doi:10.1126/science.359.6377.725 · Cited by 564
Explainable Artificial Intelligence (XAI)
Hossain, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6371435 · Cited by 541
Kochanek, K. D., Murphy, S. L., Xu, J., & Arias, E. (2019) · PubMed · Cited by 455
Intelligible Support Vector Machines for Diagnosis of Diabetes Mellitus
Barakat, N., Bradley, A. P., & Barakat, M. N. H. (2010) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2009.2039485 · Cited by 367
Challenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366
The role of artificial intelligence in precision medicine
Mesko, B. (2017) · Expert Review of Precision Medicine and Drug Development · doi:10.1080/23808993.2017.1380516 · Cited by 343
Kwon, B. C., Choi, M.-J., Kim, J. T., Choi, E., Kim, Y. B., Kwon, S., Sun, J., & Choo, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865027 · Cited by 294
Applications of deep learning to MRI images: A survey
Liu, J., Pan, Y., Li, M., Chen, Z., Tang, L., Lu, C., & Wang, J. (2018) · Big Data Mining and Analytics · doi:10.26599/bdma.2018.9020001 · Cited by 291
Self-citationShould Health Care Demand Interpretable Artificial Intelligence or Accept “Black Box” Medicine?
Wang, F., Kaushal, R., & Khullar, D. (2020) · Annals of Internal Medicine · doi:10.7326/m19-2548 · Cited by 244
Interpretable Deep Models for ICU Outcome Prediction.
Che, Z., Purushotham, S., Khemani, R. G., & Liu, Y. (2016) · PubMed · Cited by 243
Rulematrix: Visualizing and understanding classifiers with rules
Ming, Y., Qu, H., & Bertini, E. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2864812 · Cited by 243
An attention based deep learning model of clinical events in the intensive care unit
Kaji, D. A., Zech, J. R., Kim, J. S., Cho, S. K., Dangayach, N. S., Costa, A. B., & Oermann, E. K. (2019) · PLOS ONE · doi:10.1371/journal.pone.0211057 · Cited by 198
Diprose, W. K., Buist, N., Hua, N., Thurier, Q., Shand, G., & Robinson, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz229 · Cited by 188
Predicting drug side effects by multi-label learning and ensemble learning
Zhang, W., Liu, F., Luo, L., & Zhang, J. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0774-y · Cited by 187
Sokol, K., & Flach, P. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00637-y · Cited by 186
When Machines Think: Radiology’s Next Frontier
Dreyer, K. J., & Geis, J. R. (2017) · Radiology · doi:10.1148/radiol.2017171183 · Cited by 183
Accurate splice site prediction using support vector machines
Sonnenburg, S., Schweikert, G., Philips, P., Behr, J., & Rätsch, G. (2007) · BMC Bioinformatics · doi:10.1186/1471-2105-8-s10-s7 · Cited by 179
Explainable Artificial Intelligence (XAI)
Sewada, R., Jangid, A., Kumar, P., & Mishra, N. (2023) · Journal of Nonlinear Analysis and Optimization · doi:10.36893/jnao.2022.v13i02.041-047 · Cited by 172
Shickel, B., Loftus, T. J., Adhikari, L., Ozrazgat-Baslanti, T., Bihorac, A., & Rashidi, P. (2019) · Scientific Reports · doi:10.1038/s41598-019-38491-0 · Cited by 165
Zhang, J., Kowsari, K., Harrison, J. H., Lobo, J. M., & Barnes, L. E. (2018) · IEEE Access · doi:10.1109/access.2018.2875677 · Cited by 165
SpliceRover: interpretable convolutional neural networks for improved splice site prediction
Zuallaert, J., Godin, F., Kim, M., Soete, A., Saeys, Y., & Neve, W. D. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty497 · Cited by 155
Promoting Trust Between Patients and Physicians in the Era of Artificial Intelligence
Nundy, S., Montgomery, T., & Wachter, R. M. (2019) · JAMA · doi:10.1001/jama.2018.20563 · Cited by 148
Hao, J., Kim, Y. S., Kim, T.-K., & Kang, M. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2500-z · Cited by 147
Comprehensive Decision Tree Models in Bioinformatics
Štiglic, G., Kocbek, S., Pernek, I., & Kokol, P. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033812 · Cited by 142
Bernardini, M., Romeo, L., Misericordia, P., & Frontoni, E. (2019) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2019.2899218 · Cited by 136
Valdés, G., Luna, J. M., Eaton, E., Simone, C. B., Ungar, L., & Solberg, T. D. (2016) · Scientific Reports · doi:10.1038/srep37854 · Cited by 132
Readmission prediction via deep contextual embedding of clinical concepts
Xiao, C., Ma, T., Dieng, A. B., Blei, D. M., & Wang, F. (2018) · PLoS ONE · doi:10.1371/journal.pone.0195024 · Cited by 132
Laugié, M., Michel, J., Pohl, A., Poli, E., & Borgomano, J. R. F. (2019) · Scientific Reports · doi:10.1038/s41598-019-52821-2 · Cited by 127
SpliceMachine: predicting splice sites from high-dimensional local context representations
Degroeve, S., Saeys, Y., Bernard De Baets, C., Rouzé, P., & de Peer, Y. V. (2004) · Computer applications in the biosciences · doi:10.1093/bioinformatics/bti166 · Cited by 112
Brisimi, T. S., Xu, T., Wang, T., Dai, W., Adams, W. G., & Paschalidis, I. C. (2018) · Proceedings of the IEEE · doi:10.1109/jproc.2017.2789319 · Cited by 106
Mortality prediction in intensive care units (ICUs) using a deep rule-based fuzzy classifier
Davoodi, R., & Moradi, M. H. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.02.008 · Cited by 100
Gundersen, O. E., Gil, Y., & Aha, D. W. (2018) · AI Magazine · doi:10.1609/aimag.v39i3.2816 · Cited by 99
Asfour, M., Lambourne, A., Soliman, A. T., Al‐Behlani, S., Alasfoor, D., Bold, A., Mahtab, H., & King, H. (1995) · Diabetic Medicine · doi:10.1111/j.1464-5491.1995.tb00431.x · Cited by 98
Luo, G. (2016) · Health Information Science and Systems · doi:10.1186/s13755-016-0015-4 · Cited by 85
Personalized survival predictions via Trees of Predictors: An application to cardiac transplantation
Yoon, J., Zame, W. R., Banerjee, A., Cadeiras, M., Alaa, A. M., & van der Schaar, M. (2018) · PLoS ONE · doi:10.1371/journal.pone.0194985 · Cited by 80
Composition and stability of intestinal microbiota of healthy children within a Dutch population
de Meij, T. G. J., Budding, A. E., de Groot, E. F. J., Jansen, F. M., Kneepkens, C. M. F., Benninga, M. A., Penders, J., Bodegraven, A. A. V., & Savelkoul, P. H. M. (2015) · The FASEB Journal · doi:10.1096/fj.15-278622 · Cited by 75
Knijnenburg, T. A., Klau, G. W., Iorio, F., Garnett, M. J., McDermott, U., Shmulevich, I., & Wessels, L. F. A. (2016) · Scientific Reports · doi:10.1038/srep36812 · Cited by 74
An interpretable machine learning model for diagnosis of Alzheimer's disease
Das, D., Ito, J., Kadowaki, T., & Tsuda, K. (2019) · PeerJ · doi:10.7717/peerj.6543 · Cited by 72
DeepHINT: understanding HIV-1 integration via deep learning with attention
Hu, H., Xiao, A., Zhang, S., Li, Y., Shi, X., Jiang, T., Zhang, L., Zhang, L., & Zeng, J. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty842 · Cited by 71
Chen, J. H., Goldstein, M. K., Asch, S. M., Mackey, L. W., & Altman, R. B. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw136 · Cited by 63
Self-citationGe, W., Huh, J. W., Park, Y. R., Lee, J., Kim, Y., & Turchin, A. (2018) · Europe PMC (PubMed Central) · Cited by 59
Artificial Intelligence in Medicine
Cupples, A. (2019) · Routledge eBooks · doi:10.4324/9780429052071 · Cited by 54
Ghafouri‐Fard, S., Taheri, M. N., Omrani, M. D., Daaee, A., Mohammad‐Rahimi, H., & Kazazi, H. (2019) · Journal of Molecular Neuroscience · doi:10.1007/s12031-019-01311-1 · Cited by 45
Kim, S. G., Theera-Ampornpunt, N., Fang, C.-H., Harwani, M., Grama, A. Y., & Chaterji, S. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0302-3 · Cited by 44
Pan, L., Liu, G., Mao, X., Li, H., Zhang, J., Liang, H., & Li, X. (2018) · JMIR Medical Informatics · doi:10.2196/11728 · Cited by 42
Bouktif, S., Hanna, E. M., Zaki, N., & Khousa, E. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086456 · Cited by 27
Fuzzy support vector machine: an efficient rule-based classification technique for microarrays
Hajiloo, M., Rabiee, H. R., & Anooshahpour, M. (2013) · BMC Bioinformatics · doi:10.1186/1471-2105-14-s13-s4 · Cited by 27
Interpretation of microbiota-based diagnostics by explaining individual classifier decisions
Eck, A., Zintgraf, L., de Groot, E. F. J., de Meij, T. G. J., Cohen, T., Savelkoul, P. H. M., Welling, M., & Budding, A. E. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1843-1 · Cited by 27
Zhao, L. P., & Bolouri, H. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.03.001 · Cited by 22
Generating fuzzy rules for constructing interpretable classifier of diabetes disease
Settouti, N., Chikh, M. A., & Saidi, M. (2012) · Australasian Physical & Engineering Sciences in Medicine · doi:10.1007/s13246-012-0155-z · Cited by 16
Aditya, C. R., & Pande, M. B. S. (2017) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2016.12.004 · Cited by 16
Interpretable per case weighted ensemble method for cancer associations
Jalali, A., & Pfeifer, N. (2016) · BMC Genomics · doi:10.1186/s12864-016-2647-9 · Cited by 7
A transparent cancer classifier
Hartono, P. (2018) · Health Informatics Journal · doi:10.1177/1460458218817800 · Cited by 6
Cited by
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Loh, H. W., Ooi, C. P., Seoni, S., Barua, P. D., Molinari, F., & Acharya, U. R. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.107161 · Cited by 734
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Nazar, M., Alam, M. M., Yafi, E., & Su'ud, M. M. (2021) · IEEE Access · doi:10.1109/access.2021.3127881 · Cited by 312
Metadata
- Authors
- Payrovnaziri, S. N., Chen, Z., Rengifo-Moreno, P., Miller, T., Bian, J., Chen, J. H., Liu, X., & He, Z.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-05-17
- Volume
- 27
- Issue
- 7
- Pages
- 1173-1185
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3024173558
- Cited by
- 313
- References
- 82
- FWCI
- 16.48
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.