The DOI matches its DataCite record; 87 of the 93 references it cites were found in a registry or an index, and 6 are not checked yet.
What do we need to build explainable AI systems for the medical domain?
Open accessAbstract
Artificial intelligence (AI) generally and machine learning (ML) specifically demonstrate impressive practical success in many different application domains, e.g. in autonomous driving, speech recognition, or recommender systems. Deep learning approaches, trained on extremely large data sets or using reinforcement learning methods have even exceeded human performance in visual tasks, particularly on playing games such as Atari, or mastering the game of Go. Even in the medical domain there are remarkable results. The central problem of such models is that they are regarded as black-box models and even if we understand the underlying mathematical principles, they lack an explicit declarative knowledge representation, hence have difficulty in generating the underlying explanatory structures. This calls for systems enabling to make decisions transparent, understandable and explainable. A huge motivation for our approach are rising legal and privacy aspects. The new European General Data Protection Regulation entering into force on May 25th 2018, will make black-box approaches difficult to use in business. This does not imply a ban on automatic learning approaches or an obligation to explain everything all the time, however, there must be a possibility to make the results re-traceable on demand. In this paper we outline some of our research topics in the context of the relatively new area of explainable-AI with a focus on the application in medicine, which is a very special domain. This is due to the fact that medical professionals are working mostly with distributed heterogeneous and complex sources of data. In this paper we concentrate on three sources: images, *omics data and text. We argue that research in explainable-AI would generally help to facilitate the implementation of AI/ML in the medical domain, and specifically help to facilitate transparency and trust.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Unfortunately, scarce contributions have attempted to define such goals from a conceptual perspective [5, 13, 24, 30].”
From Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI (2020) · cites this paper as [30]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“While a complete understanding of the inner workings remains improbable, approaching this with explainable AI may be valuable (Holz -inger, 2018 ; Holzinger et al., 2017).”
From The Epistemic Cost of Opacity: How the Use of Artificial Intelligence Undermines the Knowledge of Medical Doctors in High-Stakes Contexts (2025) · cites this paper as Holzinger et al., 2017
In the citing paper
“there is a consensus among computer scientists that an inherent tension is between the performance and explainability of machine learning, with the best-performing algorithms being the least transparent and those that provide precise explanatory algorithms often being less accurate”
From Ethics and governance of trustworthy medical artificial intelligence (2023) · cites this paper as [46]
In the citing paper
“Toward this direction, multi-institutional collaboration is quickly becoming the vehicle driving the creation of well-curated and semantically annotated large cohorts that are further enhanced with research methods and results metadata, underpinning reproducible, extensible, and explainable research [59], [60].”
From AI in Medical Imaging Informatics: Current Challenges and Future Directions (2020) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Adaptation in Natural and Artificial Systems
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,340
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 7,330
Research works printed but not linked: 36
- Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013. doi: 10.1109/TPAMI.2013. · DOI
- Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic Web. Scientific American, 284(5):34-43, May 2001.
- Mark G. Core, H. Chad Lane, Michael Van Lent, Dave Gomboc, Steve Solomon, and Milton Rosenberg. Building explainable artificial intelligence systems. In AAAI, pages 1766-1773. MIT Press, 2006.
- Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1606-1615, Denver, Colorado, May-June 2015. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/ N15-1184.
- Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Zhoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in neural information processing systems (NIPS), pages 2672-2680, 2014.
- Geoffrey E. Hinton. A practical guide to training restricted boltzmann machines. In Neural networks: Tricks of the trade, Lecture Notes in Computer Science LNCS 7700, pages 599-619. Springer, Heidelberg, 2012. doi: 10.1007/978-3-642-35289-8 32. · DOI
- Andreas Holzinger, Markus Plass, Katharina Holzinger, Gloria Cerasela Crisan, Camelia- M. Pintea, and Vasile Palade. Towards interactive machine learning (iML): Applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. In Springer Lecture Notes in Computer Science LNCS 9817, pages 81-95. Springer, Heidelberg, Berlin, New York, 2016. doi: 10.1007/978-3-319-45507-56. · DOI
- Berthold Huppertz and Andreas Holzinger. Biobanks a source of large biological data sets: Open problems and future challenges. In Andreas Holzinger and Igor Jurisica, editors, Interactive Knowledge Discovery and Data Mining in Biomedical Informatics, Lecture Notes in Computer Science LNCS 8401, pages 317-330. Springer, Berlin, Heidelberg, 2014. doi: 10.1007/978-3-662-43968-5 18. · DOI
- W. Lewis Johnson. Agents that learn to explain themselves. In Twelfth National Conference on Artificial Intelligence (AAAI '94), pages 1257-1263. AAAI, 1994.
- Peter Kieseberg, Edgar Weippl, and Andreas Holzinger. Trust for the doctor-in-the-loop. European Research Consortium for Informatics and Mathematics (ERCIM) News: Tack- ling Big Data in the Life Sciences, 104(1):32-33, 2016.
- Iasonas Kokkinos, Georgios Evangelopoulos, and Petros Maragos. Texture analysis and segmentation using modulation features, generative models, and weighted curve evolution. IEEE transactions on pattern analysis and machine intelligence, 31(1):142-157, 2009. doi: 10.1109/TPAMI.2008. · DOI
- Zachary C. Lipton. The mythos of model interpretability. arXiv:1606.03490, 2016. · arXiv
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 142-150, 2011.
- Marc E. Maier, Brian J. Taylor, Huseyin Oktay, and David D. Jensen. Learning causal models of relational domains. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence (AAAI-10), pages 531-538. AAAI, 2010.
- Christopher D. Manning. Computational linguistics and deep learning. Computational Linguistics, 41(4):701-707, December 2015. doi: 10.1162/COLI a 00239. · DOI
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Christopher J.C. Burges, Leon Bottou, Max Welling, Zoubin Ghahramani, and Kilian Q. Weinberger, editors, Advances in neural information processing systems 26 (NIPS 2013), pages 3111- 3119. NIPS foundation, 2013b.
- Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. arXiv:1706.07979, 2017. · arXiv
- Heimo Müller, Robert Reihs, Kurt Zatloukal, Fleur Jeanquartier, Roxana Merino-Martinez, David van Enckevort, Morris A. Swertz, and Andreas Holzinger. State-of-the-art and future challenges in the integration of biobank catalogues. In Smart Health, Lecture Notes in Computer Science LNCS 8700, pages 261-273. Springer, Heidelberg, 2015. doi: 10.1007/978-3-319-16226-3 11. · DOI
- Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 427-436. IEEE, 2015.
- Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthe- sizing the preferred inputs for neurons in neural networks via deep generator networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29 (NIPS 2016), pages 3387-3395. Neural Information Processing Systems Foundation, 2016.
- Alexander Panchenko, Eugen Ruppert, Stefano Faralli, Simone Paolo Ponzetto, and Chris Biemann. Unsupervised does not mean uninterpretable: The case for word sense induction and disambiguation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 86-98, Valencia, Spain, April 2017. Association for Computational Linguistics. URL http: //www.aclweb.org/anthology/E17-1009.
- John Pavlopoulos, Prodromos Malakasiotis, and Ion Androutsopoulos. Deeper attention to abusive user content moderation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1136-1146. Association for Computa- tional Linguistics, 2017.
- Judea Pearl. Causality: Models, Reasoning, and Inference (2nd Edition). Cambridge Uni- versity Press, Cambridge, 2009.
- Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. Cambridge (MA), 2017.
- Rosalind W. Picard. Affective Computing. MIT Press, Cambridge (MA), 1997.
- Brett Poulin, Roman Eisner, Duane Szafron, Paul Lu, Russell Greiner, David S. Wishart, Alona Fyshe, Brandon Pearcy, Cam MacDonell, and John Anvik. Visual explanation of evidence with additive classifiers. In National Conference On Artificial Intelligence, pages 1822-1829. AAAI, 2006.
- Sarni Suhaila Rahim, Vasile Palade, Chrisina Jayne, Andreas Holzinger, and James Shut- tleworth. Detection of diabetic retinopathy and maculopathy in eye fundus images using fuzzy image processing. In Yike Guo, Karl Friston, Faisal Aldo, Sean Hill, and Hanchuan Peng, editors, Brain Informatics and Health, Lecture Notes in Computer Science, LNCS 9250, pages 379-388. Springer, Cham, Heidelberg, New York, Dordrecht, London, 2015. doi: 10.1007/978-3-319-23344-4 37. · DOI
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should I trust you?: Ex- plaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144. ACM, 2016b.
- Sascha Rothe, Sebastian Ebert, and Hinrich Schütze. Ultradense word embeddings by or- thogonal transformation. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pages 767-777, San Diego, California, June 2016. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/N16-1091.
- Stuart J. Russell and Peter Norvig. Artificial Intelligence: A modern approach. Prentice Hall, Englewood Cliffs, 1995.
- Deepika Singh, Erinc Merdivan, Ismini Psychoula, Johannes Kropf, Sten Hanke, Matthieu Geist, and Andreas Holzinger. Human activity recognition using recurrent neural net- works. In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar Weippl, ed- itors, Machine Learning and Knowledge Extraction: Lecture Notes in Computer Sci- ence LNCS 10410, pages 267-274. Springer International Publishing, Cham, 2017. doi: 10.1007/978-3-319-66808-6 18. · DOI
- Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi. One pixel attack for fooling deep neural networks. arXiv:1710.08864, 2017. · arXiv
- Sandra Wartner, Dominic Girardi, Manuela Wiesinger-Widi, Johannes Trenkler, Raimund Kleiser, and Andreas Holzinger. Ontology-guided principal component analysis: Reaching the limits of the doctor-in-the-loop. In Information Technology in Bio-and Medical Informatics: 7th International Conference, ITBAM 2016, Porto, Portugal, September 5-8, 2016, Proceedings, pages 22-33. Springer International Publishing, Cham, 2016. doi: 10.1007/978-3-319-43949-5 2. · DOI
- Seid Muhie Yimam, Steffen Remus, Alexander Panchenko, Andreas Holzinger, and Chris Biemann. Entity-centric information access with the human-in-the-loop for the biomedical domains. In Svetla Boytcheva, Kevin Bretonnel Cohen, Guergana Savova, and Galia Angelova, editors, Biomedical NLP Workshop, 11th International Conference on Recent Advances in Natural Language Processing (RANLP 2017), pages 42-48, 2017.
- Feng Zhou and Fernando De la Torre. Factorized graph matching. In CVPR 2012 IEEE Conference onComputer Vision and Pattern Recognition, pages 127-134. IEEE, 2012.
- Martin Zinkevich. Online convex programming and generalized infinitesimal gradient as- cent. In 20th International Conference on Machine Learning (ICML '03), pages 928-936. AAAI, 2003.
Cited by
Sorted by citations
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294
“Recently, researchers have conducted preliminary work aiming to make clinical AI-based systems explainable [37], [45]–[47].”
BackgroundKey challenges for delivering clinical impact with artificial intelligence
Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G., & King, D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1426-2 · Cited by 3,136
FusionGAN: A generative adversarial network for infrared and visible image fusion
Ma, J., Yu, W., Liang, P., Li, C., & Jiang, J. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.09.004 · Cited by 2,299
Causability and explainability of artificial intelligence in medicine
Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1312 · Cited by 1,885
Self-citationExplainable AI (XAI): Core Ideas, Techniques, and Solutions
Dwivedi, R., Dave, D., Naik, H., Singhal, S., Omer, R., Patel, P., Qian, B., Wen, Z., Shah, T., Morgan, G., et al. (2023) · ACM Computing Surveys · doi:10.1145/3561048 · Cited by 1,386
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Explainable artificial intelligence: an analytical review
Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I., & Atkinson, P. M. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1424 · Cited by 843
Deep neural network models for computational histopathology: A survey
Srinidhi, C. L., Ciga, O., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101813 · Cited by 749
Deep learning in medical imaging and radiation therapy
Sahiner, B., Pezeshk, A. X., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., Summers, R. M., & Giger, M. L. (2018) · Medical Physics · doi:10.1002/mp.13264 · Cited by 763
Yang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
Explainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718
AI in Medical Imaging Informatics: Current Challenges and Future Directions
Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., Foran, D., Do, N., Golemati, S., Kurc, T., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2991043 · Cited by 713
Self-citationAntoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629
An Introductory Review of Deep Learning for Prediction Models With Big Data
Emmert‐Streib, F., Yang, Z., Han, F., Tripathi, S., & Dehmer, M. (2020) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2020.00004 · Cited by 651
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare
Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A., Valencia, A., Rementeria, M. J., Chadha, A. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0288-5 · Cited by 553
Ethics and governance of trustworthy medical artificial intelligence
Zhang, J., & Zhang, Z.-M. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02103-9 · Cited by 507
“However, there is a consensus among computer scientists that an inherent tension is between the performance and explainability of machine learning, with the best-performing algorithms being the least transparent and those that provide precise explanatory algorithms often being less accurate [46] which seems to be a lo…”
BackgroundExplainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372
Moncada‐Torres, A., van Maaren, M. C., Hendriks, M. P., Siesling, S., & Geleijnse, G. (2021) · Scientific Reports · doi:10.1038/s41598-021-86327-7 · Cited by 358
Artificial Intelligence for a Better Future
Stahl, B. C. (2021) · SpringerBriefs in research and innovation governance · doi:10.1007/978-3-030-69978-9 · Cited by 437
Carter, S. M., Rogers, W., Win, K. T., Frazer, H., Richards, B., & Houssami, N. (2020) · The Breast · doi:10.1016/j.breast.2019.10.001 · Cited by 312
Pathologist-level interpretable whole-slide cancer diagnosis with deep learning
Zhang, Z., Chen, P., McGough, M., Xing, F., Wang, C., Bui, M., Xie, Y., Sapkota, M., Cui, L., Dhillon, J., et al. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0052-1 · Cited by 311
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 370
Artificial Intelligence and Patient-Centered Decision-Making
Bjerring, J. C., & Busch, J. (2021) · Philosophy & Technology · doi:10.1007/s13347-019-00391-6 · Cited by 274
Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.11.003 · Cited by 302
Kiseleva, A., Kotzinos, D., & De Hert, P. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.879603 · Cited by 278
Diao, J. A., Wang, J. K., Chui, W. F., Mountain, V., Gullapally, S. C., Srinivasan, R., Mitchell, R. N., Glass, B., Hoffman, S., Rao, S. K., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21896-9 · Cited by 289
The perils and pitfalls of explainable AI: Strategies for explaining algorithmic decision-making
de Bruijn, H., Warnier, M., & Janssen, M. (2022) · Government Information Quarterly · doi:10.1016/j.giq.2021.101666 · Cited by 264
Applications of Artificial Intelligence (AI) in healthcare: A reviewPreprint
Shaheen, M. Y. (2021) · doi:10.14293/s2199-1006.1.sor-.ppvry8k.v1 · Cited by 270
Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata, Z., Stumpf, S., Kieseberg, P., & Holzinger, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-99740-7_21 · Cited by 259
Self-citationExplanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251
Neupane, S., Ables, J., Anderson, W. S., Mittal, S., Rahimi, S., Banicescu, I., & Seale, M. A. (2022) · IEEE Access · doi:10.1109/access.2022.3216617 · Cited by 293
Interactive machine learning: experimental evidence for the human in the algorithmic loop
Holzinger, A., Plass, M., Kickmeier-Rust, M., Holzinger, K., Crişan, G. C., Pintea, C.-M., & Palade, V. (2018) · Applied Intelligence · doi:10.1007/s10489-018-1361-5 · Cited by 235
Self-citationExplainability in human–agent systems
Rosenfeld, A., & Richardson, A. (2019) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-019-09408-y · Cited by 249
Ghaffarian, S., Taghikhah, F. R., & Maier, H. R. (2023) · International Journal of Disaster Risk Reduction · doi:10.1016/j.ijdrr.2023.104123 · Cited by 269
Górriz, J. M., Álvarez, I., Álvarez-Marquina, A., Arco, J. E., Atzmueller, M., Ballarini, F., Barakova, E. I., Bologna, G., Bonomini, P., Castellanos-Dominguez, G., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101945 · Cited by 251
Holzinger, A., Dehmer, M., Emmert‐Streib, F., Cucchiara, R., Augenstein, I., Ser, J. D., Samek, W., Jurišica, I., & Díaz-Rodríguez, N. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.10.007 · Cited by 223
Self-citationMienye, I. D., Obaido, G., Jere, N., Mienye, E., Aruleba, K., Emmanuel, I. D., & Ogbuokiri, B. (2024) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2024.101587 · Cited by 212
Vicnesh, J., Ng, E. Y. K., Tan, R., Oh, S. L., & Acharya, U. R. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105550 · Cited by 224
A Review of Trust in Artificial Intelligence: Challenges, Vulnerabilities and Future Directions
Lockey, S., Gillespie, N. A., Holm, D., & Someh, I. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.664 · Cited by 233
A Review on Explainable Artificial Intelligence for Healthcare: Why, How, and When?
Bharati, S., Mondal, M. R. H., & Podder, P. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3266418 · Cited by 251
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Theissler, A., Spinnato, F., Schlegel, U., & Guidotti, R. (2022) · IEEE Access · doi:10.1109/access.2022.3207765 · Cited by 245
AI-driven healthcare: A review on ensuring fairness and mitigating bias
Chinta, S. V., Wang, Z., Palikhe, A., Zhang, X., Kashif, A., Smith, M. A., Liu, J., & Zhang, W. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000864 · Cited by 268
Deep learning models in genomics; are we there yet?
Koumakis, L. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.06.017 · Cited by 194
Explainable Artificial Intelligence for Human Decision Support System in the Medical Domain
Knapič, S., Malhi, A. K., Saluja, R., & Främling, K. (2021) · Machine Learning and Knowledge Extraction · doi:10.3390/make3030037 · Cited by 179
Diagnosis of focal liver lesions from ultrasound using deep learning
Schmauch, B., Hérent, P., Jehanno, P., Dehaene, O., Saillard, C., Aubé, C., Luciani, A., Lassau, N., & Jégou, S. (2019) · Diagnostic and Interventional Imaging · doi:10.1016/j.diii.2019.02.009 · Cited by 172
Pawar, U., O’Shea, D., Rea, S., & O’Reilly, R. (2020) · International Conference on Cyber Situational Awareness, Data Analytics and Assessment (CyberSA) · doi:10.1109/cybersa49311.2020.9139655 · Cited by 175
Wani, N. A., Kumar, R., Mamta, ., Bedi, J., & Rida, I. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102472 · Cited by 172
Operationalising ethics in artificial intelligence for healthcare: a framework for AI developers
Solanki, P., Grundy, J., & Hussain, W. (2023) · AI and Ethics · doi:10.1007/s43681-022-00195-z · Cited by 157
“These principles can be further operationalized to target explainable medicine by developing explainable interfaces that allowing experts to understand machine diagnosis/outcome and its influencing factors or causality [124, 125].”
BackgroundExploring chemical space using natural language processing methodologies for drug discovery
Öztürk, H., Özgür, A., Schwaller, P., Laino, T., & Özkırımlı, E. (2020) · Drug Discovery Today · doi:10.1016/j.drudis.2020.01.020 · Cited by 167
Bayesian neural networks: An introduction and survey
Goan, E., & Fookes, C. (2020) · QUT ePrints (Queensland University of Technology) · Cited by 162
Explainable Artificial Intelligence for Neuroscience: Behavioral Neurostimulation
Fellous, J., Sapiro, G., Rossi, A. F., Mayberg, H., & Ferrante, M. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01346 · Cited by 165
HCI for health and wellbeing: Challenges and opportunities
Blandford, A. (2019) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2019.06.007 · Cited by 152
Wang, C. J., Hamm, C. A., Savic, L. J., Ferrante, M., Schobert, I. T., Schlachter, T. R., Lin, M., Weinreb, J. C., Duncan, J. S., Chapiro, J., et al. (2019) · European Radiology · doi:10.1007/s00330-019-06214-8 · Cited by 154
TRUST XAI: Model-Agnostic Explanations for AI With a Case Study on IIoT Security
Zolanvari, M., Yang, Z., Khan, K. M., Jain, R. K., & Meskin, N. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3122019 · Cited by 161
The integration of AI in nursing: addressing current applications, challenges, and future directions
Wei, Q., Pan, S., Liu, X., Hong, M., Nong, C., & Zhang, W. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1545420 · Cited by 143
Artificial intelligence and primary care research: a scoping review
Kueper, J. K., Terry, A. L., Zwarenstein, M., & Lizotte, D. J. (2020) · The Annals of Family Medicine · doi:10.1370/afm.2518 · Cited by 143
An emotion and cognitive based analysis of mental health disorders from social media data
Uban, A. S., Chulvi, B., & Rosso, P. (2021) · Future Generation Computer Systems · doi:10.1016/j.future.2021.05.032 · Cited by 156
Explaining Models by Propagating Shapley Values of Local Components
Chen, H., Lundberg, S., & Lee, S. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-53352-6_24 · Cited by 150
Classification and specific primer design for accurate detection of SARS-CoV-2 using deep learning
Lopez‐Rincon, A., Tonda, A. P., Mendoza‐Maldonado, L., Mulders, D. G. J. C., Molenkamp, R., Perez-Romero, C. A., Claassen, E., Garssen, J., & Kraneveld, A. D. (2021) · Scientific Reports · doi:10.1038/s41598-020-80363-5 · Cited by 137
The effects of personality and locus of control on trust in humans versus artificial intelligence
Sharan, N. N., & Romano, D. M. (2020) · Heliyon · doi:10.1016/j.heliyon.2020.e04572 · Cited by 138
Chatbots as Conversational Healthcare Services
Jovanovic, M., Báez, M., & Casati, F. (2020) · IEEE Internet Computing · doi:10.1109/mic.2020.3037151 · Cited by 123
Ahsan, M. M., Gupta, K. D., Islam, M. M., Sen, S., Rahman, M. L., & Hossain, M. S. (2020) · Machine Learning and Knowledge Extraction · doi:10.3390/make2040027 · Cited by 146
Reviewing the Need for Explainable Artificial Intelligence (xAI)
Gerlings, J., Shollo, A., & Constantiou, I. (2021) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2021.156 · Cited by 165
Walsh, C. G., Chaudhry, B., Dua, P., Goodman, K. W., Kaplan, B., Kavuluru, R., Solomonides, A., & Subbian, V. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooz054 · Cited by 118
Rahim, N., El–Sappagh, S., Ali, S., Muhammad, K., Ser, J. D., & Abuhmed, T. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.028 · Cited by 131
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Ehsan, U., Passi, S., Liao, Q. V., Chan, L., Lee, I.-H., Muller, M., & Riedl, M. O. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642474 · Cited by 123
Trust in artificial intelligence for medical diagnoses
(2020) · Progress in Brain Research · doi:10.1016/bs.pbr.2020.06.006 · Cited by 116
Post-hoc explanation of black-box classifiers using confident itemsets
Moradi, M., & Samwald, M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113941 · Cited by 130
Luo, Y., Tseng, H.-H., Cui, S., Wei, L., Haken, R. K. T., & Naqa, I. E. (2019) · BJR|Open · doi:10.1259/bjro.20190021 · Cited by 118
Jang, M., Kim, M., Bae, S. J., Lee, S. H., Koh, J., & Kim, N. (2020) · Journal of Bone and Mineral Research · doi:10.1002/jbmr.4477 · Cited by 118
Promises and perils of artificial intelligence in dentistry
Pethani, F. (2020) · Australian Dental Journal · doi:10.1111/adj.12812 · Cited by 129
Mertes, S., Huber, T., Weitz, K., Heimerl, A., & André, E. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.825565 · Cited by 119
Hassan, S. U., Abdulkadir, S. J., Zahid, M. S. M., & Al-Selwi, S. M. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109569 · Cited by 123
Hurvitz, N., Azmanov, H., Kesler, A., & Ilan, Y. (2021) · European Journal of Human Genetics · doi:10.1038/s41431-021-00928-4 · Cited by 106
Explainable artificial intelligence and machine learning: A reality rooted perspective
Emmert‐Streib, F., Yli-Harja, O. P., & Dehmer, M. (2020) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1368 · Cited by 119
Broad-coverage biomedical relation extraction with SemRep
Halil Kilicoglu, B., Rosemblat, G., Marcelo Fiszman, B., & Shin, D. (2020) · BMC Bioinformatics · doi:10.1186/s12859-020-3517-7 · Cited by 106
A Robust Interpretable Deep Learning Classifier for Heart Anomaly Detection Without Segmentation
Dissanayake, T., Fernando, T., Denman, S., Sridharan, S., Ghaemmaghami, H., & Fookes, C. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3027910 · Cited by 105
Zheng, Q., Delingette, H., & Ayache, N. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.06.001 · Cited by 104
Demystifying Brain Tumor Segmentation Networks: Interpretability and Uncertainty Analysis
Natekar, P., Kori, A., & Krishnamurthi, G. (2020) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2020.00006 · Cited by 103
Mursch‐Edlmayr, A. S., Ng, W. S., Diniz‐Filho, A., Sousa, D. C., Arnould, L., Schlenker, M. B., Dueñas-Ángeles, K., Keane, P. A., Crowston, J. G., & Jayaram, H. (2020) · Translational Vision Science & Technology · doi:10.1167/tvst.9.2.55 · Cited by 109
Using machine learning to revisit the diversification–performance relationship
Choi, J.-H., Menon, A. R., & Tabakovic, H. (2021) · Strategic Management Journal · doi:10.1002/smj.3317 · Cited by 108
The application of artificial neural networks in metabolomics: a historical perspective
Mendez, K. M., Broadhurst, D., & Reinke, S. N. (2019) · Metabolomics · doi:10.1007/s11306-019-1608-0 · Cited by 102
Evolving Rule-Based Explainable Artificial Intelligence for Unmanned Aerial Vehicles
Keneni, B., Kaur, D., Bataineh, A. S. A., Devabhaktuni, V. K., Javaid, A. Y., Zaientz, J. D., & Marinier, R. (2019) · IEEE Access · doi:10.1109/access.2019.2893141 · Cited by 98
Tackling COVID-19 Through Responsible AI Innovation: Five Steps in the Right Direction
Leslie, D. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.4bb9d7a7 · Cited by 103
AI in medical diagnosis: AI prediction & human judgment
Göndöcs, D., & Dörfler, V. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102769 · Cited by 108
Zou, L., Goh, H. L., Liew, C. J. Y., Quah, J. L., Gu, G. T., Chew, J. J., Kumar, M. P., Ang, C. G. L., & Ta, A. W. A. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3153754 · Cited by 96
Explainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87
Self-citationSabol, P., Sinčák, P., Hartono, P., Kočan, P., Benetinová, Z., Blichárová, A., Verbóová, Ľ., Štammová, E., Antónia, S. F., & Anna, J. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103523 · Cited by 93
Visual interpretability in 3D brain tumor segmentation network
Saleem, H. A., Shahid, A. R., & Raza, B. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104410 · Cited by 92
Exploring and Promoting Diagnostic Transparency and Explainability in Online Symptom Checkers
Tsai, C.-H., You, Y., Gui, X., Kou, Y., & Carroll, J. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445101 · Cited by 86
Charting the Sociotechnical Gap in Explainable AI: A Framework to Address the Gap in XAI
Ehsan, U., Saha, K., Choudhury, M. D., & Riedl, M. O. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579467 · Cited by 113
Ladbury, C., Zarinshenas, R., Semwal, H., Tam, A., Vaidehi, N., Rodin, A. S., Liu, A., Glaser, S., Salgia, R., & Amini, A. (2022) · Translational Cancer Research · doi:10.21037/tcr-22-1626 · Cited by 86
Wani, N. A., Kumar, R., & Bedi, J. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108939 · Cited by 92
Shi, W., Li, T., Zhu, Y., & Wang, M. D. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3074893 · Cited by 84
Examining the effect of explanation on satisfaction and trust in AI diagnostic systems
Alam, L., & Mueller, S. T. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01542-6 · Cited by 89
Explaining Machine Learning Models for Clinical Gait Analysis
Slijepcevic, D., Horst, F., Lapuschkin, S., Horsak, B., Raberger, A.-M., Kranzl, A., Samek, W., Breiteneder, C., Schöllhorn, W. I., & Zeppelzauer, M. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3474121 · Cited by 82
Explainable AI, But Explainable to Whom? An Exploratory Case Study of xAI in Healthcare
Gerlings, J., Jensen, M. S., & Shollo, A. (2021) · Intelligent systems reference library · doi:10.1007/978-3-030-83620-7_7 · Cited by 95
Explainable AI: A review of applications to neuroimaging data
Farahani, F. V., Fiok, K., Lahijanian, B., Karwowski, W., & Douglas, P. K. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.906290 · Cited by 87
Metadata
- Authors
- Holzinger, A., Biemann, C., Pattichis, C. S., & Kell, D. B.
- Affiliations
- Medical University of Graz (AT)
- Universität Hamburg (DE)
- University of Cyprus (CY)
- University of Manchester (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2017
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1712.09923
- OpenAlex
- W2778796877
- Cited by
- 636
- References
- 32
- Citations by year
- 2026: 17 · 2025: 71 · 2024: 69 · 2023: 78 · 2022: 69 · 2021: 136 · 2020: 103 · 2019: 62 · 2018: 30
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- University of Cyprus
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- explainable AI, artificial intelligence in healthcare, black-box models, transparency, trust in AI, General Data Protection Regulation, privacy, medical images, omics data, medical texts, heterogeneous data sources, knowledge representation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
