Validating Automatic Concept-Based Explanations for AI-Based Digital Histopathology
Open accessAbstract
Digital histopathology poses several challenges such as label noise, class imbalance, limited availability of labelled data, and several latent biases to deep learning, negatively influencing transparency, reproducibility, and classification performance. In particular, biases are well known to cause poor generalization. Proposed tools from explainable artificial intelligence (XAI), bias detection, and bias discovery suffer from technical challenges, complexity, unintuitive usage, inherent biases, or a semantic gap. A promising XAI method, not studied in the context of digital histopathology is automated concept-based explanation (ACE). It automatically extracts visual concepts from image data. Our objective is to evaluate ACE’s technical validity following design science principals and to compare it to Guided Gradient-weighted Class Activation Mapping (Grad-CAM), a conventional pixel-wise explanation method. To that extent, we created and studied five convolutional neural networks (CNNs) in four different skin cancer settings. Our results demonstrate that ACE is a valid tool for gaining insights into the decision process of histopathological CNNs that can go beyond explanations from the control method. ACE validly visualized a class sampling ratio bias, measurement bias, sampling bias, and class-correlated bias. Furthermore, the complementary use with Guided Grad-CAM offers several benefits. Finally, we propose practical solutions for several technical challenges. In contradiction to results from the literature, we noticed lower intuitiveness in some dermatopathology scenarios as compared to concept-based explanations on real-world images.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ACE relies on manual parameter setting, which can affect outcomes, for example by producing duplicated or mixed concepts.
“However, the method relies on the manual setting of parameters, which can affect the outcome, e.g., resulting in duplicated or mixed concepts [25].”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.74 · Cited by 24,513 · see citing works
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555 · see citing works
Design Science in Information Systems Research1
Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004) · MIS Quarterly · doi:10.2307/25148625 · Cited by 10,707 · cited 2× · see citing works
“While ACE [55] represents a method from the category of global explanations on model level, Guided Grad-CAM [35] represents local explanations on image level using a pixel-wise heatmap visualization.”
MethodWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
SLIC Superpixels Compared to State-of-the-Art Superpixel Methods
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., & Susstrunk, S. E. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.120 · Cited by 9,232 · see citing works
On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., & Samek, W. (2015) · PLOS ONE · doi:10.1371/journal.pone.0130140 · Cited by 4,524 · see citing works
Random Erasing Data Augmentation
Zhong, Z., Zheng, L., Kang, G., Li, S., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.7000 · Cited by 3,617 · see citing works
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · see citing works
Positioning and Presenting Design Science Research for Maximum Impact1
Gregor, S. D., & Hevner, A. R. (2013) · MIS Quarterly · doi:10.25300/misq/2013/37.2.01 · Cited by 3,154 · see citing works
Genomic classification of cutaneous melanoma
Akbani, R., Akdemir, K. C., Aksoy, B. A., Albert, M., Ally, A., Amin, S. B., Arachchi, H., Arora, A., Auman, J. T., Ayala, B., et al. (2015) · Cell · doi:10.1016/j.cell.2015.05.044 · Cited by 3,151 · see citing works
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · see citing works
Searching for activation functions
Ramachandran, P., Zoph, B., & Le, Q. V. (2017) · arXiv · doi:10.48550/arxiv.1710.05941 · Cited by 2,335 · see citing works
A method for normalizing histology slides for quantitative analysis
Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J. T., Guan, X., Schmitt, C., & Thomas, N. E. (2009) · 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro · doi:10.1109/isbi.2009.5193250 · Cited by 1,482 · see citing works
Design Research in Information Systems: Theory and Practice
Hevner, A., & Chatterjee, S. (2010) · Integrated Series in Information Systems · doi:10.1007/978-1-4419-5653-8 · Cited by 1,382 · see citing works
Sanity Checks for Saliency Maps
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018) · arXiv · doi:10.48550/arxiv.1810.03292 · Cited by 1,294 · see citing works
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 13× · see citing works
“Anders et al. [23] recently argued that global XAI methods (such as TCAV) are inappropriate to discover unknown biases.”
MethodInfluentialKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240 · see citing works
Shall we Really do it Again? The Powerful Concept of Replication is Neglected in the Social Sciences
Schmidt, S. (2009) · Review of General Psychology · doi:10.1037/a0015108 · Cited by 1,125 · see citing works
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K.-R. (2019) · Nature Communications · doi:10.1038/s41467-019-08987-4 · Cited by 1,047 · see citing works
Explainable artificial intelligence: an analytical review
Angelov, P. P., Soares, E. A., Jiang, R., Arnold, N. I., & Atkinson, P. M. (2021) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1424 · Cited by 843 · see citing works
Improving Deep Learning with Generic Data Augmentation
Taylor, L., & Nitschke, G. (2018) · 2018 IEEE Symposium Series on Computational Intelligence (SSCI) · doi:10.1109/ssci.2018.8628742 · Cited by 796 · see citing works
Deep neural network models for computational histopathology: A survey
Srinidhi, C. L., Ciga, O., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101813 · Cited by 749 · see citing works
Tellez, D., Litjens, G., Bándi, P., Bulten, W., Bokhorst, J.-M., Ciompi, F., & van der Laak, J. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101544 · Cited by 649 · see citing works
Unmasking Clever Hans predictors and assessing what machines really learn
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., & Müller, K. (2019) · RePEc: Research Papers in Economics · Cited by 580 · see citing works
Transparency and reproducibility in artificial intelligence
Haibe-Kains, B., Adam, G. A., Hosny, A., Khodakarami, F., Massive Analysis Quality Control (MAQC) Society Board of Directors, Shraddha, T., Kusko, R., Sansone, S.-A., Tong, W., Wolfinger, R. D., et al. (2020) · Nature · doi:10.1038/s41586-020-2766-y · Cited by 540 · see citing works
Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making
Cai, C. J., Reif, E., Hegde, N., Hipp, J., Kim, B., Smilkov, D., Wattenberg, M., Viegas, F., Corrado, G. S., Stumpe, M. C., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300234 · Cited by 405 · see citing works
Medical Image Computing and Computer Assisted Intervention – MICCAI 2019
Shen, D. G., Liu, T., Peters, T. M., Staib, L. H., Essert, C., Zhou, S., Yap, P., & Khan, A. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32251-9 · Cited by 395 · see citing works
Towards Automatic Concept-based Explanations
Ghorbani, A., Wexler, J., Zou, J., & Kim, B. (2019) · arXiv · doi:10.48550/arxiv.1902.03129 · Cited by 378 · see citing works
AI in medicine must be explainable
Kundu, S. (2021) · Nature Medicine · doi:10.1038/s41591-021-01461-z · Cited by 318 · see citing works
Four principles of explainable artificial intelligence
Phillips, P. J., Hahn, C. A., Fontana, P. C., Yates, A. N., Greene, K., Broniatowski, D. A., & Przybocki, M. A. (2021) · doi:10.6028/nist.ir.8312 · Cited by 262 · see citing works
An Introduction to Design Science
Johannesson, P., & Perjons, E. (2021) · Springer eBooks · doi:10.1007/978-3-030-78132-3 · Cited by 236 · see citing works
Metascience could rescue the ‘replication crisis’
Schooler, J. W. (2014) · Nature · doi:10.1038/515009a · Cited by 226 · see citing works
Design Science Research Methods and Patterns
Vaishnavi, V. K., Vaishnavi, V. K., & Kuechler, W. L., Jr. (2015) · CRC Press eBooks · doi:10.1201/b18448 · Cited by 191 · see citing works
Hägele, M., Seegerer, P., Lapuschkin, S., Bockmayr, M. L., Samek, W., Klauschen, F., Müller, K.-R., & Binder, A. (2020) · Scientific Reports · doi:10.1038/s41598-020-62724-2 · Cited by 185 · see citing works
ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases
Stock, P., & Cissé, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01231-1_31 · Cited by 166 · see citing works
Thomas, S. M., Lefevre, J., Baxter, G. D., & Hamilton, N. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101915 · Cited by 165 · see citing works
Analyzing and Reducing the Damage of Dataset Bias to Face Recognition With Synthetic Data
Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., & Vetter, T. R. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00279 · Cited by 156 · see citing works
Determining breast cancer biomarker status and associated morphological features using deep learning
Gamble, P., Jaroensri, R., Wang, H., Tan, F., Moran, M., Brown, T., Flament-Auvigne, I., Rakha, E. A., Toss, M., Dabbs, D. J., et al. (2021) · Communications Medicine · doi:10.1038/s43856-021-00013-3 · Cited by 154 · see citing works
Similar image search for histopathology: SMILY
Hegde, N., Hipp, J. D., Liu, Y., Emmert-Buck, M., Reif, E., Smilkov, D., Terry, M., Cai, C. J., Amin, M. B., Mermel, C. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0131-z · Cited by 153 · see citing works
Maaß, W., Parsons, J., Purao, S., Storey, V. C., & WOO, C. C. (2018) · Journal of the Association for Information Systems · doi:10.17705/1jais.00526 · Cited by 149 · see citing works
On Completeness-aware Concept-Based Explanations in Deep Neural NetworksPreprint
Yeh, C., Kim, B., Arık, S. Ö., Li, C., Pfister, T., & Ravikumar, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07969 · Cited by 144 · see citing works
Finding and removing Clever Hans: Using explanation methods to debug and improve deep models
Anders, C. J., Weber, L., Neumann, D., Samek, W., Müller, K., & Lapuschkin, S. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.015 · Cited by 122 · see citing works
Srinivasan, R., & Chander, A. (2021) · Communications of the ACM · doi:10.1145/3464903 · Cited by 109 · see citing works
Examining CNN Representations With Respect to Dataset Bias
Zhang, Q., Wang, W., & Zhu, S. C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11833 · Cited by 95 · see citing works
Concept attribution: explaining cnn decisions to physicians
M., G., V., A., S., M.-M., & H., M. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103865 · Cited by 95 · see citing works
Regression Concept Vectors for Bidirectional Explanations in Histopathology
Graziani, M., Andrearczyk, V., & Müller, H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-02628-8_14 · Cited by 89 · see citing works
(De) Constructing Bias on Skin Lesion Datasets
Bissoto, A. E., Fornaciali, M. S., Junior, E. A. D. V., & Avila, S. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00335 · Cited by 89 · see citing works
Deep Neural Network or Dermatologist?
Young, K. C., Booth, G., Simpson, B., Dutton, R., & Shrapnel, S. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33850-3_6 · Cited by 89 · see citing works
Global and Local Interpretability for Cardiac MRI Classification
Clough, J. R., Oksuz, I., Puyol-Antón, E., Ruijsink, B., King, A. P., & Schnabel, J. A. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_72 · Cited by 73 · see citing works
InsideBias: Measuring Bias in Deep Networks and Application to Face Gender Biometrics
Serna, I., Peña, A., Morales, A., & Fiérrez, J. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412443 · Cited by 73 · see citing works
Thakoor, K. A., Koorathota, S., Hood, D. C., & Sajda, P. (2020) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2020.3043215 · Cited by 72 · see citing works
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
Wang, A., Narayanan, A., & Russakovsky, O. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58580-8_43 · Cited by 72 · see citing works
Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectors
Zhang, R., Madumal, P., Miller, T., Ehinger, K. A., & Rubinstein, B. I. P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i13.17389 · Cited by 70 · see citing works
Developing image analysis methods for digital pathology
Bankhead, P. (2022) · The Journal of Pathology · doi:10.1002/path.5921 · Cited by 68 · see citing works
On Interpretability of Deep Learning based Skin Lesion Classifiers using Concept Activation Vectors
Lucieri, A., Bajwa, M. N., Braun, S. A., Malik, M. I., Dengel, A. R., & Ahmed, S. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206946 · Cited by 67 · see citing works
Kim, Y., Kim, S., Cho, C. E., Song, I. H., Lee, H. J., Ahn, S., Park, S. Y., Gong, G., & Kim, N. (2020) · Scientific Reports · doi:10.1038/s41598-020-78129-0 · Cited by 64 · see citing works
Empirically Analyzing the Effect of Dataset Biases on Deep Face Recognition Systems
Kortylewski, A., Egger, B., Schneider, A., Gerig, T., Morel-Forster, A., & Vetter, T. R. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00283 · Cited by 62 · see citing works
Detecting Bias with Generative Counterfactual Face Attribute AugmentationPreprint
Denton, E., Hutchinson, B., Mitchell, M., & Gebru, T. (2019) · arXiv (Cornell University) · Cited by 61 · see citing works
Schmitt, M., Maron, R. C., Hekler, A., Stenzinger, A., Hauschild, A., Weichenthal, M., Tiemann, M., Krahl, D., Kutzner, H., Utikal, J. S., et al. (2021) · Journal of Medical Internet Research · doi:10.2196/23436 · Cited by 59 · see citing works
A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts
Ge, Y., Xiao, Y., Xu, Z., Zheng, M., Karanam, S., Chen, T., Itti, L., & Wu, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00223 · Cited by 49 · see citing works
Korbar, B., Olofson, A. M., Miraflor, A. P., Nicka, C. M., Suriawinata, M. A., Torresani, L., Suriawinata, A. A., & Hassanpour, S. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.114 · Cited by 47 · see citing works
Towards Causal Benchmarking of Bias in Face Analysis Algorithms
Balakrishnan, G., Xiong, Y., Xia, W., & Perona, P. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58523-5_32 · Cited by 45 · see citing works
Janik, A., Dodd, J. D., Ifrim, G., Sankaran, K., & Curran, K. M. (2021) · SPIE Medical Imaging: Image Processing · doi:10.1117/12.2582227 · Cited by 45 · see citing works
Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers
Palatnik de Sousa, I., Vellasco, M. M. B. R., & Costa da Silva, E. (2021) · Sensors · doi:10.3390/s21165657 · Cited by 43 · see citing works
Towards Global Explanations of Convolutional Neural Networks With Concept Attribution
Wu, W., Su, Y., Chen, X., Zhao, S., King, I., Lyu, M. R.-T., & Tai, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00868 · Cited by 41 · see citing works
Testing DNN image classifiers for confusion & bias errors
Tian, Y., Zhong, Z., Ordonez, V., Kaiser, G. E., & Ray, B. (2020) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3377811.3380400 · Cited by 39 · see citing works
Deep Learning of Histopathological Features for the Prediction of Tumour Molecular Genetics
Murchan, P., O’Brien, C. P., O’Connell, S., McNevin, C. S., Baird, A., Sheils, O. M., Broin, P. Ó., & Finn, S. P. (2021) · Diagnostics · doi:10.3390/diagnostics11081406 · Cited by 39 · see citing works
Bias Analysis on Public X-Ray Image Datasets of Pneumonia and COVID-19 Patients
Catalá, O. D. T., Igual, I. S., Pérez‐Benito, F. J., Escrivá, D. M., Castelló, V. O., Llobet, R., & Pérez-Cortés, J.-C. (2021) · IEEE Access · doi:10.1109/access.2021.3065456 · Cited by 35 · see citing works
Discover the Unknown Biased Attribute of an Image Classifier
Li, Z., & Xu, C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01470 · Cited by 34 · see citing works
Image Counterfactual Sensitivity Analysis for Detecting Unintended BiasPreprint
Denton, E., Hutchinson, B., Mitchell, M., Gebru, T., & Zaldivar, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06439 · Cited by 34 · see citing works
Mikołajczyk, A., Grochowski, M., & Kwasigroch, A. (2020) · Journal of Artificial Intelligence and Soft Computing Research · doi:10.2478/jaiscr-2021-0004 · Cited by 32 · see citing works
Diefenbach, S., & Ullrich, D. (2015) · Interacting with Computers · doi:10.1093/iwc/iwv001 · Cited by 29 · see citing works
UBS: A Dimension-Agnostic Metric for Concept Vector Interpretability Applied to Radiomics
Yèche, H., Harrison, J., & Berthier, T. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33850-3_2 · Cited by 27 · see citing works
How much off-the-shelf knowledge is transferable from natural images to pathology images?
Li, X., & Plataniotis, K. N. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240530 · Cited by 27 · see citing works
Fang, Z., Kuang, K., Lin, Y., Wu, F. Q., & Yao, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413557 · Cited by 27 · see citing works
Now You See Me (CME): Concept-based Model ExtractionPreprint
Kazhdan, D., Dimanov, B., Jamnik, M., Lió, P., & Weller, A. (2020) · arXiv (Cornell University) · doi:10.17863/cam.57507 · Cited by 27 · see citing works
Towards Causal Benchmarking of Biasin Face Analysis Algorithms
Balakrishnan, G., Xiong, Y., Xia, W., & Perona, P. (2021) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-74697-1_15 · Cited by 26 · see citing works
Improved interpretability for computer-aided severity assessment of retinopathy of prematurity
Brown, J. M., Andrearczyk, V., Yıldız, V., Deniz Erdogmus, B., Ioannidis, S., Chiang, M. F., Kalpathy–Cramer, J., Müller, H., Campbell, J. P., & Graziani, M. (2019) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2512584 · Cited by 25 · see citing works
Concept-based model explanations for electronic health records
Mincu, D., Loreaux, E., Hou, S., Baur, S., Protsyuk, I., Seneviratne, M. G., Mottram, A., Tomasev, N., Karthikesalingam, A., & Schrouff, J. (2021) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3450439.3451858 · Cited by 25 · see citing works
Now You See Me (CME): Concept-based Model ExtractionPreprint
Kazhdan, D., Dimanov, B., Jamnik, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.13233 · Cited by 22 · see citing works
PACE: Posthoc Architecture-Agnostic Concept Extractor for Explaining CNNs
Kamakshi, V., Gupta, U., & Krishnan, N. C. (2021) · IEEE International Joint Conference on Neural Network · doi:10.1109/ijcnn52387.2021.9534369 · Cited by 20 · see citing works
Determining the Relevance of Features for Deep Neural Networks
Reimers, C., Runge, J., & Denzler, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58574-7_20 · Cited by 19 · see citing works
Reimers, C., Penzel, N., Bodesheim, P., Runge, J., & Denzler, J. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00200 · Cited by 17 · see citing works
Graziani, M., de Sousa, I. P., Vellasco, M. M. B. R., da Silva, E. C., Müller, H., & Andrearczyk, V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_51 · Cited by 16 · see citing works
Debiasing Concept-based Explanations with Causal AnalysisPreprint
Bahadori, M. T., & Heckerman, D. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.11500 · Cited by 15 · see citing works
Revisiting Sanity Checks for Saliency MapsPreprint
Yona, G., & Greenfeld, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14297 · Cited by 14 · see citing works
UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models
Krishnakumar, A., Prabhu, V., Sudhakar, S., & Hoffman, J. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.102 · Cited by 12 · see citing works
Evaluation and Comparison of CNN Visual Explanations for Histopathology
Graziani, M., Lompech, T., Müller, H., & Andrearczyk, V. (2021) · ArODES (HES-SO (https://www.hes-so.ch/)) · doi:10.5281/zenodo.4545760 · Cited by 11 · see citing works
Investigating Bias in Image Classification using Model ExplanationsPreprint
Tong, S., & Kagal, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.05463 · Cited by 8 · see citing works
Hierarchical Visual Concept Interpretation for Medical Image Classification
Khaleel, M., Tavanapong, W., Wong, J., Oh, J., & de Groen, P. (2021) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms52027.2021.00012 · Cited by 8 · see citing works
Reducing the Annotation Cost of Whole Slide Histology Images using Active Learning
Xu, J., An, H., Wang, J., Wen, K., & Wu, Z. (2021) · International Conference on Image Processing and Machine Vision (IPMV) · doi:10.1145/3469951.3469960 · Cited by 7 · see citing works
Histopathology Non-Melanoma Skin Cancer Segmentation Dataset
Thomas, S. M., Hamilton, N., & Thomas, S. M. (2021) · doi:10.14264/8be4bd0 · Cited by 5 · see citing works
Sanity Checks for Saliency Metrics
Tomsett, R., Harborne, D., Chakraborty, S., Gurram, P., & Preece, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i04.6064 · Cited by 3 · see citing works
X-BaD: A Flexible Tool for Explanation-Based Bias Detection
Pacini, M., Nesti, F., Biondi, A., & Buttazzo, G. (2021) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr51186.2021.9527956 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [24]Anders C.J. Neumann D. Marinc T. Samek W. Müller K.-R. Lapuschkin S. XAI for Analyzing and Unlearning Spurious Correlations in ImageNet Proceedings of the 2020 ICML Workshop on Extending Explainable AI Beyond Deep Models and Classifiers (XXAI) Vienna, Austria 17 July 2020
- [91]Maas A.L. Hannun A.Y. Ng A.Y. Rectifier Nonlinearities Improve Neural Network Acoustic Models Proceedings of the 30th International Conference on Machine Learning Atlanta, GA, USA 16–21 June 2013
Cited by
Sorted by citations
Borys, K., Schmitt, Y. A., Nauta, M., Seifert, C., Krämer, N., Friedrich, C. M., & Nensa, F. (2023) · European Journal of Radiology · doi:10.1016/j.ejrad.2023.110786 · Cited by 233 · see citing works
Self-citationRecent Applications of Explainable AI (XAI): A Systematic Literature Review
Saarela, M., & Podgorelec, V. (2024) · Applied Sciences · doi:10.3390/app14198884 · Cited by 219 · see citing works
Explainable AI for Medical Data: Current Methods, Limitations, and Future Directions
Hossain, M. I., Zamzmi, G., Mouton, P. R., Salekin, M. S., Sun, Y., & Goldgof, D. B. (2023) · ACM Computing Surveys · doi:10.1145/3637487 · Cited by 120 · see citing works
Sauter, D., Lodde, G. C., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107083 · Cited by 33 · see citing works
Self-citationToumaj, S., Heidari, A., & Navimipour, N. J. (2025) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2025.103243 · Cited by 36 · see citing works
A concept-based interpretable model for the diagnosis of choroid neoplasias using multimodal data
Wu, Y., Liu, Y., Yang, Y., Yao, M. S., Yang, W., Shi, X., Yang, L., Li, D., Liu, Y., Yin, S., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58801-7 · Cited by 21 · see citing works
Navigating the landscape of concept-supported XAI: Challenges, innovations, and future directions
Khoozani, Z. S., Sabri, A. Q. M., Seng, W. C., Seera, M., & Eg, K. Y. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17666-y · Cited by 18 · see citing works
Pathologist-like explainable AI for interpretable Gleason grading in prostate cancer
Mittmann, G., Laiouar-Pedari, S., Mehrtens, H. A., Haggenmüller, S., Bucher, T.-C., Chanda, T., Gaisa, N. T., Wagner, M., Klamminger, G. G., Rau, T. T., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64712-4 · Cited by 12 · see citing works
Manz, R., Bäcker, J., Cramer, S., Meyer, P., Müller, D., Muzalyova, A., Rentschler, L., Wengenmayr, C., Hinske, L. C., Huss, R., et al. (2025) · The Journal of Pathology Clinical Research · doi:10.1002/2056-4538.70023 · Cited by 10 · see citing works
XAI Unveiled: Revealing the Potential of Explainable AI in Medicine - A Systematic Review
Scarpato, N., Ferroni, P., & Guadagni, F. (2024) · IEEE Access · doi:10.1109/access.2024.3514197 · Cited by 7 · see citing works
Cocuz, I. G., Niculescu, R., Popelea, M. C., Cocuz, M. E., Sabău, A.-H., Tinca, A. C., Cozac-Szőke, A. R., Chiorean, D. M., Budin, C. E., & Cotoi, O. S. (2025) · Diagnostics · doi:10.3390/diagnostics15172196 · Cited by 7 · see citing works
Borra, S., Dey, N., Fong, S., Sherratt, R. S., & Shi, F. (2026) · Cancers · doi:10.3390/cancers18091361 · Cited by 8 · see citing works
Foundational approaches to post-hoc explainability for image classification
Santis, A. D., Campi, R., Bianchi, M., Tocchetti, A., & Brambilla, M. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-340553-2.00008-3 · Cited by 5 · see citing works
Machine learning to detect melanoma exploiting nuclei morphology and Spatial organization
Veronesi, G., Curti, N., Gardini, A., Querzoli, G., Castellani, G., & Dika, E. (2025) · Scientific Reports · doi:10.1038/s41598-025-02913-z · Cited by 6 · see citing works
Allen, K., Yawson, A. K., Haggenmüller, S., Kather, J. N., & Brinker, T. J. (2024) · ESMO Real World Data and Digital Oncology · doi:10.1016/j.esmorw.2024.100077 · Cited by 6 · see citing works
A Review of Artificial Intelligence in Dermatopathology
Prestwood, C. A., Gibbs, D. C., Wolner, Z. J., & Stoff, B. K. (2025) · Dermatologic Clinics · doi:10.1016/j.det.2025.05.006 · Cited by 3 · see citing works
Artificial Intelligence in Melanoma Dermatopathology: A Review of Literature
Neimy, H., Helmy, J., Snyder, A., & Valdebran, M. (2023) · American Journal of Dermatopathology · doi:10.1097/dad.0000000000002593 · Cited by 5 · see citing works
Alharith, R., zhang, J., Ibrahim, A. O., & Wu, Z. (2026) · International Journal of Computer Vision · doi:10.1007/s11263-025-02647-5 · Cited by 2 · see citing works
Pesecan, C.-M., & Stoicu-Tivadar, L. (2024) · Studies in health technology and informatics · doi:10.3233/shti240579 · Cited by 2 · see citing works
Kakish, D. R. K., AlSamhori, J. F., Qaqish, L. N., Aburumman, L., Sarsur, R., Salkhadi, A. A., Nassif, Z. B., Akmal, M. A., & Nashwan, A. J. (2025) · Dermatological Reviews · doi:10.1002/der2.70022 · Cited by 3 · see citing works
Şengöz, N. (2022) · El-Cezeri Fen ve Mühendislik Dergisi · doi:10.31202/ecjse.1159621 · Cited by 1 · see citing works
Cramer, S., de Llanes, E. G., Bäcker, J., Manz, R., Meyer, P., Müller, D., Hinske, L. C., Raffler, J., & Soto‐Rey, I. (2025) · doi:10.2196/preprints.73124
AI-Driven Deep Learning Models for Breast Cancer Histopathology Image Classification
Oktavianiva, R., Almohab, H., & Kusnadi, A. (2025) · IEEE International Symposium on Consumer Electronics (ISCE) · doi:10.1109/isct66099.2025.11297317
Explainable AI for HCI: a systematic review of a decade of studies on XAI tools
Alaqsam, A., & Sas, C. (2026) · Interacting with Computers · doi:10.1093/iwc/iwag022
Explainable Deep Learning for Imaging‐Based Skin Lesion Diagnosis: A Systematic Literature Review
Dakhli, R., & Barhoumi, W. (2026) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.70885
Does Synthetic Image Generation Improve Explainability? A Concept-Based Case Study in Histopathology
Moreno-Sánchez, P. A., Vitoria, I., López-Saratxaga, C., Benito-Del-Valle, L., & Galdrán, A. (2026) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms69103.2026.00089
- Authors
- Sauter, D., Lodde, G., Nensa, F., Schadendorf, D., Livingstone, E., & Kukuk, M.
- Affiliations
- Dortmund University of Applied Sciences and Arts (DE)
- Essen University Hospital (DE)
- Corresponding author
- Sauter, Daniel
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2022-07-18
- Volume
- 22
- Issue
- 14
- Pages
- 5346
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC9319808
- OpenAlex
- W4285803300
- Cited by
- 26
- References
- 92
- Citations by year
- 2026: 5 · 2025: 11 · 2024: 5 · 2023: 4 · 2022: 1
- FWCI
- 1.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DuEPublico (University of Duisburg-Essen) (submittedVersion, cc-by)
- Funders
- DFG Research Training Group (WisPerMed)
- University of Duisburg-Essen (WisPerMed)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Oncology › Cutaneous Melanoma Detection and Management
- Keywords
- digital histopathology, explainable AI, skin cancer classification, dermatopathology, bias detection, measurement bias, sampling bias, Grad-CAM, convolutional neural networks, model interpretability
- MeSH terms
- Artificial Intelligence, Humans, Skin Neoplasms, Reproducibility of Results, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
