CoChat
Sign Up
Cited by 302Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationUsability and User Interface Design

The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index.

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Open access
Chen, Haomin, Gomez, Catalina, Huang, Chien-Ming, Unberath, Mathias·2022·npj Digital Medicine
DOI 10.1038/s41746-022-00699-2PMC PMC9581990
Open this paper in CoChat — free accountOpen PDF

Abstract

Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00699-2

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Schematic representation of the INTRPRT guideline within the main stages of a human-centered design process. The blue boxes demonstrate the process from understanding end users and their context to the validation of the developed system, which ultimately might result in large scale deployment. The guidelines are summarized below and are located within the design phases based on the aspects pertinent to each one and the corresponding themes of each guideline are listed on the left. Opportunities for iterative design are illustrated with the dashed arrows.
Fig. 1 Schematic representation of the INTRPRT guideline within the main stages of a human-centered design process. The blue boxes demonstrate the process from understanding end users and their context to the validation of the developed system, which ultimately might result in large scale deployment. The guidelines are summarized below and are located within the design phases based on the aspects pertinent to each one and the corresponding themes of each guideline are listed on the left. Opportunities for iterative design are illustrated with the dashed arrows.CC BY · PubMed Central
Fig. 2 Illustrative examples of different techniques used in transparent ML systems for classification and segmentation tasks from the systematic review. Within each task, a non-interpretable model generates the task outcome from the input image (top). The use of clinical knowledge or computer vision information as priors attempts to add transparency in the outcome generation process (bottom). Images retrieved from the ORIGA127 and BraTS2020 datasets128.
Fig. 2 Illustrative examples of different techniques used in transparent ML systems for classification and segmentation tasks from the systematic review. Within each task, a non-interpretable model generates the task outcome from the input image (top). The use of clinical knowledge or computer vision information as priors attempts to add transparency in the outcome generation process (bottom). Images retrieved from the ORIGA127 and BraTS2020 datasets128.CC BY · PubMed Central
Fig. 3 Relationships of components in ML systems. a Shows the relationship between the clinical scenario and requirements in non-transparent ML systems, where the system may not be human-facing, and as such, meeting quantitative performance requirements is sufficient. b Shows the relationship between the clinical scenario, requirements, and end users in transparent ML systems, as they arise in a human-centered system that seeks to enable users to accomplish a certain task.
Fig. 3 Relationships of components in ML systems. a Shows the relationship between the clinical scenario and requirements in non-transparent ML systems, where the system may not be human-facing, and as such, meeting quantitative performance requirements is sufficient. b Shows the relationship between the clinical scenario, requirements, and end users in transparent ML systems, as they arise in a human-centered system that seeks to enable users to accomplish a certain task.CC BY · PubMed Central
Fig. 4 PRISMA diagram for transparent ML in medical imaging. The flow diagram shows the number of records identified, of studies excluded and the reasons for exclusion, and of studies included in our systematic review.
Fig. 4 PRISMA diagram for transparent ML in medical imaging. The flow diagram shows the number of records identified, of studies excluded and the reasons for exclusion, and of studies included in our systematic review.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 31

  1. McCoy, L. G., Brenna, C. T., Chen, S., Vold, K. & Das, S. Believing in black boxes: Machine learning for healthcare does not need explainability to be evidence-based. J. Clin. Epidemiol 142, 252–257 (2021). · DOI · Paper page
  2. Banerjee, I. et al. Reading race: Ai recognises patient’s racial identity in medical images. preprint at https://arxiv.org/abs/2107.10356 (2021). · arXiv
  3. Lu, M. Y. et al. Deep learning-based computational pathology predicts origins for cancers of unknown primary. preprint at https://arxiv.org/abs/2006.13932 (2020). · arXiv
  4. Datta, A., Tschantz, M. C. & Datta, A. Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. preprint at https://arxiv.org/abs/1408.6491 (2014). · arXiv
  5. Afshar, P. et al. MIXCAPS: A capsule network-based mixture of experts for lung nodule malignancy prediction. Pattern Recognition 116, https://doi.org/10.1016/j.patcog.2021.107942NS (2021). · DOI
  6. Fan, M., Chakraborti, T., Chang, E. I. C., Xu, Y. & Rittscher, J. Microscopic Fine-Grained Instance Classification Through Deep Attention. 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, October 4, 2020 - October 8, 2020 12265 LNCS, 490–499 (2020).
  7. Graziani, M., Lompech, T., Muller, H., Depeursinge, A. & Andrearczyk, V. Interpretable CNN Pruning for Preserving Scale-Covariant Features in Medical Imaging. 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop o 12446 LNCS, 23–32 (2020).
  8. Hou, B., Kang, G., Xu, X. & Hu, C. Cross Attention Densely Connected Networks for Multiple Sclerosis Lesion Segmentation. 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019, November 18, 2019 - November 21, 2019 2356–2361, https://doi.org/10.1109/BIBM47256.2019.8983149NS (2019). · DOI
  9. Huang, Y. & Chung, A. C. S. Evidence localization for pathology images using weakly supervised learning. 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 11764 LNCS, 613–621 (2019).
  10. Sun, J., Darbehani, F., Zaidi, M. & Wang, B. SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation. 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, October 4, 2020 - October 8, 2020 12264 LNCS, 797–806 (2020).
  11. Giannini, V., Rosati, S., Regge, D. & Balestra, G. Texture features and artificial neural networks: A way to improve the specificity of a CAD system for multiparametric MR prostate cancer. 14th Mediterranean Conference on Medical and Biological Engineering and Computing, MEDICON 2016, March 31, 2016 - April 2, 2016 57, 296–301 (2016).
  12. Li, J., Shi, H. & Hwang, K.-S. An explainable ensemble feedforward method with Gaussian convolutional filter. Knowl.-Based Syst. 225, https://doi.org/10.1016/j.knosys.2021.107103NS (2021). · DOI
  13. Kim, S. T., Lee, H., Kim, H. G. & Ro, Y. M. ICADx: Interpretable computer aided diagnosis of breast masses. Medical Imaging 2018: Computer-Aided Diagnosis, February 12, 2018 - February 15, 2018 10575, DECTRIS Ltd.; The Society of Photo–Optical Instrum. https://doi.org/10.1117/12.2293570NS (2018). · DOI
  14. Barata, C., Celebi, M. E. & Marques, J. S. Explainable skin lesion diagnosis using taxonomies. Pattern Recognition 110, https://doi.org/10.1016/j.patcog.2020.107413NS (2021). · DOI
  15. Silva, W., Fernandes, K., Cardoso, M. J. & Cardoso, J. S. Towards complementary explanations using deep neural networks. 1st International Workshop on Machine Learning in Clinical Neuroimaging, MLCN 2018, 1st International Workshop on Deep Learning Fails, DLF 2018, and 1st International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 11038 LNCS, 133–140 (2018).
  16. Khaleel, M., Tavanapong, W., Wong, J., Oh, J. & De Groen, P. Hierarchical visual concept interpretation for medical image classification. 34th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2021, June 7, 2021 - June 9, 2021 2021-June, 25–30 (2021).
  17. Yan, K. et al. Holistic and comprehensive annotation of clinically significant findings on diverse CT images: Learning from radiology reports and label ontology. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, June 16, 2019 - June 20, 2019 2019-June, 8515–8524, https://doi.org/10.1109/CVPR.2019.00872NS (2019). · DOI
  18. Verma, A., Shukla, P. & Verma, S. An interpretable SVM based model for cancer prediction in mammograms. 1st International Conference on Communication, Networks and Computing, CNC 2018, March 22, 2018 - March 24, 2018 839, 443–451 (2019).
  19. Folke, T., Yang, S. C.-H., Anderson, S. & Shafto, P. Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian teaching. Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III 2021, April 12, 2021 - April 16, 2021 11746, The Society of Photo–Optical Instrumentation Engin. https://doi.org/10.1117/12.2585967NS (2021). · DOI
  20. Shinde, S., Chougule, T., Saini, J. & Ingalhalikar, M. HR-CAM: Precise localization of pathology using multi-level learning in CNNS. 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 11767 LNCS, 298–306 (2019).
  21. Ballard, D. H. Modular learning in neural networks. AAAI 647, 279–284 (1987).
  22. Couteaux, V., Nempont, O., Pizaine, G. & Bloch, I. Towards interpretability of segmentation networks by analyzing deepDreams. 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 11797 LNCS, 56–63 (2019).
  23. Janik, A., Dodd, J., Ifrim, G., Sankaran, K. & Curran, K. Interpretability of a deep learning model in the application of cardiac MRI segmentation with an ACDC challenge dataset. Medical Imaging 2021: Image Processing, February 15, 2021 - February 19, 2021 11596, The Society of Photo–Optical Instrumentation Engin. https://doi.org/10.1117/12.2582227NS (2021). · DOI
  24. Zhu, P. & Ogino, M. Guideline-based additive explanation for computer-aided diagnosis of lung nodules. 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 11797 LNCS, 39–47 (2019).
  25. Pirovano, A., Heuberger, H., Berlemont, S., Ladjal, S. & Bloch, I. Improving interpretability for computer-aided diagnosis tools on whole slide imaging with multiple instance learning and gradient-based explanations. 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop o 12446 LNCS, 43–53 (2020).
  26. de Sousa, I. P., Vellasco, M. M. B. R. & da Silva, E. C. Approximate explanations for classification of histopathology patches. Workshops of the 20th Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD, September 14, 2020 - September 18, 2020 1323, 517–526 (2020).
  27. Uzunova, H., Ehrhardt, J., Kepp, T. & Handels, H. Interpretable explanations of black box classifiers applied on medical images by meaningful perturbations using variational autoencoders. Medical Imaging 2019: Image Processing, February 19, 2019 - February 21, 2019 10949, The Society of Photo–Optical Instrumentation Engin. https://doi.org/10.1117/12.2511964NS (2019). · DOI
  28. Peng, T., Boxberg, M., Weichert, W., Navab, N. & Marr, C. Multi-task learning of a deep K-nearest neighbour network for histopathological image classification and retrieval. 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 11764 LNCS, 676–684 (2019).
  29. Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. preprint athttps://arxiv.org/abs/1702.08608 (2017).
  30. Yeche, H., Harrison, J. & Berthier, T. UBS: A dimension-agnostic metric for concept vector interpretability applied to radiomics. 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 11797 LNCS, 12–20 (2019).
  31. Chen, H., Miao, S., Xu, D., Hager, G. D. & Harrison, A. P. Deep hierarchical multi-label classification of chest x-ray images. In International Conference on Medical Imaging with Deep Learning, 109–120 (PMLR, 2019).

Cited by

Sorted by citations

See all 302

Related

See all

Metadata

Authors
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M.
Affiliations
Johns Hopkins University (US)
Corresponding author
Unberath, Mathias
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2022-10-19
Volume
5
Issue
1
Pages
156-156
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-022-00699-2
PMC
PMC9581990
OpenAlex
W4306790271
Cited by
300
References
101
Citations by year
2026: 97 · 2025: 94 · 2024: 67 · 2023: 38 · 2022: 4
FWCI
3.61
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
explainable AI, machine learning transparency, human-centered design, medical image analysis, user-centered design, user evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review