CoChat
Sign Up
Cited by 302Explainable Artificial Intelligence (XAI)Artificial Intelligence in Healthcare and EducationUsability and User Interface Design

Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review

Open access
Chen, Haomin, Gomez, Catalina, Huang, Chien-Ming, Unberath, Mathias·2022·npj Digital Medicine
DOI 10.1038/s41746-022-00699-2PMC PMC9581990
Open this paper in CoChat — free accountOpen PDF

Abstract

Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41746-022-00699-2

Cites

101 of 101 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. [26]McCoy LG Brenna CT Chen S Vold K Das S Believing in black boxes: Machine learning for healthcare does not need explainability to be evidence-based J. Clin. Epidemiol 2021 142 252 257 10.1016/j.jclinepi.2021.11.001 34748907 · DOI · Paper page
  2. [28]Banerjee, I. et al. Reading race: Ai recognises patient’s racial identity in medical images. preprint at https://arxiv.org/abs/2107.10356 (2021). · arXiv
  3. [30]Lu, M. Y. et al. Deep learning-based computational pathology predicts origins for cancers of unknown primary. preprint at https://arxiv.org/abs/2006.13932 (2020). · arXiv
  4. [40]Datta, A., Tschantz, M. C. & Datta, A. Automated experiments on ad privacy settings: A tale of opacity, choice, and discrimination. preprint at https://arxiv.org/abs/1408.6491 (2014). · arXiv
  5. [56]Afshar, P. et al. MIXCAPS: A capsule network-based mixture of experts for lung nodule malignancy prediction. Pattern Recognition116, 10.1016/j.patcog.2021.107942NS (2021). · DOI
  6. [57]Fan M Chakraborti T Chang EIC Xu Y Rittscher J Microscopic Fine-Grained Instance Classification Through Deep Attention 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, October 4, 2020 - October 8, 2020 2020 12265 LNCS 490 499
  7. [58]Graziani M Lompech T Muller H Depeursinge A Andrearczyk V Interpretable CNN Pruning for Preserving Scale-Covariant Features in Medical Imaging 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop o 2020 12446 LNCS 23 32
  8. [61]Hou, B., Kang, G., Xu, X. & Hu, C. Cross Attention Densely Connected Networks for Multiple Sclerosis Lesion Segmentation. 2019 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2019, November 18, 2019 - November 21, 2019 2356–2361, 10.1109/BIBM47256.2019.8983149NS (2019). · DOI
  9. [62]Huang Y Chung ACS Evidence localization for pathology images using weakly supervised learning 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 2019 11764 LNCS 613 621
  10. [67]Sun J Darbehani F Zaidi M Wang B SAUNet: Shape Attentive U-Net for Interpretable Medical Image Segmentation 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2020, October 4, 2020 - October 8, 2020 2020 12264 LNCS 797 806
  11. [72]Giannini V Rosati S Regge D Balestra G Texture features and artificial neural networks: A way to improve the specificity of a CAD system for multiparametric MR prostate cancer 14th Mediterranean Conference on Medical and Biological Engineering and Computing, MEDICON 2016, March 31, 2016 - April 2, 2016 2016 57 296 301
  12. [77]Li, J., Shi, H. & Hwang, K.-S. An explainable ensemble feedforward method with Gaussian convolutional filter. Knowl.-Based Syst.225, 10.1016/j.knosys.2021.107103NS (2021). · DOI
  13. [81]Kim, S. T., Lee, H., Kim, H. G. & Ro, Y. M. ICADx: Interpretable computer aided diagnosis of breast masses. Medical Imaging 2018: Computer-Aided Diagnosis, February 12, 2018 - February 15, 201810575, DECTRIS Ltd.; The Society of Photo–Optical Instrum. 10.1117/12.2293570NS (2018). · DOI
  14. [86]Barata, C., Celebi, M. E. & Marques, J. S. Explainable skin lesion diagnosis using taxonomies. Pattern Recognition110, 10.1016/j.patcog.2020.107413NS (2021). · DOI
  15. [87]Silva W Fernandes K Cardoso MJ Cardoso JS Towards complementary explanations using deep neural networks 1st International Workshop on Machine Learning in Clinical Neuroimaging, MLCN 2018, 1st International Workshop on Deep Learning Fails, DLF 2018, and 1st International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2018 11038 LNCS 133 140
  16. [88]Khaleel M Tavanapong W Wong J Oh J De Groen P Hierarchical visual concept interpretation for medical image classification 34th IEEE International Symposium on Computer-Based Medical Systems, CBMS 2021, June 7, 2021 - June 9, 2021 2021 2021-June 25 30
  17. [90]Yan, K. et al. Holistic and comprehensive annotation of clinically significant findings on diverse CT images: Learning from radiology reports and label ontology. 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2019, June 16, 2019 - June 20, 2019 2019-June, 8515–8524, 10.1109/CVPR.2019.00872NS (2019). · DOI
  18. [92]Verma A Shukla P Verma S An interpretable SVM based model for cancer prediction in mammograms 1st International Conference on Communication, Networks and Computing, CNC 2018, March 22, 2018 - March 24, 2018 2019 839 443 451
  19. [96]Folke, T., Yang, S. C.-H., Anderson, S. & Shafto, P. Explainable AI for medical imaging: Explaining pneumothorax diagnoses with Bayesian teaching. Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III 2021, April 12, 2021 - April 16, 202111746, The Society of Photo–Optical Instrumentation Engin. 10.1117/12.2585967NS (2021). · DOI
  20. [98]Shinde S Chougule T Saini J Ingalhalikar M HR-CAM: Precise localization of pathology using multi-level learning in CNNS 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 2019 11767 LNCS 298 306
  21. [99]Ballard DH Modular learning in neural networks AAAI 1987 647 279 284
  22. [101]Couteaux V Nempont O Pizaine G Bloch I Towards interpretability of segmentation networks by analyzing deepDreams 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 2019 11797 LNCS 56 63
  23. [103]Janik, A., Dodd, J., Ifrim, G., Sankaran, K. & Curran, K. Interpretability of a deep learning model in the application of cardiac MRI segmentation with an ACDC challenge dataset. Medical Imaging 2021: Image Processing, February 15, 2021 - February 19, 2021 11596, The Society of Photo–Optical Instrumentation Engin. 10.1117/12.2582227NS (2021). · DOI
  24. [106]Zhu P Ogino M Guideline-based additive explanation for computer-aided diagnosis of lung nodules 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 2019 11797 LNCS 39 47
  25. [107]Pirovano A Heuberger H Berlemont S Ladjal S Bloch I Improving interpretability for computer-aided diagnosis tools on whole slide imaging with multiple instance learning and gradient-based explanations 3rd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2020, the 2nd International Workshop on Medical Image Learning with Less Labels and Imperfect Data, MIL3ID 2020, and the 5th International Workshop o 2020 12446 LNCS 43 53
  26. [109]de Sousa IP Vellasco MMBR da Silva EC Approximate explanations for classification of histopathology patches Workshops of the 20th Joint European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD, September 14, 2020 - September 18, 2020 2020 1323 517 526
  27. [112]Uzunova, H., Ehrhardt, J., Kepp, T. & Handels, H. Interpretable explanations of black box classifiers applied on medical images by meaningful perturbations using variational autoencoders. Medical Imaging 2019: Image Processing, February 19, 2019 - February 21, 201910949, The Society of Photo–Optical Instrumentation Engin. 10.1117/12.2511964NS (2019). · DOI
  28. [116]Peng T Boxberg M Weichert W Navab N Marr C Multi-task learning of a deep K-nearest neighbour network for histopathological image classification and retrieval 22nd International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2019, October 13, 2019 - October 17, 2019 2019 11764 LNCS 676 684
  29. [123]Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. preprint at https://arxiv.org/abs/1702.08608 (2017). · arXiv
  30. [125]Yeche H Harrison J Berthier T UBS: A dimension-agnostic metric for concept vector interpretability applied to radiomics 2nd International Workshop on Interpretability of Machine Intelligence in Medical Image Computing, iMIMIC 2019, and the 9th International Workshop on Multimodal Learning for Clinical Decision Support, ML-CDS 2019, held in conjunction with the 22nd Interna 2019 11797 LNCS 12 20
  31. [126]Chen, H., Miao, S., Xu, D., Hager, G. D. & Harrison, A. P. Deep hierarchical multi-label classification of chest x-ray images. In International Conference on Medical Imaging with Deep Learning, 109–120 (PMLR, 2019).

Cited by

Sorted by citations

Related

Authors
Chen, H., Gomez, C., Huang, C.-M., & Unberath, M.
Affiliations
Johns Hopkins University (US)
Corresponding author
Unberath, Mathias
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2022-10-19
Volume
5
Issue
1
Pages
156-156
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-022-00699-2
PMC
PMC9581990
OpenAlex
W4306790271
Cited by
300
References
101
Citations by year
2026: 97 · 2025: 94 · 2024: 67 · 2023: 38 · 2022: 4
FWCI
3.61
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
Keywords
explainable AI, machine learning transparency, human-centered design, medical image analysis, user-centered design, user evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review