CoChat

Is It Time to Get Rid of Black Boxes and Cultivate Trust in AI?

Open access
Gastounioti, Aimilia, Kontos, Despina · 2020 · Radiology: Artificial Intelligence

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

W a short time, artificial intelligence (AI) has taken center stage in radiology, with a large number of publications that is growing at a swift pace. Radiologists routinely assess medical images and report findings to detect, characterize, and monitor diseases. Such assessment is based both on education and experience and can be, at times, subjective. To augment such qualitative interpretation, AI excels at recognizing complex patterns in imaging data and can offer novel tools for quantitative medical image interpretation in an automated fashion. Deep learning, in particular, is a subset of AI that utilizes neural network architectures loosely inspired by the human brain. Although most of the earlier AI methods led to applications with subhuman performance, deep learning has recently demonstrated performance that matches and often surpasses humans (1–3). Recognizing its power, medical researchers and scientists have been exploring the role of deep learning methods in a variety of radiologic applications (4), including early detection, risk prediction, prognosis assessment, treatment planning, as well as lesion and organ segmentation, for various diseases. However, AI models are often considered “black boxes”; although we may be able to get accurate decisions and predictions, we cannot easily or clearly explain or interpret the logic behind an AI model’s outputs. But how can we gain useful insights? Which aspects of an AI model should we strive to elucidate, and what are the computational tools we need to achieve that? These questions are important to address when the issue of AI model interpretability or explainability is raised. Although interpretability can be a broad, poorly defined concept, in AI it can be thought of as the ability to explain or to present in understandable terms to a human the cause of a model’s decision. The more interpretable or explainable an AI model is, the easier it is for someone to comprehend why certain decisions or predictions have been made. The concept of interpretability in AI is not new. Inherently interpretable AI models, such as decision trees and decision rules, have received a great deal of attention from the research community. Moreover, several methods for feature importance analysis have been developed to offer insights into what the model is learning and what factors might be relevant. However, in the case of deep learning, the situation is more complicated. A key difference is that traditional AI systems primarily rely on predefined, engineered (ie, handcrafted) features based on expert knowledge. In contrast, deep learning allows the algorithm to learn discriminative features from data automatically, offering the ability to approximate very complex nonlinear relationships through several layers and neurons. Therefore, deep learning interpretability becomes even more challenging, as we need not only to explain the relationship between features and decisions or predictions, which usually involves a vast number of parameters, but also to understand the learned features themselves. In this issue of Radiology: Artificial Intelligence, Reyes et al address this important topic by reviewing computational approaches to interpret deep learning models, and discuss gaps and challenges that need to be addressed, focusing primarily on radiologic applications and their translation into clinical practice (6). The authors are to be commended for the comprehensive summary of existing interpretability technologies, as well as for the intuitive, yet detailed, descriptions of such sophisticated computational tools. Reyes and colleagues have categorized various interpretability methods, provided explanatory figures, demonstrated related applications, and offered a thorough discussion on the advantages and limitations. If a deep learning model performs well, why shouldn’t we simply trust the model and ignore why it made a specific decision? A lot of this has to do with the impact that Is It Time to Get Rid of Black Boxes and Cultivate Trust in AI?

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1148/ryai.2020200088

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Cited by

See all 48

Metadata

Authors
Gastounioti, A., & Kontos, D.
Affiliations
  • University of Pennsylvania (US)
  • From the Department of Radiology, University of Pennsylvania, 3700 Hamilton Walk, Richards Bldg, Room D702, Philadelphia, PA 19104.
Venue
Radiology: Artificial Intelligence
Published
2020-05-01
Volume
2
Issue
3
Pages
e200088
ISSN
2638-6100 (ISSN-L)
OpenAlex
W3030973653
Cited by
47
References
6
Citations by year
2026: 3 · 2025: 12 · 2024: 11 · 2023: 8 · 2022: 6 · 2021: 4 · 2020: 3
Type
journal article
Language
en
Open access
Locations
  • Radiology Artificial Intelligence (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
Funders
National Cancer Institute (R01CA207084, R01CA161749)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
black box, explainable AI, trust in AI, algorithmic transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.