CoChat

Hidden stratification causes clinically meaningful failures in machine learning for medical imaging

Open access
Oakden-Rayner, Luke, Dunnmon, Jared, Carneiro, Gustavo, Re, Christopher · 2020 · Proceedings of the ACM Conference on Health, Inference, and Learning

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3368555.3384468

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 2

  1. Jeremy Irvin , Pranav Rajpurkar , Michael Ko , Yifan Yu , Silviana Ciurea-Ilcus , Chris Chute , Henrik Marklund , Behzad Haghgoo , Robyn Ball , Katie Shpanskaya , and Others. 2019 . Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901. 07031 (2019). Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, and Others. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901. 07031 (2019).
  2. Alex Krizhevsky , Vinod Nair , and Geoffrey Hinton . 2009. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html 6 ( 2009 ). Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html 6 (2009).

Cited by

See all 440

Metadata

Authors
Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
Affiliations
  • Australian Institute for Machine Learning, University of Adelaide
  • Stanford University (US)
  • The University of Adelaide (AU)
  • Department of Computer Science, Stanford University
Venue
Proceedings of the ACM Conference on Health, Inference, and Learning
Publisher
ACM
Published
2020-04-02
Volume
2020
Pages
151-159
OpenAlex
W2975035192
Cited by
411
References
33
Citations by year
2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
FWCI
25.17
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Google
  • Toshiba
  • Teradata
  • Ant Financial
  • Intel Corporation
  • NEC Corporation
  • Taiwan Semiconductor Manufacturing Company
  • Accenture
  • Qualcomm
  • Analog Devices
  • Facebook
  • VMWare
  • Infosys
  • National Institutes of Health (U54EB020405)
  • NXP
  • Xilinx
  • LETI-CEA
  • American Family Insurance
  • Swiss Re
  • Intelligence Community Postdoctoral Research Fellowship Program
  • National Science Foundation (CCF1763315, CCF1563078, CC1937301)
  • IBM
  • Hitachi
  • Okawa Foundation for Information and Telecommunications
  • Google Cloud
  • Office of Naval Research (N000141712266)
  • Gordon and Betty Moore Foundation
  • ARM
  • Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
  • Microsoft
  • BASF
  • Ericsson
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.