Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this example, reproduced from Oakden-Rayner and colleagues, 13 the artificial intelligence model learns to detect the artifacts of clinical care (chest drains) and fails to adequately learn the features of the pathology itself.”
From The medical algorithmic audit (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences53.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences45.”
From Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension (2020) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, this drop in performance might not be obvious at the aggregate level of typical artificial intelligence testing, but rather in subsets of the target cohort or specific strata within the input data, a concept that has been described as hidden stratification.13”
From The medical algorithmic audit (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Esteva et al. [12] is particularly notable, as this dataset is labelled for more than 2,000 diagnostic subclasses but the results presented only consider “top-level” diagnostic categories.”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Gulshan et al. [16], for instance, present variations in retinopathy detection performance on subsets with images obtained in different locations, with differing levels of disease severity, and with different degrees of pupil dilation.”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“Deep learning systems have shown remarkable promise in medical image analysis, often claiming performance rivaling that of human experts [13].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundA dendrite method for cluster analysis
Caliński, T., & Harabasz, J. (1974) · Communications in Statistics · doi:10.1080/03610927408827101 · Cited by 4,480
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“]. However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40]. As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techn…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“observed [6, 34]; such analyses may be helpful in identifying error modes via human review, but do not characterize the full space of subset performance [33].”
Background
Research works printed but not linked: 1
- Irvin Jeremy, Rajpurkar Pranav, Ko Michael, Yu Yifan, Ciurea-Ilcus Silviana, Chute Chris, Marklund Henrik, Haghgoo Behzad, Ball Robyn, Shpanskaya Katie, and Others. 2019 Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019). · arXiv
Cited by
Items 401–440 of 440
Septiani, F., Fazharyasti, V., Putranto, T. A., Maarifat, N. N., Tandionugroho, S., & Harahap, M. I. R. (2026) · Universa Medicina · doi:10.18051/univmed.2026.v45.322-331
“(31,32) Moreover, in a screening context, maximizing sensitivity is often prioritized to curtail false negatives (missed hemorrhages), even if this entails a marginal compromise in specificity (resulting in more false positives), as false positives can be subsequently filtered through expert human…”
Singh, D. B., Dangi, Y., & Pokharel, B. P. (2026) · Health Science Reports · doi:10.1002/hsr2.73117
Measuring ML Dataset Coverage and Completeness: Metrics and Visualizations
Kuhn, D. R., & Franco, J. F. D. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-18585-3_4
Refined Exclusion in Medical AI: Data Justice and Patient Safety Governance
Lee, J. H., Choi, B., Jeong, K., Suh, S. W., Kim, J. H., & Son, D. (2026) · Journal of Medical Internet Research · doi:10.2196/102359
“Hidden stratification describes a technical failure in which clinically meaningful subsets are obscured within broad labels or tasks [10].”
Wei, L., Feng, G., Tan, M., Weng, P., & Zhu, W. (2026) · International Journal of General Medicine · doi:10.2147/ijgm.s621520
“34,35 To address this, Federated Learning (FL) has emerged as a promising technical solution.”
Chung, S. W., Park, S.-J., Yoon, Y. S., Kim, D.-H., Cho, C., Kim, J.-Y., & Yoon, J. P. (2026) · Clinics in Shoulder and Elbow · doi:10.5397/cise.2026.00661
“These limitations complicate early detection of progressive mechanical failure and reduce the reproducibility of radiographic assessment across institutions [11,12].”
Deep Learning-Based Multi-Class Classification of Bone Fractures in X-Ray Radiographs
Ali, G. S. H., Al-Sharif, H., Al-Sharif, H., Al-Qershi, J., Al-Kumaim, A., & Obadi, M. (2026) · International Conference on Digital Transformation, Innovation & Sustainable Development (DTISD) · doi:10.1109/dtisd69661.2026.11668316
“Diagnosing fractures with X-rays is difficult because 2 – 9% of all X-rays taken in the emergency department are false negatives, failing to detect fractures, especially subtle ones such as hairline or nondisplaced fractures [4], [5].”
Artificial intelligence: transforming drug discovery
Mishra, S. (2026) · International Journal of Basic & Clinical Pharmacology · doi:10.18203/2319-2003.ijbcp20263250
Jebaseeli, T. J., Selvarathi, M., Brindha, D., David, D. J., Juliet, S. J. J., & Angelin, P. A. C. (2026) · Sensing and Imaging · doi:10.1007/s11220-026-00877-6
“The requirement to update and adapt the model without changing it in different contexts will make the use of ML in clinical practice more complex [ 19 – 25 ].”
Na, S. H., & Ahn, S. (2026) · Journal of Clinical Medicine · doi:10.3390/jcm15187075
“A single aggregate score also hides the subgroups in which a model fails, so that a respectable mean can coexist with a class recovered below chance [13].”
Li, H., Wang, C., Xu, Y., Wang, X., & Zhao, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7459680
Li, I. D., Correia, C., & Ung, C. Y. (2026) · Cancers · doi:10.3390/cancers18183015
“Fourth, impressive internal performance can be undermined by patient-level leakage, hidden scanner effects, class imbalance, and generalizability of models [31–34].”
Purkayastha, S., Naliyatthaliyazchayil, P., & Gichoya, J. W. (2026) · Research Square · doi:10.21203/rs.3.rs-10919324/v1
Liu, Y., Tong, S., Cai, S., & Peng, Y. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02327-x
Manurung, H. E., Budiman, F., Santoso, H. A., & Marjuni, A. (2026) · International journal of intelligent engineering and systems · doi:10.22266/ijies2026.1031.23
Konyspayev, D., Kuzenbayev, B., Abatov, N. T., & Alippayeva, D. (2026) · Вестник КазАТК · doi:10.52167/1609-1817-2026-144-3-300-312
Lee, M., Lin, W.-Y., & Chiu, Y. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-02312-4
Didi, Y., Walha, A., & Wali, A. (2026) · Multimedia Tools and Applications · doi:10.1007/s11042-026-21915-1
LLM-Steered Clinical Knowledge Discovery for CVD Outcome Modeling
Yuxuan, L., Xu, X., Wang, G., & Yan, P. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38172-9_38
Higher-dimensional embedding of time-series data for machine learning
Singh, K., Nath, P. P., Sinha, U., & Sinha, A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1874928
Jin, K., Zhao, K., Agrawal, R. V., Ying, G., & Grzybowski, A. E. (2026) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2026.101533 · Cited by 1
Evaluation Metrics for Image Classification, Segmentation and LocalizationPreprint
Kalejahi, B. K., & Rajabi, M. J. (2026) · Research Square · doi:10.21203/rs.3.rs-11205716/v1
Reproducibility vs. Representativeness: The Trade-Off in MRI AI Algorithm Development
Martínez-Licona, F. M. (2026) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-032-39674-7_45
D, S. P., Prasidhan, P., & Jacob, A. (2026) · International Journal of Information Technology Research Studies (IJITRS) · doi:10.63090/ijitrs/3139.3209.0035
Mengistie, B. T., Gete, K. Y., Mohammed, A., Zimme, Z. A., Teferi, M. G., Adugna, A., Kibret, F., Fetle, Y., Abera, B. F., & Mengistie, C. T. (2026) · Egyptian Liver Journal · doi:10.1186/s43066-026-00546-z
“Validation should span sites, hardware, software, protocols, quality strata, and clinical pathways, with subgroup error reporting to identify inequitable failures [30, 31].”
Patel, K., Beedala, P., Vora, D., & Mehta, H. (2026) · medRxiv · doi:10.64898/2026.10.03.26364646
Nam, H., Yoo, J., Lee, D.-H., Kim, J., Jang, Y. J., & Kang, K. (2026) · Research Square · doi:10.21203/rs.3.rs-10788572/v1
Liu, H., Xu, Y., Cheng, F., & Zha, J. (2026) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.70452
Yanık, H. (2026) · Biomimetics · doi:10.3390/biomimetics11100744
“Group distributionally robust optimization [29] and minimax-Pareto fairness [30] formalize worst-group objectives, and subgroup failures have been documented in ECG and other medical AI [9,31,32].”
Ye, F., Yin, M., & Sun, S. (2026) · Sustainability · doi:10.3390/su182110735
Hossain, M. S., & Borsha, M. K. N. (2026) · International Journal of Computer Science and Mobile Computing · doi:10.47760/ijcsmc.2026.v15i10.001
Yu, Y. (2026) · Research Square · doi:10.21203/rs.3.rs-11460866/v1
Ali, F., Ali, S. R., Alom, S., Kakoti, B. B., Shakya, A., Ghosh, S. K., Singh, U. P., & Bhat, H. R. (2026) · In Silico Research in Biomedicine · doi:10.1016/j.insi.2026.100705
Akash, M., Suer, S., Ferdous, K. M., Aygün, R. S., & Karakaya, M. (2026) · doi:10.2196/preprints.114649
Char, I. C. J., Sayed, A. S. A., Ichikawa, K., Sayed, Y. M. G., Milhan, A. B., Sim, D. Y. Y., & Ramanathan, A. (2026) · Frontiers in Big Data · doi:10.3389/fdata.2026.1972023
Lamrabti, A., Hassani, I. B., & Mouchawrab, S. (2026) · doi:10.1109/aice71085.2026.11736947
“Hidden subgroup failure, algorithmic bias, and poor external validation remain serious risks [5]–[7].”
TelecomAudit: Origin-Aware Benchmark Auditing and Calibration for 5G Anomaly Detection
Salavati, C., Wu, L., Wan, K., Darbari, M., & Hong, L. (2026) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3799682.3840150
Saha, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-36005-2_8
Metadata
- Authors
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
- Affiliations
- Australian Institute for Machine Learning, University of Adelaide
- Stanford University (US)
- The University of Adelaide (AU)
- Department of Computer Science, Stanford University
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 151-159
- PMC
- PMC7665161
- OpenAlex
- W2975035192
- Cited by
- 411
- References
- 33
- Citations by year
- 2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toshiba
- Teradata
- Ant Financial
- Intel Corporation
- NEC Corporation
- Taiwan Semiconductor Manufacturing Company
- Accenture
- Qualcomm
- Analog Devices
- VMWare
- Infosys
- National Institutes of Health (U54EB020405)
- NXP
- Xilinx
- LETI-CEA
- American Family Insurance
- Swiss Re
- Intelligence Community Postdoctoral Research Fellowship Program
- National Science Foundation (CCF1763315, CCF1563078, CC1937301)
- IBM
- Hitachi
- Okawa Foundation for Information and Telecommunications
- Google Cloud
- Office of Naval Research (N000141712266)
- Gordon and Betty Moore Foundation
- ARM
- Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
- Microsoft
- BASF
- Ericsson
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.