Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this example, reproduced from Oakden-Rayner and colleagues, 13 the artificial intelligence model learns to detect the artifacts of clinical care (chest drains) and fails to adequately learn the features of the pathology itself.”
From The medical algorithmic audit (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences53.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences45.”
From Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension (2020) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, this drop in performance might not be obvious at the aggregate level of typical artificial intelligence testing, but rather in subsets of the target cohort or specific strata within the input data, a concept that has been described as hidden stratification.13”
From The medical algorithmic audit (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Esteva et al. [12] is particularly notable, as this dataset is labelled for more than 2,000 diagnostic subclasses but the results presented only consider “top-level” diagnostic categories.”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Gulshan et al. [16], for instance, present variations in retinopathy detection performance on subsets with images obtained in different locations, with differing levels of disease severity, and with different degrees of pupil dilation.”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“Deep learning systems have shown remarkable promise in medical image analysis, often claiming performance rivaling that of human experts [13].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundA dendrite method for cluster analysis
Caliński, T., & Harabasz, J. (1974) · Communications in Statistics · doi:10.1080/03610927408827101 · Cited by 4,480
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“]. However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40]. As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techn…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“observed [6, 34]; such analyses may be helpful in identifying error modes via human review, but do not characterize the full space of subset performance [33].”
Background
Research works printed but not linked: 1
- Irvin Jeremy, Rajpurkar Pranav, Ko Michael, Yu Yifan, Ciurea-Ilcus Silviana, Chute Chris, Marklund Henrik, Haghgoo Behzad, Ball Robyn, Shpanskaya Katie, and Others. 2019 Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019). · arXiv
Cited by
Items 201–300 of 440
Robustness Stress Testing in Medical Image Classification
Islam, M., Li, Z., & Glocker, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44336-7_17 · Cited by 14
Critical Appraisal of a Machine Learning Paper: A Guide for the Neurologist
Vinny, P. W., Garg, R. D., Srivastava, M. P., Lal, V., & Vishnu, V. Y. (2021) · Annals of Indian Academy of Neurology · doi:10.4103/aian.aian_1120_20 · Cited by 8
“[38] A ML algorithm designed to recognize large vessel occlusion (LVO) patterns from CT scans without the need for contrast-enhanced imaging showed an area under the curve (AUC) for the identification of LVO as 0.87 (sensitivity: 83%, specificity: 71%, positive predictive value: 79%, negative…”
MethodInfluentialReliable Model Compression via Label-Preservation-Aware Loss Functions.Preprint
Joseph, V. R., Siddiqui, S. A., Bhaskara, A., Gopalakrishnan, G., Muralidharan, S., Garland, M., Ahmed, S., & Dengel, A. R. (2020) · arXiv (Cornell University) · Cited by 7
Fonseca, P. A. D. S., Suárez‐Vega, A., Esteban‐Blanco, C., Marina, H., Pelayo, R., Gutiérrez‐Gil, B., & Arranz, J.-J. (2025) · BMC Genomics · doi:10.1186/s12864-025-11520-1 · Cited by 7
“Common causes include feature selection before cross-validation [109], biased data splitting [110], and leakage of target information through the input features [111].”
Gonzalez-Valenzuela, R. E., Mettes, P., Loos, B. G., Marquering, H. A., & Berkhout, E. (2025) · Clinical Oral Investigations · doi:10.1007/s00784-025-06672-z · Cited by 11
“…early proximal caries lesions leads to biased results, as an AI model trained with suboptimal manual annotations may exhibit reduced sensitivity for early-stage caries compared to advanced lesions, particularly underperforming in detecting early-stage caries compared to advanced stages [21].”
Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models
Pavlak, M., Drenkow, N. G., Petrick, N., Farhangi, M. M., & Unberath, M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43898-1_43 · Cited by 10
Toward Real-World Multi-View Object Classification: Dataset, Benchmark, and Analysis
Wang, R., Kim, T. S., Kim, J.-S., & Lee, H.-J. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3359681 · Cited by 7
“Meanwhile, the intra-class view variance causes the hidden stratification phenomenon [76], meaning the mismatch between hard labels and various view features.”
BackgroundMethodClusterFix: A Cluster-Based Debiasing Approach without Protected-Group Supervision
Capitani, G., Bolelli, F., Porrello, A., Calderara, S., & Ficarra, E. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00480 · Cited by 10
Wongveerasin, P., Tongdee, T., & Saiviroonporn, P. (2024) · European Journal of Radiology Open · doi:10.1016/j.ejro.2024.100593 · Cited by 6
“…the limited generalizability problem [9,10], which could be due to the ICU population ’ s associated conditions, e.g., more opacifications on CXR, increased numbers of tubes and lines, etc., leading to hidden stratification, where rare subpopulations are consistently missed by the model [27].”
BackgroundGaze-Directed Vision GNN for Mitigating Shortcut Learning in Medical Image
Wu, S., Zhang, X., Wang, B., Jin, Z., Li, H., & Feng, J. H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72378-0_48 · Cited by 15
Nadkarni, P. M., & Merchant, S. A. (2022) · Artificial Intelligence in Medical Imaging · doi:10.35711/aimi.v3.i3.55 · Cited by 7
KALM: Key Area Localization Mechanism for Abnormality Detection in Musculoskeletal Radiographs
Huang, W. F., Xiong, Z., Wang, Q., & Li, X. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053768 · Cited by 6
“In [22], Dense-169 is bigger and deeper than our VGG16KALM , and there are 13,942 studies used for training, which are larger than 13,457 studies in this paper.”
BackgroundClinical Artificial Intelligence
McDermott, M. B. A., Nestor, B., & Szolovits, P. (2022) · Clinics in Laboratory Medicine · doi:10.1016/j.cll.2022.09.004 · Cited by 7
Unsupervised Brain Abnormality Detection Using High Fidelity Image Reconstruction Networks.Preprint
Kobayashi, K., Hataya, R., Kurose, Y., Bolatkan, A., Miyake, M., Watanabe, H., Takahashi, M., Mihara, N., Itami, J., Harada, T., et al. (2020) · arXiv (Cornell University) · Cited by 6
Dao, L., Harmouch, S. S., Chin, A. S.-L., Dao, T., Thian, Z., Chartrand‐Lefebvre, C., & Cohen, J. (2022) · medRxiv · doi:10.1101/2022.07.11.22277278 · Cited by 9
“naturally affected by labelling biases: discrepancy between the radiologist’s, clinician’s, and automatic labeller’s understanding of a radiology report (35); errors in labelling due in part to automatic labellers (36,37); framing bias or limitations in objectivity (33,38,39); and interobserver variability, (40) to on…”
BackgroundKrishna, A., Wang, G., & Mueller, K. D. (2025) · Physics in Medicine and Biology · doi:10.1088/1361-6560/adb9b3 · Cited by 7
Perdomo-Lampignano, J. A., Kale, A. U., Kumar, S., Shah, D., Reddy, B., & Lowe, D. J. (2026) · Journal of Medical Systems · doi:10.1007/s10916-026-02426-w · Cited by 7
Liu, X., Rivera, S. C., Moher, D., Calvert, M., Denniston, A. K., & y CONSORT-AI, G. D. T. S.-A. (2024) · Revista Panamericana de Salud Pública · doi:10.26633/rpsp.2024.13 · Cited by 6
Kale, A. U., Hogg, H. D. J., Pearson, R. V., Glocker, B., Golder, S., Coombe, A., Waring, J. J., Liu, X., Moore, D., & Denniston, A. K. (2024) · JMIR Research Protocols · doi:10.2196/51614 · Cited by 8
“…of in-depth performance error analysis including the identification of “inhuman errors” (eg, highly displaced fractures missed by AI), testing for algorithmic encoding of protected characteristics, and conducting exploratory error analyses to identify cases of hidden stratification [20-22].”
BackgroundLeveraging Neurosymbolic AI for Slice Discovery
Collevati, M., Eiter, T., & Higuera, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_22 · Cited by 6
“Several studies (Buolamwini & Gebru, 2018; DeVries et al., 2019; Koenecke et al., 2020; Oakden-Rayner et al., 2020) have shown that neural network models often make systematic errors on data slices.”
Jin, D. L. K., Rosenthal, J. H., Thompson, E. E., Dunnmon, J., Mohtashamian, A., Ward, D., Austin, R., Tetteh, H. A., & Olson, N. H. (2022) · Journal of Pathology Informatics · doi:10.1016/j.jpi.2022.100142 · Cited by 6
“This is significant because models for medical imaging commonly suffer from hidden stratification.(19) For instance, the overall performance of a cancer detection algorithm may be high, but its performance on an uncommon cancer subclassmay be extremely poor.”
BackgroundSelf-citationZhang, C., Raghu, M., Kleinberg, J. M., & Bengio, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12580 · Cited by 29
“These questions are of central importance in assessing the limitations of artificial neural networks, a topic that has gained pressing urgency as these models are applied to high-stakes settings such as medical diagnosis, which require robust reasoning capabilities (Winkler et al., 2019; Oakden-Rayner et al., 2020).”
BackgroundRedefining Bias Audits for Generative AI in Health Care
Chen, I. Y., & Alsentzer, E. (2025) · NEJM AI · doi:10.1056/aip2500015 · Cited by 8
Distributionally Robust Optimization with Probabilistic Group
Ghosal, S. S., & Li, Y. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26394 · Cited by 20
Balancing Performance and Interpretability in Medical Image Analysis: Case study of Osteopenia
Mikulić, M., Vičević, D., Nagy, E., Napravnik, M., Štajduhar, I., Tschauner, S., & Hržić, F. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01194-8 · Cited by 6
“According to a study [31], researchers have discovered that hidden stratification can lead to more than 20% performance differences in clinically important subsets.”
Efficient Bias Mitigation Without Privileged Information
Zarlenga, M. E., Sankaranarayanan, S., Andrews, J. T. A., Shams, Z., Jamnik, M., & Xiang, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73220-1_9 · Cited by 15
“It is often difficult to place the resounding success of Deep Neural Networks (DNNs) [32,60,70,79] in conjunction with their tendency to exploit biases within their training sets [12,16,27,31,51].”
BackgroundMoshe, Y. H., Teicher, M., & Artzi, M. (2026) · Technology in Cancer Research & Treatment · doi:10.1177/15330338251405180 · Cited by 4
Sim, J. Z. T., Lin, J. L. L., Fong, Q. W., Soon, A. Y. Q., Khin, L. W., Balakrishnan, S., Lin, T., Wong, S., & Tan, C. H. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-111127 · Cited by 4
Malone, K. (2020) · Harvard Data Science Review · doi:10.1162/99608f92.c2fc310d · Cited by 5
Meta-Repository of Screening Mammography Classifiers
Chłędowski, J., Stadnick, B., Witowski, J. S., Rajiv, V., Shamout, F. E., Cho, K., & Geras, K. J. (2025) · Studies in health technology and informatics · doi:10.3233/shti251263 · Cited by 6
The Details Matter: Preventing Class Collapse in Supervised Contrastive Learning
Fu, D. Y., Chen, M. F., Zhang, M., Fatahalian, K., & Ré, C. (2022) · doi:10.3390/cmsf2022003004 · Cited by 12
“Our treatment of strata is strongly inspired by [5,6], who document empirical consequences of hidden strata.”
BackgroundRivera, S. C., Liu, X., Chan, A., Denniston, A. K., Calvert, M., y CONSORT-AI, G. D. T. S.-A., y CONSORT-AI, G. D. S.-A., & y CONSORT-AI, G. D. C. S.-A. (2024) · Revista Panamericana de Salud Pública · doi:10.26633/rpsp.2024.12 · Cited by 8
Are Generative Models Fair? A Study of Racial Bias in Dermatological Image Generation
López-Pérez, M., Hauberg, S., & Feragen, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95918-9_27 · Cited by 11
Poles, I., D’Arnese, E., Buccino, F., Vergani, L. M., & Santambrogio, M. D. (2023) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc40787.2023.10341140 · Cited by 5
“Further problems arise with the hidden stratification issue: a pathology defined as a class for DL is often comprised of several distinct clinical signs affecting model training, model performance, and clinical outcomes [12].”
BackgroundSynthesis of fracture radiographs with deep neural networks
Chedid, N. R., Sadda, P., Gonchigar, A. C., Langdon, J., Porrino, J., Haims, A. H., & Taylor, R. A. (2020) · Health Information Science and Systems · doi:10.1007/s13755-020-00111-x · Cited by 4
“In particular, Oakden-Rayner et al. [7] describe a “hidden stratification” problem in medical image classifiers.”
BackgroundSubclass Knowledge Distillation with Known Subclass Labels
Sajedi, A., Lawryshyn, Y., & Plataniotis, K. N. (2022) · Multidimensional Signal Processing Workshop · doi:10.1109/ivmsp54334.2022.9816232 · Cited by 4
Robust Benchmarking for Machine Learning of Clinical Entity ExtractionPreprint
Agrawal, M., O’Connell, C. P., Fatemi, Y., Levy, A., & Sontag, D. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.16127 · Cited by 7
“In a similar vein, Oakden-Rayner et al. (2020) found that machine learning models trained on medical images suffer from hidden stratification, where performance depends on unknown confounders.”
BackgroundOn the Efficiency of Subclass Knowledge Distillation in Classification TasksPreprint
Sajedi, A., & Plataniotis, K. N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.05587 · Cited by 6
“Models trained exclusively on class labels often ignore the fine-grained knowledge of subclasses, which can have an effect on model training, particularly for clinical tasks such as cancer detection (Sohoni et al. 2020; Oakden-Rayner et al. 2020).”
InfluentialVLM@school: Evaluation of AI Image Understanding on German Middle School Knowledge
Peinl, R., & Tischler, V. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07986-2_41 · Cited by 4
Subgroup Performance Analysis in Hidden Stratifications
Bissoto, A. E., Hoang, T.-D., Flühmann, T., Sun, S., Baumgartner, C. F., & Koch, L. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05185-1_57 · Cited by 6
“Machine learning (ML) models often perform systematically differently across patient subgroups [18,6,13,15].”
Wang, X., Xiong, D., Cui, S., Duan, B., Hung, Y., Jing, H. E., Ding, G., Tang, Y., & Wang, Q. (2026) · Discover Oncology · doi:10.1007/s12672-026-04864-x · Cited by 6
“Regulators increasingly expect stress testing on out-of-distribution and adversar-ial-like cases and safeguards such as automatic flagging of low-confidence predictions for human review [117, 118].”
Causal Feature Alignment: Learning to Ignore Spurious Background Features
Venkataramani, R., Dutta, P., Melapudi, V. R., & Dukkipati, A. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00460 · Cited by 14
Evans, H., Sivakumar, N., Bhanderi, S., Graham, S., Snead, D. R. J., Patel, A., & Robinson, A. (2025) · BMJ Open Gastroenterology · doi:10.1136/bmjgast-2024-001649 · Cited by 3
“14 This needs to be overcome by incorporating these rarer subtypes into training data sets.”
Rokvic, L., Danassis, P., Karimireddy, S. P., & Faltings, B. V. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825097 · Cited by 7
“The model may rely on these shortcuts instead of looking at the actual scan [48].”
Mask of Truth: Model Sensitivity to Unexpected Regions of Medical Images
Sourget, T., Hestbek-Møller, M., Jiménez-Sánchez, A., Xu, J. J., & Cheplygina, V. (2025) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-025-01531-5 · Cited by 10
“Another interesting result is that for all the classes, models trained on Full images performed better when evaluated on images without lungs than on images with only lungs.”
Weng, C., Ward, J., Lin, W., Dong, S., Liu, Q., & Zhang, H. (2025) · Clinical and Translational Science · doi:10.1111/cts.70349 · Cited by 4
“Nothing Abnormal”: Disambiguating Medical Reports via Contrastive Knowledge Infusion
He, Z., Yan, A., Gentili, A., McAuley, J., & Hsu, C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i12.26665 · Cited by 6
Modern capabilities of artificial intelligence technologies in cardiovascular imaging
Islamgulov, A. K., Bogdanova, A. S., Sufiyarov, D. I., Chernyavskaya, A. V., Bairakaeva, E. R., Maksimova, A. A., Nemychnikov, N. V., Bikieva, D. R., Shakhmaeva, A. I., Burdina, L. A., et al. (2025) · Digital Diagnostics · doi:10.17816/dd640895 · Cited by 3
Gamified Learning of Project Business Skills
Koivisto, M. (2022) · International Journal of Advanced Corporate Learning (iJAC) · doi:10.3991/ijac.v15i1.26689 · Cited by 3
Zeng, T., Furst, E., Wang, Y., Tchoua, R., Furst, J. D., & Raicu, D. S. (2023) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2653633 · Cited by 3
“Oakden-Rayner et al. 6 identify the necessity of finding hidden stratification in the medical domain.”
BackgroundMitigating medical dataset bias by learning adaptive agreement from a biased council
Luo, L., Huang, X., Wang, M., Wan, Z., Ma, W., & Chen, H. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103629 · Cited by 9
Is Systematic Data Sharding able to Stabilize Asynchronous Parameter Server Training?
Provatas, N., Konstantinou, I., & Koziris, N. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9672001 · Cited by 3
“Emphasizing on hidden stratification, in [57] they conclude that in some cases, medical related classification models should not be used without considering any hidden patterns.”
BackgroundMaskSearch: Querying Image Masks at Scale
He, D., Zhang, J., Daum, M., Ratner, A., & Bałazińska, M. (2025) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde65448.2025.00184 · Cited by 5
“For instance, in the context of model explanation, examining saliency maps is the most common approach to understanding whether a model is relying on spurious correlations in the input data, i.e., signals that deviate from domain knowledge [8, 18, 41, 42, 46, 59].”
A Principled Approach to Failure Analysis and Model Repairment: Demonstration in Medical Imaging
Henn, T., Sakamoto, Y., Jacquet, C., Yoshizawa, S., Andou, M., Tchen, S., Saga, R., Ishihara, H., Shimizu, K., Li, Y., et al. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-87199-4_48 · Cited by 3
“For example, a critical yet rare subgroup could be missed with such an approach and go unreported [8].”
BackgroundLearning Global and Local Features of Normal Brain Anatomy for Unsupervised Abnormality DetectionPreprint
Kobayashi, K., Hataya, R., Kurose, Y., Bolatkan, A., Miyake, M., Watanabe, H., Takahashi, M., Itami, J., Harada, T., & Hamamoto, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.12573 · Cited by 5
“The most essential but hidden factor in comparison of model performance may be the intraclass variability of target disease (Oakden-Rayner et al., 2020).”
BackgroundThe Challenge Dataset – simple evaluation for safe, transparent healthcare AI deploymentPreprint
Sanayei, J. K., Abdalla, M., Ahluwalia, M., Seyyed-Kalantari, L., Minotti, S. C., & Fine, B. (2022) · medRxiv · doi:10.1101/2022.12.15.22280619 · Cited by 4
“[19 ] They also reflect only a small subset of the full gamut of pathologies encountered in practice, excluding many of the uncommon but important findings present in the “long tail” of imaging findings.”
BackgroundZhu, M., Liu, Y., Luo, Z., & Zhu, T. (2026) · npj Digital Medicine · doi:10.1038/s41746-025-02176-y · Cited by 8
“This paradigm further compromises model performance, especially given the diverse presentations and pathophysiology of rare conditions 20 .”
Raghupathi, W. (2026) · Algorithms · doi:10.3390/a19010063 · Cited by 2
Miao, S., Dong, H., Feng, J., Jiang, Y., Sun, M., Liu, Z., Wang, Q., Ding, X., & Wang, R. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106348 · Cited by 2
Tsougos, I., Valotassiou, V. J., Tsivaka, D., Σάτρα, Μ., Angelidis, G., Papatriantafyllou, J. D., Panagiotidis, E., Dardiotis, E., Hadjigeorgiou, G. M., & Georgoulias, P. (2026) · Annals of Nuclear Medicine · doi:10.1007/s12149-026-02194-0 · Cited by 3
Clinician-accessible automated machine learning in oral healthcare: A systematic review
Shujaat, S., Riaz, M., Almutairi, H., Alanazi, H., Altalhi, L., Alenazi, N., Osayl, H. B., Philip, M. R., Aboalela, A. A., & Ma, H. (2026) · Japanese Dental Science Review · doi:10.1016/j.jdsr.2026.03.001 · Cited by 2
“Few provided visual explanations such as saliency maps or systematic misclassification analyses, reflecting broader concerns about opacity in medical imaging AI [55 – 57].”
Eschweiler, D., Merodio, E. C., Ordonez, F. B., Lichev, A., Ignatova, N., von der Stück, M. S., Kuhl, C. K., Truhn, D., & Nebelung, S. (2026) · European Radiology · doi:10.1007/s00330-026-12555-y · Cited by 2
Perlmutter, J. W., & Joseph, M. (2025) · Education and human development · doi:10.5772/intechopen.1014025 · Cited by 3
Frustratingly Easy Environment Discovery for Invariant Learning
Zare, S., & Nguyen, H. V. (2024) · doi:10.3390/cmsf2024009002 · Cited by 4
“This performance gap is observed across various applications like medical imaging [8–10], and facial recognition [5,11].”
BackgroundThick Data Analytics (TDA): An Iterative and Inductive Framework for Algorithmic Improvement
Nguyen, M., Eulalio, T. Y., Marafino, B. J., Rose, C. C., Chen, J. H., & Baiocchi, M. (2024) · The American Statistician · doi:10.1080/00031305.2024.2327535 · Cited by 2
“To promote the use of feedback mechanisms in addition to retrospective analysis of model predictions, a recent shift in the literature of model assessment investigates unusual behaviors, errors, and failures (Liu et al. 2022; Raji et al. 2020; Oakden-Rayner et al. 2020, 2022).”
An Exploration of Diabetic Foot Osteomyelitis X-ray Data for Deep Learning Applications
Abela, B., Mašek, M., Abu-Khalaf, J., Suter, D., & Gupta, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66535-6_4 · Cited by 2
Teaching Machine Learning as Part of Agile Software Engineering
Chenoweth, S., & Linos, P. K. (2025) · IEEE Transactions on Education · doi:10.1109/te.2025.3572355 · Cited by 2
Kumar, N., Shrestha, R., Li, Z., & Wang, L. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-45249-9_18 · Cited by 2
Brown, K. E., Talbert, S., & Talbert, D. A. (2023) · PubMed Central · Cited by 4
A Novel Approach to Unveil Hidden Stratification in Osteoporosis SR-MicroCT Image Classification
Poles, I., D’Arnese, E., Buccino, F., Vergani, L. M., & Santambrogio, M. D. (2024) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi56570.2024.10635487 · Cited by 2
“…potential in studying osteoporosis, several criticalities emerge from interpretability limitations, which make differences among image samples at the microscale fine-grained, and, therefore, unrecognizable by human eyes to the micro-to macroscale knowledge discrepancy of the bone status [7].”
BackgroundApplications and potential of machine, learning augmented chest X-ray interpretation in cardiology
Milne, M., Ahmad, H. K., Buchlak, Q. D., Esmaili, N., Tang, C. H. M., Seah, J. C. Y., Ektas, N., Brotchie, P., Marwick, T. H., & Jones, C. M. (2024) · Minerva Cardiology and Angiology · doi:10.23736/s2724-5683.24.06288-4 · Cited by 2
Runtime Verification for Visual Deep Learning Systems With Loss Prediction
Torpmann-Hagen, B., Riegler, M. A., Halvorsen, P., & Johansen, D. (2025) · IEEE Access · doi:10.1109/access.2025.3552157 · Cited by 4
Medical Image Processing of Patients for Skin Cancer Diagnosis Using Artificial Intelligence
Gorgbandi, S., & Nazari, S. (2025) · Transactions on Machine Intelligence · doi:10.47176/tmi.2025.38 · Cited by 3
Iterative K-means clustering for disease subtype discovery
Aubert, K., Huber, C., Furst, J. D., Raicu, D. S., & Tchoua, R. (2023) · SPIE Medical Imaging: Computer-Aided Diagnosis · doi:10.1117/12.2653973 · Cited by 2
“Because of this, subtypes are often hidden or obscured by the majority of samples of their corresponding superclass, and so they are left unlearned by machine learning models, a phenomenon known as hidden stratification .(5) Within medical datasets, subtypes are heavily prevalent due to disease”
BackgroundMachine Learning Systems for Intelligent Services in the IoT: A SurveyPreprint
Toussaint, W., & Ding, A. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04950 · Cited by 4
“may exhibit predictive performance as good as, or exceeding that of human experts, they are prone to making errors that no human would make and that defy common sense [124].”
BackgroundCheXbreak: Misclassification Identification for Deep Learning Models Interpreting Chest X-raysPreprint
Chen, E., Kim, A. J., Krishnan, R., Long, J., Ng, A. Y., & Rajpurkar, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.09957 · Cited by 4
“…with high likelihood of misclassification may help protect against inaccurate diagnoses, determine when clinical decisions should be deferred to downstream experts, and enable monitoring of deep learning based models in clinical workflow (Mozannar and Sontag (2020); Oakden-Rayner et al. (2019)).”
BackgroundBird, A., McMaster, C., & Liew, D. F. L. (2024) · Arthritis Care & Research · doi:10.1002/acr.25310 · Cited by 2
Anbukkarasi, S., Hemalatha, S., Balakrishnan, A., Varadhaganapathy, S., & Easwaramoorthy, S. V. (2026) · Scientific reports · doi:10.1038/s41598-026-47146-w · Cited by 1
“Chest X ray images were subjected to standardized preprocessing and augmentation to help class imbalance and hidden stratification effects which are known for causing model failures that need not be considered further 55 .”
Course Corrections for Clinical AI
DeGrave, A. J., Janizek, J. D., & Lee, S. (2021) · Kidney360 · doi:10.34067/kid.0004152021 · Cited by 1
Caturla, S. I., Mestre, J. D. D. B., & Mozos, Ó. M. (2025) · Future Internet · doi:10.3390/fi17070292 · Cited by 2
Automated Bone Fracture Classification Using a Hybrid Quantum-Classical Neural Network Approach
Praveen, S., Maharajan, K., Yuktesh, P., Guptha, P. Y., & Kumar, R. V. (2025) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet64471.2025.11140367 · Cited by 1
“More recently, Johri et al. [18] developed a quantum-classical convolutional architecture for cancer detection in histopathology images, achieving a 5% accuracy improvement compared to classical-only models.”
Palmer, K. J., Houweling, A., Laturnus, L., Pilkington, J. F., Riera, A., Wladichuk, J. L., & Joy, R. (2026) · Marine Mammal Science · doi:10.1111/mms.70126 · Cited by 3
Measuring Contextual Reliance of Object Detection Models: A Black Box Explainability Method
Vonderhaar, L., Elvira, T., & Ochoa, O. (2026) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213026400051 · Cited by 1
In Data We Trust: Make AI Great Again
Cayce, G. I., & Bailey, C. P. (2025) · IEEE MetroCon · doi:10.1109/metrocon66796.2025.11405786 · Cited by 1
“Dataset bias, insufficient diversity, label noise, hidden subclasses, and improper train–test splits continue to limit real-world reliability across domains ranging from computer vision to medical diagnostics [3], [4], [5], [6].”
Artificial Intelligence Across the Radiology Workflow: A Nine-Stage Narrative Review
Rai, M. C. E., Far, A. B., Darweesh, M., Dhou, S., Aburaed, N., El, S., Elkhazendar, M., & Ellahham, S. (2026) · Diagnostics · doi:10.3390/diagnostics16101485 · Cited by 1
“AI can also contribute to operational management by enabling dynamic worklist prioritization, allowing urgent examinations to be escalated for timely radiologist review [38].”
Uncovering and mitigating spurious features in domain generalization
Karimi, S., & Dibeklioğlu, H. (2024) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.55730/1300-0632.4071 · Cited by 2
Linking in Style: Understanding Learned Features in Deep Learning Models
Wehrheim, M. H., Osuna-Vargas, P., & Kaschube, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73013-9_10 · Cited by 1
Clinical Reports Classification using Fine-Tuned DistilBERT Model
Abid, M. A., Rahman, A., Ali, S., & Haider, U. (2024) · International Conference on IT and Industrial Technologies (ICIT) · doi:10.1109/icit63607.2024.10859364 · Cited by 1
Achara, A., Anton, E. P., Hammers, A., King, A. P., & Initiative, A. D. N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05870-6_16 · Cited by 1
“Spurious features can be learned by DL algorithms for tasks such as classification [23, 34], segmentation [22] or generation [7, 10].”
Fairness and Robustness of CLIP-Based Models for Chest X-Rays
Sourget, T., Restrepo, D. S., Hudelot, C., Ferrante, E., Christodoulidis, S., & Vakalopoulou, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05870-6_2 · Cited by 2
BiasICL: In-Context Learning and Demographic Biases of Vision Language Models
Xu, S. H., Janizek, J. D., Jiang, Y., & Daneshjou, R. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04981-0_9 · Cited by 1
From Lesion to Decision: AI for ARIA Detection and Predictive Imaging in Alzheimer’s Disease
Christodoulou, R. C., Papageorgiou, P. S., Sarquis, M. D., Rivera, L., Gonzalez, C. M., Eller, D. W., Rivera, G., Petrou, V., Vamvouras, G., Vassiliou, E. K., et al. (2025) · Biomedicines · doi:10.3390/biomedicines13112739 · Cited by 3
“ARIA events, while clinically significant, are relatively uncommon for some agents and regimens, with ARIA-E rates ranging from 10% in particular phase 2 lecanemab cohorts to ~20–24% in phase 3 trials, and clustering early during titration [15,63,64].”
Debiased Prompt Tuning for Vision-Language Models without Annotations
Jiang, C., Yang, Y., Hu, R., & Sang, J. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227756 · Cited by 2
“Moreover, in real-world applications, relying on spurious correlations may cause serious safety issues and societal impacts, such as medical diagnosis [14], [15], autonomous driving [16], and face recognition [17].”
Label-Free Estimation of Clinically Relevant Performance Metrics Under Distribution Shifts
Flühmann, T., Bissoto, A. E., Hoang, T.-D., & Koch, L. M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06593-3_8 · Cited by 1
Dindorf, C., Dully, J., Simon, S., Perchthaler, D., Becker, S. T., Ehmann, H., Heitmann, K. R., Stetter, B. J., Diers, C., & Fröhlich, M. (2026) · Scientific Reports · doi:10.1038/s41598-025-33707-y · Cited by 1
“Similar problems have been documented in other domains, such as medical imaging, where hidden stratification—arising from unrecognized heterogeneity within a class—has significantly reduced model performance 52 .”
Quantum artificial intelligence: A comprehensive survey
Jiang, H., Pan, Y., Chen, J., Liu, Z., Sun, L., Li, Q., Zhang, L., Zhu, D., Wang, X., Liu, W., et al. (2026) · Meta-Radiology · doi:10.1016/j.metrad.2026.100205 · Cited by 2
Cabitza, F. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-17216-7_21 · Cited by 1
Ebo, T. O., Osunmakinde, A., Asaolu, A. J., Ebo, D. M., Egbon, E., & Olawade, D. B. (2026) · AI & Society · doi:10.1007/s00146-026-02930-0 · Cited by 1
“A study emphasized that subgroup analysis is often omitted in published AI models, even though such testing is critical for uncovering inequities that may be invisible at the aggregate level (Oakden-Rayner et al. 2020).”
Bruun, F. J., Omaraee, Y., Nybing, J. D., Gosvig, K. K., Boesen, M., Hansen, P., Müller, F. C., & Brejnebøl, M. W. (2026) · European Journal of Radiology · doi:10.1016/j.ejrad.2026.112778 · Cited by 1
Task-Based Sampling of Patient Data for Rigorous Machine Learning/AI Performance Assessment
Baughan, N. M., Whitney, H. M., Drukker, K., Sahiner, B., Hu, T., KIM, G. H., McNitt-Gray, M. F., Myers, K. J., & Giger, M. L. (2026) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-026-01894-3 · Cited by 1
Metadata
- Authors
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
- Affiliations
- Australian Institute for Machine Learning, University of Adelaide
- Stanford University (US)
- The University of Adelaide (AU)
- Department of Computer Science, Stanford University
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 151-159
- PMC
- PMC7665161
- OpenAlex
- W2975035192
- Cited by
- 411
- References
- 33
- Citations by year
- 2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toshiba
- Teradata
- Ant Financial
- Intel Corporation
- NEC Corporation
- Taiwan Semiconductor Manufacturing Company
- Accenture
- Qualcomm
- Analog Devices
- VMWare
- Infosys
- National Institutes of Health (U54EB020405)
- NXP
- Xilinx
- LETI-CEA
- American Family Insurance
- Swiss Re
- Intelligence Community Postdoctoral Research Fellowship Program
- National Science Foundation (CCF1763315, CCF1563078, CC1937301)
- IBM
- Hitachi
- Okawa Foundation for Information and Telecommunications
- Google Cloud
- Office of Naval Research (N000141712266)
- Gordon and Betty Moore Foundation
- ARM
- Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
- Microsoft
- BASF
- Ericsson
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.