Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this example, reproduced from Oakden-Rayner and colleagues, 13 the artificial intelligence model learns to detect the artifacts of clinical care (chest drains) and fails to adequately learn the features of the pathology itself.”
From The medical algorithmic audit (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences53.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences45.”
From Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension (2020) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, this drop in performance might not be obvious at the aggregate level of typical artificial intelligence testing, but rather in subsets of the target cohort or specific strata within the input data, a concept that has been described as hidden stratification.13”
From The medical algorithmic audit (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Esteva et al. [12] is particularly notable, as this dataset is labelled for more than 2,000 diagnostic subclasses but the results presented only consider “top-level” diagnostic categories.”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Gulshan et al. [16], for instance, present variations in retinopathy detection performance on subsets with images obtained in different locations, with differing levels of disease severity, and with different degrees of pupil dilation.”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“Deep learning systems have shown remarkable promise in medical image analysis, often claiming performance rivaling that of human experts [13].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundA dendrite method for cluster analysis
Caliński, T., & Harabasz, J. (1974) · Communications in Statistics · doi:10.1080/03610927408827101 · Cited by 4,480
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“]. However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40]. As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techn…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“observed [6, 34]; such analyses may be helpful in identifying error modes via human review, but do not characterize the full space of subset performance [33].”
Background
Research works printed but not linked: 1
- Irvin Jeremy, Rajpurkar Pranav, Ko Michael, Yu Yifan, Ciurea-Ilcus Silviana, Chute Chris, Marklund Henrik, Haghgoo Behzad, Ball Robyn, Shpanskaya Katie, and Others. 2019 Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019). · arXiv
Cited by
Items 101–200 of 440
Yi, P. H., Kim, T. K., Yu, A. C., Bennett, B., Eng, J., & Lin, C. T. (2020) · Emergency Radiology · doi:10.1007/s10140-020-01767-4 · Cited by 41
“One concern about the dataset used in this and similar studies [12, 14] is the high prevalence of chest tubes in images labeled positive for pneumothorax.”
BackgroundMethodEyuboglu, S., Angus, G., Patel, B. N., Pareek, A., Davidzon, G., Long, J., Dunnmon, J., & Lungren, M. P. (2021) · Nature Communications · doi:10.1038/s41467-021-22018-1 · Cited by 58
Self-citationModel-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution DataPreprint
Robey, A., Hassani, H., & Pappas, G. J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.10247 · Cited by 72
“Importantly, many of these domains are safety-critical, meaning that the detections, recommendations, or decisions made by deep learning systems can directly impact the well-being of humans [12].”
BackgroundThe Role of Artificial Intelligence in the Identification and Evaluation of Bone Fractures
Tieu, A., Kroen, E., Kadish, Y., Liu, Z., Patel, N. R., Zhou, A., Yilmaz, A., Lee, S., & Deyer, T. W. (2024) · Bioengineering · doi:10.3390/bioengineering11040338 · Cited by 43
“Due to the diversity of visual characteristics associated with any given pathology, even overall well-performing models may consistently misdiagnose specific subsets of cases, especially when imaging features are subtle or underrepresented [123].”
BackgroundSeedat, N., Imrie, F., & van der Schaar, M. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3345805 · Cited by 65
Reporting Guidelines for Artificial Intelligence in Medical Research
Campbell, J. P., Lee, A., Abràmoff, M. D., Keane, P. A., Ting, D. S. W., Lum, F., & Chiang, M. F. (2020) · Ophthalmology · doi:10.1016/j.ophtha.2020.09.009 · Cited by 37
“12 The recommendation is to “describe results of any analysis of performance errors and how errors were identified, where applicable.”
BackgroundCritical Bias in Critical Care Devices
Charpignon, M.-L., Byers, J., Cabral, S., Celi, L. A., Fernandes, C., Gallifant, J., Lough, M. E., Mlombwa, D., Moukheiber, L., Ong, B. A., et al. (2023) · Critical Care Clinics · doi:10.1016/j.ccc.2023.02.005 · Cited by 35
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Eriksson, M., Purificato, E., Noroozian, A., Vinagre, J., Chaslot, G., Gomez, E., & Fernandez-Llorca, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06559 · Cited by 35
“For instance, Oakden-Rayner et al. (2019) found that an X-ray image classification model predicted collapsed lungs with high accuracy, although it turned out the model only identified the presence of a chest drain (used to cure the condition) that was (unknowingly) present in a majority of the…”
Why do probabilistic clinical models fail to transport between sites
Lasko, T. A., Strobl, E. V., & Stead, W. W. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01037-4 · Cited by 64
“As models have grown in their power to use complex distributional patterns, so has their need to adapt to this shift.”
BackgroundGenerative Artificial Intelligence in Medicine
Shanmugam, D., Agrawal, M., Movva, R., Chen, I. Y., Ghassemi, M., Jacobs, M. L., & Pierson, E. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-095332 · Cited by 42
Mccradden, M., Odusi, O., Joshi, S., Akrout, I., Ndlovu, K., Glocker, B., Maicas, G., Liu, X., Mazwi, M., Garnett, T., et al. (2023) · 2023 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3593013.3594096 · Cited by 33
Self-citationNew trend in artificial intelligence-based assistive technology for thoracic imaging
Yanagawa, M., Ito, R., Nozaki, T., Fujioka, T., Yamada, A., Fujita, S., Kamagata, K., Fushimi, Y., Tsuboyama, T., Matsui, Y., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01691-w · Cited by 36
“In addition, the high degree of freedom in the design of hidden layers is a factor in the excellent expressiveness of neural networks, but hidden stratification may cause better or worse performance of CNN models on certain subsets [26, 27].”
BackgroundEvans, H., & Snead, D. R. J. (2023) · Histopathology · doi:10.1111/his.15071 · Cited by 40
Machine learning robustness: a primer
Braiek, H. B., & Khomh, F. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-323761-4.00012-2 · Cited by 78
Boyd, C., Brown, G. C., Kleinig, T. J., Dawson, J. A., McDonnell, M. D., Jenkinson, M., & Bezak, E. (2021) · Diagnostics · doi:10.3390/diagnostics11030551 · Cited by 33
Ross, J., Hammouche, S., Chen, Y., Rockall, A., Alabed, S., Chen, M., Dwivedi, K., Fascia, D., Greenhalgh, R., Hall, M., et al. (2024) · Clinical Radiology · doi:10.1016/j.crad.2024.01.026 · Cited by 35
Venkatesh, K., Mutasa, S., Moore, F., Sulam, J., & Yi, P. H. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01136-4 · Cited by 41
“These results ech-oed the take-home points from Oakden-Rayner et al. [33]: schema completion—i.e., assigning more complete subclass labels beneath the superclass—often reveals label imbalances that correlate to uneven AI performance across these labels.”
Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction
Cauchois, M., Gupta, S., & Duchi, J. (2020) · arXiv · doi:10.48550/arxiv.2004.10181 · Cited by 31
Differences between human and machine perception in medical diagnosis
Makino, T., Jastrzębski, S., Oleszkiewicz, W., Chacko, C., Ehrenpreis, R., Samreen, N., Chhor, C., Kim, E., Lee, J., Pysarenko, K., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10526-z · Cited by 30
The risk of shortcutting in deep learning algorithms for medical imaging research
Hill, B. G., Koback, F. L., & Schilling, P. L. (2024) · Scientific Reports · doi:10.1038/s41598-024-79838-6 · Cited by 50
“…a cow based on a green grassy background versus a camel on a tan sandy background 23 , healthy patients getting chest X-rays in an upright position and sicker patients in a reclined position 24 , pen marks in skin lesion classification 25 and chest drains in pneumothorax classification 26 .”
Zeng, A., Houssami, N., Noguchi, N., Nickel, B., & Marinovich, M. L. (2024) · Breast Cancer Research and Treatment · doi:10.1007/s10549-024-07353-3 · Cited by 33
“Such failures require remediation in the workflow, and systematic failures have the potential to have disproportionate impacts on different sub-populations [3].”
BackgroundKodenko, M. R., Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Leonov, D. V., Blokhin, I. A., Novik, V. P., Kulberg, N. S., Samorodov, A. V., Mokienko, O. A., et al. (2022) · Diagnostics · doi:10.3390/diagnostics12123197 · Cited by 30
“This is a well-known problem of reviews of AI studies [39] that requires regulatory attention.”
BackgroundDetecting Shortcuts in Medical Images - A Case Study in Chest X-Rays
Jiménez-Sánchez, A., Juodelyte, D., Chamberlain, B., & Cheplygina, V. (2023) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi53787.2023.10230572 · Cited by 47
“Labels were kindly provided by Lauren OakdenRayner, a board-certified radiologist and author of [15].”
InfluentialThe Role of Artificial Intelligence and Machine Learning in Cardiovascular Imaging and Diagnosis
Reza-Soltani, S., Alam, L. F., Debellotte, O., Monga, T. S., Coyalkar, V. R., Tarnate, V. C. A., Ozoalor, C. U., Allam, S. R., Afzal, M., Shah, G. K., et al. (2024) · Cureus · doi:10.7759/cureus.68472 · Cited by 28
“Moreover, the lack of standardized labeling practices and the presence of noise or artifacts in medical images can compromise the accuracy of AI algorithms [52].”
BackgroundKaya, Y., & Kuncan, F. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.104023 · Cited by 30
Pseudo Bias-Balanced Learning for Debiased Chest X-Ray Classification
Luo, L., Xu, D., Chen, H., Wong, T.-T., & Heng, P.-A. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-16452-1_59 · Cited by 26
Ahluwalia, M., Abdalla, M., Sanayei, J., Seyyed-Kalantari, L., Hussain, M., Ali, A., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220270 · Cited by 26
Yao, S., Dai, F., Sun, P., Zhang, W., Qian, B., & Lu, H. J. (2024) · Nature Communications · doi:10.1038/s41467-024-44906-y · Cited by 32
A data-driven approach to decode metabolic dysfunction-associated steatotic liver disease
Ramos, M. J., Kendall, T. J., Drozdov, I. A., & Fallowfield, J. A. (2023) · Annals of Hepatology · doi:10.1016/j.aohep.2023.101278 · Cited by 27
Rapid Assisted Visual Search: Supporting Digital Pathologists with Imperfect AI
Lindvall, M., Lundström, C. F., & Löwgren, J. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450681 · Cited by 44
“of cases in real practice can cause unpredictable errors [24, 28].”
BackgroundAbbas, A., O’Byrne, C., Fu, D. J., Moraes, G., Balaskas, K., Struyven, R., Beqiri, S., Wagner, S. K., Korot, E., & Keane, P. A. (2022) · Graefe s Archive for Clinical and Experimental Ophthalmology · doi:10.1007/s00417-021-05544-y · Cited by 33
“Barriers to clinical implementation include the lack of insight into how models reach their decisions and concerns regarding hidden stratification as a source of bias [43].”
BackgroundGround Truth Or Dare: Factors Affecting The Creation Of Medical Datasets For Training AI
Zając, H. D., Avlona, N.-R., Kensing, F., Andersen, T. O., & Shklovski, I. (2023) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3600211.3604766 · Cited by 51
“hidden stratification [52]), or limited features (an inadequate number of features represented in data) [4].”
BackgroundDistributionally Robust Segmentation of Abnormal Fetal Brain 3D MRI
Fidon, L., Aertsen, M., Mufti, N., Deprest, T., Emam, D., Guffens, F., Schwartz, E., Ebner, M., Prayer, D., Kasprian, G. J., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87735-4_25 · Cited by 25
“This problem has been referred to as hidden stratification [17].”
BackgroundInfluentialHurdles to Artificial Intelligence Deployment: Noise in Schemas and “Gold” Labels
Abdalla, M., & Fine, B. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220056 · Cited by 24
Anemia Estimation for COVID-19 Patients Using A Machine Learning Model
El‐kenawy, E. M., Eid, M. M., & Ibrahim, A. F. (2021) · Cited by 32
Subbaswamy, A., Sahiner, B., Petrick, N., Pai, V. M., Adams, R., Diamond, M. C., & Saria, S. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01275-6 · Cited by 36
Bottani, S., Burgos, N., Maire, A., Saracino, D., Ströer, S., Dormont, D., & Colliot, O. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102903 · Cited by 27
“…department (Zech et al., 2018), magnetic field strength (Thibeau-Sutre et al., 2022a), the presence of a chest drain that is actually implemented only after the diagnosis and not before (Oakden-Rayner et al., 2020) and skin markings associated with melanoma detection (Winkler et al., 2019).”
Interpreting mental state decoding with deep learning models.
Thomas, A. W., Ré, C., & Poldrack, R. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.07.003 · Cited by 28
Pfohl, S. R., Zhang, H., Xu, Y., Foryciarz, A., Ghassemi, M., & Shah, N. H. (2022) · Scientific Reports · doi:10.1038/s41598-022-07167-7 · Cited by 43
“differences in model properties for unlabeled subpopulations or for intersectional ones defined across attributes [56].”
BackgroundSelective Classification Can Magnify Disparities Across GroupsPreprint
Jones, E., Sagawa, S., Koh, P. W., Kumar, A., & Liang, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.14134 · Cited by 45
“, 2020), spurious clinical variables (Badgeley et al., 2019; Oakden-Rayner et al., 2020), or linguistic artifacts (Gururangan et al.”
BackgroundInfluentialMandoline: Model Evaluation under Distribution ShiftPreprint
Chen, M. F., Goel, K., Sohoni, N. S., Poms, F., Fatahalian, K., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.00643 · Cited by 74
“X-rays of critically ill patients with chest drains [Oakden-Rayner et al., 2020] or demographic slices when detecting toxic comments [Borkan et al.”
BackgroundFoundation models for radiology—the position of the AI for Health Imaging (AI4HI) network
de Almeida, J. G., Alberich, L. C., Tsakou, G., Marias, K., Tsiknakis, M., Lekadir, K., Martí‐Bonmatí, L., & Papanikolaou, N. K. (2025) · Insights into Imaging · doi:10.1186/s13244-025-02056-9 · Cited by 26
Mahmood, U. A., Shrestha, R., Bates, . D. B., Mannelli, L., Corrias, G., Erdi, Y. E., & Kanan, C. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.671015 · Cited by 30
“Sanity tests are occasionally used to identify if a system is unlikely to generalize (Adebayo et al., 2018; Winkler et al., 2019; Oakden-Rayner et al., 2020).”
BackgroundOn (assessing) the fairness of risk score models
Petersen, E., Ganz, M., Holm, S., & Feragen, A. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594045 · Cited by 37
“Such artificial discretization of continuous variables is well-known to confound statistical analyses [1, 73, 86], and it might also cause problems in our setting.”
BackgroundFarris, A. B., Alexander, M. P., Balis, U. J., Barisoni, L. M. C., Boor, P., Bülow, R. D., Cornell, L. D., Demetris, A. J., Farkash, E. A., Hermsen, M., et al. (2023) · Transplant International · doi:10.3389/ti.2023.11783 · Cited by 24
In the Picture: Medical Imaging Datasets, Artifacts, and their Living Review
Jiménez-Sánchez, A., Avlona, N.-R., de Boer, S., Campello, V. M., Feragen, A., Ferrante, E., Ganz, M., Gichoya, J. W., González, C., Groefsema, S., et al. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732035 · Cited by 36
“Insufficient or lack of documentation may miss information such as demographics or hospital scanner, leading to biased and inaccurate models [17, 125].”
InfluentialClinician-Driven AI: Code-Free Self-Training on Public Data for Diabetic Retinopathy Referral
Korot, E., Gonçalves, M. B., Huemer, J. C., Beqiri, S., Khalid, H., Kelly, M., Chia, M. A., Mathijs, E., Struyven, R., Moussa, M. S., et al. (2023) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2023.4508 · Cited by 25
Testing Deep Learning-based Visual Perception for Automated Driving
Abrecht, S., Gauerhof, L., Gladisch, C., Groh, K., Heinzemann, C., & Woehrle, M. (2021) · ACM Transactions on Cyber-Physical Systems · doi:10.1145/3450356 · Cited by 27
“Testing the function under controlled influence factor variations can help to avoid hidden stratification [71] as stratification is performed as a dedicated and review-able task.”
BackgroundPerspectives: A surgeon’s guide to machine learning
Kuo, R. Y. L., Harrison, C. J., Jones, B., Geoghegan, L., & Furniss, D. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.106133 · Cited by 20
Black, E., Naidu, R. C. A., Ghani, R., Rodolfa, K. T., Ho, D. E., & Heidari, H. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623259 · Cited by 44
Fathi, M., Eshraghi, R., Behzad, S., Tavasol, A., Bahrami, A., Tafazolimoghadam, A., Bhatt, V., Ghadimi, D. J., & Gholamrezanezhad, A. (2024) · Emergency Radiology · doi:10.1007/s10140-024-02278-2 · Cited by 16
“Errors and missed diagnoses from the use of AI models may be due to issues like data imbalances in training populations, especially in cases of uncommon pathology or specific demographic groups [13].”
Machine Learning Systems in the IoT: Trustworthiness Trade-offs for Edge Intelligence
Toussaint, W., & Ding, A. Y. (2020) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi50398.2020.00030 · Cited by 22
“For example, in medical image classification, it has been established that training data can contain unrecognised categories that are not in the labels but that affect predictive outcomes [19].”
BackgroundA survey on cell nuclei instance segmentation and classification: Leveraging context and attention
Nunes, J. D., Montezuma, D., Oliveira, D., Pereira, T., & Cardoso, J. S. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103360 · Cited by 24
Clinical applications of artificial intelligence in cardiology on the verge of the decade
Pieszko, K., Hiczkiewicz, J., Budzianowski, J., Musielak, B., Hiczkiewicz, D., Faron, W., Rzeźniczak, J., & Burchardt, P. (2020) · Cardiology Journal · doi:10.5603/cj.a2020.0093 · Cited by 19
“This term was coined by researchers from the University of Adelaide and Stanford University [58] and describes a situation in supervised learning when, due to the coarse labeling of the data (for example normal vs.”
BackgroundMethodInfluentialVision transformer to differentiate between benign and malignant slices in 18F-FDG PET/CT
Nishigaki, D., Suzuki, Y., Watabe, T., Katayama, D., Kato, H., Wataya, T., Kita, K., Sato, J., Tomiyama, N., & Kido, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-58220-6 · Cited by 18
“The effects of hidden stratification can be problematic in machine learning for medical imaging 31 .”
BackgroundChoi, J., Hui, J. Z., Spain, D. A., Su, Y.-S., Cheng, C., & Liao, C. (2021) · Trauma Surgery & Acute Care Open · doi:10.1136/tsaco-2021-000705 · Cited by 18
Tang, C. H. M., Seah, J. C. Y., Ahmad, H. K., Milne, M., Wardman, J. B., Buchlak, Q. D., Esmaili, N., Lambert, J. F., & Jones, C. M. (2023) · Diagnostics · doi:10.3390/diagnostics13142317 · Cited by 20
“Algorithms designed to identify pneumothoraces with strong performance across an entire test dataset perform worse in subsets of patients without an accompanying intercostal drain, due to correlation between intercostal drains and pneumothoraces on CXR [32].”
BackgroundBaughan, N. M., Whitney, H. M., Drukker, K., Sahiner, B., Hu, T., Kim, G. H. J., McNitt‐Gray, M. F., Myers, K. J., & Giger, M. L. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.6.064501 · Cited by 21
“17 The developed sequestration algorithm will provide an open-source tool for avoiding hidden stratification, through the use of a type of variable population or schema completion.”
MethodCalibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
You, C., Mint, Y., Dai, W., Sekhon, J. S., Staib, L. H., & Duncan, J. S. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02470 · Cited by 52
Nyström landmark sampling and regularized Christoffel functions
Fanuel, M., Schreurs, J., & Suykens, J. A. K. (2022) · Machine Learning · doi:10.1007/s10994-022-06165-0 · Cited by 19
“This phenomenon is especially important in stratified data sets (Valverde-Albacete and Pel´aez-Moreno, 2014; Oakden-Rayner et al., 2020; Chen et al., 2019).”
BackgroundMedFILIP: Medical Fine-Grained Language-Image Pre-Training
Liang, X., Li, X., Li, F., Jiang, J., Dong, Q., Wang, W., Wang, K., Dong, S., Luo, G., & Li, S. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3528196 · Cited by 31
Diversity Sampling is an Implicit Regularization for Kernel Methods
Fanuel, M., Schreurs, J., & Suykens, J. A. K. (2021) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m1320031 · Cited by 16
Seah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Milne, M., Holt, X., Ahmad, H., Lambert, J. F., Esmaili, N., Oakden‐Rayner, L., Brotchie, P., et al. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-053024 · Cited by 18
Self-citationDecisions are not all equal—Introducing a utility metric based on case-wise raters’ perceptions
Campagner, A., Sternini, F., & Cabitza, F. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2022.106930 · Cited by 20
Steele, L., Tan, X. L., Olabi, B., Gao, J., Tanaka, R. J., & Williams, H. C. (2023) · Journal of the European Academy of Dermatology and Venereology · doi:10.1111/jdv.18814 · Cited by 17
Critical evaluation of deep neural networks for wrist fracture detection
Raisuddin, A. M., Vaattovaara, E., Nevalainen, M. T., Nikki, M. J., Järvenpää, E., Makkonen, K., Pinola, P., Palsio, T., Niemensivu, A., Tervonen, O. A., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-85570-2 · Cited by 109
“Recent studies on hidden stratification show that performance drop can be significantly high for an unaddressed stratum 23,24 .”
BackgroundInfluentialGermain, P., Vardazaryan, A., Padoy, N., Labani, A., Roy, C., Schindler, T. H., & Ghannudi, S. E. (2021) · Diagnostics · doi:10.3390/diagnostics11091554 · Cited by 15
“This can also hint at a problem called hidden stratification [23]; however, error auditing was not able to recognize anomalous patterns in our cases.”
Teaching Machine Learning as Part of Agile Software Engineering
Chenoweth, S., & Linos, P. K. (2023) · IEEE Transactions on Education · doi:10.1109/te.2023.3337343 · Cited by 14
Evaluating deep transfer learning for whole-brain cognitive decoding
Thomas, A. W., Lindenberger, U., Samek, W., & Müller, K. (2023) · Journal of the Franklin Institute · doi:10.1016/j.jfranklin.2023.07.015 · Cited by 17
AUR-RRA Review: Logistics of Academic-Industry Partnerships in Artificial Intelligence
Spilseth, B., McKnight, C. D., Li, M., Park, C. J., Fried, J. G., Yi, P. H., Brian, J. M., Lehman, C. D., Wang, X., Phalke, V., et al. (2021) · Academic Radiology · doi:10.1016/j.acra.2021.08.002 · Cited by 15
de Matos, J. C. R. G., CALSTER, B. V., Celi, L. A., Dhiman, P., Gichoya, J. W., Riley, R. D., Russell, C., Khalid, S., & Collins, G. S. (2026) · The Lancet Digital Health · doi:10.1016/j.landig.2026.101001 · Cited by 27
“37,38 Model performance, however, is expected to vary across subgroups.”
Source Matters: Source Dataset Impact on Model Robustness in Medical Imaging
Juodelyte, D., Lu, Y., Jiménez-Sánchez, A., Bottazzi, S., Ferrante, E., & Cheplygina, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82007-6_11 · Cited by 18
Oberije, C., Currie, R., Leaver, A., Redman, A., Teh, W., Sharma, N., Fox, G., Glocker, B., Khara, G. S., Nash, J., et al. (2025) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2024-101318 · Cited by 16
Yi, P. H., Malone, P. S., Lin, C. T., & Filice, R. W. (2021) · American Journal of Roentgenology · doi:10.2214/ajr.21.26882 · Cited by 13
Towards trustworthy seizure onset detection using workflow notes
Saab, K. K., Tang, S., Taha, M., Lee‐Messer, C., Ré, C., & Rubin, D. L. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01008-9 · Cited by 16
CGBNet: A Deep Learning Framework for Compost Classification
Gangopadhyay, S., & Zhai, A. (2022) · IEEE Access · doi:10.1109/access.2022.3201099 · Cited by 15
“This problem is also known as hidden stratification [29], in which the superclass labels of green and brown compost cannot meaningfully describe intraclass variations in visual characteristics.”
BackgroundGupta, R., Sasaki, M., Taylor, S. L., Fan, S., Hoch, J. S., Zhang, Y., Crase, M., Tancredi, D. J., Adams, J. Y., & Ton, H. (2025) · Journal of General Internal Medicine · doi:10.1007/s11606-025-09462-1 · Cited by 14
Population-scale cross-sectional observational study for AI-powered TB screening on one million CXRs
Munjal, P., Mahrooqi, A. A., Rajan, R., Jeremijenko, A. M., Ahmad, I., Akhtar, M. I., Pimentel, M. A. F., & Khan, S. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01832-7 · Cited by 13
Muhammad, D., Salman, M., & Bendechache, M. (2025) · Computational and Structural Biotechnology Reports · doi:10.1016/j.csbr.2025.100066 · Cited by 13
The Role of Subgroup Separability in Group-Fair Medical Image Classification
Jones, C., Roschewitz, M., & Glocker, B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43898-1_18 · Cited by 19
FACTS: First Amplify Correlations and Then Slice to Discover Bias
Yenamandra, S., Ramesh, P., Prabhu, V., & Hoffman, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00442 · Cited by 22
“…for this watermark, it is identical to the accepted version; the final published version of the proceedings is available on IEEE Xplore. riously correlated with latent [6] attributes which may be unknown apriori [22], or not correspond to a clean, “human-interpretable” concept [23] to begin with!”
BackgroundDistributionally Robust Deep Learning using Hardness Weighted Sampling
Fidon, L., Aertsen, M., Deprest, T., Emam, D., Guffens, F., Mufti, N., Elslander, E. V., Schwartz, E., Ebner, M., Prayer, D., et al. (2022) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2022-8b6a · Cited by 17
“Hidden stratification has been shown to lead to deep learning models with good average performance but poor performance on underrepresented but clinically relevant subsets of the population (Larrazabal et al., 2020; Oakden-Rayner et al., 2020; PuyolAntón et al., 2021).”
BackgroundTrabilsy, M., Prabha, S., Gomez-Cabello, C. A., Haider, S. A., Genovese, A., Borna, S., Wood, N. G., Gopala, N., Tao, C., & Forte, A. J. (2025) · Bioengineering · doi:10.3390/bioengineering12070706 · Cited by 14
“For instance, Oakden-Rayner et al. (2020) exposed diagnostic biases in chest X-ray classifiers, revealing models that relied on confounding variables such as hospital identifiers rather than true pathology [17].”
Dunnmon, J. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.006 · Cited by 10
Self-citationGenMuNN: A mutation-based approach to repair deep neural network models
Wu, H., Li, Z., Cui, Z., & Liu, J. (2022) · Advances in Complex Systems · doi:10.1142/s1793962323410088 · Cited by 12
Failures Hiding in Success for Artificial Intelligence in Radiology
Purkayastha, S., Trivedi, H., & Gichoya, J. W. (2021) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.11.008 · Cited by 10
Fast Diffusion-Based Counterfactuals for Shortcut Removal and Generation
Weng, N., Pegios, P., Petersen, E., Feragen, A., & Bigdeli, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73016-0_20 · Cited by 52
Shew, M. A., Pavelchek, C., Michelson, A. P., Ortmann, A. J., Lefler, S. M., Walia, A., Durakovic, N., Phillips, A., Rejepova, A., Herzog, J. A., et al. (2025) · Ear and Hearing · doi:10.1097/aud.0000000000001664 · Cited by 11
The point‐of‐care use of a facial phenotyping tool in the genetics clinic: An ethics tête‐a‐tête
McCradden, M. D., Patel, E., & Chad, L. (2020) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.61985 · Cited by 10
“At the present time, a lack of diversity in genomic studies is a significant problem (Popejoy & Fullerton, 2016), which may be linked to (among other factors) the perceived trustworthiness of the data-collecting endeavor (Geneviève, Martani, Shaw, Elger, & Wangmo, 2020).”
BackgroundWho wants accurate models? Arguing for a different metrics to take classification models seriouslyPreprint
Cabitza, F., & Campagner, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09246 · Cited by 14
“datasets, a problem that has been recently denoted as hidden stratification [31]: accuracy can be high for a ML model, but this latter still misses relevant findings, that is conditions that can have a strong impact on the patient’s outcome.”
BackgroundFive principles for the intelligent use of AI in medical imaging
Colak, E., Moreland, R., & Ghassemi, M. (2021) · Intensive Care Medicine · doi:10.1007/s00134-020-06316-8 · Cited by 9
“, specific equipment, imaging protocol, patient positioning) used during training [7].”
Keerthika, K., Kannan, M. R., & Saravanan, T. (2024) · Wiley eBooks · doi:10.1002/9781394272587.ch7 · Cited by 8
Quantifying impairment and disease severity using AI models trained on healthy subjects
Yu, B., Kaku, A., Liu, K., Parnandi, A., Fokas, E. E., Venkatesan, A., Pandit, N. G., Ranganath, R., Schambra, H. M., & Fernandez‐Granda, C. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01173-x · Cited by 10
Li, Z., Wang, H., & Li, L. (2025) · Biomimetics · doi:10.3390/biomimetics10070468 · Cited by 10
“This highlights the potential limitations of purely data-driven approaches in clinical classification tasks, especially in the presence of ambiguous or borderline cases [45].”
Chang, L., Lee, K.-H., Mukundan, A., Karmakar, R., Bauravindah, A., Chen, T., Huang, C., & Wang, H. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1707854 · Cited by 8
Lakkimsetti, M., Devella, S. G., Patel, K. B., Dhandibhotla, S., Kaur, J., Mathew, M., Kataria, J., Nallani, M., Farwa, U. E., Patel, T. D., et al. (2024) · Cureus · doi:10.7759/cureus.58400 · Cited by 10
“Under-represented data used to train the AI/ML algorithm can be another concern while predicting the performance of edge cases [93].”
Olesen, V., Weng, N., Feragen, A., & Petersen, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72787-0_1 · Cited by 22
“With few exceptions [19], SDMs have not yet found widespread use in the medical imaging domain.”
InfluentialIt's Not What Machines Can Learn, It's What We Cannot TeachPreprint
Yehuda, G., Gabel, M., & Schuster, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09398 · Cited by 16
Synthetic Simplicity: Unveiling Bias in Medical Data Augmentation
Babu, K. A. R., Sathish, R., Pattanaik, M., & Venkataramani, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73748-0_7 · Cited by 9
“While the dangers of shortcut learning in the context of healthcare [4,11] have been discussed, there is no prior work in the context of spurious features manifested due to utilization of synthetic data.”
BackgroundAugmenting Chest X-ray Datasets with Non-Expert Annotations
Cheplygina, V., Damgaard, C., Eriksen, T. N., Juodelyte, D., & Jiménez-Sánchez, A. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-98688-8_10 · Cited by 11
“However, CheXpert and NIH-CXR14 datasets have been shown to suffer from bias [8, 15, 26] or shortcuts [13, 21].”
BackgroundMethod
Metadata
- Authors
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
- Affiliations
- Australian Institute for Machine Learning, University of Adelaide
- Stanford University (US)
- The University of Adelaide (AU)
- Department of Computer Science, Stanford University
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 151-159
- PMC
- PMC7665161
- OpenAlex
- W2975035192
- Cited by
- 411
- References
- 33
- Citations by year
- 2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toshiba
- Teradata
- Ant Financial
- Intel Corporation
- NEC Corporation
- Taiwan Semiconductor Manufacturing Company
- Accenture
- Qualcomm
- Analog Devices
- VMWare
- Infosys
- National Institutes of Health (U54EB020405)
- NXP
- Xilinx
- LETI-CEA
- American Family Insurance
- Swiss Re
- Intelligence Community Postdoctoral Research Fellowship Program
- National Science Foundation (CCF1763315, CCF1563078, CC1937301)
- IBM
- Hitachi
- Okawa Foundation for Information and Telecommunications
- Google Cloud
- Office of Naval Research (N000141712266)
- Gordon and Betty Moore Foundation
- ARM
- Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
- Microsoft
- BASF
- Ericsson
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.