Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this example, reproduced from Oakden-Rayner and colleagues, 13 the artificial intelligence model learns to detect the artifacts of clinical care (chest drains) and fails to adequately learn the features of the pathology itself.”
From The medical algorithmic audit (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences53.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences45.”
From Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension (2020) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, this drop in performance might not be obvious at the aggregate level of typical artificial intelligence testing, but rather in subsets of the target cohort or specific strata within the input data, a concept that has been described as hidden stratification.13”
From The medical algorithmic audit (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Esteva et al. [12] is particularly notable, as this dataset is labelled for more than 2,000 diagnostic subclasses but the results presented only consider “top-level” diagnostic categories.”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Gulshan et al. [16], for instance, present variations in retinopathy detection performance on subsets with images obtained in different locations, with differing levels of disease severity, and with different degrees of pupil dilation.”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“Deep learning systems have shown remarkable promise in medical image analysis, often claiming performance rivaling that of human experts [13].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundA dendrite method for cluster analysis
Caliński, T., & Harabasz, J. (1974) · Communications in Statistics · doi:10.1080/03610927408827101 · Cited by 4,480
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“]. However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40]. As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techn…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“observed [6, 34]; such analyses may be helpful in identifying error modes via human review, but do not characterize the full space of subset performance [33].”
Background
Research works printed but not linked: 1
- Irvin Jeremy, Rajpurkar Pranav, Ko Michael, Yu Yifan, Ciurea-Ilcus Silviana, Chute Chris, Marklund Henrik, Haghgoo Behzad, Ball Robyn, Shpanskaya Katie, and Others. 2019 Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901.07031 (2019). · arXiv
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationLiu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421
Self-citationSeyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
“This is a form of bias amplification, when a model’s predicted outputs amplify a known source of error in the process of data generation 34 or data distribution 35 .”
BackgroundCruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 933
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences 45 .”
Self-citationMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
Liu, X., Rivera, S. C., Moher, D., Calvert, M. J., & Denniston, A. K. (2020) · BMJ · doi:10.1136/bmj.m3164 · Cited by 702
“53 Therefore, reporting cases of error and defining risk-mitigation strategies are important for informing when, and for which populations, the intervention can be safely implemented.”
Ethical Machine Learning in Healthcare
Chen, I. Y., Pierson, E., Rose, S., Joshi, S., Ferryman, K., & Ghassemi, M. (2021) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-092820-114757 · Cited by 645
“Clinical analysis of sources of error in disease labels can also guide improvements (89) and identify affected groups (56).”
BackgroundInfluentialSagawa, S., Koh, P. W., Hashimoto, T., & Liang, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08731 · Cited by 331
“…shifts is important future work, existing work has identified many distributional shifts that can be expressed with pre-specified groups, e.g., batch effects in biology (Leek et al., 2010), or image artifacts (Oakden-Rayner et al., 2019) and patient demographics (Badgeley et al., 2019) in medicine.”
BackgroundLiu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., Ashrafian, H., Beam, A. L., Chan, A.-W., Collins, G. S., Deeks, A. D. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30218-1 · Cited by 555
“53 Therefore, reporting cases of error and defining risk-mitigation strategies are important for informing when, and for which populations, the intervention can be safely implemented.”
Self-citationMachine learning in medical applications: a review of state-of-the-art methods
Shehab, M., Abualigah, L., Shambour, Q., Abu-Hashem, M. A., Shambour, M. K. Y., Alsalibi, A. I., & Gandomi, A. H. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.105458 · Cited by 542
Metrics reloaded: recommendations for image analysis validation
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02151-z · Cited by 470
“Detailed failure case analysis 49 and performance assessment on relevant subgroups, for example, have been highlighted as critical components for better understanding when and where an algorithm may fail 50,51 .”
BackgroundDeep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL
Strodthoff, N., Wagner, P., Schaeffter, T., & Samek, W. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3022989 · Cited by 491
“Although identifying hidden stratification is straightforward to identify in hindsight given the hierarchical organization of the diagnostic labels, [55] also demonstrated how to identify groups of samples exhibiting hidden stratification for a given class label under consideration using an unsupervised clustering app…”
BackgroundInfluentialCruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., Ashrafian, H., Beam, A. L., Collins, G. S., Darzi, A., Deeks, J. J., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30219-3 · Cited by 486
Self-citationRivera, S. C., Liu, X., Chan, A.-W., Denniston, A. K., & Calvert, M. J. (2020) · BMJ · doi:10.1136/bmj.m3210 · Cited by 463
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
“…socially salient characteristics like skin type and gender (Buolamwini and Gebru, 2018); the ML for healthcare literature (Obermeyer et al., 2019; Oakden-Rayner et al., 2020), where examples are stratified by subpopulations; and the natural language processing and computer vision literatures…”
BackgroundArtificial intelligence for clinical oncology
Kann, B. H., Hosny, A., & Aerts, H. J. W. L. (2021) · Cancer Cell · doi:10.1016/j.ccell.2021.04.002 · Cited by 370
Armoundas, A. A., Narayan, S. M., Arnett, D. K., Spector-Bagdady, K., Bennett, D. A., Celi, L. A., Friedman, P. A., Gollob, M. H., Hall, J. L., Kwitek, A. E., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001201 · Cited by 347
Assessing the trustworthiness of saliency maps for localizing abnormalities in medical imaging
Arun, N., Gaw, N., Singh, P., Chang, K., Aggarwal, M., Chen, B., Hoebel, K., Gupta, S., Patel, J., Gidwani, M., et al. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2021200267 · Cited by 342
“For example, a chest tube in an image would imply the presence of a pneumothorax, but much of the chest tube may not be in the region of interest (34).”
BackgroundA Guide to Cross-Validation for Artificial Intelligence in Medical Imaging
Bradshaw, T. J., Huemann, Z., Hu, J., & Rahmim, A. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220232 · Cited by 250
The need to separate the wheat from the chaff in medical informatics
Cabitza, F., & Campagner, A. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104510 · Cited by 276
Diao, J. A., Wang, J. K., Chui, W. F., Mountain, V., Gullapally, S. C., Srinivasan, R., Mitchell, R. N., Glass, B., Hoffman, S., Rao, S. K., et al. (2021) · Nature Communications · doi:10.1038/s41467-021-21896-9 · Cited by 269
PanNuke Dataset Extension, Insights and Baselines
Gamper, J., Koohbanani, N. A., Benes, K., Graham, S., Jahanifar, M., Khurram, S. A., Azam, A. S., Hewitt, K. J., & Rajpoot, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10778 · Cited by 146
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251
“read medical images [66, 132], and misdiagnose [180, 203].”
A systematic review of the prediction of hospital length of stay: Towards a unified framework
Stone, K., Zwiggelaar, R., Jones, P., & Mac Parthaláin, N. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000017 · Cited by 231
Groh, M., Harris, C., Soenksen, L., Lau, F., Han, R., Kim, A., Koochek, A., & Badri, O. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00201 · Cited by 229
“…within a population has been shown to lead deep neural networks to produce large accuracy disparities across gender and skin color for facial recognition [11], across images with and without surgical markings in dermatology [56, 10], and across treated and untreated conditions in radiology [40].”
BackgroundLiu, X., Glocker, B., McCradden, M. M., Ghassemi, M., Denniston, A. K., & Oakden-Rayner, L. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00003-6 · Cited by 266
Self-citationSeah, J. C. Y., Tang, C. H. M., Buchlak, Q. D., Holt, X. G., Wardman, J. B., Aimoldin, A., Esmaili, N., Ahmad, H., Pham, H., Lambert, J. F., et al. (2021) · The Lancet Digital Health · doi:10.1016/s2589-7500(21)00106-0 · Cited by 229
Self-citationA Survey of Deep Learning for Scientific Discovery
Raghu, M., & Schmidt, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11755 · Cited by 56
Distributionally Robust Neural Networks
Sagawa, S., Koh, P. W., Hashimoto, T., & Liang, P. (2020) · International Conference on Learning Representations · Cited by 780
Maleki, F., Ovens, K., Gupta, R., Reinhold, C., Spatz, A., & Forghani, R. (2023) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220028 · Cited by 150
Effect of data leakage in brain MRI classification using 2D convolutional neural networks
Yagis, E., Atnafu, S. W., García Seco de Herrera, A., Marzi, C., Scheda, R., Giannelli, M., Tessa, C., Citi, L., & Diciotti, S. (2021) · Scientific Reports · doi:10.1038/s41598-021-01681-w · Cited by 161
Just Train Twice: Improving Group Robustness without Training Group InformationPreprint
Liu, E. Z., Haghgoo, B., Chen, A., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., & Finn, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09044 · Cited by 322
Characterising Bias in Compressed Models
Hooker, S., Moorosi, N., Clark, G., Bengio, S., & Denton, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03058 · Cited by 41
“…constrained but sensitive domains such as hiring (Dastin, 2018; Harwell, 2019), health care diagnostics (Xie et al., 2019; Gruetzemacher et al., 2018; Badgeley et al., 2019; Oakden-Rayner et al., 2019), self-driving cars (NHTSA, 2017) and facial recognition software (Buolamwini & Gebru, 2018b).”
BackgroundAini, Q., Manongga, D., Rahardja, U., Sembiring, I., & Li, Y.-M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2357860 · Cited by 361
“MGA is a valuable tool for gauging the substantial distinctions within different groups within the dataset when dealing with a categorical moderator (Allen et al., 2019; Oakden-Rayner et al., 2020).”
Transparent medical image AI via an image–text foundation model grounded in medical literature
Kim, C., Gadgil, S. U., DeGrave, A. J., Omiye, J. A., Cai, Z. R., Daneshjou, R., & Lee, S.-I. (2024) · Nature Medicine · doi:10.1038/s41591-024-02887-x · Cited by 125
“However, existing approaches for examining datasets for irregularities, which require manual labeling 39 or inspection 21,40 , are labor intensive and thus not scalable to large datasets (Supplementary Discussion).”
The Pitfalls of Simplicity Bias in Neural Networks
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., & Netrapalli, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07710 · Cited by 93
McCoy, L. G., Brenna, C. T. A., Chen, S. S., Vold, K., & Das, S. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.11.001 · Cited by 118
Ianni, J. D., Soans, R. E., Sankarapandian, S., Chamarthi, R. V., Ayyagari, D., Olsen, T. G., Bonham, M. J., Stavish, C. C., Motaparthi, K., Cockerell, C. J., et al. (2020) · Scientific Reports · doi:10.1038/s41598-020-59985-2 · Cited by 112
Gradient Starvation: A Learning Proclivity in Neural NetworksPreprint
Pezeshki, M., Kaba, S.-O., Bengio, Y., Courville, A. C., Precup, D., & Lajoie, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09468 · Cited by 187
“Recent work aims at identifying the reasons behind this phenomenon [97, 70, 22, 73, 51, 76, 100, 92, 83, 105, 42, 79, 4], while complementary work quantifies resulting shortcomings, including poor generalization to out-of-distribution (OOD) test data, reliance upon spurious correlations, and lack of robustness [30, 68…”
BackgroundMcCradden, M. D., Joshi, S., Anderson, J. A., Mazwi, M., Goldenberg, A., & Zlotnik Shaul, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa085 · Cited by 111
Explaining the Black-box Smoothly- A Counterfactual Approach
Singla, S., Eslami, M., Pollack, B., Wallace, S., & Batmanghelich, K. N. (2021) · Medical Image Analysis · doi:10.1016/j.media.2022.102721 · Cited by 156
Tomašev, N., Harris, N., Baur, S., Mottram, A., Glorot, X., Rae, J. W., Zielinski, M., Askham, H., Saraiva, A., Magliulo, V., et al. (2021) · Nature Protocols · doi:10.1038/s41596-021-00513-5 · Cited by 130
Oakden-Rayner, L., Gale, W., Bonham, T. A., Lungren, M. P., Carneiro, G., Bradley, A. P., & Palmer, L. J. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00004-8 · Cited by 93
Self-citationRisk of Bias in Chest Radiography Deep Learning Foundation Models
Glocker, B., Jones, C., Roschewitz, M., & Winzeck, S. (2023) · Radiology Artificial Intelligence · doi:10.1148/ryai.230060 · Cited by 142
“Finetuning is unlikely to be able to ‘unlearn’ biases and may potentially exploit shortcuts for making predictions due to the presence of undesirable correlations in the training data (29,30).”
BackgroundSurgical data science – from concepts toward clinical translation
Maier‐Hein, L., Eisenmann, M., Sarıkaya, D., März, K., Collins, T., Malpani, A., Fallert, J., Feußner, H., Giannarou, S., Mascagni, P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102306 · Cited by 515
“Other examples of confounders (also called hidden variables) are chest drains and skin markings in the context of pneumothorax (Oakden-Rayner et al., 2020) and melanoma diagnosis (Winkler et al.”
BackgroundAlgorithmic encoding of protected characteristics in chest X-ray disease detection models
Glocker, B., Jones, C., Bernhardt, M., & Winzeck, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104467 · Cited by 77
A causal perspective on dataset bias in machine learning for medical imaging
Jones, C., Castro, D. C., Ribeiro, F. D. S., Oktay, O., McCradden, M. D., & Glocker, B. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00797-8 · Cited by 151
Preventing dataset shift from breaking machine-learning biomarkers
Dockès, J., Varoquaux, G., & Poline, J.-B. (2021) · GigaScience · doi:10.1093/gigascience/giab055 · Cited by 86
“Not only can these shifts reduce overall predictive performance, they can also lead to discriminative clinical decisions for poorly represented populations [30, 31, 32, 33, 34].”
Artificial Intelligence and Deep Learning for Rheumatologists
McMaster, C., Bird, A., Liew, D. F. L., Buchanan, R. R. C., Owen, C., Chapman, W. W., & Pires, D. E. V. (2022) · Arthritis & Rheumatology · doi:10.1002/art.42296 · Cited by 98
“Three steps are need for safe implementation: testing on external data sets is needed to ensure models generalize to different population, performance should be evaluated in subgroups to eliminate bias (89), and, ultimately, models should be tested in randomized control trials.”
BackgroundHomeyer, A., Geißler, C., Schwen, L. O., Zakrzewski, F., Evans, T., Strohmenger, K., Westphal, M., Bülow, R. D., Kargl, M., Karjauv, A., et al. (2022) · Modern Pathology · doi:10.1038/s41379-022-01147-y · Cited by 95
“Despite working well for smaller datasets containing similar correlations, such methods fail in more challenging real-world scenarios in ways humans never would [22].”
BackgroundInfluentialPetersen, E., Potdevin, Y., Mohammadi, E., Zidowitz, S., Breyer, S., Nowotka, D., Henn, S., Pechmann, L., Leucker, M., Rostalski, P., et al. (2022) · IEEE Access · doi:10.1109/access.2022.3178382 · Cited by 113
“groups that are relevant for fairness considerations in the first place [283], [393].”
BackgroundInfluentialOsteoporosis Prediction for Trabecular Bone using Machine Learning: A Review
Anam, M., Ponnusamy, V., Hussain, M., Nadeem, M. W., Awan, M. J., Goh, H. G., & Qadeer, S. (2021) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2021.013159 · Cited by 91
Age and sex affect deep learning prediction of cardiometabolic risk factors from retinal images
Gerrits, N., Elen, B., Craenendonck, T. V., Triantafyllidou, D., Petropoulos, I. N., Malik, R. A., & Boever, P. D. (2020) · Scientific Reports · doi:10.1038/s41598-020-65794-4 · Cited by 89
Karande, P., Gallagher, B., & Han, T. Y.-J. (2022) · Chemistry of Materials · doi:10.1021/acs.chemmater.2c01333 · Cited by 91
Estimating Example Difficulty using Variance of Gradients
Agarwal, C., D'souza, D., & Hooker, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01012 · Cited by 65
“Over the past decade, machine learning models are increasingly deployed to high-stake decision applications such as healthcare [4, 20, 52, 70], self-driving cars [51] and finance [53].”
BackgroundModel Patching: Closing the Subgroup Performance Gap with Data AugmentationPreprint
Goel, K., Gu, A., Li, Y., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.06775 · Cited by 137
Medical artificial intelligence for clinicians: the lost cognitive perspective
Tikhomirov, L., Semmler, C., McCradden, M., Searston, R., Ghassemi, M., & Oakden-Rayner, L. (2024) · The Lancet Digital Health · doi:10.1016/s2589-7500(24)00095-5 · Cited by 63
Self-citationDemographic bias of expert-level vision-language foundation models in medical imaging
Yang, Y., Liu, Y., Liu, X., Gulhane, A., Mastrodicasa, D., Wu, W., Wang, E. J., Sahani, D. V., & Patel, S. (2025) · Science Advances · doi:10.1126/sciadv.adq0305 · Cited by 127
“Whether demographic variables should be encoded as proxies for causal factors is a decision that should align with its actual clinical use 18,31,38 .”
Artificial intelligence and machine learning for clinical pharmacology
Ryan, D. K., Maclean, R., Balston, A. J., Scourfield, A., Shah, A. D., & Ross, J. (2023) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15930 · Cited by 78
Metrics reloaded: Recommendations for image analysis validation
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.01653 · Cited by 59
“Detailed failure case analysis 49 and performance assessment on relevant subgroups, for example, have been highlighted as critical components for better understanding when and where an algorithm may fail 50,51 .”
BackgroundShamji, M. H., Ollert, M. W., Adcock, I. M., Bennett, O., Favaro, A., Sarama, R., Riggioni, C., Annesi‐Maesano, I., Čustović, A., Fontanella, S., et al. (2023) · Allergy · doi:10.1111/all.15667 · Cited by 65
Expectations for Artificial Intelligence (AI) in Psychiatry
Monteith, S., Glenn, T., Geddes, J., Whybrow, P. C., Achtyes, E., & Bauer, M. (2022) · Current Psychiatry Reports · doi:10.1007/s11920-022-01378-5 · Cited by 55
“…a chest X-ray machine when diagnosing pneumonia [39], a ruler present in an image when diagnosing malignant skin cancer [40, 41], the scanner model and brand, and orders marked “urgent” when diagnosing a hip fracture [42], and the chest tube used for treatment when identifying a pneumothorax [43].”
BackgroundClinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
Huang, K., Singh, A., Chen, S., Moseley, E., Deng, C.-Y., George, N., & Lindvall, C. (2020) · Proceedings of the 3rd Clinical Natural Language Processing Workshop · doi:10.18653/v1/2020.clinicalnlp-1.11 · Cited by 54
A framework for understanding label leakage in machine learning for health care
Davis, S. E., Matheny, M. E., Balu, S., & Sendak, M. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad178 · Cited by 76
The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models
d'Eon, G., d’Eon, J., Wright, J. R., & Leyton‐Brown, K. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533240 · Cited by 110
“We round out this section with two quantitative experiments, where we compared the Spotlight, GEORGE, and a standard Gaussian mixture model [32].”
MethodResultKoch, L. M., Baumgartner, C. F., & Berens, P. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01085-w · Cited by 53
The cases for and against artificial intelligence in the medical school curriculum
Ngo, B., Nguyen, D., & vanSonnenberg, E. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220074 · Cited by 53
Large-scale methods for distributionally robust optimization
Levy, D., Carmon, Y., Duchi, J. C., & Sidford, A. (2020) · arXiv · doi:10.48550/arxiv.2010.05893 · Cited by 52
“The growing role of machine learning in high-stakes decision-making raises the need to train reliable models that perform robustly across subpopulations and environments [10, 25, 63, 51, 32, 47, 35].”
BackgroundZeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
Cabrera, Á. A., Fu, E., Bertucci, D., Holstein, K., Talwalkar, A., Hong, J. I., & Perer, A. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581268 · Cited by 51
“Practitioners and researchers continue to discover signifcant limitations and failures in state-of-the-art models, from systematic misclassifcation of certain medical images [44] to racial biases in pedestrian detection models [58].”
BackgroundPetrick, N., Chen, W., Delfino, J. G., Gallas, B. D., Kang, Y., Krainak, D. M., Sahiner, B., & Samala, R. K. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.5.051804 · Cited by 66
“31,45 The subgroup analyses discussed above generally require prior knowledge of the important subgroups but some more automated techniques for identifying important subgroups, based on model audits 46 and schema completion 47 have been reported in the literature.”
MethodSymphony: Composing Interactive Interfaces for Machine Learning
Bäuerle, A., Cabrera, Á. A., Hohman, F., Maher, M., Koski, D., Suau, X., Barik, T., & Moritz, D. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3502102 · Cited by 69
“This underuse can lead to missed data errors and model failures, a lack of shared team understanding of model behavior, and, ultimately, deployed ML systems that may be biased [8] or unsafe [47].”
BackgroundArtificial intelligence and the future of radiographic scoring in rheumatoid arthritis: a viewpoint
Bird, A., Oakden‐Rayner, L., McMaster, C., Smith, L., Zeng, M., Wechalekar, M. D., Ray, S., Proudman, S., & Palmer, L. J. (2022) · Arthritis Research & Therapy · doi:10.1186/s13075-022-02972-x · Cited by 57
“Models can suffer from poor performance in sub-sets of a population, and mistakes in certain subsets can have differing clinical significance [54].”
BackgroundSelf-citationA normative framework for artificial intelligence as a sociotechnical system in healthcare
McCradden, M. D., Joshi, S. D., Anderson, J. A., & London, A. J. (2023) · Patterns · doi:10.1016/j.patter.2023.100864 · Cited by 74
Evaluating Model Robustness and Stability to Dataset ShiftPreprint
Subbaswamy, A., Adams, R., & Saria, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.15100 · Cited by 95
“Oakden-Rayner et al. (2020) considered broad, sometimes manual, strategies to evaluate the performance of a model in subpopulations.”
BackgroundThe future of MRI in radiation therapy: Challenges and opportunities for the MR community
Goodburn, R. J., Philippens, M. E. P., Lefebvre, T. L., Khalifa, A., Bruijnen, T., Freedman, J. N., Waddington, D. E. J., Younus, E., Aliotta, E., Meliadò, G., et al. (2022) · Magnetic Resonance in Medicine · doi:10.1002/mrm.29450 · Cited by 61
Klontzas, M. E., Lipman, K. B. W. G., D’Antonoli, T. A., Андрейченко, А. Е., Cuocolo, R., Dietzel, M., Gitto, S., Huisman, H. J., Santinha, J., Vernuccio, F., et al. (2025) · European Radiology · doi:10.1007/s00330-025-11890-w · Cited by 57
Computer vision defect detection on unseen backgrounds for manufacturing inspection
Mezher, A. M., & Marble, A. E. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122749 · Cited by 55
A Dempster-Shafer Approach to Trustworthy AI With Application to Fetal Brain MRI Segmentation
Fidon, L., Aertsen, M., Kofler, F., Bink, A., David, A. L., Deprest, T., Emam, D., Guffens, F., András, J., Kasprian, G. J., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3346330 · Cited by 67
“However, the segmentation performance of deep learning methods typically degrades when images from a different center or a different scanner vendor as the one used for training are used or when evaluating the segmentation performance on abnormal anatomy [2, 30, 37, 42, 44, 47].”
Best practices for authors of healthcare-related artificial intelligence manuscripts
Kakarmath, S., Esteva, A., Arnaout, R., Harvey, H., Kumar, S., Muse, E., Dong, F., Wedlund, L., & Kvedar, J. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00336-w · Cited by 46
Artificial Intelligence and Healthcare Regulatory and Legal Concerns
Ganapathy, K. (2021) · Telehealth and Medicine Today · doi:10.30953/tmt.v6.252 · Cited by 66
“Such errors if undetected during a regulatory approval process may lead to catastrophic consequences when AI models are allowed to deploy at scale (10).”
BackgroundMurphy, Z. R., Venkatesh, K., Sulam, J., & Yi, P. H. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220012 · Cited by 59
Ahmad, F., & Muhmood, T. (2024) · Colloids and Surfaces B Biointerfaces · doi:10.1016/j.colsurfb.2024.114041 · Cited by 48
As if sand were stone. New concepts and metrics to probe the ground on which to build trustable AI
Cabitza, F., Campagner, A., & Sconfienza, L. M. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01224-9 · Cited by 48
“…reliable (in the sense of “true”) Gold Standard from their ratings, nevertheless one should be wary of the likelihood of systematic errors by the raters or potential hidden stratifications [54] (e.g. by case complexity) which would have an impact on the result of the reductions performed.”
BackgroundAI in Orthopedic Research: A Comprehensive Review
Mısır, A., & Yuce, A. (2025) · Journal of Orthopaedic Research® · doi:10.1002/jor.26109 · Cited by 51
Sensitivity analysis for interpretation of machine learning based segmentation models in cardiac MRI
Ankenbrand, M. J., Shainberg, L., Hock, M., Lohr, D., & Schreiber, W. (2021) · BMC Medical Imaging · doi:10.1186/s12880-021-00551-1 · Cited by 54
“Furthermore, there might be the effect of hidden stratification [12], there is usually some kind of bias in sampling the training data [13] and networks might learn shortcuts [14] using unintended features to boost performance on the training set.”
BackgroundDeep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review
Couture, H. D. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12122022 · Cited by 49
Cheng, V., Suriyakumar, V. M., Dullerud, N., Joshi, S. D., & Ghassemi, M. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445879 · Cited by 78
Nyman, J., Denize, T., Bakouny, Z., Labaki, C., Titchen, B. M., Bi, K., Hari, S. N., Rosenthal, J., Mehta, N., Jiang, B., et al. (2023) · Cell Reports Medicine · doi:10.1016/j.xcrm.2023.101189 · Cited by 52
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
Chen, V. S., Wu, S., Weng, Z., Ratner, A., & Ré, C. (2019) · PubMed · doi:10.48550/arxiv.1909.06349 · Cited by 40
“bone X-rays associated with degenerative joint disease [27]).”
BackgroundThe CellPhe toolkit for cell phenotyping using time-lapse imaging and pattern recognition
Wiggins, L., Lord, A. E., Murphy, K., Lacy, S., O’Toole, P. J., Brackenbury, W. J., & Wilson, J. C. (2023) · Nature Communications · doi:10.1038/s41467-023-37447-3 · Cited by 54
Jacques, T., Cardot, N., Ventre, J., Demondion, X., & Cotten, A. (2023) · European Radiology · doi:10.1007/s00330-023-10380-1 · Cited by 45
“The fact that certain fractures are overlooked on X-ray can lead to “hidden stratifica-tion” that can impact the model, its performance, and its clinical results [17].”
Leveraging combinatorial testing for safety-critical computer vision datasets
Gladisch, C., Heinzemann, C., Herrmann, M., & Woehrle, M. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00170 · Cited by 48
An empirical framework for domain generalization in clinical settings
Zhang, H., Dullerud, N., Seyyed-Kalantari, L., Morris, Q., Joshi, S., & Ghassemi, M. (2021) · Proceedings of the Conference on Health, Inference, and Learning · doi:10.1145/3450439.3451878 · Cited by 39
“Here, the protected group would be an example of a hidden stratification [55].”
BackgroundEffect of image resolution on automated classification of chest X-rays
Haque, M. I. U., Dubey, A., Danciu, I., Justice, A. C., Ovchinnikova, O. S., & Hinkle, J. D. (2023) · Journal of medical imaging · doi:10.1117/1.jmi.10.4.044503 · Cited by 55
“Automated CXR classification has been previously studied considering a phenomenon called hidden stratification.(39) This term refers to a situation in which labels for a classification task are coarse and represent multiple subpopulations for which there are no labels.”
BackgroundChest radiographs and machine learning – Past, present and future
Jones, C. M., Buchlak, Q. D., Oakden‐Rayner, L., Milne, M., Seah, J. C. Y., Esmaili, N., & Hachey, B. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13274 · Cited by 58
“This may be difficult if labelling of the training and test data sets is not sufficiently detailed to distinguish between subsets of cases(46) and if there is inadequate disease variation in the training data set.(44) Recent evidence suggests that training a machine learning model to both classify the presence of a fi…”
BackgroundSelf-citationWhat Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 62
“Having a thorough understanding of how an AI behaves is especially important to detect and mitigate serious concerns such as fairness [38] and safety [61] issues.”
BackgroundMoore, S., Liao, Q. V., & Subramonyam, H. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581242 · Cited by 51
“A common example of such a schema for model outputs is confusion matrices for classifcation problems [58].”
BackgroundRandomness In Neural Network Training: Characterizing The Impact of ToolingPreprint
Zhuang, D., Zhang, X., Song, S. L., & Hooker, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11872 · Cited by 78
“54 Our results suggest that deterministic tooling is critical for ensuring AI safety in sensitive domains 55 such as credit scoring, health care diagnostics [82, 19, 4, 59] and autonomous driving [54].”
BackgroundZech, J. R., Ezuma, C. O., Patel, S., Edwards, C. R., Posner, R., Hannon, E., Williams, F., Lala, S. V., Ahmad, Z. Y., Moy, M. P., et al. (2024) · Skeletal Radiology · doi:10.1007/s00256-024-04698-0 · Cited by 47
“Attention should be given to subgroups of particular concern as AI is deployed and refined, as previously discussed by Oakden-Rayner et al. [26].”
Deep learning to detect left ventricular structural abnormalities in chest X-rays
Bhave, S., Rodriguez, V., Poterucha, T., Mutasa, S., Aberle, D., Capaccione, K. M., Chen, Y., Dsouza, B., Dumeer, S., Goldstein, J., et al. (2024) · European Heart Journal · doi:10.1093/eurheartj/ehad782 · Cited by 38
“This can cause failures 27 when models are deployed in populations where such correlations may not exist.”
Background
Metadata
- Authors
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
- Affiliations
- Australian Institute for Machine Learning, University of Adelaide
- Stanford University (US)
- The University of Adelaide (AU)
- Department of Computer Science, Stanford University
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 151-159
- PMC
- PMC7665161
- OpenAlex
- W2975035192
- Cited by
- 411
- References
- 33
- Citations by year
- 2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toshiba
- Teradata
- Ant Financial
- Intel Corporation
- NEC Corporation
- Taiwan Semiconductor Manufacturing Company
- Accenture
- Qualcomm
- Analog Devices
- VMWare
- Infosys
- National Institutes of Health (U54EB020405)
- NXP
- Xilinx
- LETI-CEA
- American Family Insurance
- Swiss Re
- Intelligence Community Postdoctoral Research Fellowship Program
- National Science Foundation (CCF1763315, CCF1563078, CC1937301)
- IBM
- Hitachi
- Okawa Foundation for Information and Telecommunications
- Google Cloud
- Office of Naval Research (N000141712266)
- Gordon and Betty Moore Foundation
- ARM
- Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
- Microsoft
- BASF
- Ericsson
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.