Hidden stratification causes clinically meaningful failures in machine learning for medical imaging
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Machine learning models for medical image analysis often suffer from poor performance on important subsets of a population that are not identified during training or testing. For example, overall performance of a cancer detection model may be high, but the model may still consistently miss a rare but aggressive cancer subtype. We refer to this problem as hidden stratification, and observe that it results from incompletely describing the meaningful variation in a dataset. While hidden stratification can substantially reduce the clinical efficacy of machine learning models, its effects remain difficult to measure. In this work, we assess the utility of several possible techniques for measuring hidden stratification effects, and characterize these effects both via synthetic experiments on the CIFAR-100 benchmark dataset and on multiple real-world medical imaging datasets. Using these measurement techniques, we find evidence that hidden stratification can occur in unidentified imaging subsets with low prevalence, low label quality, subtle distinguishing features, or spurious correlates, and that it can result in relative performance differences of over 20% on clinically important subsets. Finally, we discuss the clinical implications of our findings, and suggest that evaluation of hidden stratification should be a critical component of any machine learning deployment in medical imaging.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this example, reproduced from Oakden-Rayner and colleagues, 13 the artificial intelligence model learns to detect the artifacts of clinical care (chest drains) and fails to adequately learn the features of the pathology itself.”
From The medical algorithmic audit (2022) · cites this paper as [13]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences53.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences45.”
From Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension (2020) · cites this paper as [45]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, this drop in performance might not be obvious at the aggregate level of typical artificial intelligence testing, but rather in subsets of the target cohort or specific strata within the input data, a concept that has been described as hidden stratification.13”
From The medical algorithmic audit (2022) · cites this paper as [13]
The cited source says what the claim attributes to it.
In the citing paper
“It has been shown that AI systems may be systematically biased toward different outputs, which may lead to different or even unfair treatment, on the basis of extant features53,57–59.”
From Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension (2020) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
Rousseeuw, P. J. (1987) · Journal of Computational and Applied Mathematics · doi:10.1016/0377-0427(87)90125-7 · Cited by 21,392 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Esteva et al. [12] is particularly notable, as this dataset is labelled for more than 2,000 diagnostic subclasses but the results presented only consider “top-level” diagnostic categories.”
BackgroundAggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodGulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697 · cited 1×
“Gulshan et al. [16], for instance, present variations in retinopathy detection performance on subsets with images obtained in different locations, with differing levels of disease severity, and with different degrees of pupil dilation.”
BackgroundA guide to deep learning in healthcare
Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., Cui, C., Corrado, G., Thrun, S., & Dean, J. (2019) · Nature Medicine · doi:10.1038/s41591-018-0316-z · Cited by 5,312 · cited 1×
“Deep learning systems have shown remarkable promise in medical image analysis, often claiming performance rivaling that of human experts [13].”
BackgroundWang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · cited 2×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundA dendrite method for cluster analysis
Caliński, T., & Harabasz, J. (1974) · Communications in Statistics · doi:10.1080/03610927408827101 · Cited by 4,480
A systematic study of the class imbalance problem in convolutional neural networks
Buda, M., Maki, A., & Mazurowski, M. A. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.011 · Cited by 3,088 · cited 1×
“]. However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40]. As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techn…”
BackgroundClinical-grade computational pathology using weakly supervised deep learning on whole slide images
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019) · Nature Medicine · doi:10.1038/s41591-019-0508-1 · Cited by 2,799 · cited 1×
“observed [6, 34]; such analyses may be helpful in identifying error modes via human review, but do not characterize the full space of subset performance [33].”
BackgroundZech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719
Haenssle, H. A., Fink, C., Schneiderbauer, R., Toberer, F., Buhl, T., Blum, A., Kalloo, A., Hassen, A. B. H., Thomas, L., Enk, A., et al. (2018) · Annals of Oncology · doi:10.1093/annonc/mdy166 · Cited by 1,571
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449 · cited 1×
“labeled for one of 14 different thoracic pathologies. In our analysis, we leverage a pretrained Densenet-121 model provided by Zech [39] which reproduces the procedure and results of Rajpurkar et al. [26] on this dataset. During error auditing, where examples of false positive and false negative predictions from the p…”
MethodCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219 · cited 1×
“While overall pneumothorax ROC-AUC closely matches that reported in Rajpurkar et al. [28] at 0.87, pneumothorax ROC-AUC was 0.94 on the subclass with chest drains, but only 0.77 on the subclass without chest drains.”
ResultInfluentialPopulation stratification and spurious allelic association
Cardon, L. R., & Palmer, L. J. (2003) · The Lancet · doi:10.1016/s0140-6736(03)12520-2 · Cited by 1,204
Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study
Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N. G., Venugopal, V. K., Mahajan, V., Rao, P., & Warier, P. (2018) · The Lancet · doi:10.1016/s0140-6736(18)31645-3 · Cited by 1,020 · cited 1×
“Chilamkurthy et al. [9] present a subset analysis for different diagnostic categories of intracranial hemorrhage (e.g. sub-dural vs. subarachnoid) when designing a deep learning model for abnormality detection on head CT, but do not analyze differences in performance related to bleed size,…”
BackgroundWang, P., Berzin, T. M., Glissen Brown, J. R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D., et al. (2019) · Gut · doi:10.1136/gutjnl-2018-317500 · Cited by 865 · cited 2×
“[34] perform an excellent subset analysis of a colonoscopy polyp detector, with comparative performance analysis presented by polyp size, location, shape, and underlying pathology (e.”
BackgroundMethodMazurowski, M. A., Habas, P. A., Zurada, J. M., Lo, J. Y., Baker, J. A., & Tourassi, G. D. (2008) · Neural Networks · doi:10.1016/j.neunet.2007.12.031 · Cited by 737 · cited 1×
“served stratification and dataset imbalance in medical machine learning often center on gathering more data on underperforming subsets, either via additional labelling, selective data augmentation, or oversampling [22].”
BackgroundBien, N., Rajpurkar, P., Ball, R. L., Irvin, J., Park, A., Jones, E., Bereket, M., Patel, B. N., Yeom, K. W., Shpanskaya, K., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002699 · Cited by 734 · cited 1×
“Of course, there also exist multiple studies that do not directly address the effects of hidden stratification [3, 17].”
BackgroundDo CIFAR-10 Classifiers Generalize to CIFAR-10?Preprint
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00451 · Cited by 528 · cited 1×
“Problems similar to hidden stratification have been observed or postulated in many domains, including traditional computer vision [30], fine-grained image recognition [39], genomics [7], and epidemiology (often termed “spectrum effects”) [23].”
BackgroundAgniel, D., Kohane, I. S., & Weber, G. M. (2018) · BMJ · doi:10.1136/bmj.k1479 · Cited by 391 · cited 1×
“. Non-causal confounding features such as healthcare process quantities can also contribute substantially to high model performance on data subsets heavily associated with these confounding variables [35, 2, 1, 40]. Instead of analyzing subsets defined a priori, Mahajan et al. [21] describe algorithmic audits, where de…”
BackgroundAggregated Residual Transformations for Deep Neural NetworksPreprint
Xie, S., Girshick, R., Dollár, P., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05431 · Cited by 381 · cited 1×
“Figure 1(a) presents the performance of a ResNeXt-29, 8x64d Convolutional Neural Network (CNN) trained on the 20 CIFAR-100 superclasses using the training schedule reported in [37] and the implementation provided by [38].”
MethodWinkler, J. K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., et al. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.1735 · Cited by 365 · cited 1×
“Non-causal confounding features such as healthcare process quantities can also contribute substantially to high model performance on data subsets heavily associated with these confounding variables [1, 2, 36, 41].”
BackgroundSpectrum Bias or Spectrum Effect? Subgroup Variation in Diagnostic Test Evaluation
Mulherin, S. A., & Miller, W. C. (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-7-200210010-00011 · Cited by 327 · cited 1×
“Problems similar to hidden stratification have been observed or postulated in many domains, including traditional computer vision [30], fine-grained image recognition [39], genomics [7], and epidemiology (often termed “spectrum effects”) [23].”
BackgroundDeep learning predicts hip fracture using confounding patient and healthcare variables
Badgeley, M. A., Zech, J. R., Oakden-Rayner, L., Glicksberg, B. S., Liu, M., Gale, W., McConnell, M. V., Percha, B., Snyder, T. M., & Dudley, J. T. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0105-1 · Cited by 290 · cited 1×
“. Non-causal confounding features such as healthcare process quantities can also contribute substantially to high model performance on data subsets heavily associated with these confounding variables [35, 2, 1, 40]. Instead of analyzing subsets defined a priori, Mahajan et al. [21] describe algorithmic audits, where de…”
BackgroundSelf-citationCombining randomization and discrimination for fine-grained image categorization
Yao, B., Khosla, A., & Fei-Fei, L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995368 · Cited by 288 · cited 1×
“Problems similar to hidden stratification have been observed or postulated in many domains, including traditional computer vision [30], fine-grained image recognition [39], genomics [7], and epidemiology (often termed “spectrum effects”) [23].”
BackgroundExploring Large-scale Public Medical Image Datasets
Oakden-Rayner, L. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.10.006 · Cited by 250
Self-citationMURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs
Rajpurkar, P., Irvin, J., Bagul, A., Ding, D. Y., Duan, T., Mehta, H., Yang, B., Zhu, K., Laird, D., Ball, R. L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.06957 · Cited by 243
Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P. (2019) · Radiology · doi:10.1148/radiol.2018181422 · Cited by 206 · cited 1×
“Similarly, Dunnmon et al. [11] report the performance of their chest radiograph triage system by pathology subtype, finding that models trained on binary triage labels achieved substantially lower performance on fracture than on other diseases.”
BackgroundSelf-citationDisparate Impact in Big Data Policing
Selbst, A. D. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2819182 · Cited by 192 · cited 1×
“teristics that contribute to degraded model performance: (1) low subclass prevalence, (2) reduced label accuracy within the subclass, (3) subtle discriminative features, and (4) spurious correlations [8]. We first use schema completion to evaluate clinically important hidden stratification effects in radiograph datasets…”
BackgroundLearning to Compose Domain-Specific Transformations for Data Augmentation
Ratner, A., Ehrenberg, H. R., Hussain, Z., Dunnmon, J., & Ré, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.01643 · Cited by 180 · cited 1×
“However, the cost of manual labelling is often prohibitive, appropriate augmentation transforms can be difficult to define, and oversampling an underperforming subset can cause degradation on others [14, 28, 4, 40].”
BackgroundSelf-citationRaji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020) · arXiv · doi:10.48550/arxiv.2001.00973 · Cited by 179 · cited 1×
“There has been particular interest in formalizing algorithmic audit methods recently [25], although these initiatives have yet to be tailored to the medical setting.”
BackgroundDetecting hip fractures with radiologist-level performance using deep neural networksPreprint
Gale, W., Oakden‐Rayner, L., Carneiro, G., Bradley, A. P., & Palmer, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06504 · Cited by 135 · cited 3×
“We show a similar trend in Adelaide Hip Fracture Schema completion also shows hidden stratification on a large, high quality pelvic x-ray dataset from the Royal Adelaide Hospital [15].”
MethodResultInfluentialSelf-citationWeakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences
Fries, J., Varma, P., Chen, V. S., Xiao, K., Tejeda, H., Saha, P., Dunnmon, J., Chubb, H., Maskatia, S. A., Fiterau, M., et al. (2019) · Nature Communications · doi:10.1038/s41467-019-11012-3 · Cited by 133
Self-citationMURA Dataset: Towards Radiologist-Level Abnormality Detection in Musculoskeletal RadiographsPreprint
Rajpurkar, P., Irvin, J., Bagul, A., Ding, D. Y., Duan, T., Mehta, H., Yang, B., Zhu, K., Laird, D., Ball, R. L., et al. (2017) · arXiv (Cornell University) · Cited by 83
Cross-Modal Data Programming Enables Rapid Medical Machine Learning
Dunnmon, J., Ratner, A., Saab, K. K., Khandwala, N., Markert, M. S., Sagreiya, H., Goldman, R. E., Lee‐Messer, C., Lungren, M. P., Rubin, D. L., et al. (2020) · Patterns · doi:10.1016/j.patter.2020.100019 · Cited by 82 · cited 1×
“As a result, medical imagery analysts have commonly begun either to use semi-automated labelling techniques [34, 14, 18, 10, 14] or to apply human expertise to produce a narrow or incomplete set of visual labels [25] rather than exhaustively labelling all possible findings and variations.”
BackgroundSelf-citationThe Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms—How We Do It
Mahajan, V., Venugopal, V. K., Murugavel, M., & Mahajan, H. (2019) · Academic Radiology · doi:10.1016/j.acra.2019.09.009 · Cited by 67
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
Chen, V. S., Wu, S., Weng, Z., Ratner, A., & Ré, C. (2019) · PubMed · doi:10.48550/arxiv.1909.06349 · Cited by 42 · cited 1×
“lass performance, particularly given the failure of this method in the detection of clinically relevant subclasses in the MURA task. More advanced semi-supervised methods such as those of Chen et al. [8] may ultimately be required to tackle this problem, or it may be the case that both unsupervised and semi-supervised…”
MethodConfounding variables can degrade generalization performance of radiological deep learning models
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · arXiv · doi:10.48550/arxiv.1807.00431 · Cited by 26
Liu, J., Yao, J., Bagheri, M. H., Sandfort, V., & Summers, R. M. (2019) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2019.8759189 · Cited by 16 · cited 1×
“While these results support other findings that demonstrate the utility of hidden-state clustering in model development [20], the relatively simple technique presented here should be considered only a first attempt at unsupervised identification of hidden stratification [5, 32].”
Result
Research works printed but not linked: 2
- Jeremy Irvin , Pranav Rajpurkar , Michael Ko , Yifan Yu , Silviana Ciurea-Ilcus , Chris Chute , Henrik Marklund , Behzad Haghgoo , Robyn Ball , Katie Shpanskaya , and Others. 2019 . Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901. 07031 (2019). Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, and Others. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. arXiv preprint arXiv:1901. 07031 (2019).
- Alex Krizhevsky , Vinod Nair , and Geoffrey Hinton . 2009. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html 6 ( 2009 ). Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html 6 (2009).
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-citationLiu, X., Cruz Rivera, S., Moher, D., Calvert, M. J., Denniston, A. K., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Chan, A.-W., Darzi, A., Holmes, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1034-x · Cited by 1,421
Self-citationSeyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
“This is a form of bias amplification, when a model’s predicted outputs amplify a known source of error in the process of data generation 34 or data distribution 35 .”
BackgroundCruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K., Calvert, M. J., The SPIRIT-AI and CONSORT-AI Working Group, SPIRIT-AI and CONSORT-AI Steering Group, Darzi, A., Holmes, C., Yau, C., et al. (2020) · Nature Medicine · doi:10.1038/s41591-020-1037-7 · Cited by 933
“AI systems can make errors that may be hard to foresee but that, if allowed to be deployed at scale, could have catastrophic consequences 45 .”
Self-citationMachine learning for medical imaging: methodological failures and recommendations for the future
Varoquaux, G., & Cheplygina, V. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00592-y · Cited by 656
Metadata
- Authors
- Oakden-Rayner, L., Dunnmon, J., Carneiro, G., & Re, C.
- Affiliations
- Australian Institute for Machine Learning, University of Adelaide
- Stanford University (US)
- The University of Adelaide (AU)
- Department of Computer Science, Stanford University
- Venue
- Proceedings of the ACM Conference on Health, Inference, and Learning
- Publisher
- ACM
- Published
- 2020-04-02
- Volume
- 2020
- Pages
- 151-159
- PMC
- PMC7665161
- OpenAlex
- W2975035192
- Cited by
- 411
- References
- 33
- Citations by year
- 2026: 86 · 2025: 66 · 2024: 60 · 2023: 53 · 2022: 49 · 2021: 54 · 2020: 40
- FWCI
- 25.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Health, Inference, and Learning (CHIL) (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Toshiba
- Teradata
- Ant Financial
- Intel Corporation
- NEC Corporation
- Taiwan Semiconductor Manufacturing Company
- Accenture
- Qualcomm
- Analog Devices
- VMWare
- Infosys
- National Institutes of Health (U54EB020405)
- NXP
- Xilinx
- LETI-CEA
- American Family Insurance
- Swiss Re
- Intelligence Community Postdoctoral Research Fellowship Program
- National Science Foundation (CCF1763315, CCF1563078, CC1937301)
- IBM
- Hitachi
- Okawa Foundation for Information and Telecommunications
- Google Cloud
- Office of Naval Research (N000141712266)
- Gordon and Betty Moore Foundation
- ARM
- Defense Advanced Research Projects Agency (FA87501720095, FA86501827865, FA86501827882)
- Microsoft
- BASF
- Ericsson
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- medical image analysis, rare diseases, spurious correlations, label quality, model evaluation, clinical implementation of AI, CIFAR-100
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.