The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index.
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Open accessAbstract
Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“To establish ground truth labels for each imaging study, we extracted 14 pathology labels from the associated report for each image using the CheXpert NLP algorithm 31 , an approach we validated in previous work on our dataset 6 .”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [31]
In the citing paper
“The pre-trained TorchXRayVision autoencoder was trained on numerous chest radiograph datasets (PadChest 56 , Chestx-ray8 57 , CheXpert 31 , and MIMIC-CXR 55 ).”
From Empirical data drift detection experiments on real-world medical imaging data (2024) · cites this paper as [31]
In the citing paper
“However, the application of AI to image interpretation tasks has often been limited to the identification of a handful of individual pathologies,8,9,10 representing an over-simplification of the image interpretation task.”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [8]
In the citing paper
“To address this gap, the CheXbert labeler (which is improved from the CheXpert labeler)8,9 and RadGraph,24 were developed to parse radiology reports.”
From Evaluating progress in automatic chest X-ray radiology report generation (2023) · cites this paper as [8]
In the citing paper
“Large-scale datasets, such as MIMIC-CXR [1] and CheXpert [2], use automated radiology report labelers [2-6] to extract common medical conditions from reports.”
From RadGraph: Extracting Clinical Entities and Relations from Radiology Reports (2021) · cites this paper as [2]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988) · Biometrics · doi:10.2307/2531595 · Cited by 22,649
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Gulshan, V., Peng, L., Coram, M., Stumpe, M. C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., et al. (2016) · JAMA · doi:10.1001/jama.2016.17216 · Cited by 7,697
Probabilistic Outputs for Support vector Machines and Comparisons to Regularized Likelihood Methods
Platt, J. C. (1999) · Cited by 6,385
Missing data analysis: making it work in the real world
Graham, J. W. (2009) · Annual Review of Psychology · doi:10.1146/annurev.psych.58.110405.085530 · Cited by 6,163
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Semi-Supervised Learning Literature Survey
Zhu, X. (2005) · Minds at UW (University of Wisconsin) · Cited by 4,506
Fleischner Society: Glossary of Terms for Thoracic Imaging
Hansell, D. M., Bankier, A. A., MacMahon, H., McLoud, T. C., Müller, N. L., & Remy, J. (2008) · Radiology · doi:10.1148/radiol.2462070712 · Cited by 4,198
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the 33rd annual meeting on Association for Computational Linguistics - · doi:10.3115/981658.981684 · Cited by 2,385
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280
CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219
Self-citationPreparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164
Coarse-to-fine n-best parsing and MaxEnt discriminative reranking
Charniak, E., & Johnson, M. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219862 · Cited by 872
Sun, X., & Xu, W. (2014) · IEEE Signal Processing Letters · doi:10.1109/lsp.2014.2337313 · Cited by 630
Wang, X., Peng, Y., Lu, L., Lu, Z., & Summers, R. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00943 · Cited by 545
Gohagan, J. K., Prorok, P. C., Hayes, R. B., & Kramer, B.-S. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00097-0 · Cited by 544
Learning from Noisy Labels with Distillation
Li, Y., Yang, J., Song, Y., Cao, L., Luo, J., & Li, L.-J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.211 · Cited by 541
Universal Stanford dependencies: A cross-linguistic typology
de Marneffe, M.-C., Dozat, T., Silveira, N., Haverinen, K., Ginter, F., Nivre, J., & Manning, C. D. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4ok3zdjx8t96 · Cited by 456
Data Distillation: Towards Omni-Supervised Learning
Radosavovic, I., Dollar, P., Girshick, R., Gkioxari, G., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00433 · Cited by 421
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2019) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-030-13969-8_18 · Cited by 371
Grad-CAM: Why did you say that?
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07450 · Cited by 309
Guan, Q., Huang, Y., Zhong, Z., Zheng, Z., Zheng, L., & Yang, Y. P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.09927 · Cited by 226
Learning to diagnose from scratch by exploiting dependencies among labels
Li, Y., Poblenz, E., Dagunts, D., Covington, B., Bernard, D., & Lyman, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10501 · Cited by 181
NegBio: a high-performance tool for negation and uncertainty detection in radiology reports
Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., & Lu, Z. (2017) · arXiv · doi:10.48550/arxiv.1712.05898 · Cited by 175
Boosted Cascaded Convnets for Multilabel Classification of Thoracic Diseases in Chest Radiographs
Kumar, P., Grewal, M., & Srivastava, M. M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93000-8_62 · Cited by 149
Weakly Supervised Medical Diagnosis and Localization from Multiple ResolutionsPreprint
Yao, L., Prosky, J., Problez, E., Covington, B., & Lyman, K. (2018) · arXiv (Cornell University) · doi:10.13140/rg.2.2.30419.68645 · Cited by 139
Multi-label learning with missing labels for image annotation and facial action unit recognition
Wu, B., Lyu, S., Hu, B.-G., & Ji, Q. (2015) · Pattern Recognition · doi:10.1016/j.patcog.2015.01.022 · Cited by 118
Clinical Prediction Models. A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2009) · EUR Research Repository (Erasmus University Rotterdam) · Cited by 97
Weakly Supervised Medical Diagnosis and Localization from Multiple ResolutionsPreprint
Li, Y., Prosky, J., Poblenz, E., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.07703 · Cited by 94
Thoracic Disease Identification and Localization with Limited Supervision
Li, Z., Wang, C., Han, M., Xue, Y., Wei, W., Li, L.-J., & Fei-Fei, L. (2019) · Advances in computer vision and pattern recognition · doi:10.1007/978-3-030-13969-8_7 · Cited by 75
Learning from Noisy Labels with DistillationPreprint
Li, Y., Yang, J., Song, Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02391 · Cited by 58
Wang, X., Peng, Y., Lü, L., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04334 · Cited by 57
Data Distillation: Towards Omni-Supervised LearningPreprint
Radosavovic, I., Dollár, P., Girshick, R., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.04440 · Cited by 43
The Effect of Comparison Films Upon Resident Interpretation of Pediatric Chest Radiographs
Berbaum, K. S., Franken, E. A., & Smith, W. L. (1985) · Investigative Radiology · doi:10.1097/00004424-198503000-00004 · Cited by 22
On Multilabel Classification Methods of Incompletely Labeled Biomedical Text Data
Kolesov, A., Kamyshenkov, D., Litovchenko, M., Smekalova, E. M., Golovizin, A., & Zhavoronkov, A. A. (2014) · Computational and Mathematical Methods in Medicine · doi:10.1155/2014/781807 · Cited by 20
Any Domain Parsing: Automatic Domain Adaptation for Natural Language Parsing
McClosky, D. (2009) · Brown Digital Repository · doi:10.7301/z08k77bf · Cited by 10
Cited by
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-021-01614-0 · Cited by 3,212
Self-citationA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Generative adversarial network in medical imaging: A review
Yi, X., Walia, E., & Babyn, P. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101552 · Cited by 1,863
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816
Self-citationDeep learning-enabled medical computer vision
Esteva, A., Chou, K., Yeung, S., Naik, N., Madani, A., Mottaghi, A., Liu, Y., Topol, E., Dean, J., & Socher, R. (2021) · npj Digital Medicine · doi:10.1038/s41746-020-00376-2 · Cited by 1,406
Metadata
- Authors
- Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., & Ng, A. Y.
- Affiliations
- Stanford University (US)
- ORCID
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 590-597
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2904183610
- Cited by
- 392
- References
- 35
- Citations by year
- 2026: 5 · 2025: 50 · 2024: 54 · 2023: 64 · 2022: 68 · 2021: 82 · 2020: 58 · 2019: 10
- FWCI
- 82.52
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- www.aaai.org/ (unspecified)
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- CheXpert, chest radiograph, radiology report annotation, radiologist comparison, convolutional neural networks, cardiomegaly detection, benchmark dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.