Performance of AI to exclude normal chest radiographs to reduce radiologists' Workload
Open accessAbstract
INTRODUCTION: This study investigates the performance of a commercially available artificial intelligence (AI) system to identify normal chest radiographs and its potential to reduce radiologist workload. METHODS: Retrospective analysis included consecutive chest radiographs from two medical centers between Oct 1, 2016 and Oct 14, 2016. Exclusions comprised follow-up exams within the inclusion period, bedside radiographs, incomplete images, imported radiographs, and pediatric radiographs. Three chest radiologists categorized findings into normal, clinically irrelevant, clinically relevant, urgent, and critical. A commercial AI system processed all radiographs, scoring 10 chest abnormalities on a 0-100 confidence scale. AI system performance was evaluated using the area under the ROC curve (AUC), assessing the detection of normal radiographs. Sensitivity was calculated for the default and a conservative operating point. the detection of negative predictive value (NPV) for urgent and critical findings, as well as the potential workload reduction, was calculated. RESULTS: A total of 2603 radiographs were acquired in 2141 unique patients. Post-exclusion, 1670 radiographs were analyzed. Categories included 479 normal, 332 clinically irrelevant, 339 clinically relevant, 501 urgent, and 19 critical findings. The AI system achieved an AUC of 0.92. Sensitivity for normal radiographs was 92% at default and 53% at the conservative operating point. At the conservative operating point, NPV was 98% for urgent and critical findings, and could result in a 15% workload reduction. CONCLUSION: A commercially available AI system effectively identifies normal chest radiographs and holds the potential to lessen radiologists' workload by omitting half of the normal exams from reporting. CLINICAL RELEVANCE STATEMENT: The AI system is able to detect half of all normal chest radiographs at a clinically acceptable operating point, thereby potentially reducing the workload for the radiologists by 15%. KEY POINTS: The AI system reached an AUC of 0.92 for the detection of normal chest radiographs. Fifty-three percent of normal chest radiographs were identified with a NPV of 98% for urgent findings. AI can reduce the workload of chest radiography reporting by 15%.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The high volume of chest x-rays, combined with the need for detailed narrative reporting, places significant pressure on radiologists, especially during night shifts and peak admission periods.
“The high volume of chest x-rays, combined with the need for detailed narrative reporting, places significant pressure on radiologists, especially during night shifts and peak admission periods (9, 10).”
From Swin-Qwen3: a three-stage vision–language framework for automated radiology report generation with multi-agent verification (2026) · cites this paper as [10]
The Introduction describes increasing radiologist workload and the potential to save time by reducing chest-radiograph reporting; the Discussion explicitly cites the high volume of chest radiographs. Neither section says that detailed narrative reports are required or that pressure is especially high during night shifts or peak admission periods. Page numbers are not available in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
STARD 2015: An Updated List of Essential Items for Reporting Diagnostic Accuracy Studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C., Glasziou, P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · Radiology · doi:10.1148/radiol.2015151516 · Cited by 1,349 · see citing works
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 531 · see citing works
Self-citationvan Leeuwen, K. G., Schalekamp, S., Rutten, M. J. C. M., van Ginneken, B., & de Rooij, M. (2021) · European Radiology · doi:10.1007/s00330-021-07892-z · Cited by 461 · see citing works
Self-citationHwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447 · see citing works
Physician workforce in the United States of America: forecasting nationwide shortages
Zhang, X., Lin, D., Pforsich, H., & Lin, V. W.-H. (2020) · Human Resources for Health · doi:10.1186/s12960-020-0448-3 · Cited by 433 · see citing works
Miss Rate of Lung Cancer on the Chest Radiograph in Clinical Practice
Quekel, L. G. B. A., Kessels, A. G. H., Goei, R., & van Engelshoven, J. M. A. (1999) · CHEST Journal · doi:10.1378/chest.115.3.720 · Cited by 310 · see citing works
Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks
Annarumma, M., Withey, S. J., Bakewell, R. J., Pesce, E., Goh, V., & Montana, G. (2019) · Radiology · doi:10.1148/radiol.2018180921 · Cited by 303 · see citing works
Interpretation of Plain Chest Roentgenogram
Raoof, S., Feigin, D. S., Sung, A. W., Raoof, S., Irugulpati, L., & Rosenow, E. C. (2012) · CHEST Journal · doi:10.1378/chest.10-1302 · Cited by 218 · see citing works
Artificial intelligence applications for thoracic imaging
Chassagnon, G., Vakalopoulou, M., Nikos Paragios, A., & Revel, M. (2019) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.108774 · Cited by 208 · see citing works
Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P. (2019) · Radiology · doi:10.1148/radiol.2018181422 · Cited by 206 · see citing works
Workload for radiologists during on-call hours: dramatic increase in the past 15 years
Bruls, R. J. M., & Kwee, R. M. (2020) · Insights into Imaging · doi:10.1186/s13244-020-00925-z · Cited by 192 · see citing works
Deep Learning for Chest Radiograph Diagnosis in the Emergency Department
Hwang, E. J., Nam, J. G., Lim, W. H., Park, S. J., Jeong, Y. S., Kang, J. H., Hong, E. K., Kim, T. M., Goo, J. M., Park, S., et al. (2019) · Radiology · doi:10.1148/radiol.2019191225 · Cited by 164 · see citing works
Ahn, J. S., Ebrahimian, S., McDermott, S., Lee, S., Naccarato, L., Di Capua, J. F., Wu, M. Y., Zhang, E. W., Muse, V., Miller, B., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.29289 · Cited by 138 · see citing works
Bradley, S. H., Abraham, S., Callister, M. E. J., et al. (2019) · British Journal of General Practice · doi:10.3399/bjgp19x706853 · Cited by 126 · see citing works
Kwee, T. C., & Kwee, R. M. (2021) · Insights into Imaging · doi:10.1186/s13244-021-01031-4 · Cited by 116 · see citing works
Homayounieh, F., Digumarthy, S., Ebrahimian, S., Rueckel, J., Hoppe, B. F., Sabel, B. O., Conjeti, S., Ridder, K., Sistermanns, M., Wang, L., et al. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.41096 · Cited by 113 · see citing works
Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact
Plesner, L. L., Müller, F. C., Nybing, J. D., Laustrup, L. C., Rasmussen, F., Nielsen, O. W., Boesen, M., & Andersen, M. B. (2023) · Radiology · doi:10.1148/radiol.222268 · Cited by 89 · see citing works
Trend in radiologist workload compared to number of admissions in the emergency department
Dan Lantsman, C., Barash, Y., Klang, E., Guranda, L., Konen, E., & Tau, N. (2022) · European Journal of Radiology · doi:10.1016/j.ejrad.2022.110195 · Cited by 85 · see citing works
Schalekamp, S., van Ginneken, B., Koedam, E., Snoeren, M. M., Tiehuis, A. M., Wittenberg, R., Karssemeijer, N., & Schaefer‐Prokop, C. M. (2014) · Radiology · doi:10.1148/radiol.14131315 · Cited by 81 · see citing works
Self-citationAutomated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks
Annarumma, M., Withey, S. J., Bakewell, R. J., Pesce, E., Goh, V., & Montana, G. (2019) · Radiology · doi:10.1148/radiol.2019194005 · Cited by 63 · see citing works
Nabulsi, Z., Sellergren, A., Jamshy, S., Lau, C. T., Santos, E. R. G., Kiraly, A. P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-93967-2 · Cited by 55 · see citing works
Diagnosis of normal chest radiographs using an autonomous deep-learning algorithm
Dyer, T., Dillard, L., Harrison, M., et al. (2021) · Clinical Radiology · doi:10.1016/j.crad.2021.01.015 · Cited by 38 · see citing works
Shin, H. J., Lee, S., Kim, S., Son, N., & Kim, E. (2023) · PLoS ONE · doi:10.1371/journal.pone.0282123 · Cited by 27 · see citing works
Dillon, D., & Rodriguez, R. M. (2021) · The American Journal of Emergency Medicine · doi:10.1016/j.ajem.2021.06.034 · Cited by 18 · see citing works
Fanni, S. C., Marcucci, A., Volpi, F., Valentino, S., Neri, E., & Romei, C. (2023) · Diagnostics · doi:10.3390/diagnostics13122020 · Cited by 18 · see citing works
Kaviani, P., Kalra, M. K., Digumarthy, S. R., Gupta, R. V., Dasegowda, G., Jagirdar, A., Gupta, S., Putha, P., Mahajan, V., Reddy, B., et al. (2022) · Diagnostics · doi:10.3390/diagnostics12102382 · Cited by 14 · see citing works
Geographic Variations in Growth of Radiologists and Medicare Enrollees From 2012 to 2019
Khurana, A., Patel, B., & Sharpe, R. B. (2022) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2022.06.009 · Cited by 12 · see citing works
Keski-Filppula, T., Nikki, M. J., Haapea, M., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08123 · Cited by 7 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Abdaoui, H., Barbaria, S., Al Kuwaiti, A., Raouafi, N., Ben Salah, R., & Boussi Rahmouni, H. (2026) · Frontiers in Radiology · doi:10.3389/fradi.2026.1875905
- Authors
- Schalekamp, S., van Leeuwen, K., Calli, E., Murphy, K., Rutten, M., Geurts, B., Peters-Bax, L., van Ginneken, B., & Prokop, M.
- Affiliations
- Radboud University Nijmegen (NL)
- Corresponding author
- Schalekamp, Steven
- Venue
- European Radiology
- Published
- 2024-05-17
- Volume
- 34
- Issue
- 11
- Pages
- 7255-7263
- ISSN
- 0938-7994 (ISSN-L)
- 1432-1084
- PMC
- PMC11519193
- OpenAlex
- W4397013354
- Cited by
- 28
- References
- 28
- Citations by year
- 2026: 19 · 2025: 8 · 2024: 1
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- European Radiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Keywords
- artificial intelligence, chest radiograph, radiologist workload, automated reporting, negative predictive value, radiology reporting
- MeSH terms
- Radiologists, Adolescent, Adult, Aged, Aged, 80 and over, Artificial Intelligence, Female, Humans, Male, Middle Aged, Radiographic Image Interpretation, Computer-Assisted, Retrospective Studies, Sensitivity and Specificity, Radiography, Thoracic, Workload, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
