Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Purpose To develop and test an artificial intelligence (AI) system, based on deep convolutional neural networks (CNNs), for automated real-time triaging of adult chest radiographs on the basis of the urgency of imaging appearances. Materials and Methods An AI system was developed by using 470 388 fully anonymized institutional adult chest radiographs acquired from 2007 to 2017. The free-text radiology reports were preprocessed by using an in-house natural language processing (NLP) system modeling radiologic language. The NLP system analyzed the free-text report to prioritize each radiograph as critical, urgent, nonurgent, or normal. An AI system for computer vision using an ensemble of two deep CNNs was then trained by using labeled radiographs to predict the clinical priority from radiologic appearances only. The system's performance in radiograph prioritization was tested in a simulation by using an independent set of 15 887 radiographs. Prediction performance was assessed with the area under the receiver operating characteristic curve; sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were also determined. Nonparametric testing of the improvement in time to final report was determined at a nominal significance level of 5%. Results Normal chest radiographs were detected by our AI system with a sensitivity of 71%, specificity of 95%, PPV of 73%, and NPV of 94%. The average reporting delay was reduced from 11.2 to 2.7 days for critical imaging findings (P < .001) and from 7.6 to 4.1 days for urgent imaging findings (P < .001) in the simulation compared with historical data. Conclusion Automated real-time triaging of adult chest radiographs with use of an artificial intelligence system is feasible, with clinically acceptable performance. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Auffermann in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [110]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 1×
“An artificial neural network consists of a set of simple processing units, artificial neurons, connected in a network, organized in layers, and trained with a backpropagation algorithm (3).”
MethodDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Deep neural networks have been shown to achieve excellent performance on many natural computer vision tasks, which is advantageous for medical specialties such as radiology and dermatology (4,5).”
BackgroundLakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775 · cited 1×
“T he application of deep neural networks to medical imaging is an evolving research field (1,2).”
BackgroundCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219 · cited 2×
“Previously published studies have investigated the potential of NLP and computer vision techniques in the classification of radiographs (6,19,20) but not for real-time prioritization, which was our primary aim.”
BackgroundResultThrall, J. H., Li, X., Li, Q., Cruz, C., Do, S., Dreyer, K., & Brink, J. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.026 · Cited by 710
Cognitive and System Factors Contributing to Diagnostic Errors in Radiology
Lee, C. S., Nagy, P. G., Weaver, S. J., & Newman-Toker, D. E. (2013) · American Journal of Roentgenology · doi:10.2214/ajr.12.10375 · Cited by 415 · cited 1×
“Over such a large data set, approximately 3%–5% of labels extracted can be expected to contain an error (21).”
BackgroundTowards automatic pulmonary nodule management in lung cancer screening with deep learning
Ciompi, F., Chung, K., van Riel, S. J., Setio, A. A. A., Gerke, P. K., Jacobs, C., Scholten, E. T., Schaefer-Prokop, C., Wille, M. M. W., Marchianò, A., et al. (2017) · Scientific Reports · doi:10.1038/srep46479 · Cited by 304 · cited 1×
“…work has indicated that the performance of deep learning algorithms is comparable to or even exceeds the performance of radiologists in detecting consolidation on chest radiographs (6), segmenting cysts in polycystic kidney disease on CT scans (7), and detecting pulmonary nodules on CT scans (8).”
ResultCicero, M., Bilbily, A., Colak, E., Dowdell, T., Gray, B., Perampaladas, K., & Barfett, J. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000341 · Cited by 247 · cited 3×
“Another study classifying chest radiographs as normal or as showing cardiomegaly, consolidation, pleural effusion, pulmonary edema, or pneumothorax found a sensitivity and specificity of 91% for normal radiographs (20).”
BackgroundHigh-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks
Geras, K. J., Wolfson, S., Shen, Y., Wu, N., Kim, S.-G., Kim, E. S., Heacock, L. C., Parikh, U., Moy, L. P., & Cho, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07047 · Cited by 185 · cited 1×
“T he application of deep neural networks to medical imaging is an evolving research field (1,2).”
Background
Research works printed but not linked: 7
- Siemens. Medical imaging in the age of artificial intelligence . https://www.siemens.com/press/pool/de/events/2017/healthineers/2017-11-rsna/white-paper-medical-imaging-in-the-age-of-artificial-intelligence.pdf. Published 2017. Accessed June 19, 2018.
- Royal College of Radiologists . Clinical radiology UK workforce census 2016 report. Contract no.: BFCR(17)6. London, England: Royal College of Radiologists, 2017. [Google Scholar]
- Clinical Excellence Commission . Final report: recommendations of the Clinical Advisory Committee—plain x-ray image reporting backlog. Sydney, Australia: Clinical Excellence Committee, 2014. [Google Scholar]
- Royal College of Radiologists . Unreported x-rays, computed tomography (CT) and magnetic resonance imaging (MRI) scans: results of a snapshot survey of English National Health Service (NHS) trusts. London, England: Royal College of Radiologists, 2015. [Google Scholar]
- World Health Organization . Scientific background. In: Communicating radiation risks in paediatric imaging: information to support healthcare discussions about benefit and risk. http://www.who.int/ionizing_radiation/pub_meet/radiation-risks-paediatric-imaging/en/. Published 2016. Accessed December 28, 2017. [Google Scholar]
- Cornegruta S, Bakewell R, Withey S, Montana G. Modelling radiological language with bidirectional long short-term memory networks. Cornell University Library. https://arxiv.org/abs/1609.08409. Accessed May 20, 2018. [Google Scholar] · arXiv
- Pesce E, Ypsilantis PP, Withey S, Bakewell R, Goh V, Montana G. Learning to detect chest radiographs containing lung nodules using visual attention networks. Cornell University Library. https://arxiv.org/abs/1712.00996. Accessed May 20, 2018. [DOI] [PubMed] [Google Scholar] · arXiv
Cited by
Items 101–101 of 101
A dataset of chest X-ray reports annotated with Spatial Role Labeling annotations
Datta, S., & Roberts, K. (2020) · Data in Brief · doi:10.1016/j.dib.2020.106056 · Cited by 14
Metadata
- Authors
- Annarumma, M., Withey, S. J., Bakewell, R. J., Pesce, E., Goh, V., & Montana, G.
- Affiliations
- King's College London (GB)
- Venue
- Radiology
- Published
- 2019-04
- Volume
- 291
- Issue
- 1
- Pages
- 196-202
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC6438359
- OpenAlex
- W2911410812
- Cited by
- 303
- References
- 14
- Citations by year
- 2026: 42 · 2025: 34 · 2024: 32 · 2023: 38 · 2022: 34 · 2021: 47 · 2020: 57 · 2019: 19
- FWCI
- 12.56
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Funders
- King’s Health Partners
- Guy's & St Thomas' NHS Foundation Trust
- King's College London
- Cancer Research UK
- Medical Research Council (C1519/A16463)
- Wellcome (WT 203148/Z/16/Z)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- chest radiograph, deep convolutional neural networks, delayed reporting
- MeSH terms
- Deep Learning, Adult, Artificial Intelligence, Humans, Retrospective Studies, ROC Curve, Sensitivity and Specificity, Radiography, Thoracic, Triage, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.