Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Purpose To develop and test an artificial intelligence (AI) system, based on deep convolutional neural networks (CNNs), for automated real-time triaging of adult chest radiographs on the basis of the urgency of imaging appearances. Materials and Methods An AI system was developed by using 470 388 fully anonymized institutional adult chest radiographs acquired from 2007 to 2017. The free-text radiology reports were preprocessed by using an in-house natural language processing (NLP) system modeling radiologic language. The NLP system analyzed the free-text report to prioritize each radiograph as critical, urgent, nonurgent, or normal. An AI system for computer vision using an ensemble of two deep CNNs was then trained by using labeled radiographs to predict the clinical priority from radiologic appearances only. The system's performance in radiograph prioritization was tested in a simulation by using an independent set of 15 887 radiographs. Prediction performance was assessed with the area under the receiver operating characteristic curve; sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were also determined. Nonparametric testing of the improvement in time to final report was determined at a nominal significance level of 5%. Results Normal chest radiographs were detected by our AI system with a sensitivity of 71%, specificity of 95%, PPV of 73%, and NPV of 94%. The average reporting delay was reduced from 11.2 to 2.7 days for critical imaging findings (P < .001) and from 7.6 to 4.1 days for urgent imaging findings (P < .001) in the simulation compared with historical data. Conclusion Automated real-time triaging of adult chest radiographs with use of an artificial intelligence system is feasible, with clinically acceptable performance. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Auffermann in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Only eight studies in ophthalmology imaging14,21,32,33,43,55,108,109, ten studies in respiratory imaging64,66,70,72,75,79,82,87,89,110 and six studies in breast imaging62,91,97,104,106,111 mentioned adherence to the STARD-2015 guidelines or had a STARD flow diagram in the manuscript.”
From Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis (2021) · cites this paper as [110]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 1×
“An artificial neural network consists of a set of simple processing units, artificial neurons, connected in a network, organized in layers, and trained with a backpropagation algorithm (3).”
MethodDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1×
“Deep neural networks have been shown to achieve excellent performance on many natural computer vision tasks, which is advantageous for medical specialties such as radiology and dermatology (4,5).”
BackgroundLakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775 · cited 1×
“T he application of deep neural networks to medical imaging is an evolving research field (1,2).”
BackgroundCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning
Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D. Y., Bagul, A., Langlotz, C. P., Shpanskaya, K. S., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.05225 · Cited by 1,219 · cited 2×
“Previously published studies have investigated the potential of NLP and computer vision techniques in the classification of radiographs (6,19,20) but not for real-time prioritization, which was our primary aim.”
BackgroundResultThrall, J. H., Li, X., Li, Q., Cruz, C., Do, S., Dreyer, K., & Brink, J. (2018) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2017.12.026 · Cited by 710
Cognitive and System Factors Contributing to Diagnostic Errors in Radiology
Lee, C. S., Nagy, P. G., Weaver, S. J., & Newman-Toker, D. E. (2013) · American Journal of Roentgenology · doi:10.2214/ajr.12.10375 · Cited by 415 · cited 1×
“Over such a large data set, approximately 3%–5% of labels extracted can be expected to contain an error (21).”
BackgroundTowards automatic pulmonary nodule management in lung cancer screening with deep learning
Ciompi, F., Chung, K., van Riel, S. J., Setio, A. A. A., Gerke, P. K., Jacobs, C., Scholten, E. T., Schaefer-Prokop, C., Wille, M. M. W., Marchianò, A., et al. (2017) · Scientific Reports · doi:10.1038/srep46479 · Cited by 304 · cited 1×
“…work has indicated that the performance of deep learning algorithms is comparable to or even exceeds the performance of radiologists in detecting consolidation on chest radiographs (6), segmenting cysts in polycystic kidney disease on CT scans (7), and detecting pulmonary nodules on CT scans (8).”
ResultCicero, M., Bilbily, A., Colak, E., Dowdell, T., Gray, B., Perampaladas, K., & Barfett, J. (2017) · Investigative Radiology · doi:10.1097/rli.0000000000000341 · Cited by 247 · cited 3×
“Another study classifying chest radiographs as normal or as showing cardiomegaly, consolidation, pleural effusion, pulmonary edema, or pneumothorax found a sensitivity and specificity of 91% for normal radiographs (20).”
BackgroundHigh-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks
Geras, K. J., Wolfson, S., Shen, Y., Wu, N., Kim, S.-G., Kim, E. S., Heacock, L. C., Parikh, U., Moy, L. P., & Cho, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07047 · Cited by 185 · cited 1×
“T he application of deep neural networks to medical imaging is an evolving research field (1,2).”
Background
Research works printed but not linked: 7
- Siemens. Medical imaging in the age of artificial intelligence . https://www.siemens.com/press/pool/de/events/2017/healthineers/2017-11-rsna/white-paper-medical-imaging-in-the-age-of-artificial-intelligence.pdf. Published 2017. Accessed June 19, 2018.
- Royal College of Radiologists . Clinical radiology UK workforce census 2016 report. Contract no.: BFCR(17)6. London, England: Royal College of Radiologists, 2017. [Google Scholar]
- Clinical Excellence Commission . Final report: recommendations of the Clinical Advisory Committee—plain x-ray image reporting backlog. Sydney, Australia: Clinical Excellence Committee, 2014. [Google Scholar]
- Royal College of Radiologists . Unreported x-rays, computed tomography (CT) and magnetic resonance imaging (MRI) scans: results of a snapshot survey of English National Health Service (NHS) trusts. London, England: Royal College of Radiologists, 2015. [Google Scholar]
- World Health Organization . Scientific background. In: Communicating radiation risks in paediatric imaging: information to support healthcare discussions about benefit and risk. http://www.who.int/ionizing_radiation/pub_meet/radiation-risks-paediatric-imaging/en/. Published 2016. Accessed December 28, 2017. [Google Scholar]
- Cornegruta S, Bakewell R, Withey S, Montana G. Modelling radiological language with bidirectional long short-term memory networks. Cornell University Library. https://arxiv.org/abs/1609.08409. Accessed May 20, 2018. [Google Scholar] · arXiv
- Pesce E, Ypsilantis PP, Withey S, Bakewell R, Goh V, Montana G. Learning to detect chest radiographs containing lung nodules using visual attention networks. Cornell University Library. https://arxiv.org/abs/1712.00996. Accessed May 20, 2018. [DOI] [PubMed] [Google Scholar] · arXiv
Cited by
Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis
Aggarwal, R., Sounderajah, V., Martin, G., Ting, D. S. W., Karthikesalingam, A., King, D., Ashrafian, H., & Darzi, A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00438-z · Cited by 946
Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging
Najjar, R. (2023) · Diagnostics · doi:10.3390/diagnostics13172760 · Cited by 886
“7 days, reinforcing the potency of automated triaging systems in streamlining healthcare workflows and amplifying patient care standards [42].”
BackgroundSit, C., Srinivasan, R., Amlani, A., Muthuswamy, K., Azam, A., Monzon, L., & Poon, D. S. (2020) · Insights into Imaging · doi:10.1186/s13244-019-0830-7 · Cited by 507
A deep learning approach to detect Covid-19 coronavirus with X-Ray images
Jain, G., Mittal, D., Thakur, D., & Mittal, M. K. (2020) · Biocybernetics and Biomedical Engineering · doi:10.1016/j.bbe.2020.08.008 · Cited by 324
The Effect of Image Resolution on Deep Learning in Radiography
Sabottke, C. F., & Spieler, B. M. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019190015 · Cited by 312
“Figure 2 shows the effects of varying the image resolution on an AUC for six distinct diagnosis labels: emphysema, The Effect of Image Resolution on Deep Learning in Radiography or if artificial intelligence can help prioritize the reading order for a study list in a triage scenario (15).”
BackgroundMajkowska, A., Mittal, S., Steiner, D. F., Reicher, J. J., McKinney, S. M., Duggan, G. E., Eswaran, K., Cameron Chen, P.-H., Liu, Y., Kalidindi, S. R., et al. (2020) · Radiology · doi:10.1148/radiol.2019191293 · Cited by 277
Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., & Lungren, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.117 · Cited by 262
Artificial intelligence in diagnostic imaging: impact on the radiography profession
Hardy, M., & Harvey, H. (2020) · The British Journal of Radiology · doi:10.1259/bjr.20190840 · Cited by 255
Automated abnormality classification of chest radiographs using deep convolutional neural networks
Tang, Y.-X., Tang, Y.-B., Peng, Y., Yan, K., Bagheri, M., Redd, B. A., Brandon, C. J., Lu, Z., Han, M., Xiao, J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0273-z · Cited by 248
“Very recently, Annarumma et al.(23) used about 0.”
BackgroundInfluentialIntroduction to artificial intelligence in ultrasound imaging in obstetrics and gynecology
Drukker, L., Noble, J. A., & Papageorghiou, A. T. (2020) · Ultrasound in Obstetrics & Gynecology · doi:10.1002/uog.22122 · Cited by 240
“Promising medical AI applications are emerging in the areas of screening 5,6 , prediction 7–9 , triage 10,11 , diagnosis 12,13 , drug development 14,15 , treatment 16,17 , monitoring 18 and imaging interpretation 19,20 .”
BackgroundPham, H. H., Le, T. T., Tran, D. Q., Ngo, D. T., & Nguyen, H. Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.03.127 · Cited by 214
COVID-19 on Chest Radiographs: A Multireader Evaluation of an Artificial Intelligence System
Murphy, K., Smits, H., Knoops, A. J. G., Korst, M. B. J. M., Samson, T., Scholten, E. T., Schalekamp, S., Schaefer-Prokop, C. M., Philipsen, R. H. H. M., Meijers, A., et al. (2020) · Radiology · doi:10.1148/radiol.2020201874 · Cited by 214
“An artificial intelligence (AI) system may be a helpful tool as an adjunct for radiologists or, in the common case that radiologic expertise is not available, for the medical team (7,8).”
BackgroundAutomated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm
Weikert, T., Winkel, D. J., Bremerich, J., Stieltjes, B., Parmar, V., Sauter, A. W., & Sommer, G. (2020) · European Radiology · doi:10.1007/s00330-020-06998-0 · Cited by 163
“In addition, this approach of using clinically approved reports as standard of reference has also been applied in a range of other recent studies [7, 43, 44].”
ResultWu, J. T., Wong, K. C. L., Gur, Y., Ansari, N., Karargyris, A., Sharma, A., Morris, M., Saboury, B., Ahmad, H., Boyko, O., et al. (2020) · JAMA Network Open · doi:10.1001/jamanetworkopen.2020.22779 · Cited by 149
Ouyang, X., Karanam, S., Wu, Z., Chen, T., Huo, J., Zhou, X. S., Wang, Q., & Cheng, J. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3042773 · Cited by 149
A review of explainable and interpretable AI with applications in COVID‐19 imaging
Fuhrman, J. D., Gorre, N., Hu, Q., Li, H., El Naqa, I., & Giger, M. L. (2022) · Medical Physics · doi:10.1002/mp.15359 · Cited by 120
Choi, J. W., Cho, Y. J., Lee, S., Lee, J. S., Lee, S., Choi, Y. H., Cheon, J., & Ha, J. Y. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000615 · Cited by 119
Nam, J. G., Kim, M., Park, J., Hwang, E. J., Lee, J. H., Hong, J. H., Goo, J. M., & Park, C. M. (2021) · European Respiratory Journal · doi:10.1183/13993003.03061-2020 · Cited by 103
“We categorised the 10 abnormalities covered by DLAD-10 according to their clinical urgency following a previous study [20].”
MethodSmit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A. Y., & Lungren, M. P. (2020) · arXiv · doi:10.48550/arxiv.2004.09167 · Cited by 103
“Many natural language processing systems have been developed to extract structured labels from free-text radiology reports (Pons et al., 2016; Yadav et al., 2016; Hassanpour et al., 2017; Annarumma et al., 2019; Savova et al., 2010; Wang et al., 2018; Chen et al., 2018; Bozkurt et al., 2019).”
BackgroundDraelos, R. L., Dov, D., Mazurowski, M. A., Lo, J. Y.-C., Henao, R., Rubin, G. D., & Carin, L. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101857 · Cited by 165
Applications of artificial intelligence and machine learning in respiratory medicine
Gonem, S., Janssens, W., Das, N., & Topalovic, M. (2020) · Thorax · doi:10.1136/thoraxjnl-2020-214556 · Cited by 99
Autonomous Chest Radiograph Reporting Using AI: Estimation of Clinical Impact
Plesner, L. L., Müller, F. C., Nybing, J. D., Laustrup, L. C., Rasmussen, F., Nielsen, O. W., Boesen, M., & Andersen, M. B. (2023) · Radiology · doi:10.1148/radiol.222268 · Cited by 95
“In other larger consecutive patient samples from hospital-based radiology clinics, 13% and 20% of chest radiographs were labeled as normal (12,17).”
Goncharov, M., Pisov, M., Shevtsov, A. E., Shirokikh, B., Kurmukov, A. G., Blokhin, I. A., Chernina, V., Solovev, A. V., Gombolevskiy, V. А., Morozov, S. P., et al. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102054 · Cited by 98
Artificial intelligence in radiography: Where are we now and what does the future hold?
Malamateniou, C., KNAPP, K. M., Pergola, M. B., Woznitza, N., & Hardy, M. (2021) · Radiography · doi:10.1016/j.radi.2021.07.015 · Cited by 119
Topff, L., Ranschaert, E. R., Bartels-Rutten, A., Negoita, A., Menezes, R., Beets-Tan, R. G. H., & Visser, J. J. (2023) · Radiology: Cardiothoracic Imaging · doi:10.1148/ryct.220163 · Cited by 81
Deep learning applications in pulmonary medical imaging: recent updates and insights on COVID-19
Farhat, H. J., Sakr, G. E., & Kilany, R. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01101-5 · Cited by 93
“[128] 2019 28 CXR General thoracic diseases”
BackgroundInfluentialPierre, K., Haneberg, A. G., Kwak, S., Peters, K. R., Hochhegger, B., Sananmuang, T., Tunlayadechanont, P., Tighe, P. J., Mancuso, A., & Forghani, R. (2023) · Seminars in Roentgenology · doi:10.1053/j.ro.2023.02.003 · Cited by 79
Baltruschat, I., Steinmeister, L., Nickisch, H., Saalbach, A., Grass, M., Adam, G., Knopp, T., & Ittrich, H. (2021) · European Radiology · doi:10.1007/s00330-020-07480-7 · Cited by 78
“For chest X-rays (CXRs), a potential benefit of real-time triaging by CNNs has been reported by Annarumma et al [9], but they focused mostly on the development of an AI system without a real clinical simulation and do not present the maximum RTAT.”
BackgroundPark, S., Lee, S. M., Lee, K. H., Jung, K.-H., Bae, W., Choe, J., & Seo, J. B. (2019) · European Radiology · doi:10.1007/s00330-019-06532-x · Cited by 85
Artificial intelligence in paediatric radiology: Future opportunities
Davendralingam, N., Sebire, N. J., Arthurs, O. J., & Shelmerdine, S. C. (2020) · British Journal of Radiology · doi:10.1259/bjr.20200975 · Cited by 87
“Workflow prioritisation with AI tools to facilitate urgent radiology reporting have been explored in the adult literature, predominantly for urgent findings on chest radiography 63 and CT heads.”
MethodHwang, E. J., & Park, C. M. (2020) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0821 · Cited by 71
Mastrodicasa, D., van Assen, M., Huisman, M., Leiner, T., Williamson, E. E., Nicol, E. D., Allen, B. D., Saba, L., Vliegenthart, R., & Hanneman, K. (2025) · Radiology · doi:10.1148/radiol.240516 · Cited by 69
Machine learning based natural language processing of radiology reports in orthopaedic trauma
Olthof, A. W., Shouche, P., Fennema, E. M., IJpma, F. F. A., Koolstra, R. H. C., Stirler, V. M. A., van Ooijen, P. M. A., & Cornelissen, L. J. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106304 · Cited by 70
“The results for the chest radiograph reports appear to be good enough to apply for the future purpose of automated labeling of chest radiographs to train an imaged based neural network using weak labels, because the F1 score of the NLP labels is in the same range as in the study of Annarumma [42] .”
ResultAutomatic classification of canine thoracic radiographs using deep learning
Banzato, T., Wodziński, M., Burti, S., Osti, V. L., Rossoni, V., Atzori, M., & Zotti, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-83515-3 · Cited by 87
Artificial Intelligence in Emergency Radiology: Where Are We Going?
Cellina, M., Cè, M., Irmici, G., Ascenti, V., Caloro, E., Bianchi, L., Pellegrino, G., D’Amico, N., Papa, S., & Carrafiello, G. (2022) · Diagnostics · doi:10.3390/diagnostics12123223 · Cited by 62
Schmuelling, L., Franzeck, F. C., Nickel, C. H., Mansella, G., Bingisser, R. M., Schmidt, N., Stieltjes, B., Bremerich, J., Sauter, A. W., Weikert, T., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109816 · Cited by 71
Chest X-ray in Emergency Radiology: What Artificial Intelligence Applications Are Available?
Irmici, G., Cè, M., Caloro, E., Khenkina, N., Pepa, G. D., Ascenti, V., Martinenghi, C. M. A., Papa, S., Oliva, G., & Cellina, M. I. (2023) · Diagnostics · doi:10.3390/diagnostics13020216 · Cited by 76
Zhang, S., Sun, J., Liu, C., Fang, J., Xie, H., & Ning, B. (2020) · The Bone & Joint Journal · doi:10.1302/0301-620x.102b11.bjj-2020-0712.r2 · Cited by 61
Deep learning to automate the labelling of head MRI datasets for computer vision applications
Wood, D. A., Kafiabadi, S., Busaidi, A. A., Guilhem, E., Lynch, J., Townend, M., Montvila, A., Kiik, M., Siddiqui, J., Gadapa, N., et al. (2021) · European Radiology · doi:10.1007/s00330-021-08132-0 · Cited by 61
The Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms—How We Do It
Mahajan, V., Venugopal, V. K., Murugavel, M., & Mahajan, H. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.09.009 · Cited by 57
Tolkachev, A., Sirazitdinov, I., Kholiavchenko, M., Mustafaev, T., & Ibragimov, B. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3023476 · Cited by 57
Kwon, Y. J. (., Toussie, D., Finkelstein, M., Cedillo, M. A., Maron, S. Z., Manna, S., Voutsinas, N., Eber, C., Jacobi, A., Bernheim, A., et al. (2021) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200098 · Cited by 53
“A intelligence (AI) has demonstrated promise in facilitating triage in radiology departments owing to its ability to rapidly extract key features from imaging studies and perform high-throughput analysis, especially in institutions with high volumes of disease (1).”
BackgroundAldhafeeri, F. M. (2025) · Diagnostics · doi:10.3390/diagnostics15182300 · Cited by 83
“…[15–17]; (2) segmentation and image analysis systems, including tumor contouring, organ segmentation, and quantitative imaging biomarkers [17,18]; (3) workflow optimization applications, such as triage systems, prioritization of urgent cases, and structured reporting tools [19,20]; and (4)…”
A Practical Guide to Artificial Intelligence–Based Image Analysis in Radiology
Weikert, T., Cyriac, J., Yang, S., Nešić, I., Parmar, V., & Stieltjes, B. (2019) · Investigative Radiology · doi:10.1097/rli.0000000000000600 · Cited by 59
“That AI algorithms can handle a certain amount of wrong labels has been demonstrated among others by Annarumma and colleagues, who used NLP-derived labels of over 470,000 adult chest radiographs to successfully train 2 DCNNs for triaging based on imaging information.(37) For many projects, the labeling of reports on t…”
BackgroundMethodInfluentialNabulsi, Z., Sellergren, A., Jamshy, S., Lau, C. T., Santos, E. R. G., Kiraly, A. P., Ye, W., Yang, J., Pilgrim, R., Kazemzadeh, S., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-93967-2 · Cited by 74
Artificial Intelligence (AI) for Lung Nodules, From the AJR Special Series on AI Applications
Liu, J. A., Yang, I. Y., & Tsai, E. B. (2022) · American Journal of Roentgenology · doi:10.2214/ajr.22.27487 · Cited by 60
Applications of Artificial Intelligence in Breast Imaging
Morgan, M. B., & Mates, J. (2020) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2020.08.007 · Cited by 59
Tadavarthi, Y., Makeeva, V., Wagstaff, W., Zhan, H., Podlasek, A., Bhatia, N., Heilbrun, M., Krupinski, E., Safdar, N., Banerjee, I., et al. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.210114 · Cited by 48
“Annarumma et al (63) tested such a system to simulate a triage system for retrospective adult chest radio-graphs, resulting in a theoretical reduction in reporting delay for critical studies from 11.2 to 2.7 days.”
BackgroundInfluentialUllah, N., Khan, J. A., Almakdi, S., Khan, M. S., Alshehri, M., Alboaneen, D., & Raza, A. (2022) · Applied Sciences · doi:10.3390/app12126269 · Cited by 53
“When experienced radiologists are unavailable, these computer-aided detection (CADs) systems can help with imaging-based patient classification in resource-constrained situations and increase practitioner’s chest radiograph interpretation accuracy and inter-reader variability [28,29].”
BackgroundHwang, E. J., Kim, K. B., Kim, J. Y., Lim, J., Nam, J. G., Choi, H., Kim, H., Yoon, S. H., Goo, J. M., & Park, C. M. (2021) · PLoS ONE · doi:10.1371/journal.pone.0252440 · Cited by 51
“These computer-aided detection systems (CADs) may help imaging-based triage of patients in resource-constrained environments where expert radiologists are not available, and improve accuracy and inter-reader variability of physicians’ CXR interpretations [25, 26].”
BackgroundHong, W., Hwang, E. J., Lee, J. H., Park, J., Goo, J. M., & Park, C. M. (2022) · Radiology · doi:10.1148/radiol.211706 · Cited by 49
An Efficient Deep Learning Model to Detect COVID-19 Using Chest X-ray Images
Chakraborty, S., Murali, B., & Mitra, A. K. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19042013 · Cited by 55
Deep learning models for triaging hospital head MRI examinations
Wood, D. A., Kafiabadi, S., Busaidi, A. A., Guilhem, E., Montvila, A., Lynch, J., Townend, M., Agarwal, S., Mazumder, A., Barker, G. J., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102391 · Cited by 57
Imaging Quality Control in the Era of Artificial Intelligence
Larson, D. B., & Boland, G. W. (2019) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2019.05.048 · Cited by 45
Hwang, E. J., Goo, J. M., Yoon, S. H., Beck, K. S., Seo, J. B., Choi, B. W., Chung, M. J., Park, C. M., Jin, K. N., & Lee, S. M. (2021) · Korean Journal of Radiology · doi:10.3348/kjr.2021.0544 · Cited by 43
“Using AI-SaMDs in screening for images with findings of emergency disease requiring timely interpretation, automated assignment of interpreting radiologists according to the presence of abnormality or difficulty of interpretation, and automated notification of suspected interpretive errors have also been proposed [18,…”
MethodHwang, E. J., Hong, J. H., Lee, K. H., Kim, J. I., Nam, J. G., Kim, D., Choi, H., Yoo, S., Goo, J. M., & Park, C. M. (2020) · European Radiology · doi:10.1007/s00330-020-06771-3 · Cited by 47
Habib, M. M., Hoodbhoy, Z., & Siddiqui, M. A. R. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000443 · Cited by 41
Wu, Y., Kong, Q., Zhang, L., Castiglione, A., Nappi, M., & Wan, S. (2023) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2023.3258455 · Cited by 44
Wang, X., Yang, S., Lan, J., Fang, Y., He, J., Wang, M., Zhang, J., & Han, X. (2020) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2020.3035572 · Cited by 44
“Large pneumothorax is classified as a critical abnormality that requires immediate medical treatment to remove the excess air [1], e.g., by using a needle or a chest tube.”
BackgroundMachine learning outperforms clinical experts in classification of hip fractures
Murphy, E., Ehrhardt, B., Gregson, C. L., Arx, O. A. V., Hartley, A. E., Whitehouse, M. R., Thomas, M., Stenhouse, G., Chesser, T. J. S., Budd, C. J., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-06018-9 · Cited by 49
Cadamuro, J. (2021) · Diagnostics · doi:10.3390/diagnostics11081399 · Cited by 47
“Normal chest radiographs were detected with a negative predictive value (NPV) of 94% and a positive predictive value (PPV) of 73% (sensitivity of 71%, specificity of 95%) [19].”
BackgroundAutopilots in the Operating Room: Safe Use of Automated Medical Technology
Ruskin, K. J., Corvin, C., Rice, S. C., & Winter, S. R. (2020) · Anesthesiology · doi:10.1097/aln.0000000000003385 · Cited by 38
“Another recently published study demonstrated that computers can triage chest radiographs in real time.(17) These developments are only the first examples of how advances in technology will expand the potential for intelligent medical devices.”
BackgroundArtificial Intelligence and the Trainee Experience in Radiology
Simpson, S. A., & Cook, T. S. (2020) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2020.09.028 · Cited by 38
Wang, Q., Liu, Q., Luo, G., Liu, Z., Huang, J., Zhou, Y., Zhou, Y., Xu, W., & Cheng, J. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01325-5 · Cited by 46
Buls, N., Watté, N., Nieboer, K., Ilsen, B., & Mey, J. D. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.015 · Cited by 40
“Also, an AI solution with a high false-negative rate can be more harmful, especially in outpatients [20-23].”
BackgroundOuyang, X., Xue, Z., Zhan, Y., Zhou, X. S., Wang, Q., Zhou, Y., Wang, Q., & Cheng, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32226-7_68 · Cited by 44
Guo, L., Zhou, C., Xu, J., Huang, C., Yu, Y., & Lu, G. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-00990-6 · Cited by 47
Rueckel, J., Kunz, W. G., Hoppe, B. F., Patzig, M., Notohamiprodjo, M., Meinel, F. G., Cyran, C. C., Ingrisch, M., Ricke, J., & Sabel, B. O. (2020) · Critical Care Medicine · doi:10.1097/ccm.0000000000004397 · Cited by 37
“These approaches could also serve as clinical decision support for nonradiologists in clinical settings without 24/7 radiology coverage or long turnaround times for radiology reports (11, 12).”
BackgroundDwivedi, K., Sharkey, M. J., Condliffe, R. A., Uthoff, J., Alabed, S., Metherall, P., Lu, H., Wild, J. M., Hoffman, E. A., Swift, A. J., et al. (2021) · Diagnostics · doi:10.3390/diagnostics11040679 · Cited by 41
“Triaging of scans is another area which has seen significant research [66].”
BackgroundDetection of COVID-19 from chest x-ray images using transfer learning
Manokaran, J., Zabihollahy, F., Hamilton-Wright, A., & Ukwatta, E. (2021) · Journal of medical imaging · doi:10.1117/1.jmi.8.s1.017503 · Cited by 37
“3,4 Though CT images are extensively used in the COVID-19 diagnosis, 5 – 7 cost 8 and radiation exposure are major concerns.”
MethodSupervised and unsupervised language modelling in Chest X-Ray radiological reports
Drozdov, I. A., Forbes, D., Szubert, B., Hall, M., Carlin, C. M., & Lowe, D. J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0229963 · Cited by 42
“Multi-label annotation of abnormal reports has been the primary aim of radiological language models [2, 4, 18, 19].”
BackgroundFeng, S., Liu, Q., Patel, A., Bazai, S. U., Jin, C., Kim, J. S., Sarrafzadeh, M., Azzollini, D., Yeoh, J., Kim, E., et al. (2022) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13393 · Cited by 40
“This is comparable to that of other departments globally.(3,4) Despite these delays, no effective CXR prioritisation process has been implemented in a typical New Zealand radiology department.”
MethodResultA Systematic Literature Review of 3D Deep Learning Techniques in Computed Tomography Reconstruction
Rahman, H., Khan, A. R., Sadiq, T., Farooqi, A. H., Khan, I. U., & Lim, W. H. (2023) · Tomography · doi:10.3390/tomography9060169 · Cited by 40
Real-World evaluation of an AI triaging system for chest X-rays: A prospective clinical study
Sridharan, S., Hui, A. S. X., Venkataraman, N., Tirukonda, P. S., Jeyaratnam, R. P., John, S., Babu, S. S., Liew, P., Francis, J., Tsai, K. T., et al. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111783 · Cited by 40
Datta, S., Si, Y., Rodriguez, L. M., Shooshan, S. E., Demner‐Fushman, D., & Roberts, K. E. (2020) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103473 · Cited by 44
Artificial Intelligence in Radiology: Opportunities and Challenges
Flory, M. N., Napel, S., & Tsai, E. B. (2024) · Seminars in Ultrasound, CT and MRI · doi:10.1053/j.sult.2024.02.004 · Cited by 29
Performance of AI to exclude normal chest radiographs to reduce radiologists' Workload
Schalekamp, S., van Leeuwen, K., Calli, E., Murphy, K., Rutten, M., Geurts, B., Peters-Bax, L., van Ginneken, B., & Prokop, M. (2024) · European Radiology · doi:10.1007/s00330-024-10794-5 · Cited by 28
“Therefore, several studies focused on triaging chest radiographs, prioritizing urgent fi ndings in the radiologist ’ s worklist, and ensuring that they are reported fi rst [12 – 14].”
BackgroundCuration of the CANDID-PTX Dataset with Free-Text Reports
Feng, S., Azzollini, D., Kim, J. S., Jin, C.-K., Gordon, S. P., Yeoh, J., Kim, E., Han, M., Lee, A., Patel, A., et al. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021210136 · Cited by 49
Use of artificial intelligence in triaging of chest radiographs to reduce radiologists’ workload
Yoon, S. H., Park, S.-Y., Jang, S., Kim, J., Lee, K. W., Lee, W., Lee, S., Yun, G., & Lee, K. H. (2023) · European Radiology · doi:10.1007/s00330-023-10124-1 · Cited by 35
COVID-19 prediction based on genome similarity of human SARS-CoV-2 and bat SARS-CoV-like coronavirus
Arslan, H. (2021) · Computers & Industrial Engineering · doi:10.1016/j.cie.2021.107666 · Cited by 28
Artificial intelligence in cardiac computed tomography
Aromiwura, A. A., Settle, T., Umer, M., Joshi, J. K., Shotwell, M., Mattumpuram, J., Vorla, M., Sztukowska, M. N., Contractor, S. G., Amini, A. A., et al. (2023) · Progress in Cardiovascular Diseases · doi:10.1016/j.pcad.2023.09.001 · Cited by 34
AI Applications for Thoracic Imaging: Considerations for Best Practice
Hwang, E. J., Goo, J. M., & Park, C. M. (2025) · Radiology · doi:10.1148/radiol.240650 · Cited by 39
Kuo, P., Tsai, C. C., López, D. M., Karargyris, A., Pollard, T., Johnson, A. E. W., & Celi, L. A. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00393-9 · Cited by 32
Deep-Learning-Based Diagnosis of Bedside Chest X-ray in Intensive Care and Emergency Medicine
Niehues, S. M., Adams, L. C., Gaudin, R., Erxleben, C., Keller, S., Makowski, M. R., Vahldiek, J. L., & Bressem, K. K. (2021) · Investigative Radiology · doi:10.1097/rli.0000000000000771 · Cited by 30
“32 In our institution, we observed that the “ time to report ” varied particularly during after-hours (8 PM to 6 AM ), with a median “ time to report ” of more than 60 minutes.”
BackgroundWoolley, J. L. (2023) · Industry and Innovation · doi:10.1080/13662716.2023.2168519 · Cited by 31
Yoo, H., Kim, E. Y., Kim, H., Choi, Y. R., Kim, M. Y., Hwang, S. H., Kim, Y. J., Cho, Y. J., & Jin, K. N. (2022) · Korean Journal of Radiology · doi:10.3348/kjr.2022.0189 · Cited by 27
Chen, M., Cai, R., Zhang, A., Chi, X., & Qian, J. (2024) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-024-05003-4 · Cited by 33
Siira, E., Johansson, H., & Nygren, J. (2025) · Journal of Medical Internet Research · doi:10.2196/53741 · Cited by 28
“Examples of these studies include triage of adult chest radiographs [36], triage using digitalized patient histories in primary care [49], and triage of patients with COVID-19 under limited health care resources [69].”
Clinical Artificial Intelligence Applications in Radiology
Lee, S., & Summers, R. M. (2021) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2021.07.001 · Cited by 28
Deep Learning for Chest Radiology: A Review
Feng, Y., Teh, H. S., & Cai, Y. (2019) · Current Radiology Reports · doi:10.1007/s40134-019-0333-9 · Cited by 30
Kim, Y.-G., Lee, S. M., Lee, K. H., Jang, R., Seo, J. B., & Kim, N. (2020) · European Radiology · doi:10.1007/s00330-020-06892-9 · Cited by 27
“However, Annarumma et al [13] used the large size of the input matrix with 1211×1083 that will not miss information about the small size of input matrix.”
MethodHastuti, E. T., Bustamam, A., Anki, P., Amalia, R., & Salma, A. (2021) · IEEE International Conference on Health, Instrumentation & Measurement, and Natural Sciences (InHeNce) · doi:10.1109/inhence52833.2021.9537261 · Cited by 26
Zhang, Y., Liu, M., Zhang, L., Wang, L., Zhao, K., Hu, S., Chen, X., & Xie, X. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.55113 · Cited by 20
AI-Based CXR First Reading: Current Limitations to Ensure Practical Value
Vasilev, Y. A., Vladzymyrskyy, A. V., Omelyanskaya, O. V., Blokhin, I. A., Kirpichev, Y., & Arzamasov, K. M. (2023) · Diagnostics · doi:10.3390/diagnostics13081430 · Cited by 30
“Furthermore, an AI system can serve as a clinical decision support system (CDSS) [5,6].”
BackgroundLi, A., Javidan, A. P., Namazi, B., Madani, A., & Forbes, T. L. (2023) · Annals of Vascular Surgery · doi:10.1016/j.avsg.2023.08.027 · Cited by 25
Computer‐Aided Detection AI Reduces Interreader Variability in Grading Hip Abnormalities With MRI
Tibrewala, R., Ozhinsky, E., Shah, R., Flament, I., Crossley, K. M., Srinivasan, R., Souza, R. B., Link, T. M., Pedoia, V., & Majumdar, S. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27164 · Cited by 25
“These results suggest that if incorporated in a radiology workflow, the network could detect patients with higher lesion probability and triage abnormal cases for review by the radiologist, therefore freeing up the radiologist to work on more complicated cases.(25,26) Additionally, the results of the all the networks…”
Mosquera, C., Díaz, F. N., Binder, F., Rabellino, J. M., Benítez, S., Beresñak, A. D., Seehaus, A., Ducrey, G., Ocantos, J., & Luna, D. R. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106130 · Cited by 22
Artificial Intelligence in Lung Imaging
Choe, J., Lee, S. M., Hwang, H. J., Yun, J., Kim, N., & Seo, J. B. (2022) · Seminars in Respiratory and Critical Care Medicine · doi:10.1055/s-0042-1755571 · Cited by 26
The top 100 most cited articles on artificial intelligence in radiology: a bibliometric analysis
Hughes, H., O’Reilly, M., McVeigh, N., & Ryan, R. (2023) · Clinical Radiology · doi:10.1016/j.crad.2022.09.133 · Cited by 24
Humans and machines: Moving towards a more symbiotic approach to learning clinical reasoning
Pinnock, R., McDonald, J., Ritchie, D., & Durning, S. J. (2019) · Medical Teacher · doi:10.1080/0142159x.2019.1679361 · Cited by 23
“Relieving radiologists from the task of reading routine chest x-rays will allow them more time to spend on interventional procedures (Annarumma et al. 2019).”
Background
Metadata
- Authors
- Annarumma, M., Withey, S. J., Bakewell, R. J., Pesce, E., Goh, V., & Montana, G.
- Affiliations
- King's College London (GB)
- Venue
- Radiology
- Published
- 2019-04
- Volume
- 291
- Issue
- 1
- Pages
- 196-202
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- PMC
- PMC6438359
- OpenAlex
- W2911410812
- Cited by
- 303
- References
- 14
- Citations by year
- 2026: 42 · 2025: 34 · 2024: 32 · 2023: 38 · 2022: 34 · 2021: 47 · 2020: 57 · 2019: 19
- FWCI
- 12.56
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Research Portal (King's College London) (submittedVersion, cc-by-nc-nd)
- Warwick Research Archive Portal (University of Warwick) (submittedVersion)
- Funders
- King’s Health Partners
- Guy's & St Thomas' NHS Foundation Trust
- King's College London
- Cancer Research UK
- Medical Research Council (C1519/A16463)
- Wellcome (WT 203148/Z/16/Z)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Keywords
- chest radiograph, deep convolutional neural networks, delayed reporting
- MeSH terms
- Deep Learning, Adult, Artificial Intelligence, Humans, Retrospective Studies, ROC Curve, Sensitivity and Specificity, Radiography, Thoracic, Triage, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.