The Added Effect of Artificial Intelligence on Physicians’ Performance in Detecting Thoracic Pathologies on CT and Chest X-ray: A Systematic Review
Open accessThe DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Abstract
Our systematic review investigated the additional effect of artificial intelligence-based devices on human observers when diagnosing and/or detecting thoracic pathologies using different diagnostic imaging modalities, such as chest X-ray and CT. Peer-reviewed, original research articles from EMBASE, PubMed, Cochrane library, SCOPUS, and Web of Science were retrieved. Included articles were published within the last 20 years and used a device based on artificial intelligence (AI) technology to detect or diagnose pulmonary findings. The AI-based device had to be used in an observer test where the performance of human observers with and without addition of the device was measured as sensitivity, specificity, accuracy, AUC, or time spent on image reading. A total of 38 studies were included for final assessment. The quality assessment tool for diagnostic accuracy studies (QUADAS-2) was used for bias assessment. The average sensitivity increased from 67.8% to 74.6%; specificity from 82.2% to 85.4%; accuracy from 75.4% to 81.7%; and Area Under the ROC Curve (AUC) from 0.75 to 0.80. Generally, a faster reading time was reported when radiologists were aided by AI-based devices. Our systematic review showed that performance generally improved for the physicians when assisted by AI-based devices compared to unaided interpretation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zhang, D., Liu, X., Shao, M., et al. (2021) · Personal and Ubiquitous Computing · doi:10.1007/s00779-021-01522-7 · Cited by 13
RetractedQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152 · cited 1×
“No clinical trials have been published using AI-based devices on thoracic CT or chest X-rays, whether it be as a stand-alone diagnostic tool or as an additional reader to humans [61].”
BackgroundPreparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020 · cited 9×
“To be able to include a larger number of studies, we allowed the possibility of some inter-study variability in the performance of the AI-based devices because of different AI-algorithms being used.”
BackgroundInfluentialDo as AI say: susceptibility in deployment of clinical decision-aids
Gaube, S., Suresh, H., Raue, M., Merritt, A., Berkowitz, S. J., Lermer, E., Coughlin, J. F., Guttag, J. V., Colak, E., & Ghassemi, M. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00385-9 · Cited by 543 · cited 2×
“[51] suggested that physicians with less experience were more likely to accept and deploy the suggested advice given to them by AI.”
BackgroundNam, J. G., Park, S., Hwang, E. J., Lee, J. H., Jin, K.-N., Lim, K. Y., Vu, T. H., Sohn, J. H., Hwang, S., Goo, J. M., et al. (2019) · Radiology · doi:10.1148/radiol.2018180237 · Cited by 539
Hwang, E. J., Park, S., Jin, K.-N., Kim, J. I., Choi, S. Y., Lee, J. H., Goo, J. M., Aum, J., Yim, J.-J., Cohen, J. G., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.1095 · Cited by 447
Wang, P., Liu, X., Berzin, T. M., Glissen Brown, J. R., Liu, P., Zhou, C., Lei, L., Li, L., Guo, Z., Lei, S., et al. (2020) · The Lancet Gastroenterology & Hepatology · doi:10.1016/s2468-1253(19)30411-x · Cited by 415 · cited 1×
“However, few prospective clinical trials have been published where AI-based devices have been used, in a more dynamic and clinically realistic environment [59,60].”
BackgroundStrohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C., & Moors, E. H. M. (2020) · European Radiology · doi:10.1007/s00330-020-06946-y · Cited by 329
Non-invasive decision support for NSCLC treatment using PET/CT radiomics
Mu, W., Jiang, L., Zhang, J., Shi, Y., Gray, J. E., Tunali, I., Gao, C., Sun, Y., Tian, J., Zhao, X., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-19116-x · Cited by 289
Bias in Radiology: The How and Why of Misses and Misinterpretations
Busby, L. P., Courtier, J. L., & Glastonbury, C. M. (2018) · RadioGraphics · doi:10.1148/rg.2018170107 · Cited by 277
Awai, K., Murao, K., Ozawa, A., Komi, M., Hayakawa, H., Hori, S., & Nishimura, Y. (2004) · Radiology · doi:10.1148/radiol.2302030049 · Cited by 272
Sim, Y., Chung, M. J., Kotter, E., Yune, S., Kim, M., Do, S., Han, K., Kim, H., Yang, S., Lee, D.-J., et al. (2020) · Radiology · doi:10.1148/radiol.2019182465 · Cited by 248 · cited 1×
“[46] 2020 Biopsy, surgery, CT, and pathology Commercially available (ALND, version 1.”
BackgroundKobayashi, T., Xu, X. W., MacMahon, H., Metz, C. E., & Doi, K. (1996) · Radiology · doi:10.1148/radiology.199.3.8638015 · Cited by 244
Lin, H., Li, R., Liu, Z., Chen, J., Yang, Y., Chen, H., Lin, Z., Lai, W., Long, E., Wu, X., et al. (2019) · EClinicalMedicine · doi:10.1016/j.eclinm.2019.03.001 · Cited by 199
Kakeda, S., Moriya, J., Sato, H., Aoki, T., Watanabe, H., Nakata, H., Oda, N., Katsuragawa, S., Yamamoto, K., & Doi, K. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.182.2.1820505 · Cited by 141
Assessment of Medical Imaging Systems and Computer Aids: A Tutorial Review
Wagner, R. F., Metz, C. E., & Campbell, G. (2007) · Academic Radiology · doi:10.1016/j.acra.2007.03.001 · Cited by 139
Rajpurkar, P., O’Connell, C., Schechter, A., Asnani, N., Li, J., Kiani, A., Ball, R. L., Mendelson, M., Maartens, G., van Hoving, D. J., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00322-2 · Cited by 137 · cited 2×
“[31] 2020 Positive culture or Xpert MTB/RIF test Convolutional Neural Network Tuberculosis 114 13 physicians (6 months–25 years of experience) Chest X-ray”
MethodCT Colonography with Computer-aided Detection as a Second Reader: Observer Performance Study
Petrick, N., Haider, M., Summers, R. M., Yeshwant, S. C., Brown, L., Iuliano, E. M., Louie, A., Choi, J. R., & Pickhardt, P. J. (2008) · Radiology · doi:10.1148/radiol.2453062161 · Cited by 134
Matsuki, Y., Nakamura, K., Watanabe, H., Aoki, T., Nakata, H., Katsuragawa, S., & Doi, K. (2002) · American Journal of Roentgenology · doi:10.2214/ajr.178.3.1780657 · Cited by 127
Pehrson, L. M., Nielsen, M. B., & Ammitzbøl Lauridsen, C. (2019) · Diagnostics · doi:10.3390/diagnostics9010029 · Cited by 125
Self-citationLiu, K., Li, Q., Ma, J., Zhou, Z., Sun, M., Deng, Y., Tu, W., Wang, Y., Fan, L., Xia, C., et al. (2019) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2019180084 · Cited by 110
Nam, J. G., Kim, M., Park, J., Hwang, E. J., Lee, J. H., Hong, J. H., Goo, J. M., & Park, C. M. (2021) · European Respiratory Journal · doi:10.1183/13993003.03061-2020 · Cited by 103 · cited 2×
“[29] 2021 RT-PCR and CT Deep learning-based algorithm (Deep convolutional neural network) Pneumonia, pulmonary edema, active tuberculosis, interstitial lung disease, nodule/mass, pleural effusion, acute aortic syndrome, pneumoperitoneum, rib fracture, pneumothorax, mediastinal mass.”
MethodBeyer, F., Zierott, L., Fallenberg, E. M., Juergens, K. U., Stoeckel, J., Heindel, W. L., & Wormanns, D. (2007) · European Radiology · doi:10.1007/s00330-007-0667-1 · Cited by 99
Awai, K., Murao, K., Ozawa, A., Nakayama, Y., Nakaura, T., Liu, D., Kawanaka, K., Funama, Y., Morishita, S., & Yamashita, Y. (2006) · Radiology · doi:10.1148/radiol.2383050167 · Cited by 85
Computer-aided Detection of Lung Cancer on Chest Radiographs: Effect on Observer Performance
de Hoop, B., De Boo, D. W., Gietema, H. A., van Hoorn, F., Mearadji, B., Schijf, L., van Ginneken, B., Prokop, M., & Schaefer-Prokop, C. (2010) · Radiology · doi:10.1148/radiol.10092437 · Cited by 82
Schalekamp, S., van Ginneken, B., Koedam, E., Snoeren, M. M., Tiehuis, A. M., Wittenberg, R., Karssemeijer, N., & Schaefer-Prokop, C. M. (2014) · Radiology · doi:10.1148/radiol.14131315 · Cited by 80
Roos, J. E., Paik, D., Olsen, D., Liu, E. G., Chow, L. C., Leung, A. N., Mindelzun, R., Choudhury, K. R., Naidich, D. P., Napel, S., et al. (2010) · European Radiology · doi:10.1007/s00330-009-1596-y · Cited by 79
Abé, H., MacMahon, H., Engelmann, R. M., Li, Q., Shiraishi, J., Katsuragawa, S., Aoyama, M., Ishida, T., Ashizawa, K., Metz, C. E., et al. (2003) · Radiographics · doi:10.1148/rg.231025129 · Cited by 75
Biases in Radiologic Reasoning
Gunderman, R. B. (2009) · American Journal of Roentgenology · doi:10.2214/ajr.08.1220 · Cited by 74
Sung, J., Park, S., Lee, S. M., Bae, W., Park, B., Jung, E., Seo, J. B., & Jung, K.-H. (2021) · Radiology · doi:10.1148/radiol.2021202818 · Cited by 71
Li, D., Mikela Vilmun, B., Frederik Carlsen, J., Albrecht-Beste, E., Ammitzbøl Lauridsen, C., Bachmann Nielsen, M., & Lindskov Hansen, K. (2019) · Diagnostics · doi:10.3390/diagnostics9040207 · Cited by 66
Self-citationLi, F., Engelmann, R. M., Pesce, L. L., et al. (2011) · Radiology · doi:10.1148/radiol.11110192 · Cited by 64
Oda, S., Awai, K., Suzuki, K., Yanaga, Y., Funama, Y., MacMahon, H., & Yamashita, Y. (2009) · American Journal of Roentgenology · doi:10.2214/ajr.09.2431 · Cited by 63
Bogoni, L., Ko, J. P., Alpert, J. B., Anand, V. C., Fantauzzi, J. P., Florin, C., Koo, C. W., Mason, D., Rom, W. N., Shiau, M. C., et al. (2012) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-012-9496-0 · Cited by 58
Artificial Intelligence in Diagnostic Imaging
Sharma, P., Suehling, M., Flohr, T. G., & Comaniciu, D. (2020) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000499 · Cited by 57
Kozuka, T., Matsukubo, Y., Kadoba, T., Oda, T., Suzuki, A., Hyodo, T., Im, S., Kaida, H., Yagyu, Y., Tsurusaki, M., et al. (2020) · Japanese Journal of Radiology · doi:10.1007/s11604-020-01009-0 · Cited by 54
Trebeschi, S., Bodalal, Z., Boellaard, T. N., Tareco Bucho, T. M., Drago, S. G., Kurilova, I., Calin-Vainak, A. M., Delli Pizzi, A., Muller, M., Hummelink, K., et al. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.609054 · Cited by 54
Li, F., Hara, T., Shiraishi, J., et al. (2011) · American Journal of Roentgenology · doi:10.2214/ajr.10.4816 · Cited by 48
Fukushima, A., Ashizawa, K., Yamaguchi, T., et al. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.183.2.1830297 · Cited by 45
Abé, H., Ashizawa, K., Li, F., Matsuyama, N., Fukushima, A., Shiraishi, J., MacMahon, H., & Doi, K. (2003) · Academic Radiology · doi:10.1016/s1076-6332(03)00572-5 · Cited by 44
Kim, J. H., Kim, J. Y., Kim, G. H., Kang, D., Kim, I. J., Seo, J., Andrews, J. R., & Park, C. M. (2020) · Journal of Clinical Medicine · doi:10.3390/jcm9061981 · Cited by 42
Kligerman, S., Cai, L., & White, C. S. (2013) · Journal of Thoracic Imaging · doi:10.1097/rti.0b013e31826c29ec · Cited by 41
Chae, K. J., Jin, G. Y., Ko, S. B., Wang, Y., Zhang, H., Choi, E. J., & Choi, H. (2020) · Academic Radiology · doi:10.1016/j.acra.2019.05.018 · Cited by 40 · cited 15×
“Some studies suggest that physicians with less radiological task experience benefit more from AI-assistance [30,32,35,36,48,50], while others showed that physicians with greater radiological task experience benefitted the most from AI-assistance [44,49].”
BackgroundInfluentialGroup sequential design for comparative diagnostic accuracy studies
Mazumdar, M., & Liu, A. (2003) · Statistics in Medicine · doi:10.1002/sim.1386 · Cited by 38
Lee, K. H., Goo, J. M., Park, C. M., Lee, H. J., & Jin, K. N. (2012) · Korean Journal of Radiology · doi:10.3348/kjr.2012.13.5.564 · Cited by 37
Bai, H. X., Wang, R., Xiong, Z., Hsieh, B., Chang, K., Halsey, K., Tran, T. M. L., Choi, J. W., Wang, D., Shi, L., et al. (2021) · Radiology · doi:10.1148/radiol.2021219004 · Cited by 36
Singh, R., Kalra, M. K., Homayounieh, F., Nitiwarangkul, C., McDermott, S., Little, B. P., Lennes, I. T., Shepard, J.-A. O., & Digumarthy, S. R. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-20-630 · Cited by 35
Chen, H., Wang, X., Ma, D., et al. (2007) · Chinese Medical Journal · doi:10.1097/00029330-200707020-00001 · Cited by 32
Yang, Y., Lure, F. Y. M., Miao, H., Zhang, Z., Jaeger, S., Liu, J., & Guo, L. (2020) · Journal of X-Ray Science and Technology · doi:10.3233/xst-200735 · Cited by 31
Kasai, S., Li, F., Shiraishi, J., & Doi, K. (2008) · American Journal of Roentgenology · doi:10.2214/ajr.07.3091 · Cited by 30
Laino, M. E., Ammirabile, A., Posa, A., Cancian, P., Shalaby, S. Y., Savevski, V., & Neri, E. (2021) · Diagnostics · doi:10.3390/diagnostics11081317 · Cited by 27
Martini, K., Blüthgen, C., Eberhard, M., Schönenberger, A., Martini, I. D., Huber, F. A., Barth, B. K., Euler, A., & Frauenfelder, T. (2020) · Academic Radiology · doi:10.1016/j.acra.2020.01.014 · Cited by 26
COVID-19 pneumonia accurately detected on chest radiographs with artificial intelligence
Dorr, F., Chaves, H., Serra, M., Ramirez, A., Costa, M. E., Seia, J. O., Cejas, C. P., Castro, M., Eyheremendy, E., Slezak, D. F., et al. (2020) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2020.100014 · Cited by 25
Rao, R. B., Bi, J., Fung, G., et al. (2007) · ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/1281192.1281306 · Cited by 18
Koo, Y. H., Shin, K. E., Park, J. S., Lee, J. W., Byun, S., & Lee, H. (2020) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13105 · Cited by 17
Liu, J., Zhao, L., Han, X., Ji, H., Liu, L., & Wen, H. (2020) · Asia-Pacific Journal of Clinical Oncology · doi:10.1111/ajco.13362 · Cited by 10
Objectively Measuring and Comparing Performance Levels of Diagnostic Imaging Systems and Practices
Gur, D. (2007) · Academic Radiology · doi:10.1016/j.acra.2007.04.007 · Cited by 9
Liu, P., Tsai, Y., Chen, P., Tsai, H., Hsu, L.-W., Wang, C.-S., Lee, N., Huang, M., Wu, Y.-C., Ko, W., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/19878 · Cited by 7
To err is human; analysis finds radiologists very human
Brice, J. (2010) · Cited by 1
Cited by
Non-task expert physicians benefit from correct explainable AI advice when reviewing X-rays
Gaube, S., Suresh, H., Raue, M., Lermer, E., Koch, T. K., Hudecek, M. F. C., Ackery, A. D., Grover, S. C., Coughlin, J. F., Frey, D., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-28633-w · Cited by 92
Metadata
- Authors
- Li, D., Pehrson, L. M., Lauridsen, C. A., Tøttrup, L., Fraccaro, M., Elliott, D., Zając, H. D., Darkner, S., Carlsen, J. F., & Nielsen, M. B.
- Affiliations
- University of Copenhagen (DK)
- Copenhagen University Hospital (DK)
- Unumed Aps, 1055 Copenhagen, Denmark
- Corresponding author
- Li, Dana
- ORCID
- Venue
- Diagnostics
- Publisher
- MDPI AG
- Published
- 2021-11-26
- Volume
- 11
- Issue
- 12
- Pages
- 2206
- ISSN
- 2075-4418 (ISSN-L)
- PMC
- PMC8700414
- OpenAlex
- W3215796869
- Cited by
- 35
- References
- 60
- Citations by year
- 2026: 2 · 2025: 9 · 2024: 8 · 2023: 12 · 2022: 4
- FWCI
- 0.49
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Diagnostics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Innovation Fund Denmark (0176-00013B)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Keywords
- artificial intelligence, chest radiography, computed tomography, radiologist performance, diagnostic accuracy, AI-assisted diagnosis, sensitivity and specificity, lung imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.