Deep Learning Algorithm Trained with COVID-19 Pneumonia Also Identifies Immune Checkpoint Inhibitor Therapy-Related Pneumonitis
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
Abstract
Background: Coronavirus disease 2019 (COVID-19) pneumonia and immune checkpoint inhibitor (ICI) therapy-related pneumonitis share common features. The aim of this study was to determine on chest computed tomography (CT) images whether a deep convolutional neural network algorithm is able to solve the challenge of differential diagnosis between COVID-19 pneumonia and ICI therapy-related pneumonitis. Methods: We enrolled three groups: a pneumonia-free group (n = 30), a COVID-19 group (n = 34), and a group of patients with ICI therapy-related pneumonitis (n = 21). Computed tomography images were analyzed with an artificial intelligence (AI) algorithm based on a deep convolutional neural network structure. Statistical analysis included the Mann–Whitney U test (significance threshold at p < 0.05) and the receiver operating characteristic curve (ROC curve). Results: The algorithm showed low specificity in distinguishing COVID-19 from ICI therapy-related pneumonitis (sensitivity 97.1%, specificity 14.3%, area under the curve (AUC) = 0.62). ICI therapy-related pneumonitis was identified by the AI when compared to pneumonia-free controls (sensitivity = 85.7%, specificity 100%, AUC = 0.97). Conclusions: The deep learning algorithm is not able to distinguish between COVID-19 pneumonia and ICI therapy-related pneumonitis. Awareness must be increased among clinicians about imaging similarities between COVID-19 and ICI therapy-related pneumonitis. ICI therapy-related pneumonitis can be applied as a challenge population for cross-validation to test the robustness of AI models used to analyze interstitial pneumonias of variable etiology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“CNN has become the primary tool for automatic COVID-19 diagnosis in many hospitals around the world using chest X-ray images [256–260].”
From Review of deep learning: concepts, CNN architectures, challenges, applications, future directions (2021) · cites this paper as [260]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Coronaviridae Study Group of the International Committee on Taxonomy of Viruses, Gorbalenya, A. E., Baker, S. C., Baric, R. S., de Groot, R. J., Drosten, C., Gulyaeva, A. A., Haagmans, B. L., Lauber, C., Leontovich, A. M., et al. (2020) · Nature Microbiology · doi:10.1038/s41564-020-0695-z · Cited by 8,403
Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning
Kermany, D. S., Goldbaum, M., Cai, W., Valentim, C. C. S., Liang, H., Baxter, S. L., McKeown, A., Yang, G., Wu, X., Yan, F., et al. (2018) · Cell · doi:10.1016/j.cell.2018.02.010 · Cited by 4,842
Coronaviruses: An Overview of Their Replication and Pathogenesis
Fehr, A. R., & Perlman, S. (2015) · Methods in Molecular Biology · doi:10.1007/978-1-4939-2438-7_1 · Cited by 3,787 · cited 1×
“Coronaviruses are composed of four major structural proteins, including the envelope (E) protein, the membrane (M) protein, the nucleocapsid (N) protein, and the spike (S) protein, the latter being of paramount importance since it mediates viral attachment to the host cell membrane receptor [4].”
BackgroundCT Imaging Features of 2019 Novel Coronavirus (2019-nCoV)
Chung, M., Bernheim, A., Mei, X., Zhang, N., Huang, M., Zeng, X., Cui, J., Xu, W., Yang, Y., Fayad, Z. A., et al. (2020) · Radiology · doi:10.1148/radiol.2020200230 · Cited by 2,751
Covid-19 — Navigating the Uncharted
Fauci, A. S., Lane, H. C., & Redfield, R. R. (2020) · New England Journal of Medicine · doi:10.1056/nejme2002387 · Cited by 1,969 · cited 1×
“SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) is a novel coronavirus that was first identified in Wuhan, China, and that is responsible for a highly contagious respiratory disease named coronavirus disease 2019 (COVID-19) [1,2].”
BackgroundLi, L., Qin, L., Xu, Z., Yin, Y., Wang, X., Kong, B., Bai, J., Lu, Y., Fang, Z., Song, Q., et al. (2020) · Radiology · doi:10.1148/radiol.2020200905 · Cited by 1,927 · cited 3×
“Artificial intelligence (AI), using deep learning technology, is highly promising in the medical imaging field due to its capability of feature extraction and analysis [9,21,22].”
BackgroundResultGorbalenya, A. E., Baker, S. C., Baric, R. S., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.02.07.937862 · Cited by 1,408 · cited 1×
“SARS-CoV-2 (severe acute respiratory syndrome coronavirus 2) is a novel coronavirus that was first identified in Wuhan, China, and that is responsible for a highly contagious respiratory disease named coronavirus disease 2019 (COVID-19) [1,2].”
BackgroundLung pattern classification for interstitial lung diseases using a deep convolutional neural network
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., & Mougiakakou, S. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2535865 · Cited by 1,343 · cited 1×
“It has been also applied to detect various imaging features of chest CT [23,24], allowing qualitative and quantitative analysis, which could provide an estimation of the disease burden, facilitating and expediting imaging interpretation [25,26].”
MethodThe Role of Chest Imaging in Patient Management During the COVID-19 Pandemic
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J.-J., Martin, I. B. K., et al. (2020) · Chest · doi:10.1016/j.chest.2020.04.003 · Cited by 1,245 · cited 1×
“Computed tomography (CT) of the chest can detect lung manifestations, which are often associated with COVID-19 [6–8].”
BackgroundPneumonitis in Patients Treated With Anti–Programmed Death-1/Programmed Death Ligand 1 Therapy
Naidoo, J., Wang, X., Woo, K. M., Iyriboz, T., Halpenny, D., Cunningham, J., Chaft, J. E., Segal, N. H., Callahan, M. K., Lesokhin, A. M., et al. (2017) · Journal of Clinical Oncology · doi:10.1200/jco.2016.68.2005 · Cited by 1,215 · cited 1×
“ICI therapy-related pneumonitis [13,14] is an uncommon but important immune-related adverse event, with potential significant morbidity and mortality.”
BackgroundRubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J.-J., Martin, I. B. K., et al. (2020) · Radiology · doi:10.1148/radiol.2020201365 · Cited by 1,101
Temporal Changes of CT Findings in 90 Patients with COVID-19 Pneumonia: A Longitudinal Study
Wang, Y., Dong, C., Hu, Y., Li, C., Ren, Q., Zhang, X., Shi, H., & Zhou, M. (2020) · Radiology · doi:10.1148/radiol.2020200843 · Cited by 949 · cited 1×
“The patients with RT-PCR confirmed diagnosis of COVID-19 pneumonia, showing the typical CT findings reported in several studies [6,7,28–31].”
ResultSimpson, S., Kay, F. U., Abbara, S., Bhalla, S., Chung, J. H., Chung, M., Henry, T. S., Kanne, J. P., Kligerman, S., Ko, J. P., et al. (2020) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000524 · Cited by 773 · cited 1×
“The patients with RT-PCR confirmed diagnosis of COVID-19 pneumonia, showing the typical CT findings reported in several studies [6,7,28–31].”
ResultNishino, M., Giobbie-Hurder, A., Hatabu, H., Ramaiya, N. H., & Hodi, F. S. (2016) · JAMA Oncology · doi:10.1001/jamaoncol.2016.2453 · Cited by 758 · cited 1×
“ICI therapy-related pneumonitis [13,14] is an uncommon but important immune-related adverse event, with potential significant morbidity and mortality.”
BackgroundSimpson, S., Kay, F. U., Abbara, S., Bhalla, S., Chung, J. H., Chung, M., Henry, T. S., Kanne, J. P., Kligerman, S., Ko, J. P., et al. (2020) · Radiology: Cardiothoracic Imaging · doi:10.1148/ryct.2020200152 · Cited by 752
How Do Cytotoxic Lymphocytes Kill Cancer Cells?
Martínez-Lostao, L., Anel, A., & Pardo, J. (2015) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-15-0685 · Cited by 717 · cited 1×
“ICIs act through immune system-mediated destruction of tumor cells [11,12].”
BackgroundChest CT Severity Score: An Imaging Tool for Assessing Severe COVID-19
Yang, R., Li, X., Liu, H., Zhen, Y., Zhang, X., Xiong, Q., Luo, Y., Gao, C., & Zeng, W. (2020) · Radiology: Cardiothoracic Imaging · doi:10.1148/ryct.2020200047 · Cited by 685
Gabay, C., Emery, P., van Vollenhoven, R., Dikranian, A., Alten, R., Pavelka, K., Klearman, M., Musselman, D., Agarwal, S., Green, J., et al. (2013) · The Lancet · doi:10.1016/s0140-6736(13)60250-0 · Cited by 672
Chest CT Features of COVID-19 in Rome, Italy
Caruso, D., Zerunian, M., Polici, M., Pucciarelli, F., Polidori, T., Rucci, C., Guido, G., Bracci, B., De Dominicis, C., & Laghi, A. (2020) · Radiology · doi:10.1148/radiol.2020201237 · Cited by 614
Autopsy in suspected COVID-19 cases
Hanley, B., Lucas, S. B., Youd, E., Swift, B., & Osborn, M. (2020) · Journal of Clinical Pathology · doi:10.1136/jclinpath-2020-206522 · Cited by 478 · cited 1×
“In real clinical settings, lung consolidations can be the results of more complex clinical patterns, especially in the later stages of disease [32].”
BackgroundSerial Quantitative Chest CT Assessment of COVID-19: A Deep Learning Approach
Huang, L., Han, R., Ai, T., Yu, P., Kang, H., Tao, Q., & Xia, L. (2020) · Radiology: Cardiothoracic Imaging · doi:10.1148/ryct.2020200075 · Cited by 431 · cited 1×
“It has been also applied to detect various imaging features of chest CT [23,24], allowing qualitative and quantitative analysis, which could provide an estimation of the disease burden, facilitating and expediting imaging interpretation [25,26].”
Method(2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24571-3 · Cited by 421
Tocilizumab for the management of immune mediated adverse events secondary to PD-1 blockade
Stroud, C. R., Hegde, A., Cherry, C., Naqash, A. R., Sharma, N., Addepalli, S., Cherukuri, S., Parent, T., Hardin, J., & Walker, P. (2019) · Journal of Oncology Pharmacy Practice · doi:10.1177/1078155217745144 · Cited by 331 · cited 1×
“Indeed, for both COVID-19 pneumonia [18] and ICI therapy pneumonitis [19], the use of tocilizumab, a recombinant humanized monoclonal antibody inhibiting the human IL-6 receptor, originally developed for the treatment of rheumatoid arthritis, is under scrutiny [20].”
BackgroundMechanisms of action and rationale for the use of checkpoint inhibitors in cancer
Granier, C., De Guillebon, E., Blanc, C., Roussel, H., Badoual, C., Colin, E., Saldmann, A., Gey, A., Oudard, S., & Tartour, E. (2017) · ESMO Open · doi:10.1136/esmoopen-2017-000213 · Cited by 324 · cited 1×
“ICIs act through immune system-mediated destruction of tumor cells [11,12].”
BackgroundControversies About COVID-19 and Anticancer Treatment with Immune Checkpoint Inhibitors
Bersanelli, M. (2020) · Immunotherapy · doi:10.2217/imt-2020-0067 · Cited by 213 · cited 1×
“Interestingly, COVID-19 pneumonia and ICI therapy-related pneumonitis have been suggested to share critical biological mechanisms, including the hyperactivation of immune cells associated with a significant increase in proinflammatory cytokines [17].”
BackgroundKaehler, K. C., Piel, S., Livingstone, E., Schilling, B., Hauschild, A., & Schadendorf, D. (2010) · Seminars in Oncology · doi:10.1053/j.seminoncol.2010.09.003 · Cited by 170 · cited 1×
“Importantly, given the expanding population of cancer patients exposed to ICIs, the number of therapy-related pneumonitis is expected to escalate in the near future [15,16].”
BackgroundImmune Checkpoint Inhibitor Therapy–related Pneumonitis: Patterns and Management
Kalisz, K. R., Ramaiya, N. H., Laukamp, K. R., & Gupta, A. (2019) · RadioGraphics · doi:10.1148/rg.2019190036 · Cited by 159 · cited 2×
“ICI therapy-related pneumonitis might present with several patterns that include organizing pneumonia, nonspecific interstitial pneumonia, and acute interstitial pneumonia–acute respiratory distress syndrome [10].”
BackgroundOff‐label use of tocilizumab in patients with SARS‐CoV‐2 infection
Di Giambenedetto, S., Ciccullo, A., Borghetti, A., Gambassi, G., Landi, F., Visconti, E., Zileri Dal Verme, L., Bernabei, R., Tamburrini, E., Cauda, R., et al. (2020) · Journal of Medical Virology · doi:10.1002/jmv.25897 · Cited by 124 · cited 1×
“Indeed, for both COVID-19 pneumonia [18] and ICI therapy pneumonitis [19], the use of tocilizumab, a recombinant humanized monoclonal antibody inhibiting the human IL-6 receptor, originally developed for the treatment of rheumatoid arthritis, is under scrutiny [20].”
BackgroundDepeursinge, A., Chin, A. S.-L., Leung, A. N. C., Terrone, D. G., Bristow, M., Rosen, G., & Rubin, D. L. (2014) · Investigative Radiology · doi:10.1097/rli.0000000000000127 · Cited by 92 · cited 1×
“It has been also applied to detect various imaging features of chest CT [23,24], allowing qualitative and quantitative analysis, which could provide an estimation of the disease burden, facilitating and expediting imaging interpretation [25,26].”
MethodWang, M., Xia, C., Huang, L., Xu, S., Qin, C., Liu, J., Cao, Y., Yu, P., Zhu, T., Zhu, H., et al. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30199-0 · Cited by 86
Choi, H., Qi, X., Yoon, S. H., Park, S. J., Lee, K. H., Kim, J. Y., Lee, Y. K., Ko, H., Kim, K. H., Park, C. M., et al. (2020) · Radiology: Cardiothoracic Imaging · doi:10.1148/ryct.2020204001 · Cited by 75
Immune‑related adverse events during anticancer immunotherapy: Pathogenesis and management (Review)
Stucci, S., Palmirotta, R., Passarelli, A., Silvestris, E., Argentiero, A., Lanotte, L., Acquafredda, S., Todisco, A., & Silvestris, F. (2017) · Oncology Letters · doi:10.3892/ol.2017.6919 · Cited by 66 · cited 1×
“Importantly, given the expanding population of cancer patients exposed to ICIs, the number of therapy-related pneumonitis is expected to escalate in the near future [15,16].”
BackgroundImmunotherapy Associated Pulmonary Toxicity: Biology Behind Clinical and Radiological Features
Porcu, M., Silva, P. D., Solinas, C., Battaglia, A., Schena, M., Scartozzi, M., Bron, D., Suri, J. S., Willard-Gallo, K. E., Sangiolo, D., et al. (2019) · Cancers · doi:10.3390/cancers11030305 · Cited by 65 · cited 1×
“Since ICI therapy-related pneumonitis has been reported to frequently affect the lower lobes [33], the result of the left lower lobe reported in the present study might be influenced by the small sample size.”
ResultRussano, M., Citarella, F., Napolitano, A., Dell’Aquila, E., Cortellini, A., Pantano, F., Vincenzi, B., Tonini, G., & Santini, D. (2020) · Expert Opinion on Biological Therapy · doi:10.1080/14712598.2020.1789097 · Cited by 24 · cited 1×
“This is particularly important, as COVID-19 pneumonia cases are expected to decrease in the future and might become epidemiologically less relevant with the development and distribution of adequate vaccination therapies, whereas the number of ICI therapy-related pneumonitis is predicted to increase according to the in…”
BackgroundSelf-citationQuattrocchi, C. C., Mallio, C. A., Presti, G., Zobel, B. B., Cardinale, J., Iozzino, M., & Sala, S. W. D. (2020) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-20-782 · Cited by 12 · cited 1×
“It has been also applied to detect various imaging features of chest CT [23,24], allowing qualitative and quantitative analysis, which could provide an estimation of the disease burden, facilitating and expediting imaging interpretation [25,26].”
MethodSelf-citationMallio, C. A., Quattrocchi, C. C., Zobel, B. B., & Parizel, P. M. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-20-1306 · Cited by 12 · cited 1×
“Lesion distribution and other radiomics-based data, including clinical information, are promising to further improve the performance of AI algorithms [35].”
BackgroundSelf-citation
Cited by
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906
Novel Transfer Learning Approach for Medical Imaging with Limited Labeled Data
Alzubaidi, L., Al-Amidie, M., Al-Asadi, A., Humaidi, A. J., Al-Shamma, O., Fadhel, M. A., Zhang, J., Santamaría, J., & Duan, Y. (2021) · Cancers · doi:10.3390/cancers13071590 · Cited by 292
Radiation-Induced Lung Injury—Current Perspectives and Management
Rahi, M. S., Parekh, J., Pednekar, P., Parmar, G., Abraham, S., Nasir, S., Subramaniyam, R., Jeyashanmugaraja, G. P., & Gunasekaran, K. (2021) · Clinics and Practice · doi:10.3390/clinpract11030056 · Cited by 104
“, a similar deep learning algorithm was unable to distinguish COVID-19 pneumonia and ICI treatment-related pneumonitis [83].”
ResultArtificial intelligence and radiomics: fundamentals, applications, and challenges in immunotherapy
Dercle, L., McGale, J., Sun, S. H., Marabelle, A., Yeh, R., Deutsch, É., Mokrane, F., Farwell, M. D., Ammari, S., Schöder, H. M., et al. (2022) · Journal for ImmunoTherapy of Cancer · doi:10.1136/jitc-2022-005292 · Cited by 109
Mallio, C. A., Sertorio, A. C., Bernetti, C., & Zobel, B. B. (2023) · La radiologia medica · doi:10.1007/s11547-023-01651-4 · Cited by 86
“The organization of information and the standardization of terminology, together with artificial intelligence tools, could allow automated data mining, assist longitudinal data tracking and adjuvate the implementation of imaging biomarkers [4, 5].”
MethodSelf-citation
Metadata
- Authors
- Mallio, C. A., Napolitano, A., Castiello, G., Giordano, F. M., D'Alessio, P., Iozzino, M., Sun, Y., Angeletti, S., Russano, M., Santini, D., Tonini, G., Zobel, B. B., Vincenzi, B., & Quattrocchi, C. C.
- Affiliations
- Università Campus Bio-Medico (IT)
- Departmental Faculty of Medicine and Surgery, Unit of Medical Oncology, 00128 Rome, Italy
- Departmental Faculty of Medicine and Surgery, Unit of Diagnostic Imaging and Interventional Radiology, Università Campus Bio-Medico di Roma, 00128 Rome, Italy
- Ospedale Santa Maria Goretti (IT)
- Infervision Europe GmbH, Mainzer Strasse 75, D-65189 Wiesbaden, Germany
- Corresponding author
- Mallio, Carlo Augusto
- Napolitano, Andrea
- ORCID
- Venue
- Cancers
- Publisher
- MDPI AG
- Published
- 2021-02-06
- Volume
- 13
- Issue
- 4
- Pages
- 652
- ISSN
- 2072-6694 (ISSN-L)
- PMC
- PMC7914551
- OpenAlex
- W3126423192
- Cited by
- 30
- References
- 36
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 3 · 2023: 8 · 2022: 5 · 2021: 7
- FWCI
- 3.04
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Cancers (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- IRIS Research product catalog (Sapienza University of Rome) (submittedVersion, cc-by)
- Iris (University of Trento) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- COVID-19 pneumonia, immune-related pneumonitis, differential diagnosis, chest CT, deep convolutional neural networks, artificial intelligence, interstitial lung disease, model robustness, cross-validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.