A critic evaluation of methods for COVID-19 automatic detection from X-ray images
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
In this paper, we compare and evaluate different testing protocols used for automatic COVID-19 diagnosis from X-Ray images in the recent literature. We show that similar results can be obtained using X-Ray images that do not contain most of the lungs. We are able to remove the lungs from the images by turning to black the center of the X-Ray scan and training our classifiers only on the outer part of the images. Hence, we deduce that several testing protocols for the recognition are not fair and that the neural networks are learning patterns in the dataset that are not correlated to the presence of COVID-19. Finally, we show that creating a fair testing protocol is a challenging task, and we provide a method to measure how fair a specific testing protocol is. In the future research we suggest to check the fairness of a testing protocol using our tools and we encourage researchers to look for better techniques than the ones that we propose.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
Contrast Limited Adaptive Histogram Equalization
Zuiderveld, K. (1994) · Graphics Gems · doi:10.1016/b978-0-12-336156-1.50061-6 · Cited by 4,192 · cited 1×
“In the third one we used a 240 sized square, but we also preprocess the image by applying contrast limited adaptive histogram equalization [29] and by cropping the image by cutting its upper and lower part so that the height of the new image is between the 90% and the 100% of its width.”
COVID-19 and Italy: what next?
Remuzzi, A., & Remuzzi, G. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30627-9 · Cited by 3,119
Apostolopoulos, I. D., & Mpesiana, T. A. (2020) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-020-00865-4 · Cited by 2,338 · cited 1×
“[11] combined Cohen repository and many other sources to create a larger dataset containing 224 images of COVID-19, 700 of pneumonia and 504 negative.”
Narin, A., Kaya, C., & Pamuk, Z. (2021) · Pattern Analysis and Applications · doi:10.1007/s10044-021-00984-y · Cited by 2,054 · cited 1×
“[10] created a small dataset with 50 COVID-19 cases coming from Cohen repository and 50 heathy cases coming from Kaggle (https://www.”
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“The problem of generalizing to new datasets or of learning statistical features independent on the content of the images is not new in the deep learning community [15–17].”
Comparison of Deep Learning Approaches for Multi-Label Chest X-Ray Classification
Baltruschat, I. M., Nickisch, H., Grass, M., Knopp, T., & Saalbach, A. (2019) · Scientific Reports · doi:10.1038/s41598-019-42294-8 · Cited by 455 · cited 1×
“We are not the first to report potential biases in chest x-ray image classification [17,18].”
CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.07031 · Cited by 392 · cited 4×
“Irvin et al. collected and labelled Chexpert [28], a large dataset containing 224,316 chest radiographs of 65,240 patients divided into 14 classes.”
BackgroundInfluential
Cited by
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044
COVID-19 Image Data Collection: Prospective Predictions are the Future
Cohen, J. P., Morrison, P., Dao, L., Roth, K., Duong, T., & Ghassem, M. (2020) · Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2020-48g7 · Cited by 927
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 895
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
COVID-19 identification in chest X-ray images on flat and hierarchical classification scenarios
Pereira, R. M., Bertolini, D., Teixeira, L. O., Silla, C. N., & Costa, Y. M. G. (2020) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2020.105532 · Cited by 509
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 506
Vats, S., Sharma, V., Singh, K., Pratap Singh, D., Yazid Bajuri, M., Taniar, D., Innab, N., Mouldi, A., & Ahmadian, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124464 · Cited by 391
COVIDGR Dataset and COVID-SDNet Methodology for Predicting COVID-19 Based on Chest X-Ray Images
Tabik, S., Gomez-Rios, A., Martin-Rodriguez, J. L., Sevillano-Garcia, I., Rey-Area, M., Charte, D., Guirado, E., Suarez, J. L., Luengo, J., Valero-Gonzalez, M. A., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3037127 · Cited by 375
A Review on Deep Learning Techniques for the Diagnosis of Novel Coronavirus (COVID-19)
Islam, M. M., Karray, F., Alhajj, R., & Zeng, J. (2021) · IEEE Access · doi:10.1109/access.2021.3058537 · Cited by 292
Garcia Santa Cruz, B., Bossa, M. N., Sölter, J., & Husch, A. D. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102225 · Cited by 89
Deep Ensemble Model for Classification of Novel Coronavirus in Chest X‐Ray Images
Ahmad, F., Farooq, A., & Ghani, M. U. (2021) · Computational Intelligence and Neuroscience · doi:10.1155/2021/8890226 · Cited by 65
A systematic collection of medical image datasets for deep learning
Li, J., Zhu, G., Hua, C., Feng, M., BasheerBennamoun, Li, P., Lu, X., Song, J., Shen, P., Xu, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.12864 · Cited by 61
Cao, L., & Liu, Q. (2022) · doi:10.1101/2022.08.22.22279022 · Cited by 49
Díaz-Rodríguez, N., Lamas, A., Sanchez, J., Franchi, G., Donadello, I., Tabik, S., Filliat, D., Cruz, P., Montes, R., & Herrera, F. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.09.022 · Cited by 2
Metadata
- Authors
- Maguolo, G., & Nanni, L.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2021-12
- Volume
- 76
- Pages
- 1-7
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W3017744567
- Cited by
- 250
- References
- 14
- FWCI
- 25.63
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.