Retinopathy Online Challenge: Automatic Detection of Microaneurysms in Digital Color Fundus Photographs
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
The detection of microaneurysms in digital color fundus photographs is a critical first step in automated screening for diabetic retinopathy (DR), a common complication of diabetes. To accomplish this detection numerous methods have been published in the past but none of these was compared with each other on the same data. In this work we present the results of the first international microaneurysm detection competition, organized in the context of the Retinopathy Online Challenge (ROC), a multiyear online competition for various aspects of DR detection. For this competition, we compare the results of five different methods, produced by five different teams of researchers on the same set of data. The evaluation was performed in a uniform manner using an algorithm presented in this work. The set of data used for the competition consisted of 50 training images with available reference standard and 50 test images where the reference standard was withheld by the organizers (M. Niemeijer, B. van Ginneken, and M. D. Abràmoff). The results obtained on the test data was submitted through a website after which standardized evaluation software was used to determine the performance of each of the methods. A human expert detected microaneurysms in the test set to allow comparison with the performance of the automatic methods. The overall results show that microaneurysm detection is a challenging task for both the automatic methods as well as the human expert. There is room for improvement as the best performing system does not reach the performance of the human expert. The data associated with the ROC microaneurysm detection competition will remain publicly available and the website will continue accepting submissions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms
Scharstein, D., & Szeliski, R. (2002) · International Journal of Computer Vision · doi:10.1023/a:1014573219977 · Cited by 6,740
Ridge-based vessel segmentation in color images of the retina
Staal, J., Abramoff, M. D., Niemeijer, M., Viergever, M. A., & van Ginneken, B. (2004) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2004.825627 · Cited by 4,202
Self-citationLocating blood vessels in retinal images by piecewise threshold probing of a matched filter response
Hoover, A. D., Kouznetsova, V., & Goldbaum, M. (2000) · IEEE Transactions on Medical Imaging · doi:10.1109/42.845178 · Cited by 2,582
Morphological grayscale reconstruction in image analysis: applications and efficient algorithms
Vincent, L. (1993) · IEEE Transactions on Image Processing · doi:10.1109/83.217222 · Cited by 1,923
ROC Methodology in Radiologic Imaging
METZ, C. E. (1986) · Investigative Radiology · doi:10.1097/00004424-198609000-00009 · Cited by 1,642
Comparison and evaluation of methods for liver segmentation from CT datasets
Heimann, T., van Ginneken, B., Styner, M. A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., et al. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2013851 · Cited by 1,135
Comparative study of retinal vessel segmentation methods on a new publicly available database
Niemeijer, M., Staal, J., van Ginneken, B., Loog, M., & Abramoff, M. D. (2004) · SPIE Proceedings · doi:10.1117/12.535349 · Cited by 790
Self-citationLocating the optic nerve in a retinal image using the fuzzy convergence of the blood vessels
Hoover, A., & Goldbaum, M. (2003) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2003.815900 · Cited by 787
Fluorescein Angiography Complication Survey
Yannuzzi, L. A., Rohrer, K. T., Tindel, L. J., Sobel, R. S., Costanza, M. A., Shields, W., & Zang, E. (1986) · Ophthalmology · doi:10.1016/s0161-6420(86)33697-2 · Cited by 606
Automated detection of diabetic retinopathy on digital fundus images
Sinthanayothin, C., Boyce, J. F., Williamson, T. H., Cook, H. L., Mensah, E., Lal, S., & Usher, D. (2002) · Diabetic Medicine · doi:10.1046/j.1464-5491.2002.00613.x · Cited by 509
Cited by
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Retinal Imaging and Image Analysis
Abramoff, M. D., Garvin, M. K., & Sonka, M. (2010) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2010.2084567 · Cited by 1,439
Self-citationFEEDBACK ON A PUBLICLY DISTRIBUTED IMAGE DATABASE: THE MESSIDOR DATABASE
Decencière, E., Zhang, X., Cazuguel, G., Lay, B., Cochener, B., Trone, C., Gain, P., Ordonez, R., Massin, P., Erginay, A., et al. (2014) · Image Analysis & Stereology · doi:10.5566/ias.1155 · Cited by 1,424
Self-citationBlood vessel segmentation methodologies in retinal images – A survey
Fraz, M. M., Remagnino, P., Hoppe, A., Uyyanonvara, B., Rudnicka, A. R., Owen, C. G., & Barman, S. A. (2012) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2012.03.009 · Cited by 1,001
Evaluation of prostate segmentation algorithms for mri: The promise12 challenge
Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.12.002 · Cited by 857
“…been organized to solve similar issues in the fields of liver segmentation on CT (Heimann et al., 2009), coronary image analysis (Schaap et al., 2009), brain segmentation on MR (Shattuck et al., 2009), retinal image analysis (Niemeijer et al., 2010) and lung registration on CT (Murphy et al., 2011).”
BackgroundOrlando, J. I., Fu, H., Barbosa Breda, J., van Keer, K., Bathula, D. R., Diaz-Pinto, A., Fang, R., Heng, P.-A., Kim, J., Lee, J., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101570 · Cited by 855
Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening
Li, T., Gao, Y., Wang, K., Guo, S., Liu, H., & Kang, H. (2019) · Information Sciences · doi:10.1016/j.ins.2019.06.011 · Cited by 663
Blood vessel segmentation algorithms — Review of methods, datasets and evaluation metrics
Moccia, S., De Momi, E., El Hadji, S., & Mattos, L. S. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2018.02.001 · Cited by 567
Automated Analysis of Retinal Images for Detection of Referable Diabetic Retinopathy
Abràmoff, M. D., Folk, J. C., Han, D. P., Walker, J. D., Williams, D. F., Russell, S. R., Massin, P., Cochener, B., Gain, P., Tang, L., et al. (2013) · JAMA Ophthalmology · doi:10.1001/jamaophthalmol.2013.1743 · Cited by 463
Self-citationComputer-aided diagnosis of diabetic retinopathy: A review
Mookiah, M. R. K., Acharya, U. R., Chua, C. K., Lim, C. M., Ng, E. Y. K., & Laude, A. (2013) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2013.10.007 · Cited by 430
Applications of deep learning in fundus images: A review
Li, T., Bo, W., Hu, C., Kang, H., Liu, H., Wang, K., & Fu, H. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101971 · Cited by 396
Why rankings of biomedical image analysis competitions should be interpreted with care
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07619-7 · Cited by 380
Self-citationIDRiD: Diabetic Retinopathy – Segmentation and Grading Challenge
Porwal, P., Pachade, S., Kokare, M., Deshmukh, G., Son, J., Bae, W., Liu, L., Wang, J., Liu, X., Gao, L., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101561 · Cited by 372
Self-citationAn Ensemble-Based System for Microaneurysm Detection and Diabetic Retinopathy Grading
Antal, B., & Hajdu, A. (2012) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2012.2193126 · Cited by 335
Deep learning for diabetic retinopathy detection and classification based on fundus images: A review
Tsiknakis, N., Theodoropoulos, D., Manikis, G., Ktistakis, E., Boutsora, O., Berto, A., Scarpa, F., Scarpa, A., Fotiadis, D. I., & Marias, K. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104599 · Cited by 328
Exudate-based diabetic macular edema detection in fundus images using publicly available datasets
Giancardo, L., Meriaudeau, F., Karnowski, T. P., Li, Y., Garg, S., Tobin, K. W., & Chaum, E. (2012) · Medical Image Analysis · doi:10.1016/j.media.2011.07.004 · Cited by 322
An ensemble deep learning based approach for red lesion detection in fundus images
Orlando, J. I., Prokofyeva, E., del Fresno, M., & Blaschko, M. B. (2018) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2017.10.017 · Cited by 293
Detection and classification of retinal lesions for grading of diabetic retinopathy
Usman Akram, M., Khalid, S., Tariq, A., Khan, S. A., & Azam, F. (2014) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2013.11.014 · Cited by 291
Identification and classification of microaneurysms for early detection of diabetic retinopathy
Akram, M. U., Khalid, S., & Khan, S. A. (2013) · Pattern Recognition · doi:10.1016/j.patcog.2012.07.002 · Cited by 289
Automated Early Detection of Diabetic Retinopathy
Abràmoff, M. D., Reinhardt, J. M., Russell, S. R., Folk, J. C., Mahajan, V. B., Niemeijer, M., & Quellec, G. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.03.046 · Cited by 287
Self-citationRed Lesion Detection Using Dynamic Shape Features for Diabetic Retinopathy Screening
Seoud, L., Hurtut, T., Chelbi, J., Cheriet, F., & Langlois, J. M. P. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2509785 · Cited by 277
RETOUCH: The Retinal OCT Fluid Detection and Segmentation Benchmark and Challenge
Bogunovic, H., Venhuizen, F., Klimscha, S., Apostolopoulos, S., Bab-Hadiashar, A., Bagci, U., Beg, M. F., Bekalo, L., Chen, Q., Ciller, C., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2901398 · Cited by 273
Self-citationDeep Learning Techniques for Diabetic Retinopathy Classification: A Survey
Atwany, M. Z., Sahyoun, A. H., & Yaqub, M. (2022) · IEEE Access · doi:10.1109/access.2022.3157632 · Cited by 266
Gómez-Valverde, J. J., Antón, A., Fatti, G., Liefers, B., Herranz, A., Santos, A., Sánchez, C. I., & Ledesma-Carbayo, M. J. (2019) · Biomedical Optics Express · doi:10.1364/boe.10.000892 · Cited by 265
A systematic collection of medical image datasets for deep learning
Li, J., Zhu, G., Hua, C., Feng, M., BasheerBennamoun, Li, P., Lu, X., Song, J., Shen, P., Xu, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.12864 · Cited by 61
Metadata
- Authors
- Niemeijer, M., van Ginneken, B., Cree, M. J., Mizutani, A., Quellec, G., Sanchez, C. I., Zhang, B., Hornero, R., Lamard, M., Muramatsu, C., Wu, X., Cazuguel, G., You, J., Mayo, A., Li, Q., Hatanaka, Y., Cochener, B., Roux, C., Karray, F., Garcia, M., Fujita, H., & Abramoff, M. D.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2010-01
- Volume
- 29
- Issue
- 1
- Pages
- 185-195
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2107871626
- Cited by
- 516
- References
- 25
- FWCI
- 16.91
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.