The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge
Open accessAbstract
HighlightsA novel objective evaluation framework for nodule detection algorithms using the largest publicly available LIDC‐IDRI data set.The impact of combining individual systems on the detection performance was investigated.The combination of classical candidate detectors and a combination of deep learning architectures generates excellent results, better than any individual system.Our observer study has shown that CAD detects nodules that were missed by expert readers.We released this set of additional nodules for further development of CAD systems. Graphical abstract Figure. No caption available. ABSTRACT Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC‐IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC‐IDRI data. We released this set of additional nodules for further development of CAD systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
(2010) · Cited by 14,134
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic Screening
The National Lung Screening Trial Research Team (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1102873 · Cited by 11,064
(1968) · American Association of Industrial Nurses Journal · doi:10.1177/216507996801601208 · Cited by 8,798
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · cited 1×
“The data set was collected from the largest publicly available reference database for lung nodules: the LIDC-IDRI (Armato et al. (2011); Clark et al. (2013); Armato III et al. (2015)).”
MethodA Discriminative Feature Learning Approach for Deep Face Recognition
Wen, Y., Zhang, K., Li, Z., & Qiao, Y. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_31 · Cited by 4,077
(2013) · Cited by 3,740
An Introduction to the Bootstrap.
Buckland, S. T., Efron, B., & Tibshirani, R. J. (1994) · Biometrics · doi:10.2307/2532810 · Cited by 3,103
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887
(2015) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-24553-9 · Cited by 2,285
An Introduction to the Bootstrap
Grimshaw, S. D., Efron, B., & Tibshirani, R. J. (1995) · Technometrics · doi:10.2307/1269918 · Cited by 2,095
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,387
Setio, A. A. A., Ciompi, F., Litjens, G., Gerke, P., Jacobs, C., van Riel, S. J., Wille, M. M. W., Naqibullah, M., Sanchez, C. I., & van Ginneken, B. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2536809 · Cited by 1,258
Self-citationTheano: new features and speed improvements
Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I., Bergeron, A., Bouchard, N., Warde-Farley, D., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5590 · Cited by 1,005 · cited 6×
“The system was implemented using the Lasagne and Theano libraries (Diele-man et al. (2015); Bastien et al. (2012)).”
MethodInfluentialNaidich, D. P., Bankier, A. A., MacMahon, H., Schaefer-Prokop, C. M., Pistolesi, M., Goo, J. M., Macchiarini, P., Crapo, J. D., Herold, C. J., Austin, J. H., et al. (2013) · Radiology · doi:10.1148/radiol.12120628 · Cited by 1,000
Self-citationCT Screening for Lung Cancer: Frequency and Significance of Part-Solid and Nonsolid Nodules
Henschke, C. I., Yankelevitz, D. F., Mirtcheva, R., McGuinness, G., McCauley, D., & Miettinen, O. S. (2002) · American Journal of Roentgenology · doi:10.2214/ajr.178.5.1781053 · Cited by 909
Deep Feature Learning for Knee Cartilage Segmentation Using a Triplanar Convolutional Neural Network
Prasoon, A., Petersen, K., Igel, C., Lauze, F., Dam, E., & Nielsen, M. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-40763-5_31 · Cited by 582
Multilevel Contextual 3-D CNNs for False Positive Reduction in Pulmonary Nodule Detection
Dou, Q., Chen, H., Yu, L., Qin, J., & Heng, P.-A. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2613502 · Cited by 582
Self-citationA new computationally efficient CAD system for pulmonary nodule detection in CT imagery
Messay, T., Hardie, R. C., & Rogers, S. K. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.02.004 · Cited by 439
(2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9 · Cited by 394
Li, Q., Sone, S., & Doi, K. (2003) · Medical Physics · doi:10.1118/1.1581411 · Cited by 349
Murphy, K., van Ginneken, B., Schilham, A. M. R., de Hoop, B. J., Gietema, H. A., & Prokop, M. (2009) · Medical Image Analysis · doi:10.1016/j.media.2009.07.001 · Cited by 331
Self-citationvan Ginneken, B., Armato, S. G., de Hoop, B., van Amelsvoort-van de Vorst, S., Duindam, T., Niemeijer, M., Murphy, K., Schilham, A., Retico, A., Fantacci, M. E., et al. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.05.005 · Cited by 298
Self-citationKuhnigk, J.-M., Dicken, V., Bornemann, L., Bakai, A., Wormanns, D., Krass, S., & Peitgen, H.-O. (2006) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2006.871547 · Cited by 288
Dieleman, S., Schlüter, J., Raffel, C., Olson, E. M., Sønderby, S. K., Nouri, D., Maturana, D., Thoma, M., Battenberg, E., Kelly, J., et al. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.27878 · Cited by 279
van Ginneken, B., Setio, A. A. A., Jacobs, C., & Ciompi, F. (2015) · 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2015.7163869 · Cited by 278
Self-citationAutomatic detection of subsolid pulmonary nodules in thoracic computed tomography images
Jacobs, C., van Rikxoort, E. M., Twellmann, T., Scholten, E. T., de Jong, P. A., Kuhnigk, J.-M., Oudkerk, M., de Koning, H. J., Prokop, M., Schaefer-Prokop, C., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.12.001 · Cited by 262
Self-citationvan Rikxoort, E. M., de Hoop, B., Viergever, M. A., Prokop, M., & van Ginneken, B. (2009) · Medical Physics · doi:10.1118/1.3147146 · Cited by 228
Self-citationTan, M., Deklerck, R., Jansen, B., Bister, M., & Cornelis, J. (2011) · Medical Physics · doi:10.1118/1.3633941 · Cited by 221
Self-citation(2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-48219-9 · Cited by 197
Kazerooni, E. A., Austin, J. H. M., Black, W. C., Dyer, D. S., Hazelton, T. R., Leung, A. N., McNitt-Gray, M. F., Munden, R. F., & Pipavath, S. (2014) · Journal of Thoracic Imaging · doi:10.1097/rti.0000000000000097 · Cited by 175
Firmino, M., de Morais, A. H. F., Mendoça, R. M., Ribeiro-Dantas, M. D. C., Hékis, H. R., & Valentim, R. A. D. M. (2014) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-13-41 · Cited by 161
Automatic detection of lung nodules in CT datasets based on stable 3D mass–spring models
Cascio, D., Magro, R., Fauci, F., Iacomi, M., & Raso, G. (2012) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2012.09.002 · Cited by 153
Armato, S. G., III, McLennan, G., Bidaut, L., McNitt-Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2015) · The Cancer Imaging Archive · doi:10.7937/k9/tcia.2015.lo9ql9sx · Cited by 150
Automatic detection of large pulmonary solid nodules in thoracic CT images
Setio, A. A. A., Jacobs, C., Gelderblom, J., & van Ginneken, B. (2015) · Medical Physics · doi:10.1118/1.4929562 · Cited by 136
Self-citationOn Combining Computer-Aided Detection Systems
Niemeijer, M., Loog, M., Abràmoff, M. D., Viergever, M. A., Prokop, M., & van Ginneken, B. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2072789 · Cited by 131
Self-citationJacobs, C., van Rikxoort, E. M., Murphy, K., Prokop, M., Schaefer‐Prokop, C. M., & van Ginneken, B. (2015) · European Radiology · doi:10.1007/s00330-015-4030-7 · Cited by 130
Self-citationLarge scale validation of the M5L lung CAD on heterogeneous CT datasets
Torres, E. L., Fiorina, E., Pennazio, F., Peroni, C., Saletta, M., Camarlinghi, N., Fantacci, M. E., & Cerello, P. (2015) · Medical Physics · doi:10.1118/1.4907970 · Cited by 128
Self-citationFast lung nodule detection in chest CT images using cylindrical nodule-enhancement filter
Teramoto, A., & Fujita, H. (2012) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-012-0767-5 · Cited by 117
How Swarms Build Cognitive Maps
Chialvo, D. R., & Millonas, M. M. (1995) · Springer eBooks · doi:10.1007/978-3-642-79629-6_20 · Cited by 103
Lung nodule detection in low-dose and thin-slice computed tomography
Retico, A., Delogu, P., Fantacci, M. E., Gori, I., & Martinez, A. P. (2008) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2008.02.001 · Cited by 94
Self-citationChoi, W., & Choi, T. (2013) · Entropy · doi:10.3390/e15020507 · Cited by 94
The Ladder: A Reliable Leaderboard for Machine Learning Competitions
Blum, A., & Hardt, M. (2015) · arXiv · doi:10.48550/arxiv.1502.04585 · Cited by 92
A novel multithreshold method for nodule detection in lung CT
Golosio, B., Masala, G. L., Piccioli, A., Oliva, P., Carpinelli, M., Cataldo, R., Cerello, P., Carlo, F. D., Falaschi, F., Fantacci, M. E., et al. (2009) · Medical Physics · doi:10.1118/1.3160107 · Cited by 86
Self-citationAssessment of Radiologist Performance in the Detection of Lung Nodules
Armato, S. G., Roberts, R. Y., Kocherginsky, M., Aberle, D. R., Kazerooni, E. A., MacMahon, H., van Beek, E. J. R., Yankelevitz, D. F., McLennan, G., McNitt‐Gray, M. F., et al. (2008) · Academic Radiology · doi:10.1016/j.acra.2008.05.022 · Cited by 82
The Lung Reporting and Data System (LU-RADS): A Proposal for Computed Tomography Screening
Manos, D., Seely, J. M., Taylor, J. L., Borgaonkar, J. N., Roberts, H. C., & Mayo, J. R. (2014) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2014.03.004 · Cited by 71
Combination of computer-aided detection algorithms for automatic lung nodule identification
Camarlinghi, N., Gori, I., Retico, A., Bellotti, R., Bosco, P., Cerello, P., Gargano, G., Torres, E. L., Megna, R., Peccarisi, M., et al. (2011) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-011-0637-6 · Cited by 66
Self-citationRiccardi, A., Petkov, T., Ferri, G., Masotti, M., & Campanini, R. (2011) · Medical Physics · doi:10.1118/1.3560427 · Cited by 64
Toward clinically usable CAD for lung cancer screening with computed tomography
Brown, M. S., Lo, P., Goldin, J. G., Barnoy, E. A., Kim, G. H. J., McNitt‐Gray, M. F., & Aberle, D. R. (2014) · European Radiology · doi:10.1007/s00330-014-3329-0 · Cited by 63
Pulmonary nodule detection using a cascaded SVM classifier
Bergtholdt, M., Wiemker, R., & Klinder, T. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2216747 · Cited by 35
Tan, M., Deklerck, R., Cornelis, J. P., & Jansen, B. (2013) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2013.07.002 · Cited by 28
Self-citationHigh performance lung nodule detection schemes in CT using local and global information
Guo, W., & Li, Q. (2012) · Medical Physics · doi:10.1118/1.4737109 · Cited by 23
Research works printed but not linked: 6
- Cerello, 2010, 3-D object segmentation using ant colonies, Pattern Recognit., 43, 1476, 10.1016/j.patcog.2009.10.007 · doi:10.1016/j.patcog.2009.10.007
- Dou, 2016, Multi-level contextual 3D CNNs for false positive reduction in pulmonary nodule detection, IEEE Trans. Biomed. Eng.
- Efron, 1994, vol. 57
- International Commission on Radiation Units and Measurements, 2008, Receiver operating characteristic analysis in medical imaging, J. ICRU, 8, 1, 10.1093/jicru_ndn003 · doi:10.1093/jicru_ndn003
- Retico, 2009, Pleural nodule identification in low-dose and thin-slice lung computed tomography, Comput. Biol. Med., 39, 1137, 10.1016/j.compbiomed.2009.10.005 · doi:10.1016/j.compbiomed.2009.10.005
- Tieleman, 2012, Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude, COURSERA, 4
Cited by
MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation
Ibtehaz, N., & Rahman, M. S. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.08.025 · Cited by 2,311
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M., Thai, T. C., Moore, K., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,080
Yan, K., Wang, X., Lu, L., & Summers, R. M. (2018) · Journal of medical imaging · doi:10.1117/1.jmi.5.3.036501 · Cited by 573
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Self-citation
Metadata
- Authors
- Setio, A. A. A., Traverso, A., de Bel, T., Berens, M. S. N., Bogaard, C. V. D., Cerello, P., Chen, H., Dou, Q., Fantacci, M. E., Geurts, B., Gugten, R. V. D., Heng, P. A., Jansen, B., de Kaste, M. M. J., Kotov, V., Lin, J. Y.-H., Manders, J. T. M. C., Sóñora-Mengana, A., García-Naranjo, J. C., Papavasileiou, E., Prokop, M., Saletta, M., Schaefer-Prokop, C. M., Scholten, E. T., Scholten, L., Snoeren, M. M., Torres, E. L., Vandemeulebroucke, J., Walasek, N., Zuidhof, G. C. A., Ginneken, B. V., & Jacobs, C.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2017-12
- Volume
- 42
- Pages
- 1-13
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2584017349
- Cited by
- 1,374
- References
- 63
- FWCI
- 52.97
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.