The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge
Open accessAbstract
HighlightsA novel objective evaluation framework for nodule detection algorithms using the largest publicly available LIDC‐IDRI data set.The impact of combining individual systems on the detection performance was investigated.The combination of classical candidate detectors and a combination of deep learning architectures generates excellent results, better than any individual system.Our observer study has shown that CAD detects nodules that were missed by expert readers.We released this set of additional nodules for further development of CAD systems. Graphical abstract Figure. No caption available. ABSTRACT Automatic detection of pulmonary nodules in thoracic computed tomography (CT) scans has been an active area of research for the last two decades. However, there have only been few studies that provide a comparative performance evaluation of different systems on a common database. We have therefore set up the LUNA16 challenge, an objective evaluation framework for automatic nodule detection algorithms using the largest publicly available reference database of chest CT scans, the LIDC‐IDRI data set. In LUNA16, participants develop their algorithm and upload their predictions on 888 CT scans in one of the two tracks: 1) the complete nodule detection track where a complete CAD system should be developed, or 2) the false positive reduction track where a provided set of nodule candidates should be classified. This paper describes the setup of LUNA16 and presents the results of the challenge so far. Moreover, the impact of combining individual systems on the detection performance was also investigated. It was observed that the leading solutions employed convolutional networks and used the provided set of nodule candidates. The combination of these solutions achieved an excellent sensitivity of over 95% at fewer than 1.0 false positives per scan. This highlights the potential of combining algorithms to improve the detection performance. Our observer study with four expert readers has shown that the best system detects nodules that were missed by expert readers who originally annotated the LIDC‐IDRI data. We released this set of additional nodules for further development of CAD systems.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
(2010) · Cited by 14,134
Proceedings of IEEE International Conference on Computer Vision
(1995) · Proceedings of IEEE International Conference on Computer Vision · doi:10.1109/iccv.1995.466933 · Cited by 12,860
Reduced Lung-Cancer Mortality with Low-Dose Computed Tomographic Screening
The National Lung Screening Trial Research Team (2011) · New England Journal of Medicine · doi:10.1056/nejmoa1102873 · Cited by 11,064
(1968) · American Association of Industrial Nurses Journal · doi:10.1177/216507996801601208 · Cited by 8,798
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408
Research works printed but not linked: 6
- Cerello, 2010, 3-D object segmentation using ant colonies, Pattern Recognit., 43, 1476, 10.1016/j.patcog.2009.10.007 · DOI
- Dou, 2016, Multi-level contextual 3D CNNs for false positive reduction in pulmonary nodule detection, IEEE Trans. Biomed. Eng.
- Efron, 1994, vol. 57
- International Commission on Radiation Units and Measurements, 2008, Receiver operating characteristic analysis in medical imaging, J. ICRU, 8, 1, 10.1093/jicru_ndn003 · doi:10.1093/jicru_ndn003
- Retico, 2009, Pleural nodule identification in low-dose and thin-slice lung computed tomography, Comput. Biol. Med., 39, 1137, 10.1016/j.compbiomed.2009.10.005 · doi:10.1016/j.compbiomed.2009.10.005
- Tieleman, 2012, Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude, COURSERA, 4
Cited by
MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation
Ibtehaz, N., & Rahman, M. S. (2020) · Neural Networks · doi:10.1016/j.neunet.2019.08.025 · Cited by 2,311
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
Recent advances and clinical applications of deep learning in medical image analysis
Chen, X., Wang, X., Zhang, K., Fung, K.-M., Thai, T. C., Moore, K., Mannel, R. S., Liu, H., Zheng, B., & Qiu, Y. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102444 · Cited by 1,080
Convolutional Neural Networks for Radiologic Images: A Radiologist’s Guide
Soffer, S., Ben-Cohen, A., Shimon, O., Amitai, M. M., Greenspan, H., & Klang, E. (2019) · Radiology · doi:10.1148/radiol.2018180547 · Cited by 577
Yan, K., Wang, X., Lu, L., & Summers, R. M. (2018) · Journal of medical imaging · doi:10.1117/1.jmi.5.3.036501 · Cited by 573
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Self-citationfastMRI: An Open Dataset and Benchmarks for Accelerated MRI
Žbontar, J., Knöll, F., Sriram, A., Murrell, T., Huang, Z., Muckley, M. J., Defazio, A., Stern, R., Johnson, P., Bruno, M. T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08839 · Cited by 528
Segment anything model for medical images?
Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al. (2024) · Medical Image Analysis · doi:10.1016/j.media.2023.103061 · Cited by 536
“Medical images cover a wide range of object types, such as brain organs and tumors [29], [35], [42], [47], [66], lungs and hearts [23], [61], [64], [70], abdomen [22], [24], [57], [63], [66], [80], spine [65], [76], [77], cells [32], and polyps [38], [59], etc.”
Evaluate the Malignancy of Pulmonary Nodules Using the 3-D Deep Leaky Noisy-OR Network
Liao, F., Liang, M., Li, Z., Hu, X., & Song, S. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2892409 · Cited by 491
Adams, S. J., Stone, E., Baldwin, D. R., Vliegenthart, R., Lee, P., & Fintelmann, F. J. (2023) · The Lancet · doi:10.1016/s0140-6736(22)01694-4 · Cited by 467
Automated pulmonary nodule detection in CT images using deep convolutional neural networks
Xie, H., Yang, D., Sun, N., Chen, Z., & Zhang, Y. (2019) · Pattern Recognition · doi:10.1016/j.patcog.2018.07.031 · Cited by 459
Medical Image Data Augmentation: Techniques, Comparisons and Interpretations
Goceri, E. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10453-z · Cited by 459
“Lung images have been provided from Lung Image Database Consortium and Image Database Resource Initiative (LIDC-IDRI) database (Armato et al. 2011; McNitt-Gray et al. 2007; Wang et al. 2015) by the LUNA16 challenge (Setio et al. 2017).”
MethodDeepLung: Deep 3D Dual Path Nets for Automated Pulmonary Nodule Detection and Classification
Zhu, W., Liu, C., Fan, W., & Xie, X. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00079 · Cited by 449
Holzinger, A., Carrington, A., & Müller, H. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00636-z · Cited by 406
Hybrid deep learning for detecting lung diseases from X-ray images
Bharati, S., Podder, P., & Mondal, M. R. H. (2020) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2020.100391 · Cited by 404
Ding, J., Li, A., Hu, Z., & Wang, L. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_64 · Cited by 385
Why rankings of biomedical image analysis competitions should be interpreted with care
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunovic, H., Bradley, A. P., Carass, A., et al. (2018) · Nature Communications · doi:10.1038/s41467-018-07619-7 · Cited by 380
Nasrullah, N., Sang, J., Alam, M. S., Mateen, M., Cai, B., & Hu, H. (2019) · Sensors · doi:10.3390/s19173722 · Cited by 359
Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101840 · Cited by 328
What the radiologist should know about artificial intelligence – an ESR white paper
European Society of Radiology (ESR) (2019) · Insights into Imaging · doi:10.1186/s13244-019-0738-2 · Cited by 320
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
A survey on incorporating domain knowledge into deep learning for medical image analysis
Xie, X., Niu, J., Liu, X., Chen, Z., Tang, S., & Yu, S. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.101985 · Cited by 311
Going Deep in Medical Image Analysis: Concepts, Methods, Challenges, and Future Directions
Altaf, F., Islam, S. M. S., Akhtar, N., & Janjua, N. K. (2019) · IEEE Access · doi:10.1109/access.2019.2929365 · Cited by 286
Hussein, S., Kandel, P., Bolan, C. W., Wallace, M. B., & Bagci, U. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2894349 · Cited by 285
Vision Transformers in medical computer vision—A contemplative retrospection
Parvaiz, A., Khalid, M. A., Zafar, R., Ameer, H., Ali, M., & Fraz, M. M. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106126 · Cited by 276
Ozdemir, O., Russell, R. L., & Berlin, A. A. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2947595 · Cited by 265
Self-supervised learning methods and applications in medical imaging analysis: a survey
Shurrab, S., & Duwairi, R. (2022) · PeerJ Computer Science · doi:10.7717/peerj-cs.1045 · Cited by 254
van Velzen, S. G. M., Lessmann, N., Velthuis, B. K., Bank, I. E. M., van den Bongard, D. H. J. G., Leiner, T., de Jong, P. A., Veldhuis, W. B., Correa, A., Terry, J. G., et al. (2020) · Radiology · doi:10.1148/radiol.2020191621 · Cited by 244
“Despite many efforts supporting the development of robust deep learning (DL) software (16,17), an established limitation of DL algorithms is that performance can rapidly degrade and fail with what appears to humans as only slight variations in the input data (18).”
BackgroundFoundation Model for Cancer Imaging Biomarkers
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00807-9 · Cited by 199
UniverSeg: Universal Medical Image Segmentation
Butoi, V. I., Gonzalez Ortiz, J. J., Ma, T., Sabuncu, M. R., Guttag, J., & Dalca, A. V. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01960 · Cited by 195
Zhang, C., Sun, X., Dang, K., Li, K., Guo, X.-W., Chang, J., Yu, Z.-Q., Huang, F.-Y., Wu, Y.-S., Liang, Z., et al. (2019) · The Oncologist · doi:10.1634/theoncologist.2018-0908 · Cited by 168
FocalMix: Semi-Supervised Learning for 3D Medical Image Detection
Wang, D., Zhang, Y., Zhang, K., & Wang, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00401 · Cited by 157
Yan, K., Wang, X., Lu, L., Zhang, L., Harrison, A. P., Bagheri, M., & Summers, R. M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00965 · Cited by 150
Application of Artificial Intelligence in Lung Cancer
Chiu, H.-Y., Chao, H.-S., & Chen, Y.-M. (2022) · Cancers · doi:10.3390/cancers14061370 · Cited by 138
“LUNA16 [23] 2016 LDCT 888 Adapted from LIDC, with additional nodules found during model training.”
BackgroundMethodInfluentialM3T: three-dimensional Medical image classifier using Multi-plane and Multi-slice Transformer
Jang, J., & Hwang, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02006 · Cited by 133
A deep 3D residual CNN for false‐positive reduction in pulmonary nodule detection
Jin, H., Li, Z., Tong, R., & Lin, L. (2018) · Medical Physics · doi:10.1002/mp.12846 · Cited by 118
Dynamic Memory to Alleviate Catastrophic Forgetting in Continual Learning with Medical Imaging
Perkonigg, M., Hofmanninger, J., Herold, C. J., Brink, J. A., Pianykh, O., Prosch, H., & Langs, G. (2021) · Nature Communications · doi:10.1038/s41467-021-25858-z · Cited by 106
Liu, M., Wu, J., Wang, N., Zhang, X., Bai, Y., Guo, J., Zhang, L., Liu, S., & Tao, K. (2023) · PLOS ONE · doi:10.1371/journal.pone.0273445 · Cited by 84
An effective WSSENet-based similarity retrieval method of large lung CT image databases
(2022) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2022.07.013 · Cited by 79
Machine learning based medical image deepfake detection: A comparative study
Solaiyappan, S., & Wen, Y. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100298 · Cited by 73
A systematic collection of medical image datasets for deep learning
Li, J., Zhu, G., Hua, C., Feng, M., BasheerBennamoun, Li, P., Lu, X., Song, J., Shen, P., Xu, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2106.12864 · Cited by 61
Xiang, Y., Huang, C., He, Y., & Zhang, Q. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.644150 · Cited by 46
Kim, Y., Park, J. Y., Hwang, E. J., Lee, S. M., & Park, C. M. (2021) · Journal of Thoracic Disease · doi:10.21037/jtd-21-1342 · Cited by 31
Pre-training in Medical Data: A Survey
Qiu, Y., Lin, F., Chen, W., & Xu, M. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1382-8 · Cited by 30
Ozdemir, O., Woodward, B., & Berlin, A. A. (2017) · arXiv · doi:10.48550/arxiv.1712.00497 · Cited by 30
Improving Breast Cancer Detection Using Symmetry Information with Deep Learning
Brhane Hagos, Y., Gubern Mérida, A., & Teuwen, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00946-5_10 · Cited by 27
Artificial intelligence for low-dose CT lung cancer screening: Comparison of utilization scenarios
Lee, M., Hwang, E. J., Lee, J. H., Nam, J. G., Lim, W. H., Park, H., Park, C. M., Choi, H., Park, J., & Goo, J. M. (2025) · American Journal of Roentgenology · doi:10.2214/ajr.25.32829 · Cited by 24
Lessons learned from RadiologyNET foundation models for transfer learning in medical radiology
Napravnik, M., Hržić, F., Urschler, M., Miletić, D., & Štajduhar, I. (2025) · Scientific Reports · doi:10.1038/s41598-025-05009-w · Cited by 15
“When observing each challenge separately, the LUNA dataset showed interesting results.”
InfluentialHolzinger, A., Carrington, A., & Müller, H. (2019) · arXiv · doi:10.48550/arxiv.1912.09024 · Cited by 11
BIAS: Transparent reporting of biomedical image analysis challenges
Maier-Hein, L., Reinke, A., Kozubek, M., Martel, A. L., Arbel, T., Eisenmann, M., Hanbury, A., Jannin, P., Müller, H., Onogur, S., et al. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101796 · Cited by 9
“They are often published in prestigious journals [11,12,13,14,15,16], are associated with significant amounts of prize money (up to €1 million on platforms like Kaggle [17]) and receive a huge amount of attention, indicated by the number of downloads, citations and views.”
Metadata
- Authors
- Setio, A. A. A., Traverso, A., de Bel, T., Berens, M. S. N., Bogaard, C. V. D., Cerello, P., Chen, H., Dou, Q., Fantacci, M. E., Geurts, B., Gugten, R. V. D., Heng, P. A., Jansen, B., de Kaste, M. M. J., Kotov, V., Lin, J. Y.-H., Manders, J. T. M. C., Sóñora-Mengana, A., García-Naranjo, J. C., Papavasileiou, E., Prokop, M., Saletta, M., Schaefer-Prokop, C. M., Scholten, E. T., Scholten, L., Snoeren, M. M., Torres, E. L., Vandemeulebroucke, J., Walasek, N., Zuidhof, G. C. A., Ginneken, B. V., & Jacobs, C.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2017-12
- Volume
- 42
- Pages
- 1-13
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2584017349
- Cited by
- 1,374
- References
- 63
- FWCI
- 52.97
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.