Multi-Scale Deep Reinforcement Learning for Real-Time 3D-Landmark Detection in CT Scans
Abstract
Robust and fast detection of anatomical structures is a prerequisite for both diagnostic and interventional medical image analysis. Current solutions for anatomy detection are typically based on machine learning techniques that exploit large annotated image databases in order to learn the appearance of the captured anatomy. These solutions are subject to several limitations, including the use of suboptimal feature engineering techniques and most importantly the use of computationally suboptimal search-schemes for anatomy detection. To address these issues, we propose a method that follows a new paradigm by reformulating the detection problem as a behavior learning task for an artificial agent. We couple the modeling of the anatomy appearance and the object search in a unified behavioral framework, using the capabilities of deep reinforcement learning and multi-scale image analysis. In other words, an artificial agent is trained not only to distinguish the target anatomical object from the rest of the body but also how to find the object by learning and following an optimal navigation path to the target object in the imaged volumetric space. We evaluated our approach on 1487 3D-CT volumes from 532 patients, totaling over 500,000 image slices and show that it significantly outperforms state-of-the-art solutions on detecting several anatomical structures with no failed cases from a clinical acceptance perspective, while also achieving a 20-30 percent higher detection accuracy. Most importantly, we improve the detection-speed of the reference methods by 2-3 orders of magnitude, achieving unmatched real-time performance on large 3D-CT scans.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Artificial intelligence: a modern approach
Kumar, D. A., Balasubramanian, S., Byeon, D. H., & Manerkar, P. G. V. (1995) · Choice Reviews Online · doi:10.5860/choice.33-1577 · Cited by 22,182 · see citing works
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300 · see citing works
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001 · see citing works
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · see citing works
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083 · see citing works
3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation
Çiçek, Ö., Abdulkadir, A., Lienkamp, S. S., Brox, T., & Ronneberger, O. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_49 · Cited by 7,945 · see citing works
Introduction to Reinforcement Learning
Sutton, R. S., & Barto, A. G. (1998) · The MIT Press eBooks · Cited by 6,855 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
R-FCN: Object Detection via Region-based Fully Convolutional NetworksPreprint
Dai, J., Li, Y., He, K., & Sun, J. Z. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.06409 · Cited by 5,135 · see citing works
Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966 · see citing works
Region-based convolutional networks for accurate object detection and segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2437384 · Cited by 2,912 · see citing works
Static Analysis of Shape in TensorFlow ProgramsPreprint
The Theano Development Team, Al‐Rfou, R., Alain, G., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,629 · see citing works
Target-driven visual navigation in indoor scenes using deep reinforcement learning
Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., & Farhadi, A. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989381 · Cited by 1,533 · see citing works
(2012) · neural information processing systems · Cited by 1,291 · see citing works
Scalable Object Detection Using Deep Neural Networks
Erhan, D., Szegedy, C., Toshev, A., & Anguelov, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.276 · Cited by 1,161 · cited 2× · see citing works
“dom regression forests [11], [12], [13], [14], [30], [31], random-ferns [32], deep convolutional neural networks [33] or modern spatial-transformer neural networks [34] to learn the mapping.”
Reinforcement learning for robots using neural networks
Lin, L.-J. (1992) · Defense Technical Information Center (DTIC) · Cited by 836 · see citing works
Learning a Deep Compact Image Representation for Visual Tracking
Wang, N., & Yeung, D. (2013) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 836 · see citing works
Shape and arrangement of columns in cat's striate cortex
Hubel, D. H., & Wiesel, T. N. (1963) · The Journal of Physiology · doi:10.1113/jphysiol.1963.sp007079 · Cited by 834 · see citing works
Learning to Track: Online Multi-object Tracking by Decision Making
Yu, X., Alahi, A., & Savarese, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.534 · Cited by 684 · see citing works
Scale-space for discrete signals
Tony Lindeberg, C. (1990) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.49051 · Cited by 621 · see citing works
Zheng, Y., Barbu, A., Georgescu, B., Scheuering, M., & Comaniciu, D. (2008) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2008.2004421 · Cited by 546 · see citing works
Self-citationActive Object Localization with Deep Reinforcement Learning
Caicedo, J. C., & Lazebnik, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.286 · Cited by 454 · see citing works
Tu, Z. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.194 · Cited by 397 · see citing works
Išgum, I., Staring, M., Rutten, A., Prokop, M., Viergever, M. A., & van Ginneken, B. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2008.2011480 · Cited by 361 · see citing works
Regression Forests for Efficient Anatomy Detection and Localization in CT Studies
Criminisi, A., Shotton, J., Robertson, D., & Ender Konukoglu, A. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-18421-5_11 · Cited by 275 · see citing works
Robust and Accurate Shape Model Matching Using Random Forest Regression-Voting
Lindner, C., Bromiley, P. A., Ionita, M. C., et al. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2382106 · Cited by 269 · see citing works
Regressing Heatmaps for Multiple Landmark Localization Using CNNs
Payer, C., Štern, D., Bischof, H., & Urschler, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_27 · Cited by 248 · see citing works
How precise can bony landmarks be determined on a CT scan of the knee?
Victor, J. M. K., Doninck, D. V., Labey, L., et al. (2009) · The Knee · doi:10.1016/j.knee.2009.01.001 · Cited by 228 · see citing works
Reinforcement Learning for Visual Object Detection
Mathe, S., Pirinen, A., & Sminchisescu, C. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.316 · Cited by 168 · see citing works
An Artificial Agent for Anatomical Landmark Detection in Medical Images
Ghesu, F. C., Georgescu, B., Mansi, T., Neumann, D., Joachim Hornegger, A., & Comaniciu, D. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46726-9_27 · Cited by 160 · see citing works
Self-citationAutomatic Detection and Segmentation of Kidneys in 3D CT Images Using Random Forests
Cuingnet, R., Prevost, R., Lesage, D., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33454-2_9 · Cited by 156 · see citing works
Chien, P., Et, P., Eraso, F. E., et al. (2009) · Dentomaxillofacial Radiology · doi:10.1259/dmfr/81889955 · Cited by 156 · see citing works
3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data
Zheng, Y., Liu, D., Georgescu, B., Nguyen, H. V., & Comaniciu, D. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_69 · Cited by 151 · see citing works
Self-citationMarginal space deep learning: efficient architecture for volumetric image parsing
Ghesu, F. C., Krubasik, E., Georgescu, B., Singh, V., Zheng, Y., Hornegger, J., & Comaniciu, D. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2538802 · Cited by 142 · see citing works
Cebral, J. R., & Löhner, R. (2005) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2005.844172 · Cited by 134 · see citing works
Chu, C., Belavý, D. L., Armbrecht, G., Bansmann, M. P., Felsenberg, D., & Guoyan Zheng, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0143327 · Cited by 130 · see citing works
Deep Reinforcement Learning for Visual Object Tracking in VideosPreprint
Zhang, D., Maei, H. R., Wang, X., & Wang, Y.-F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.08936 · Cited by 110 · see citing works
Tree-Structured Reinforcement Learning for Sequential Object LocalizationPreprint
Jie, Z., Liang, X., Feng, J., Jin, X., Lu, W. F., & Yan, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.02710 · Cited by 108 · see citing works
Fast Multiple Organ Detection and Localization in Whole-Body MR Dixon Sequences
Pauly, O., Glocker, B., Criminisi, A., et al. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23626-6_30 · Cited by 102 · see citing works
Trends in Computed Tomography Utilization Rates
Hess, E. P., Haas, L. R., Shah, N. D., Stroebel, R. J., Denham, C. R., & Swensen, S. J. (2013) · Journal of Patient Safety · doi:10.1097/pts.0b013e3182948b1a · Cited by 102 · see citing works
2D image classification for 3D anatomy localization: employing deep convolutional neural networks
de Vos, B. D., Wolterink, J. M., de Jong, P. A., Viergever, M. A., & Išgum, I. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2216971 · Cited by 98 · see citing works
Hierarchical Object Detection with Deep Reinforcement LearningPreprint
Bellver, M., Giró-i-Nieto, X., Marqués, F., & Torres, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.03718 · Cited by 98 · see citing works
Donner, R., Menze, B. H., Bischof, H., & Langs, G. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.02.004 · Cited by 87 · see citing works
Pouch, A. M., Wang, H., Takabe, M., Jackson, B. M., Gorman, J. H., Gorman, R. C., Yushkevich, P. A., & Sehgal, C. M. (2013) · Medical Image Analysis · doi:10.1016/j.media.2013.10.001 · Cited by 83 · see citing works
Active Scheduling of Organ Detection and Segmentation in Whole-Body Medical Images
Zhan, Y., Zhou, X. S., Peng, Z., & Krishnan, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85988-8_38 · Cited by 68 · see citing works
Multi-organ localization with cascaded global-to-local regression and shape prior
Gauriau, R., Cuingnet, R., Lesage, D., & Isabelle Bloch, A. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.04.007 · Cited by 67 · see citing works
Yang, L., Georgescu, B., Zheng, Y., Wang, Y., Meer, P., & Comaniciu, D. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2158440 · Cited by 59 · see citing works
Self-citationHierarchical Object Detection with Deep Reinforcement Learning
Miriam, B. B., Giró-i-Nieto, X., Ferran, M., et al. (2017) · Advances in parallel computing · doi:10.3233/978-1-61499-822-8-164 · Cited by 57 · see citing works
Search strategies for multiple landmark detection by submodular maximization
Liu, D., Zhou, K., Bernhardt, D., et al. (2010) · doi:10.1109/cvpr.2010.5540016 · Cited by 52 · see citing works
From Local to Global Random Regression Forests: Exploring Anatomical Landmark Localization
Štern, D., Ebner, T., & Urschler, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_26 · Cited by 47 · see citing works
Automatic Labeling of Anatomical Structures in MR FastView Images Using a Statistical Atlas
Fenchel, M., Thesen, S., & Schilling, A. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85988-8_69 · Cited by 34 · see citing works
Personalized Graphical Models for Anatomical Landmark Localization in Whole-Body Medical Images
Potěšil, V., Kadir, T., Platsch, G., et al. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-014-0731-7 · Cited by 22 · see citing works
Robust Registration of Longitudinal Spine CT
Glocker, B., Zikic, D., & Haynor, D. R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10404-1_32 · Cited by 22 · see citing works
Anatomy Detection and Localization in 3D Medical Images
Criminisi, A., Robertson, D., Pauly, O., et al. (2013) · Advances in computer vision and pattern recognition · doi:10.1007/978-1-4471-4929-3_14 · Cited by 13 · see citing works
Robust 5DOF Transesophageal Echo Probe Tracking at Fluoroscopic Frame Rates
Hatt, C. R., Speidel, M. A., & Raval, A. N. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24553-9_36 · Cited by 11 · see citing works
Scale-space for N-D discrete signals
Lindeberg, T. (1994) · Springer eBooks · doi:10.1007/978-1-4757-6465-9_4 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Deep learning in medical image registration: a review
Fu, Y., Lei, Y., Wang, T., Curran, W. J., Liu, T., & Yang, X. (2020) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ab843e · Cited by 661 · see citing works
Lian, C., Liu, M., Zhang, J., & Shen, D. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889096 · Cited by 544 · see citing works
Reinforcement learning for intelligent healthcare applications: A survey
Coronato, A., Naeem, M., De Pietro, G., & Paragliola, G. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101964 · Cited by 320 · see citing works
Skin disease diagnosis with deep learning: a review
Li, H., Pan, Y., Zhao, J., & Zhang, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.08.096 · Cited by 178 · see citing works
The artificial intelligence and machine learning in lung cancer immunotherapy
Gao, Q., Yang, L., Lu, M., Jin, R., Ye, H., & Ma, T. (2023) · Journal of Hematology & Oncology · doi:10.1186/s13045-023-01456-y · Cited by 148 · see citing works
Hanneman, K., Playford, D., Dey, D., van Assen, M., Mastrodicasa, D., Cook, T. S., Gichoya, J. W., Williamson, E. E., Rubin, G. D., & on behalf of the American Heart Association Council on Cardiovascular Radiology and Intervention; and Council on Lifelong Congenital Heart Disease and Heart Health in the Young (2024) · Circulation · doi:10.1161/cir.0000000000001202 · Cited by 72 · see citing works
Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views
Bier, B., Goldmann, F., Zaech, J.-N., Fotouhi, J., Hegeman, R., Grupp, R., Armand, M., Osgood, G., Navab, N., Maier, A., et al. (2019) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-019-01975-5 · Cited by 43 · see citing works
Self-citationDeep reinforcement learning and its applications in medical imaging and radiation therapy: a survey
Xu, L., Zhu, S., & Wen, N. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9cb3 · Cited by 35 · see citing works
Agent with Warm Start and Adaptive Dynamic Termination for Plane Localization in 3D Ultrasound
Yang, X., Dou, H., Huang, R., Xue, W., Huang, Y., Qian, J., Zhang, Y., Luo, H., Guo, H., Wang, T., et al. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3069663 · Cited by 32 · see citing works
- Authors
- Ghesu, F.-C., Georgescu, B., Zheng, Y., Grbic, S., Maier, A., Hornegger, J., & Comaniciu, D.
- Affiliations
- Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
- Siemens (United States) (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-01-01
- Volume
- 41
- Issue
- 1
- Pages
- 176-189
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2771590529
- Cited by
- 310
- References
- 63
- Citations by year
- 2026: 12 · 2025: 26 · 2024: 37 · 2023: 50 · 2022: 38 · 2021: 51 · 2020: 53 · 2019: 35 · 2018: 7
- FWCI
- 11.69
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Anatomy and Medical Technology
- Keywords
- anatomical structure detection, computed tomography, deep reinforcement learning, multiscale image analysis, real-time detection, object search, medical image analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
