A Deep Step Pattern Representation for Multimodal Retinal Image Registration
Abstract
This paper presents a novel feature-based method that is built upon a convolutional neural network (CNN) to learn the deep representation for multimodal retinal image registration. We coined the algorithm deep step patterns, in short DeepSPa. Most existing deep learning based methods require a set of manually labeled training data with known corresponding spatial transformations, which limits the size of training datasets. By contrast, our method is fully automatic and scale well to different image modalities with no human intervention. We generate feature classes from simple step patterns within patches of connecting edges formed by vascular junctions in multiple retinal imaging modalities. We leverage CNN to learn and optimize the input patches to be used for image registration. Spatial transformations are estimated based on the output possibility of the fully connected layer of CNN for a pair of images. One of the key advantages of the proposed algorithm is its robustness to non-linear intensity changes, which widely exist on retinal images due to the difference of acquisition modalities. We validate our algorithm on extensive challenging datasets comprising poor quality multimodal retinal images which are adversely affected by pathologies (diseases), speckle noise and low resolutions. The experimental results demonstrate the robustness and accuracy over state-of-the-art multimodal image registration algorithms.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lee and colleagues proposed a CNN-based retinal image registration method that learns multimodal features simultaneously from several imaging modalities.
“In [142], Lee et al. proposed a new CNN-based retinal image registration method to learn multimodal features simultaneously from several imaging modalities.”
From A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2022) · cites this paper as [142]
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Fischler, M. A., & Bolles, R. C. (1981) · Communications of the ACM · doi:10.1145/358669.358692 · Cited by 25,893 · see citing works
A Combined Corner and Edge Detector
Harris, C., & Stephens, M. (1988) · Procedings of the Alvey Vision Conference 1988 · doi:10.5244/c.2.23 · Cited by 12,901 · see citing works
Multidimensional binary search trees used for associative searching
Bentley, J. L. (1975) · Communications of the ACM · doi:10.1145/361002.361007 · Cited by 7,659 · see citing works
Image registration methods: a survey
Zitová, B., & Flusser, J. (2003) · Image and Vision Computing · doi:10.1016/s0262-8856(03)00137-9 · Cited by 6,423 · see citing works
Speckle in optical coherence tomography
Schmitt, J. M., Xiang, S., & Yung, K.-W. (1999) · Journal of Biomedical Optics · doi:10.1117/1.429925 · Cited by 841 · see citing works
A Review of 3d/2d Registration Methods for Image-Guided Interventions
Markelj, P., Tomaževič, D., Likar, B., & Pernuš, F. (2012) · Medical Image Analysis · doi:10.1016/j.media.2010.03.005 · Cited by 746 · see citing works
A CNN Regression Approach for Real-Time 2D/3D Registration
Miao, S., Wang, Z. J., & Liao, R. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2521800 · Cited by 514 · see citing works
The dual-bootstrap iterative closest point algorithm with application to retinal image registration
Stewart, C. V., Tsai, C.-L., & Roysam, B. (2003) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2003.819276 · Cited by 347 · see citing works
A Partial Intensity Invariant Feature Descriptor for Multimodal Retinal Image Registration
Chen, J., Tian, J., Lee, N., Zheng, J., Smith, R. T., & Laine, A. F. (2010) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2010.2042169 · Cited by 312 · see citing works
Guorong Wu, A., Kim, M., Wang, Q., Munsell, B. C., & Shen, D. G. (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2015.2496253 · Cited by 277 · see citing works
Can, A., Stewart, C. V., Roysam, B., & Tanenbaum, H. L. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.990136 · Cited by 275 · see citing works
Registration of Challenging Image Pairs: Initialization, Estimation, and Decision
Yang, G., Stewart, C. V., Sofka, M., & Tsai, C.-L. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.1116 · Cited by 260 · see citing works
An artificial agent for robust image registration
Liao, R., Miao, S., De Tournemire, P., Grbic, S., Kamen, A., Mansi, T., & Comaniciu, D. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11230 · Cited by 199 · see citing works
Automatic retinal image registration scheme using global optimization techniques
Matsopoulos, G. K., Mouravliansky, N. A., Delibasis, K. K., & Nikita, K. S. (1999) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/4233.748975 · Cited by 189 · see citing works
2‐D and 3‐D Image Registration For Medical, Remote Sensing, and Industrial Applications.
Behan, A. (2006) · The Photogrammetric Record · doi:10.1111/j.1477-9730.2006.00375_1.x · Cited by 177 · see citing works
Registration and fusion of retinal images-an evaluation study
Laliberte, F., Gagnon, L., & Sheng, Y. (2003) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2003.812263 · Cited by 155 · see citing works
Diabetic Retinopathy Grading by Digital Curvelet Transform
Alipour, S. H. M., Rabbani, H., & Akhlaghi, M. (2012) · Computational and Mathematical Methods in Medicine · doi:10.1155/2012/761901 · Cited by 139 · see citing works
Tsai, C.-L., Stewart, C. V., Tanenbaum, H. L., & Roysam, B. (2004) · IEEE Transactions on Information Technology in Biomedicine · doi:10.1109/titb.2004.826733 · Cited by 119 · see citing works
Tsai, C.-L., Li, C.-Y., Yang, G., & Lin, K.-S. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2030324 · Cited by 111 · see citing works
Ghassabi, Z., Shanbehzadeh, J., Sedaghat, A., & Fatemizadeh, E. (2013) · EURASIP Journal on Image and Video Processing · doi:10.1186/1687-5281-2013-25 · Cited by 106 · see citing works
Kelman, A., Sofka, M., & Stewart, C. V. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383426 · Cited by 90 · see citing works
Multimodal Registration of Retinal Images Using Self Organizing Maps
Matsopoulos, G. K., Asvestas, P. A., Mouravliansky, N. A., et al. (2004) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2004.836547 · Cited by 86 · see citing works
Dynamic two-strip algorithm in curve fitting
Leung, M. K. H., & Yang, Y. (1990) · Pattern Recognition · doi:10.1016/0031-3203(90)90049-q · Cited by 84 · see citing works
Registration of digital retinal images using landmark correspondence by expectation maximization
Ryan, N., Heneghan, C., & de Chazal, P. (2004) · Image and Vision Computing · doi:10.1016/j.imavis.2004.04.004 · Cited by 72 · see citing works
Yoo, I., Hildebrand, D. G. C., Tobin, W. F., Lee, W.-C. A., & Jeong, W. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_29 · Cited by 69 · see citing works
Lee, J. A., Cheng, J. T., Lee, B. H., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298707 · Cited by 39 · see citing works
Self-citationA novel registration method for retinal images based on local features
Chen, J., Smith, R. T., Tian, J., et al. (2008) · doi:10.1109/iembs.2008.4649642 · Cited by 30 · see citing works
Registration of multimodal fluorescein images sequence of the retina
Choe, T. E., & Cohen, I. (2005) · doi:10.1109/iccv.2005.206 · Cited by 22 · see citing works
Automatic geometrical registration of fluorescein retinal angiograms
Jagoe, R., Blauth, C. I., Smith, P., et al. (1990) · Computers and Biomedical Research · doi:10.1016/0010-4809(90)90030-g · Cited by 18 · see citing works
A Robust Feature-Based Method for Mosaic of the Curved Human Color Retinal Images
Li, J., Chen, H., Yao, C., et al. (2008) · International Conference on Biomedical Engineering and Informatics (BMEI) · doi:10.1109/bmei.2008.76 · Cited by 17 · see citing works
A registration algorithm of eye fundus images using a Bayesian Hough transform
Zana, F. (1999) · IEE International Conference on Image Processing and its Applications (IPA) · doi:10.1049/cp:19990368 · Cited by 9 · see citing works
Geometric corner extraction in retinal fundus images
Lee, J. A., Lee, B. H., Xu, G., Ong, E. P., Wong, D. W. K., Liu, J., & Lim, T. H. (2014) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2014.6943553 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Bayoudh, K., Knani, R., Hamdaoui, F., & Mtibaa, A. (2022) · The Visual Computer · doi:10.1007/s00371-021-02166-7 · Cited by 456 · see citing works
- Authors
- Lee, J., Liu, P., Cheng, J., & Fu, H.
- Affiliations
- University of Electronic Science and Technology of China (CN)
- Chinese Academy of Sciences (CN)
- Inception Institute of Artificial Intelligence (AE)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 5076-5085
- OpenAlex
- W2988267930
- Cited by
- 56
- References
- 35
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 10 · 2023: 10 · 2022: 11 · 2021: 7 · 2020: 6
- FWCI
- 9.04
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- feature-based registration, convolutional neural networks, speckle noise, low-resolution images
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
