Real-time visual tracking of complex structures
Abstract
Presents a framework for three-dimensional model-based tracking. Graphical rendering technology is combined with constrained active contour tracking to create a robust wire-frame tracking system. It operates in real time at video frame rate (25 Hz) on standard hardware. It is based on an internal CAD model of the object to be tracked which is rendered using a binary space partition tree to perform hidden line removal. A Lie group formalism is used to cast the motion computation problem into simple geometric terms so that tracking becomes a simple optimization problem solved by means of iterative reweighted least squares. A visual servoing system constructed using this framework is presented together with results showing the accuracy of the tracker. The paper then describes how this tracking system has been extended to provide a general framework for tracking in complex configurations. The adjoint representation of the group is used to transform measurements into common coordinate frames. The constraints are then imposed by means of Lagrange multipliers. Results from a number of experiments performed using this framework are presented and discussed.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
As opposed to PnP solvers, the technique used in GNFIR and ULTOR makes use of the Lie group SO(3) to find and measure the distance between a rendered model of the target and matching nearby edges in the image.
“As opposed to PnP solvers, this technique makes use of the Lie group SO(3) to find and measure the distance between a rendered model of the target and the matching nearby edges in the image.”
From Review of the robustness and applicability of monocular pose estimation systems for relative navigation with an uncooperative spacecraft (2019) · cites this paper as [47]
Other authors [12,22] implemented the technique proposed in [47] and the ULTOR engine [48] in their GNFIR and ULTOR algorithms, respectively, for pose tracking.
“Aside from the listed solvers adopted to solve the pose initialization problem, other authors [12,22] implemented the technique proposed in [47] and the ULTOR engine [48] in their Goddard Natural Feature Image Recognition (GNFIR) and UL-415 TOR algorithms, respectively, for the pose tracking.”
From Review of the robustness and applicability of monocular pose estimation systems for relative navigation with an uncooperative spacecraft (2019) · cites this paper as [47]
Cites
Shi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · see citing works
Huber, P. J. (1981) · Wiley Series in Probability and Statistics · doi:10.1002/0471725250 · Cited by 7,066 · see citing works
CONDENSATION—Conditional Density Propagation for Visual Tracking
Isard, M., & Blake, A. (1998) · International Journal of Computer Vision · doi:10.1023/a:1008078328650 · Cited by 4,886 · see citing works
Wiley Series in Probability and Statistics
(2008) · Wiley series in probability and statistics · doi:10.1002/series1345 · Cited by 4,270 · see citing works
A tutorial on visual servo control
Hutchinson, S., Hager, G. D., & Corke, P. (1996) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.538972 · Cited by 3,546 · see citing works
Three-dimensional computer vision: a geometric viewpoint
(1994) · Choice Reviews Online · doi:10.5860/choice.32-0357 · Cited by 3,030 · see citing works
A New Approach to Visual Servoing in Robotics
Espiau, B., Chaumette, F., & Rives, P. (1992) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.143350 · Cited by 1,384 · see citing works
A new approach to visual servoing in robotics
Espiau, B., Chaumette, F., & Rives, P. (1993) · Lecture notes in computer science · doi:10.1007/3-540-57132-9_8 · Cited by 1,350 · see citing works
EigenTracking: Robust Matching and Tracking of Articulated Objects Using a View-Based Representation
Black, M. J., & Jepson, A. D. (1998) · International Journal of Computer Vision · doi:10.1023/a:1007939232436 · Cited by 1,038 · see citing works
Fitting parameterized three-dimensional models to images
Lowe, D. G. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.134043 · Cited by 837 · see citing works
Dynamic sensor-based control of robots with visual feedback
Weiss, L. E., Sanderson, A. C., & Neuman, C. P. (1987) · IEEE Journal on Robotics and Automation · doi:10.1109/jra.1987.1087115 · Cited by 832 · see citing works
Tracking people with twists and exponential maps
Bregler, C., & Malik, J. (2002) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1998.698581 · Cited by 719 · see citing works
3-D model-based tracking of humans in action: a multi-view approach
Gavrila, D. M., & Davis, L. S. (1996) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1996.517056 · Cited by 707 · see citing works
Relative end-effector control using Cartesian position based visual servoing
Wilson, W. J., Hulls, C. C. W., & Bell, G. (1996) · IEEE Transactions on Robotics and Automation · doi:10.1109/70.538974 · Cited by 625 · see citing works
EigenTracking: Robust matching and tracking of articulated objects using a view-based representation
Black, M. J., & Jepson, A. D. (1996) · Lecture notes in computer science · doi:10.1007/bfb0015548 · Cited by 414 · see citing works
Lie Groups and Algebras with Applications to Physics, Geometry, and Mechanics
Sattinger, D. H., & Weaver, O. L. (1986) · Applied mathematical sciences · doi:10.1007/978-1-4757-1910-9 · Cited by 376 · see citing works
Terzopoulos, D., & Szeliski, R. (1993) · Cited by 318 · see citing works
Robust model-based motion tracking through the integration of search and estimation
Lowe, D. G. (1992) · International Journal of Computer Vision · doi:10.1007/bf00127170 · Cited by 311 · see citing works
A multiple hypothesis approach to figure tracking
Cham, T., & Rehg, J. M. (2003) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.1999.784636 · Cited by 283 · see citing works
Efficient binary space partitions for hidden-surface removal and solid modeling
Paterson, M. S., & Yao, F. F. (1990) · Discrete & Computational Geometry · doi:10.1007/bf02187806 · Cited by 195 · see citing works
3D articulated models and multi-view tracking with silhouettes
Delamarre, Q., & Faugeras, O. D. (1999) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.790292 · Cited by 188 · see citing works
Harris, C. (1993) · Cited by 184 · see citing works
Robust real-time visual tracking using a 2D-3D model-based approach
Marchand, É., Bouthémy, P., Chaumette, F., et al. (1999) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.791229 · Cited by 156 · see citing works
Nonrigid Motion Analysis: Articulated and Elastic Motion
Aggarwal, J. K., Cai, Q., Liao, W., & Sabata, B. (1998) · Computer Vision and Image Understanding · doi:10.1006/cviu.1997.0620 · Cited by 134 · see citing works
Harris, C. (1993) · Cited by 126 · see citing works
The dynamic analysis of apparent contours
Cipolla, R., & Blake, A. (2002) · International Conference on Computer Vision · doi:10.1109/iccv.1990.139606 · Cited by 119 · see citing works
Self-citationArmstrong, M. (1995) · Asian Conference on Computer Vision · Cited by 114 · see citing works
Real-time visual tracking of 3d objects with dynamic handling of occlusion
Wunsch, P., & Hirzinger, G. (2002) · Proceedings of International Conference on Robotics and Automation · doi:10.1109/robot.1997.606722 · Cited by 93 · see citing works
Application of Lie Algebras to Visual Servoing
Drummond, T. W., & Cipolla, R. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008125412549 · Cited by 68 · see citing works
Self-citationVisual homing: Surfing on the epipoles
Basri, R., Rivlin, E., & Shimshoni, I. (2007) · Springer eBooks · doi:10.1007/bfb0109660 · Cited by 54 · see citing works
Real-Time Tracking of Multiple Articulated Structures in Multiple Views
Drummond, T. W., & Cipolla, R. (2000) · Lecture notes in computer science · doi:10.1007/3-540-45053-x_2 · Cited by 46 · see citing works
Self-citationSelection of features and evaluation of visual measurements during robotic visual servoing tasks
Papanikolopoulos, N. P. (1995) · Journal of Intelligent & Robotic Systems · doi:10.1007/bf01424011 · Cited by 46 · see citing works
Real-time tracking of multiple articulated structures in multiple views
Drummond, T. W., & Cipolla, R. (2000) · Cited by 43 · see citing works
Self-citationVisual homing: surfing on the epipoles
Basri, R., Rivlin, E., & Shimshoni, I. (2002) · International Conference on Computer Vision · doi:10.1109/iccv.1998.710818 · Cited by 40 · see citing works
Image Divergence and Deformation from Closed Curves
Cipolla, R., & Blake, A. (1997) · The International Journal of Robotics Research · doi:10.1177/027836499701600106 · Cited by 36 · see citing works
Self-citationPose refinement of active models using forces in 3D
Worrall, A. D., Sullivan, G. D., & Baker, K. D. (1994) · Lecture notes in computer science · doi:10.1007/3-540-57956-7_38 · Cited by 26 · see citing works
Modelled Object Pose Estimation and Tracking by Monocular Vision
Daucher, N., Dhome, M., Lapresté, J., et al. (1993) · British Machine Vision Conference (BMVC) · doi:10.5244/c.7.25 · Cited by 25 · see citing works
Spatial dependence in the observation of visual contours
MacCormick, J., & Blake, A. (1998) · Lecture notes in computer science · doi:10.1007/bfb0054778 · Cited by 24 · see citing works
Feature-based visual servoing and its application to telerobotics
Hager, G. D., Grunwald, G., & Hirzinger, G. (2002) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.1994.407395 · Cited by 24 · see citing works
Tracking of complex driving manoeuvres in traffic image sequences
Haag, M., & Nagel, H.-H. (1998) · Image and Vision Computing · doi:10.1016/s0262-8856(98)00096-1 · Cited by 22 · see citing works
Feature-Based Visual Servoing and its Application to Telerobotics
Hager, G. D., Grunwald, G., & Toyama, K. (1995) · Elsevier eBooks · doi:10.1016/b978-044482250-5/50028-1 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SSD-6D: Making RGB-Based 3D Detection and 6D Pose Estimation Great Again
Kehl, W., Manhardt, F., Tombari, F., Ilic, S., & Navab, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.169 · Cited by 1,047 · see citing works
Pose estimation for augmented reality: a hands-on survey
Marchand, E., Uchiyama, H., & Spindler, F. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2513408 · Cited by 746 · see citing works
Monocular model-based 3D tracking of rigid objects: a survey
Lepetit, V., & Fua, P. (2005) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000001 · Cited by 484 · see citing works
Opromolla, R., Fasano, G., Rufino, G., & Grassi, M. (2017) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2017.07.001 · Cited by 330 · see citing works
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197244 · Cited by 229 · see citing works
“As for URSO, we collected one dataset for the dragon spacecraft and two datasets for the soyuz model with different operating ranges: soyuz_easy with [10-20] m and soyuz_hard with [10-40]”
BackgroundMethodPasqualetto Cassinis, L., Fonod, R., & Gill, E. (2019) · Progress in Aerospace Sciences · doi:10.1016/j.paerosci.2019.05.008 · Cited by 156 · see citing works
Vision-Based Relative Pose Estimation for Autonomous Rendezvous And Docking
Kelsey, J. M., Byrne, J., Cosgrove, M., Seereeram, S., & Mehra, R. K. (2006) · 2006 IEEE Aerospace Conference · doi:10.1109/aero.2006.1655916 · Cited by 130 · see citing works
Relative pose estimation for cylinder-shaped spacecrafts using single image
Liu, C., & Hu, W. (2014) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2014.120757 · Cited by 101 · see citing works
Augmented reality based on edge computing using the example of remote live support
Schneider, M., Rambach, J., & Stricker, D. (2017) · 2017 IEEE International Conference on Industrial Technology (ICIT) · doi:10.1109/icit.2017.7915547 · Cited by 96 · see citing works
Deep Learning for Spacecraft Pose Estimation from Photorealistic Rendering
Proenca, P. F., & Gao, Y. (2019) · arXiv · Cited by 251 · see citing works
Naasz, B. J., Burns, R. D., Queen, S. Z., Van Eepoel, J., Hannah, J., & Skelton, E. (2009) · The Journal of the Astronautical Sciences · doi:10.1007/bf03321512 · Cited by 37 · see citing works
On ground validation of debris removal technologies
Colmenarejo, P., Graziano, M., Novelli, G., Mora, D., Serra, P., Tomassini, A., Seweryn, K., Prisco, G., & Fernandez, J. G. (2019) · Acta Astronautica · doi:10.1016/j.actaastro.2018.01.026 · Cited by 33 · see citing works
ChiNet: Deep recurrent convolutional learning for multimodal spacecraft pose estimation
Rondao, D., Aouf, N., & Richardson, M. A. (2022) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2022.3193085 · Cited by 24 · see citing works
- Authors
- Drummond, T., & Cipolla, R.
- Affiliations
- University of Cambridge (GB)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2002-07
- Volume
- 24
- Issue
- 7
- Pages
- 932-946
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2158890580
- Cited by
- 598
- References
- 41
- Citations by year
- 2026: 11 · 2025: 7 · 2024: 8 · 2023: 10 · 2022: 13 · 2021: 18 · 2020: 20 · 2019: 24 · 2018: 24 · 2017: 22 · 2016: 28 · 2015: 22 · 2014: 23 · 2013: 37 · 2012: 28
- FWCI
- 26.98
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- 3D model-based tracking, hidden line removal, iteratively reweighted least squares, visual servoing, adjoint representation, Lagrange multipliers, real-time video tracking, CAD models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
