An integrated approach to visual perception of articulated objects
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
We present an integrated approach for perception of unknown articulated objects. To robustly perceive objects and understand interactions, our method tightly integrates pose tracking, shape reconstruction, and the estimation of their kinematic structure. The key insight of our method is that these sub-problems complement each other: for example, tracking is greatly facilitated by knowing the shape of the object, whereas the shape and the kinematic structure can be more easily reconstructed if the motion of the object is known. Our combined method leverages these synergies to improve the performance of perception. We analyze the proposed method in average cases and difficult scenarios using a variety of rigid and articulated objects. The results show that our integrated solution achieves better results than solutions for the individual problems. This demonstrates the benefits of approaching robot perception problems in an integrated manner.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Efficient Graph-Based Image Segmentation
Felzenszwalb, P. F., & Huttenlocher, D. P. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000022288.19776.77 · Cited by 6,176
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 2×
“Other methods [6, 17] build on top of KinectFusion [15].”
BackgroundMethodSegmentation of Moving Objects by Long Term Video Analysis
Ochs, P., Malik, J., & Brox, T. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.242 · Cited by 585
Voxel Cloud Connectivity Segmentation - Supervoxels for Point Clouds
Papon, J., Abramov, A., Schoeler, M., & Worgotter, F. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.264 · Cited by 545 · cited 1×
“assuming surface continuity as exploited by conditional random fields [9] graph-cuts, [10], and supervoxel region growing [11], or exploiting object location as in active segmentation [12].”
BackgroundEfficient moving object segmentation algorithm using background registration technique
Chien, S.-Y., Ma, S.-Y., & Chen, L.-G. (2002) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2002.800516 · Cited by 355 · cited 1×
“To detect which points moved due to object motion, image differencing [13, 14] can be applied.”
BackgroundManipulator and object tracking for in-hand 3D object modeling
Krainin, M., Henry, P., Ren, X., & Fox, D. (2011) · The International Journal of Robotics Research · doi:10.1177/0278364911403178 · Cited by 166 · cited 2×
“Partial object views can be obtained using image segmentation and pose information by controlling camera and object motion [1].”
BackgroundRGB-D object tracking: A particle filter approach on GPU
Choi, C., & Christensen, H. I. (2013) · 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2013.6696485 · Cited by 117 · cited 1×
“On the other hand, to track the pose of an object, methods commonly rely on knowledge of the object’s shape and its segmentation in the image [2, 3].”
Interactive segmentation for manipulation in unstructured environments
Kenney, J., Buckley, T., & Brock, O. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152393 · Cited by 99 · cited 1×
“To detect which points moved due to object motion, image differencing [13, 14] can be applied.”
BackgroundSelf-citationA Probabilistic Framework for Learning Kinematic Models of Articulated Objects
Sturm, J., Stachniss, C., & Burgard, W. (2011) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3229 · Cited by 96 · cited 1×
“Based on the tracked poses one can infer the kinematic structure restricting their relative motion [8].”
BackgroundProbabilistic object tracking using a range camera
Wuthrich, M., Pastor, P., Kalakrishnan, M., Bohg, J., & Schaal, S. (2013) · 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2013.6696810 · Cited by 79 · cited 2×
“We distinguish two main approaches to visual tracking: based on a known shape model of the object [2], and based on sufficient texture using point features [4] or dense optical flow [5, 7].”
Method
Cited by
Co-fusion: Real-time segmentation, tracking and fusion of multiple objects
Runz, M., & Agapito, L. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989518 · Cited by 227
“[12] propose a combined approach for estimating pose, shape, and the kinematic structure of articulated objects based on motion segmentation.”
BackgroundInteractive perception: Leveraging action in perception and perception in action
Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2721939 · Cited by 227
“7 ision) 56, 44, 49], [109] , [49], [109] II 48] APR AP M P 7 ision) EU - I [57] CNS PM M P 7 ision) DOM - II [58] APR AP G P 7 ision) EU Esti- gmen- tation [43] CNS O 7 P 7 ision) SOM Esti- I [100], [103, 95, 110] CNS 7 95, 110] [100] P 7 ision) dynamics [103] SDM [99, 110] EU 99, 110] [100] Esti- II 102, 98, 101] AP…”
Self-citation(2021) · Journal of Machine Learning Research · Cited by 224
“text parameters to accurately model the degrees of freedom. Passive perception can be used to estimate the kinematic chains of articulated objects, especially if the object is being moved by a person [258,278,299,42]. Interactive perception is also well suited for estimating the articulation model parameters and the r…”
BackgroundCategory-Level Articulated Object Pose Estimation
Li, X., Wang, H., Yi, L., Guibas, L. J., Abbott, A. L., & Song, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00376 · Cited by 196
Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation
Mittal, M., Hoeller, D., Farshidian, F., Hutter, M., & Garg, A. (2022) · 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981779 · Cited by 86
“A common approach is estimating the object’s kinematic structure from passive observations [20]– [23] or through interactive perception [24], [25].”
BackgroundA-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation
Mu, J., Qiu, W., Kortylewski, A., Yuille, A., Vasconcelos, N. M., & Wang, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01276 · Cited by 156
Ditto: Building Digital Twins of Articulated Objects from Interaction
Jiang, Z., Hsu, C.-C., & Zhu, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00553 · Cited by 92
“As articulation can be ambiguous to infer from a single observation, interactive perception methods [4, 14, 21, 29, 30] have been employed to estimate articulation from actiongenerated visual stimuli.”
MethodAKB-48: A Real-World Articulated Object Knowledge Base
Liu, L., Xu, W., Fu, H., Qian, S., Yu, Q., Yang, H., & Lu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01439 · Cited by 137
“In terms of robotics community, researchers usually solve interaction or manipulation tasks to achieve articulation inference such as robot interactive perception [12], feedback by visual observation [9] and task integration [21].”
BackgroundScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Jain, A., Lioutikov, R., Chuck, C., & Niekum, S. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561132 · Cited by 61
“In more recent works, [23] and [10] have further extended the approach and used hierarchical recursive Bayesian filters to develop online algorithms from articulation model estimation from RGB images.”
MethodVAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated ObjectsPreprint
Wu, R., Zhao, Y. L., Mo, K., Guo, Z., Wang, Y., Wu, T., Fan, Q., Chen, X., Guibas, L. J., & Dong, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14440 · Cited by 113
“A vast literature (Yan & Pollefeys, 2006; Katz et al., 2008; Sturm et al., 2009; 2011; Huang et al., 2012; Katz et al., 2013; Martin & Brock, 2014; Höfer et al., 2014; Katz et al., 2014; Schmidt et al., 2014; Hausman et al., 2015; Martı́n-Martı́n et al., 2016; Tzionas & Gall, 2016; Paolillo et al., 2017; Martın-Martın…”
BackgroundScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Jain, A., Lioutikov, R., Chuck, C., & Niekum, S. (2020) · arXiv · doi:10.48550/arxiv.2008.10518 · Cited by 63
“In more recent works, [23] and [10] have further extended the approach and used hierarchical recursive Bayesian filters to develop online algorithms from articulation model estimation from RGB images.”
MethodCLA-NeRF: Category-Level Articulated Neural Radiance Field
Tseng, W.-C., Liao, H.-J., Lin, Y.-C., & Sun, M. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812272 · Cited by 74
Single-view robot pose and joint angle estimation via render & compare
Labbé, Y., Carpentier, J., Aubry, M., & Šivic, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00170 · Cited by 77
“The first class of methods aims at discovering properties of the kinematic chain through active manipulation [22, 23, 12, 36] using depth as input and unlike our approach cannot be applied to a single image.”
The RBO dataset of articulated objects and interactions
Martín-Martín, R., Eppner, C., & Brock, O. (2019) · The International Journal of Robotics Research · doi:10.1177/0278364919844314 · Cited by 44
“Our dataset will help evaluate algorithms for tracking articulated objects (e.g. Schmidt et al. (2014)) and building models of articulated objects (e.g. Martı́n-Martı́n et al. (2016)).”
MethodSelf-citationVisuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter
Kurenkov, A., Taglic, J., Kulkarni, R., Dominguez-Kuhne, M., Garg, A., Martín-Martín, R., & Savarese, S. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341545 · Cited by 51
“Interactive perception approaches [11] have considered the perceptual problem of segmenting images of a pile of objects into coherent components that move together [12–15].”
BackgroundFour aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015
Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9761-2 · Cited by 40
“2017; Katz and Brock 2011; MartínMartín and Brock 2014), and by integrating interdependent perceptual tasks that were traditionally tackled in isolation (Martín-Martín et al. 2016).”
BackgroundSelf-citationNeural Implicit Representation for Building Digital Twins of Unknown Articulated Objects
Weng, Y., Wen, B., Tremblay, J., Blukis, V., Fox, D., Guibas, L. J., & Birchfield, S. T. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00303 · Cited by 88
Visual Identification of Articulated Object Parts
Zeng, V., Lee, T. E., Liang, J., & Kroemer, O. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636054 · Cited by 54
Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose EstimationPreprint
Liu, Q., Qiu, W., Wang, W., Hager, G. D., & Yuille, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00088 · Cited by 30
“One line of work uses a robot manipulator to interact with the articulated objects, then use the 3D points to infer the pose[18, 17, 26].”
BackgroundReconstructing Articulated Rigged Models from RGB-D Videos
Tzionas, D., & Gall, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-49409-8_53 · Cited by 26
“Due to the popularity of different image sensors over the years, research focus has gradually shifted from reconstructing 2D skeletons from RGB data [8–11] to 3D skeletons from RGB [12–15] or RGB-D data [16, 4, 17].”
BackgroundMethodInterlinked Visual Tracking and Robotic Manipulation of Articulated Objects
Paolillo, A., Chappellet, K., Bolotnikova, A., & Kheddar, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2835515 · Cited by 25
Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
Hsu, C.-C., Jiang, Z., & Zhu, Y. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161431 · Cited by 21
“Some following works extend it with hierarchical recursive Bayesian filters [25], probabilistic models [38], and feature tracking [33].”
BackgroundObject and relation centric representations for push effect prediction
Tekden, A. E., Erdem, A., Erdem, E., Asfour, T., & Uğur, E. (2024) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104632 · Cited by 23
Opening a lockbox through physical exploration
Baum, M., Bernstein, M. A., Martín-Martín, R., Höfer, S., Kulick, J., Toussaint, M., Kacelnik, A., & Brock, O. (2017) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2017.8246913 · Cited by 20
“our prior work on perceiving articulation [17] and generating interaction [18]) and problem representation can also influence exploration behavior.”
BackgroundSelf-citationCoupled learning of action parameters and forward models for manipulation
Höfer, S., & Brock, O. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759573 · Cited by 14
“Although the physical realization of the perceptual and motor capabilities is outside the scope of this paper, we provide references to prior works that implement them on a physical system [19], [20].”
BackgroundMethodSelf-citationBelief Regulated Dual Propagation Nets for Learning Action Effects on Groups of Articulated Objects
Tekden, A. E., Erdem, A., Erdem, E., Imre, M., Seker, M. Y., & Uğur, E. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196878 · Cited by 15
“t from these approaches, our model is verified in a real robotic setup. While the studies above focused on predicting the next states of the objects given relations among them, researchers [19], [20], [21], [22] also studied on estimating the joint relations between objects for real-time tracking and prediction of the…”
MethodA Multiview Approach to Learning Articulated Motion Models
Daniele, A. F., Howard, T. M., & Walter, M. R. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_30 · Cited by 14
“The authors improve upon this work by integrating shape reconstruction, pose estimation, and kinematic estimation [31].”
MethodDistributional Depth-Based Estimation of Object Articulation ModelsPreprint
Jain, A., Giguere, S., Lioutikov, R., & Niekum, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05875 · Cited by 17
“[8] used hierarchical recursive Bayesian filters to make estimation more robust and developed online methods for articulation model estimation from RGB images [7–9].”
MethodMartín-Martín, R., & Brock, O. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206165 · Cited by 11
“This strategy has proved successful for object segmentation and recognition [4, 5], shape reconstruction [6, 7], and the perception of dynamic [8, 9] and kinematic properties [10, 11] of articulated objects.”
BackgroundSelf-citationInteraction-Driven Active 3D Reconstruction with Object Interiors
Yan, Z., Su, F., Wang, M., Hu, R., Zhang, H., & Huang, H. F. (2023) · ACM Transactions on Graphics · doi:10.1145/3618327 · Cited by 11
Physics-Based Selection of Informative Actions for Interactive Perception
Eppner, C., Martín-Martín, R., & Brock, O. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460596 · Cited by 9
“Our method can be applied to articulated objects with single-DoF joints, but it inherits the need for an initial interaction from our perceptual algorithm for the estimation of kinematic models [19], [20].”
MethodSelf-citationOnline Unsupervised Learning of the 3D Kinematic Structure of Arbitrary Rigid Bodies
Nunes, U. M., & Demiris, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00391 · Cited by 7
“Thus, it is an active research topic in computer vision [5, 17, 23] and robotics [14].”
BackgroundA Multi-Step CNN-Based Estimation of Aircraft Landing Gear Angles
Li, F., Wu, Z., Li, J., Lai, Z., Zhao, B., & Chen, M. (2021) · Sensors · doi:10.3390/s21248440 · Cited by 6
Visual estimation of articulated objects configuration during manipulation with a humanoid
Paolillo, A., Bolotnikova, A., Chappellet, K., & Kheddar, A. (2017) · IEEE/SICE International Symposium on System Integration (SII) · doi:10.1109/sii.2017.8279234 · Cited by 6
“In [21], RGB-D data is processed in a unified framework able to estimate the pose, the shape and the structure of the observed object.”
BackgroundSingle-view robot pose and joint angle estimation via render & comparePreprint
Labbé, Y., Carpentier, J., Aubry, M., & Šivic, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.09359 · Cited by 7
“The first class of methods aims at discovering properties of the kinematic chain through active manipulation [22, 23, 12, 36] using depth as input and unlike our approach cannot be applied to a single image.”
Object and Relation Centric Representations for Push Effect PredictionPreprint
Tekden, A. E., Erdem, A., Erdem, E., Asfour, T., & Uğur, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4386221 · Cited by 5
A Helping (Human) Hand in Kinematic Structure Estimation
Pfisterer, A., Li, X., Mengers, V., & Brock, O. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127847 · Cited by 4
“Combining image features with shape-based segmentation can also assist in tracking object parts [9], [10].”
Self-citationKinematic Structure Estimation of Arbitrary Articulated Rigid Objects for Event Cameras
Nunes, U. M., & Demiris, Y. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812430 · Cited by 4
Active Planar Mass Distribution Estimation with Robotic Manipulation
Yuan, J., Choi, C., Tadmor, E. B., & Isler, V. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10342506 · Cited by 5
“Previous studies have shown that robots can estimate such mechanical properties of objects through feedback from various sensor modalities including vision [1], audio [2], or touch [3].”
BackgroundThe Rbo Dataset Of Articulated Objects And Interactions
Martín-Martín, R., Eppner, C., & Brock, O. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1036659 · Cited by 1
Self-citationAnnotation scaffolds for manipulating articulated objects
Frank-Bolton, P., Leontie, R., Drumwright, E., & Simha, R. (2021) · Autonomous Robots · doi:10.1007/s10514-021-09983-8
“These annotated objects might be hand-crafted using Computer Assisted Design software or CAD (such as Solidworks, or 3DsMax), or constructed by integrating features from repeated interactions (Huang et al. 2014; Sturm et al. 2010; Martín-Martín et al. 2016).”
MethodDiscovering Robotic Interaction Modes with Discrete Representation LearningPreprint
Wang, L., Goyal, A., Xu, H., & Garg, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.20258 · Cited by 3
“Common methods deduce the object’s kinematic structure via passive observation [15, 16, 17] or interactive perception [18, 19].”
MethodChappellet, K. (2021) · theses.fr (ABES) · doi:10.70675/479ab81dz71e3z408az8260z8822692dcd5c
Metadata
- Authors
- Martin-Martin, R., Hofer, S., & Brock, O.
- Affiliations
- Technische Universität Berlin (DE)
- ORCID
- Brock, Oliver
- Venue
- 2016 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2016-05
- Pages
- 5091-5097
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2419427454
- Cited by
- 41
- References
- 22
- Citations by year
- 2025: 1 · 2024: 2 · 2023: 4 · 2022: 5 · 2021: 10 · 2020: 4 · 2019: 4 · 2018: 3 · 2017: 6 · 2016: 2
- FWCI
- 8.46
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Keywords
- articulated objects, visual perception, pose tracking, shape recovery, robot perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.