The DOI matches its DataCite record; 51 of the 59 references it cites were found in a registry or an index.
Where2Act: From Pixels to Actions for Articulated 3D Objects
Open accessAbstract
One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. Check the website for code and data release: https://cs.stanford.edu/~kaichun/where2act/
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Another line of research acquires active training data with simulated or real robots for affordance learning [16, 19, 26, 28, 41].”
From Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception (2023) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpacePreprint
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02413 · Cited by 10,099
Self-citation3D ShapeNets: A deep representation for volumetric shapes
Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298801 · Cited by 5,413
A Point Set Generation Network for 3D Object Reconstruction from a Single Image
Fan, H., Su, H., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.264 · Cited by 2,513
Self-citationShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330
Self-citationPoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes
Xiang, Y., Schmidt, T., Narayanan, V., & Fox, D. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.019 · Cited by 2,139
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T. A., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2023) · doi:10.5281/zenodo.10089705 · Cited by 1,774
Self-citationMedical Image Computing and Computer-Assisted Intervention — MICCAI’98
Roche, A., Malandain, G., Pennec, X., & Ayache, N. (1998) · Lecture notes in computer science · doi:10.1007/bfb0056181 · Cited by 1,578
On the Continuity of Rotation Representations in Neural Networks
Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00589 · Cited by 1,255
A scalable active framework for region annotation in 3D shape collections
Yi, L., Kim, V. G., Ceylan, D., Shen, I., Yan, M., Su, H., Lu, C., Qi-xing Huang, A., Sheffer, A., & Guibas, L. J. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980238 · Cited by 1,209
Self-citationSupersizing self-supervision: Learning to grasp from 50K tries and 700 robot hours
Pinto, L., & Gupta, A. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487517 · Cited by 1,099
Self-citationNormalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation
Wang, H., Sridhar, S., Huang, J., Valentin, J., Song, S., & Guibas, L. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00275 · Cited by 797
Self-citationAutomatic grasp planning using shape primitives
Miller, A. T., Knoop, S., Christensen, H. I., & Allen, P. K. (2004) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2003.1241860 · Cited by 711
The Replica Dataset: A Digital Replica of Indoor SpacesPreprint
Straub, J., Whelan, T. J., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C. Y., Verma, S., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05797 · Cited by 711
Mo, K., Zhu, S., Chang, A. X., Yi, L., Tripathi, S., Guibas, L. J., & Su, H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00100 · Cited by 666
Self-citationLearning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning
Zeng, A., Song, S., Welker, S., Lee, J., Rodriguez, A., & Funkhouser, T. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593986 · Cited by 606
Implicit 3D Orientation Learning for 6D Object Detection from RGB Images
Sundermeyer, M., Marton, Z.-C., Durner, M., Brucker, M., & Triebel, R. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_43 · Cited by 564
Deep Object Pose Estimation for Semantic Robotic Grasping of Household ObjectsPreprint
Tremblay, J., To, T., Sundaralingam, B., Yu, X., Fox, D., & Birchfield, S. T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10790 · Cited by 510
SAPIEN: A SimulAted Part-Based Interactive ENvironment
Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01111 · Cited by 406
Self-citationPCPN et Learning Local Shape Properties from Raw Point Clouds
Guerrero, P., Kleiman, Y., Ovsjanikov, M., & Mitra, N. J. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13343 · Cited by 387
Mahler, J., Pokorny, F. T., Hou, B., Roderick, M., Laskey, M., Aubry, M., Kohlhoff, K., Kröger, T., Kuffner, J., & Goldberg, K. Y. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487342 · Cited by 361
ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging
Brahmbhatt, S., Ham, C., Kemp, C. C., & Hays, J. H. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00891 · Cited by 223
6-DOF Grasping for Target-driven Object Manipulation in Clutter
Murali, A., Mousavian, A., Eppner, C., Paxton, C., & Fox, D. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197318 · Cited by 204
Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation
Chen, D., Li, J., Wang, Z., & Xu, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01199 · Cited by 202
Category-Level Articulated Object Pose Estimation
Li, X., Wang, H., Yi, L., Guibas, L. J., Abbott, A. L., & Song, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00376 · Cited by 196
Self-citationGrounded Human-Object Interaction Hotspots From Video
Nagarajan, T., Feichtenhofer, C., & Grauman, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00878 · Cited by 147
Evaluation of 3D interest point detection techniques via human-generated ground truth
Dutağacı, H., Cheung, C. P., & Godil, A. (2012) · The Visual Computer · doi:10.1007/s00371-012-0746-4 · Cited by 142
Shape2Pose: human-centric shape analysis
Kim, V. G., Chaudhuri, S., Guibas, L., & Funkhouser, T. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601117 · Cited by 112
Self-citationLearning to push by grasping: Using multiple tasks for effective learning
Pinto, L., & Gupta, A. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989249 · Cited by 106
Self-citationSemantically-enriched 3D models for common-sense knowledge
Savva, M., Chang, A. X., & Hanrahan, P. (2015) · IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301289 · Cited by 104
Shape2Motion: Joint Analysis of Motion Parts and Attributes From 3D Shapes
Wang, X., Zhou, B., Shi, Y., Chen, X., Zhao, Q., & Xu, K. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00908 · Cited by 104
Deep part induction from articulated object pairs
Yi, L., Huang, H., Liu, D., Kalogerakis, E., Su, H., & Guibas, L. (2018) · ACM Transactions on Graphics · doi:10.1145/3272127.3275027 · Cited by 81
Self-citationPeople Watching: Human Actions as a Cue for Single View Geometry
Fouhey, D. F., Delaitre, V., Gupta, A., Efros, A. A., Ivan Laptev, A., & Šivic, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_53 · Cited by 81
Self-citationKeypointNet: A Large-Scale 3D Keypoint Dataset Aggregated From Numerous Human Annotations
You, Y., Lou, Y., Li, C., Cheng, Z., Li, L., Ma, L., Lu, C., & Wang, W. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01366 · Cited by 72
Volumetric Hierarchical Approximate Convex Decomposition
Lengyel, E. (2016) · A K Peters/CRC Press eBooks · doi:10.1201/b21177-15 · Cited by 71
ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Jain, A., Lioutikov, R., Chuck, C., & Niekum, S. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561132 · Cited by 63
An object-dependent hand pose prior from sparse training data
Hamer, H., Gall, J., Weise, T., & Gool, L. V. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540150 · Cited by 62
Yan, Z., Hu, R., Yan, X., Chen, L., van Kaick, O. M., Zhang, H., & Huang, H. F. (2019) · ACM Transactions on Graphics · doi:10.1145/3355089.3356573 · Cited by 56
Learning to predict part mobility from a single static snapshot
Hu, R., Li, W., van Kaick, O. M., Shamir, A., Zhang, H., & Huang, H. F. (2017) · ACM Transactions on Graphics · doi:10.1145/3130800.3130811 · Cited by 40
Learning Instance Segmentation by Interaction
Pathak, D., Shentu, Y., Chen, D., Agrawal, P., Darrell, T., Levine, S., & Malik, J. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00276 · Cited by 38
Learning Affordance Landscapes for Interaction Exploration in 3D Environments
Nagarajan, T., & Grauman, K. (2020) · arXiv · doi:10.48550/arxiv.2008.09241 · Cited by 38
The RBO dataset of articulated objects and interactions
Martín-Martín, R., Eppner, C., & Brock, O. (2019) · The International Journal of Robotics Research · doi:10.1177/0278364919844314 · Cited by 38
Functionality Representations and Applications for Shape Analysis
Hu, R., Savva, M., & van Kaick, O. (2018) · Computer Graphics Forum · doi:10.1111/cgf.13385 · Cited by 33
Action Image Representation: Learning Scalable Deep Grasping Policies with Zero Real World Data
Khansari, M., Kappler, D., Luo, J., Bingham, J., & Kalakrishnan, M. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197415 · Cited by 23
Predictive and generative neural networks for object functionality
Hu, R., Yan, Z., Zhang, J., Kaick, O. V., Shamir, A., Zhang, H., & Huang, H. (2018) · ACM Transactions on Graphics · doi:10.1145/3197517.3201287 · Cited by 21
Learning Material-Aware Local Descriptors for 3D Shapes
Lin, H., Averkiou, M., Kalogerakis, E., Kovács, B., Ranade, S., Kim, V. G., Chaudhuri, S., & Bala, K. (2018) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00027 · Cited by 15
Learning About Objects by Learning to Interact with ThemPreprint
Lohmann, M., Salvador, J., Kembhavi, A., & Mottaghi, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09306 · Cited by 13
Cross-Modal Attribute Transfer for Rescaling 3D Models
Shao, L., Chang, A. X., Su, H., Savva, M., & Guibas, L. J. (2017) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2017.00078 · Cited by 4
Self-citationXu, X., Charatan, D., Raychaudhuri, S., Jiang, H., Heitmann, M., Kim, V. G., Chaudhuri, S., Savva, M., Chang, A. X., & Ritchie, D. (2020) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00071 · Cited by 3
Research works printed but not linked: 1
- Zihao Yan, Ruizhen Hu, Xingguang Yan, Luanmin Chen, Oliver Van Kaick, Hao Zhang, and Hui Huang. Rpm-net: Recurrent prediction of motion and parts from point cloud. ACM Transactions on Graphics (TOG), 38(6):240, 2019. 2
Cited by
Sorted by citations
OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction
Yang, L., Li, K., Zhan, X., Wu, F., Xu, A., Liu, L., & Lu, C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02028 · Cited by 99
Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91
Articulated Object Interaction in Unknown Scenes with Whole-Body Mobile Manipulation
Mittal, M., Hoeller, D., Farshidian, F., Hutter, M., & Garg, A. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9981779 · Cited by 91
A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation
Mu, J., Qiu, W., Kortylewski, A., Yuille, A., Vasconcelos, N. M., & Wang, X. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01276 · Cited by 85
Geng, H., Xu, H., Zhao, C., Xu, C., Yi, L., Huang, S., & Wang, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00684 · Cited by 121
Metadata
- Authors
- Mo, K., Guibas, L., Mukadam, M., Gupta, A., & Tulsiani, S.
- Affiliations
- Stanford University (US)
- Meta (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- Pages
- 6793-6803
- arXiv
- 2101.02692
- OpenAlex
- W3120441392
- Cited by
- 132
- References
- 59
- Citations by year
- 2026: 19 · 2025: 29 · 2024: 29 · 2023: 33 · 2022: 15 · 2021: 7
- FWCI
- 31.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1763268)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- articulated objects, actionable information, pushing and pulling, movable parts, learning from interaction, simulation-based training, depth information, human-robot interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
