KETO: Learning Keypoint Representations for Tool Manipulation
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
We aim to develop an algorithm for robots to manipulate novel objects as tools for completing different task goals. An efficient and informative representation would facilitate the effectiveness and generalization of such algorithms. For this purpose, we present KETO, a framework of learning keypoint representations of tool-based manipulation. For each task, a set of task-specific keypoints is jointly predicted from 3D point clouds of the tool object by a deep neural network. These keypoints offer a concise and informative description of the object to determine grasps and subsequent manipulation actions. The model is learned from self-supervised robot interactions in the task environment without the need for explicit human annotations. We evaluate our framework in three manipulation tasks with tool use. Our model consistently outperforms state-of-the-art methods in terms of task success rates. Qualitative results of keypoint prediction and tool generation are shown to visualize the learned representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
The Ecological Approach to Visual Perception: Classic Edition
Gibson, J. J. (2014) · doi:10.4324/9781315740218 · Cited by 12,132
ORB: An efficient alternative to SIFT or SURF
Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126544 · Cited by 11,037
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · cited 1×
“Keypoint-based methods have been extensively studied in facial recognition [6], [36], [35], human pose estimation [5], [9], [40], [51], [8], [53] and tracking [39], [20], [43], [21], [10] in computer vision and pattern recognition applications.”
BackgroundStacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · cited 1×
“Keypoint-based methods have been extensively studied in facial recognition [6], [36], [35], human pose estimation [5], [9], [40], [51], [8], [53] and tracking [39], [20], [43], [21], [10] in computer vision and pattern recognition applications.”
BackgroundDomain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124
Human Pose Estimation with Iterative Error Feedback
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.512 · Cited by 805 · cited 1×
“Keypoint-based methods have been extensively studied in facial recognition [6], [36], [35], human pose estimation [5], [9], [40], [51], [8], [53] and tracking [39], [20], [43], [21], [10] in computer vision and pattern recognition applications.”
BackgroundDeep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599
Maitin-Shepard, J., Cusumano-Towner, M., Lei, J., & Abbeel, P. (2010) · 2010 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2010.5509439 · Cited by 418 · cited 1×
“In robotic manipulation tasks with visual inputs, keypoints can provide compact information of the environment and the objects [11], [17], [54], [49], [33], [37], [34].”
Background
Cited by
You Only Demonstrate Once: Category-Level Manipulation from Single Visual Demonstration
Wen, B., Lian, W., Bekris, K., & Schaal, S. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.044 · Cited by 70
Wang, Y., Wu, R., Mo, K., Ke, J., Fan, Q., Guibas, L. J., & Dong, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19818-2_6 · Cited by 40
ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
Mu, T., Ling, Z., Xiang, F., Yang, D., Li, X., Tao, S., Huang, Z., Jia, Z., & Su, H. (2021) · arXiv · doi:10.48550/arxiv.2107.14483 · Cited by 34
“On the other hand, learning-based grasping [44, 45, 46, 47, 48, 49, 50, 51] has gained increasing popularity and can propose novel grasp poses on novel objects.”
BackgroundRoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., & Fox, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10721 · Cited by 7
“Affordance can be represented in many ways such as part segmentation [38], dense image feature descriptors [39] and keypoints [40, 41, 12].”
Background
Metadata
- Authors
- Qin, Z., Fang, K., Zhu, Y., Fei-Fei, L., & Savarese, S.
- Venue
- 2020 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2020-05
- Pages
- 7278-7285
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3091470805
- Cited by
- 78
- References
- 12
- FWCI
- 19.25
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: KETO: Learning Keypoint Representations for Tool Manipulation. https://app.cochat.ai/papers/10.1109/icra40945.2020.9196971
- The paper
- Qin, Z., Fang, K., Zhu, Y., Fei-Fei, L., & Savarese, S. (2020). KETO: Learning Keypoint Representations for Tool Manipulation. 2020 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra40945.2020.9196971
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.