Understanding tools: Task-oriented object modeling, learning and recognition
The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
In this paper, we present a new framework - task-oriented modeling, learning and recognition which aims at understanding the underlying functions, physics and causality in using objects as “tools”. Given a task, such as, cracking a nut or painting a wall, we represent each object, e.g. a hammer or brush, in a generative spatio-temporal representation consisting of four components: i) an affordance basis to be grasped by hand; ii) a functional basis to act on a target object (the nut), iii) the imagined actions with typical motion trajectories; and iv) the underlying physical concepts, e.g. force, pressure, etc. In a learning phase, our algorithm observes only one RGB-D video, in which a rational human picks up one object (i.e. tool) among a number of candidates to accomplish the task. From this example, our algorithm learns the essential physical concepts in the task (e.g. forces in cracking nuts). In an inference phase, our algorithm is given a new set of objects (daily objects or stones), and picks the best choice available together with the inferred affordance basis, functional basis, imagined human actions (sequence of poses), and the expected physical quantity that it will produce. From this new perspective, any objects can be viewed as a hammer or a shovel, and object recognition is not merely memorizing typical appearance examples for each category but reasoning the physical mechanisms in various tasks to achieve generalization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“We apply the KinectFusion algorithm [27] to generate a 3D reconstruction of the static scene, including a tool and an object.”
MethodA survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
How to grow a mind: Statistics, structure, and abstraction
Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011) · Science · doi:10.1126/science.1192788 · Cited by 1,815
Shape2Pose: human-centric shape analysis
Kim, V. G., Chaudhuri, S., Guibas, L., & Funkhouser, T. (2014) · ACM Transactions on Graphics · doi:10.1145/2601097.2601117 · Cited by 112
Cited by
Explainable machine learning for scientific insights and discoveries
Roscher, R., Bohn, B., Duarte, M. F., & Garcke, J. (2020) · IEEE Access · doi:10.1109/access.2020.2976199 · Cited by 1,040
“Reference [45] introduces a framework that calculates physical concepts from color-depth videos that estimates tool and tool-use such as cracking a nut.”
BackgroundGrounded Human-Object Interaction Hotspots From Video
Nagarajan, T., Feichtenhofer, C., & Grauman, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00878 · Cited by 147
“They assume access to training images with manually drawn masks or keypoints [45, 10, 12] and some leverage additional sensors like depth [31, 65, 66] or force gloves [3], all of which restrict scalability.”
Background3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding
Deng, S., Xu, X., Wu, C., Chen, K., & Jia, K. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00182 · Cited by 124
High-Fidelity Grasping in Virtual Reality using a Glove-based System
Liu, H., Zhang, Z., Xie, X., Zhu, Y., Liu, Y., Wang, Y., & Zhu, S.-C. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794230 · Cited by 73
“This grasping data collected could be used for different domains of research ranging from studying human utilities [33] or robotics grasp planning [34].”
MethodSelf-citationA glove-based system for studying hand-object manipulation via joint pose and force sensing
Liu, H., Xie, X., Millar, M., Edmonds, M., Gao, F., Zhu, Y., Santos, V. J., Rothrock, B., & Zhu, S.-C. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206575 · Cited by 60
“While researchers can track hand pose based on perception [3], force estimation from vision using numerical differentiation methods [4], [5], or sophisticated physics-based soft-body simulation [6], [7], glove-based devices still have their own advantages, presenting convenient, integrated solutions that can be natura…”
BackgroundSelf-citation
Metadata
- Authors
- Zhu, Y., Zhao, Y., & Zhu, S.-C.
- Venue
- 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2015-06
- Pages
- 2855-2864
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W1920293286
- Cited by
- 144
- References
- 4
- FWCI
- 20.64
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Understanding tools: Task-oriented object modeling, learning and recognition. https://app.cochat.ai/papers/10.1109/cvpr.2015.7298903
- The paper
- Zhu, Y., Zhao, Y., & Zhu, S.-C. (2015). Understanding tools: Task-oriented object modeling, learning and recognition. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2015.7298903
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.