Demo2Vec: Reasoning Object Affordances from Online Videos
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Watching expert demonstrations is an important way for humans and robots to reason about affordances of unseen objects. In this paper, we consider the problem of reasoning object affordances through the feature embedding of demonstration videos. We design the Demo2Vec model which learns to extract embedded vectors of demonstration videos and predicts the interaction region and the action label on a target image of the same object. We introduce the Online Product Review dataset for Affordance (OPRA) by collecting and labeling diverse YouTube product review videos. Our Demo2Vec model outperforms various recurrent neural network baselines on the collected dataset.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · cited 12×
“CNN+ConvLSTM+Deconv: A ConvLSTM [27] is used as the demonstration encoder to better capture temporally correlated spatial information, the demonstration embedding and the image feature are concatenated directly in their respective spatial dimensions.”
MethodInfluentialGenerative adversarial imitation learning
Ho, J., & Ermon, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03476 · Cited by 1,942
A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
Ross, S., Gordon, G. J., & Bagnell, J. A. (2018) · Carnegie Mellon University · doi:10.1184/r1/6550949 · Cited by 1,941
Anticipating Human Activities Using Object Affordances for Reactive Robotic Response
Koppula, H. S., & Saxena, A. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2430335 · Cited by 596 · cited 1×
“To teach a robot about how humans manipulate and interact with objects, previous methods learn this knowledge from simulated agent-object interactions [33, 23, 29], demonstrations observed by the robot camera in the robot workspace [22, 17] or demonstrations observed from a third-person viewpoint [13, 14].”
BackgroundLearning from Demonstrations Through the Use of Non-rigid Registration
Schulman, J., Ho, J., Lee, C., & Abbeel, P. (2016) · Springer Tracts in Advanced Robotics · doi:10.1007/978-3-319-28872-7_20 · Cited by 133 · cited 1×
“To teach a robot about how humans manipulate and interact with objects, previous methods learn this knowledge from simulated agent-object interactions [33, 23, 29], demonstrations observed by the robot camera in the robot workspace [22, 17] or demonstrations observed from a third-person viewpoint [13, 14].”
Background
Cited by
Understanding Human Hands in Contact at Internet Scale
Shan, D., Geng, J., Shu, M., & Fouhey, D. F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00989 · Cited by 256
Hand-Object Contact Consistency Reasoning for Human Grasps Generation
Jiang, H., Liu, S., Wang, J., & Wang, X. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01092 · Cited by 165
“Predicting scene and object affordance plays an important role in visual understanding [27, 52, 6, 10, 53, 12, 25, 15, 51].”
BackgroundGrounded Human-Object Interaction Hotspots From Video
Nagarajan, T., Feichtenhofer, C., & Grauman, K. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00878 · Cited by 147
“They assume access to training images with manually drawn masks or keypoints [45, 10, 12] and some leverage additional sensors like depth [31, 65, 66] or force gloves [3], all of which restrict scalability.”
BackgroundMethodInfluentialAugmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
Nasiriany, S., Liu, H., & Zhu, Y. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9812140 · Cited by 91
Wang, Y., Wu, R., Mo, K., Ke, J., Fan, Q., Guibas, L. J., & Dong, H. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19818-2_6 · Cited by 40
Metadata
- Authors
- Fang, K., Wu, T.-L., Yang, D., Savarese, S., & Lim, J. J.
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 2139-2147
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2798897443
- Cited by
- 105
- References
- 8
- FWCI
- 14.01
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.