Grounded Human-Object Interaction Hotspots From Video
Open accessAbstract
Learning how to interact with objects is an important step towards embodied visual intelligence, but existing techniques suffer from heavy supervision or sensing requirements. We propose an approach to learn human-object interaction "hotspots" directly from video. Rather than treat affordances as a manually supervised semantic segmentation task, our approach learns about interactions by watching videos of real human behavior and anticipating afforded actions. Given a novel image or video, our model infers a spatial hotspot map indicating where an object would be manipulated in a potential interaction even if the object is currently at rest. Through results with both first and third person video, we show the value of grounding affordances in real human-object interactions. Not only are our weakly supervised hotspots competitive with strongly supervised affordance methods, but they can also anticipate object interaction for novel object categories. Project page: http://vision.cs.utexas.edu/projects/interaction-hotspots/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior methods for affordance learning are tested primarily on human video datasets with no physical robot or in-the-wild experiments.
“prior methods for affordance learning are tested primarily on human video datasets with no physical robot or in-the-wild experiments.”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [80]
The Experiments section evaluates this paper’s method on two human-video datasets, OPRA and EPIC-Kitchens, with no physical-robot experiment described. It does not establish that prior methods were tested primarily on such datasets; the Introduction instead describes some prior exemplars as captured in artificial lab tabletop environments. EPIC-Kitchens also contains unscripted activities, so the blanket claim of no in-the-wild experiments is not supported. (LLM, conf=0.84)
Experimental — accuracy not yet measuredWith both k-NN and BC, VRB outperforms prior methods on 7 out of 8 tasks, with an average success rate of 57%, while the runner-up method (Hotspots) achieves 25%.
“With both k -NN and BC, our method outperforms prior tasks on 7 out of 8 tasks, with an average success rate of 57 %, with the runner-up method (Hotspots [80]) only getting 25 %.”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [80]
The Hotspots baseline was trained on EpicKitchens, the same dataset as the approach.
“Specficially, for Hotspots [80], we employ the model trained on EpicKitchens [20], as this is what our approach is also trained on.”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [80]
VRB consistently outperforms the Hotspots baseline for grasping held-out rare objects.
“VRB consistently outperforms our most competitive baseline, Hotspots [80].”
From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [80]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 79 · Cited by 147
Research works printed but not linked: 8
- D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. ECCV, 2018. 2, 6
- C. Finn, I. Goodfellow, and S. Levine. Unsupervised learn- ing for physical interaction through video prediction. In NeurIPS, 2016. 3
- T. Hermans, J. M. Rehg, and A. Bobick. Affordance pre- diction via learned object attributes. In ICRA: Workshop on Semantic Perception, Mapping, and Exploration, 2011. 1
- D. Jayaraman, F. Ebert, A. A. Efros, and S. Levine. Time- agnostic prediction: Predicting predictable video frames. ICLR, 2019. 3
- M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. ICLR, 2016. 3
- J. Oh, X. Guo, H. Lee, R. L. Lewis, and S. Singh. Action- conditional video prediction using deep networks in atari games. In NeurIPS, 2015. 3
- R. Villegas, J. Yang, Y. Zou, S. Sohn, X. Lin, and H. Lee. Learning to generate long-term future via hierarchical pre- diction. In ICML, 2017. 3
- B. Yao, J. Ma, and L. Fei-Fei. Discovering object function- ality. In ICCV, 2013. 2
- Authors
- Nagarajan, T., Feichtenhofer, C., & Grauman, K.
- Affiliations
- The University of Texas at Austin (US)
- Meta (United States) (US)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 8687-8696
- OpenAlex
- W2904070579
- Cited by
- 147
- References
- 79
- Citations by year
- 2026: 11 · 2025: 23 · 2024: 38 · 2023: 25 · 2022: 19 · 2021: 17 · 2020: 12 · 2019: 2
- FWCI
- 19.34
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- human-object interaction, affordances, weakly supervised learning, video-based learning, object manipulation, novel object categories, egocentric video
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
