CoChat
Sign Up
Cited by 596Context-Aware Activity Recognition SystemsHuman Pose and Action RecognitionRobot Manipulation and Learning

Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

Koppula, Hema S., Saxena, Ashutosh·2016·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2015.2430335
Open this paper in CoChat — free account

Abstract

An important aspect of human perception is anticipation, which we use extensively in our day-to-day activities when interacting with other humans as well as with our surroundings. Anticipating which activities will a human do next (and how) can enable an assistive robot to plan ahead for reactive responses. Furthermore, anticipation can even improve the detection accuracy of past activities. The challenge, however, is two-fold: We need to capture the rich context for modeling the activities and object affordances, and we need to anticipate the distribution over a large space of future human activities. In this work, we represent each possible future using an anticipatory temporal conditional random field (ATCRF) that models the rich spatial-temporal relations through object affordances. We then consider each ATCRF as a particle and represent the distribution over the potential futures using a set of particles. In extensive evaluation on CAD-120 human activity RGB-D dataset, we first show that anticipation improves the state-of-the-art detection results. We then show that for new subjects (not seen in the training set), we obtain an activity anticipation accuracy (defined as whether one of top three predictions actually happened) of 84.1, 74.4 and 62.2 percent for an anticipation time of 1, 3 and 10 seconds respectively. Finally, we also show a robot using our algorithm for performing a few reactive responses.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2015.2430335

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Environment change, i.e., change in the position of objects, is a task-agnostic metric for exploration.

    “We utilize environment change, i.e ., change in the position of objects as a task-agnostic metric for exploration [6].”

    From Affordances from Human Videos as a Versatile Representation for Robotics (2023) · cites this paper as [6]

    The Overview (p. 2) discusses predicting object locations, and Experiments (p. 6) measures errors in anticipated trajectories. Neither passage describes changes in object positions as a task-agnostic metric for exploration. Searches for that phrase and for an exploration metric based on object-position changes found no supporting passage in our copy. (LLM, conf=0.73)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 32

  1. [2][2] Rosen Diankov. Automated Construction of Robotic Manipulation Programs . PhD thesis, CMU, Robotics Institute, 2010. -
  2. [3][3] A. Doucet, N. de Freitas, K. P. Murphy, and S. J. Russell. Raoblackwellised particle filtering for dynamic bayesian networks. In UAI , 2000. -
  3. [6][6] D. Fox. Kld-sampling: Adaptive particle filters. In NIPS , 2001. -
  4. [7][7] S. Gong and T. Xiang. Recognition of group activities using dynamic probabilistic networks. In ICCV , 2003. -
  5. [8][8] E. Guizzo and E. Ackerman. The rise of the robot worker. Spectrum, IEEE , 49(10):34 -41, October 2012. -
  6. [9][9] A. Gupta, A. Kembhavi, and L. S. Davis. Observing humanobject interactions: Using spatial and functional compatibility for recognition. IEEE PAMI , 31(10):1775-1789, 2009. -
  7. [10][10] F. Gustafsson, F. Gunnarsson, N. Bergman, U. Forssell, J. Jansson, R. Karlsson, and P.-J. Nordlund. Particle filters for positioning, navigation, and tracking. IEEE Trans. SP , 2002. -
  8. [11][11] R. Hess and A. Fern. Discriminatively trained particle filters for complex multi-object tracking. In CVPR , 2009. -
  9. [12][12] M. Hoai and F. De la Torre. Max-margin early event detectors. In CVPR , 2012. -
  10. [13][13] S. Hongeng and R. Nevatia. Large-scale event detection using semi-hidden markov models. In ICCV , 2003. -
  11. [14][14] P. Jensfelt, D. Austin, O. Wijk, and M. Andersson. Feature based condensation for mobile robot localization. In ICRA , 2000. -
  12. [15][15] Y. Jiang, M. Lim, and A. Saxena. Learning object arrangements in 3d scenes using human context. In ICML , 2012. -
  13. [16][16] Y. Jiang, H. S. Koppula, and A. Saxena. Hallucinated humans as the hidden context for labeling 3d scenes. In CVPR , 2013. -
  14. [17][17] Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. IEEE PAMI , 2005. -
  15. [18][18] K. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert. Activity forecasting. In ECCV , 2012. -
  16. [19][19] H. S. Koppula and Ashutosh Saxena. Learning spatio-temporal structure from rgb-d videos for human activity detection and anticipation. In ICML , 2013. -
  17. [20][20] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena. Semantic labeling of 3d point clouds for indoor scenes. In NIPS , 2011. -
  18. [24][24] M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit. Fastslam: A factored solution to the simultaneous localization and mapping problem. In AAAI , 2002. -
  19. [25][25] M. Montemerlo, S. Thrun, and W. Whittaker. Conditional particle filters for simultaneous mobile robot localization and people-tracking. In ICRA , 2002. -
  20. [29][29] H. Pirsiavash and D. Ramanan. Detecting activities of daily living in first-person camera views. In CVPR , 2012. -
  21. [30][30] A. Quattoni, S. Wang, L.-P. Morency, M. Collins, and T. Darrell. Hidden-state conditional random fields. IEEE PAMI , 2007. -
  22. [31][31] M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele. A database for fine grained activity detection of cooking activities. In CVPR , 2012. -
  23. [32][32] M.S. Ryoo. Human activity prediction: Early recognition of ongoing activities from streaming videos. In ICCV , 2011. -
  24. [33][33] S. Sarawagi and W. W. Cohen. Semi-markov conditional random fields for information extraction. In NIPS , 2004. -
  25. [34][34] D. Schulz, D. Fox, and J. Hightower. People tracking with anonymous and id-sensors using rao-blackwellised particle filters. In IJCAI , 2003. -
  26. [35][35] C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas. Conditional models for contextual human motion recognition. In ICCV , 2005. -
  27. [36][36] J. Sung, C. Ponce, B. Selman, and A. Saxena. Unstructured human activity detection from rgbd images. In ICRA , 2012. -
  28. [37][37] K. Tang, L. Fei-Fei, and D. Koller. Learning latent temporal structure for complex event detection. In CVPR , 2012. -
  29. [39][39] W. Yang, Y. Wang, and G. Mori. Recognizing human actions from still images with latent poses. In CVPR , 2010. -
  30. [40][40] B. Yao and L. Fei-Fei. Modeling mutual context of object and human pose in human-object interaction activities. In CVPR , 2010. -
  31. [41][41] H. Zhang and L. E. Parker. 4-dimensional local spatio-temporal features for human activity recognition. In IROS , 2011. -
  32. [42][42] B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. Dey, and S. Srinivasa. Planningbased prediction for pedestrians. In IROS , 2009.

Cited by

Sorted by citations

Authors
Koppula, H. S., & Saxena, A.
Affiliations
Cornell University (US)
ORCID
Saxena, Ashutosh
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2016-01-01
Volume
38
Issue
1
Pages
14-29
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
DOI
10.1109/tpami.2015.2430335
OpenAlex
W1483019628
Cited by
596
References
3
Citations by year
2026: 20 · 2025: 35 · 2024: 49 · 2023: 55 · 2022: 55 · 2021: 70 · 2020: 67 · 2019: 61 · 2018: 65 · 2017: 57 · 2016: 30 · 2015: 19 · 2014: 9 · 2013: 4
FWCI
49.50
Type
journal article
Language
en
Open access
closed
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html (vor)
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
Funders
  • ARO (W911NF-12-1-0267)
  • US National Science Foundation (NSF)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
object affordances, assistive robots, human activity recognition, spatio-temporal modeling
MeSH terms
Machine Learning, Algorithms, Humans, Human Activities, Movement, Perception, Robotics, Video Recording, Models, Statistical, Imaging, Three-Dimensional, Anticipation, Psychological

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review