Recognizing object affordances in terms of spatio-temporal object-object relationships
Open accessAbstract
In this paper we describe a probabilistic framework that models the interaction between multiple objects in a scene. We present a spatio-temporal feature encoding pairwise interactions between each object in the scene. By the use of a kernel representation we embed object interactions in a vector space which allows us to define a metric comparing interactions of different temporal extent. Using this metric we define a probabilistic model which allows us to represent and extract the affordances of individual objects based on the structure of their interaction. In this paper we focus on the presented pairwise relationships but the model can naturally be extended to incorporate additional cues related to a single object or multiple objects. We compare our approach with traditional kernel approaches and show a significant improvement.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Spatio-temporal graphs are a popular general tool for representing high-level spatio-temporal structures.
“Spatio-temporal graphs (st-graphs) are a popular [39,37,4,11,32,65,22] general tool for representing such high-level spatio-temporal structures.”
From Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2016) · cites this paper as [65]
Cites
The Ecological Approach to Visual Perception
Bornstein, M. H., & Gibson, J. J. (1980) · The Journal of Aesthetics and Art Criticism · doi:10.2307/429816 · Cited by 22,935 · see citing works
Dynamic Programming Algorithm Optimization for Spoken Word Recognition
Sakoe, H., & Chiba, S. (1978) · IEEE Transactions on Acoustics Speech and Signal Processing · doi:10.1109/tassp.1978.1163055 · Cited by 6,727 · see citing works
Harris, Z. S. (1954) · <i>WORD</i> · doi:10.1080/00437956.1954.11659520 · Cited by 4,807 · see citing works
An Introduction to Support Vector Machines
Cristianini, N., & Shawe-Taylor, J. S. (2000) · ePrints Soton (University of Southampton) · Cited by 4,115 · see citing works
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908 · see citing works
Bar, M. (2004) · Nature Reviews Neuroscience · doi:10.1038/nrn1476 · Cited by 1,613 · see citing works
Pyramidal implementation of the Lucas Kanade feature tracker description of the algorithm
Bouguet, J.-Y. (1999) · Cited by 1,557 · see citing works
Is imitation learning the route to humanoid robots?
Schaal, S. (1999) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(99)01327-3 · Cited by 1,325 · see citing works
The role of context in object recognition
Oliva, A., & Torralba, A. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.009 · Cited by 1,063 · see citing works
Robotic Grasping of Novel Objects using Vision
Saxena, A., Driemeyer, J., & Ng, A. Y. (2008) · The International Journal of Robotics Research · doi:10.1177/0278364907087172 · Cited by 950 · see citing works
Contextual Priming for Object Detection
Torralba, A. B. (2003) · International Journal of Computer Vision · doi:10.1023/a:1023052124951 · Cited by 777 · see citing works
Detecting activities of daily living in first-person camera views
Pirsiavash, H., & Ramanan, D. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248010 · Cited by 666 · see citing works
Markov fields on finite graphs and lattices
Hammersley, J. M., & Clifford, P. (1971) · Cited by 579 · see citing works
Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition
Gupta, A. K., Kembhavi, A., & Davis, L. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.83 · Cited by 494 · see citing works
Learning Object Affordances: From Sensory--Motor Coordination to Imitation
Montesano, L., Lopes, M. C., Bernardino, A. J. M., & Santos-Victor, J. (2008) · IEEE Transactions on Robotics · doi:10.1109/tro.2007.914848 · Cited by 356 · see citing works
A Scalable Approach to Activity Recognition based on Object Use
Wu, J., Osuntogun, A., Choudhury, T., Philipose, M., & Rehg, J. M. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408865 · Cited by 328 · see citing works
Exploiting human actions and object context for recognition tasks
Moore, D., Essa, I. A., & Hayes, M. H., III (1999) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.1999.791201 · Cited by 281 · see citing works
Visual object-action recognition: Inferring object affordances from human demonstration
Kjellström, H., Romero, J., & Kragić, D. (2010) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2010.08.002 · Cited by 226 · see citing works
Gräbner, H., Gall, J., & Gool, L. V. (2011) · doi:10.1109/cvpr.2011.5995327 · Cited by 224 · see citing works
From 3D scene geometry to human workspace
Gupta, A., Satkin, S., Efros, A. A., & Hebert, M. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995448 · Cited by 220 · see citing works
Recognition by Functional Parts
Rivlin, E., Dickinson, S., & Rosenfeld, A. (1995) · Computer Vision and Image Understanding · doi:10.1006/cviu.1995.1048 · Cited by 137 · see citing works
Context models and out-of-context objects
Choi, M. J., Torralba, A. B., & Willsky, A. S. (2012) · Pattern Recognition Letters · doi:10.1016/j.patrec.2011.12.004 · Cited by 126 · see citing works
Learning relational affordance models for robots in multi-object manipulation tasks
Moldovan, B., Moreno, P., van Otterlo, M., Santos-Victor, J., & Raedt, L. D. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225042 · Cited by 108 · see citing works
Integrated Contextual Representation for Objects' Identities and Their Locations
Gronau, N., Neta, M., & Bar, M. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20027 · Cited by 107 · see citing works
Functional Object Class Detection Based on Learned Affordance Cues
Stark, M. M., Lies, P., Zillich, M., Wyatt, J., & Schiele, B. (2008) · Springer eBooks · doi:10.1007/978-3-540-79547-6_42 · Cited by 95 · see citing works
Categorizing object-action relations from semantic scene graphs
Aksoy, E. E., Abramov, A., Wörgötter, F., & Dellen, B. K. (2010) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2010.5509319 · Cited by 80 · see citing works
Beyond active noun tagging: Modeling contextual interactions for multi-class active learning
Siddiquie, B., & Gupta, A. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540044 · Cited by 72 · see citing works
Confidence-based cue integration for visual place recognition
Pronob, A., & Caputo, B. (2007) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2007.4399493 · Cited by 70 · see citing works
Peursum, P., WEST, G. A. W., & Venkatesh, S. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.57 · Cited by 60 · see citing works
Functional categorization of objects using real-time markerless motion capture
Gall, J., Fossati, A., & Gool, L. V. (2011) · doi:10.1109/cvpr.2011.5995582 · Cited by 58 · see citing works
From object categories to grasp transfer using probabilistic reasoning
Madry, M., Song, D., & Kragić, D. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6225052 · Cited by 50 · see citing works
Unsupervised Learning of Functional Categories in Video Scenes
Turek, M. W., Hoogs, A. J., & Collins, R. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15552-9_48 · Cited by 45 · see citing works
Recognizing primitive interactions by exploring actor-object states
Filipovych, R., & Ribeiro, E. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587726 · Cited by 44 · see citing works
Functional object descriptors for human activity modeling
Pieropan, A., Ek, C. H., & Kjellström, H. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630736 · Cited by 43 · see citing works
Self-citationImproving generalization for 3D object categorization with Global Structure Histograms
Madry, M., Ek, C. H., Detry, R. J., et al. (2012) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2012.6385874 · Cited by 35 · see citing works
Self-citationGrasping familiar objects using shape context
Bohg, J., & Kragić, D. (2009) · Cited by 34 · see citing works
Non-probabilistic cellular automata-enhanced stereo vision simultaneous localization and mapping
Nalpantidis, L., Sirakoulis, G. C., & Gasteratos, A. C. (2011) · Measurement Science and Technology · doi:10.1088/0957-0233/22/11/114027 · Cited by 30 · see citing works
Generic object recognition using form & function
Stark, L., & Bowyer, K. W. (1996) · Medical Entomology and Zoology · Cited by 29 · see citing works
Generic Object Recognition Using Form and Function
Stark, L., & Bowyer, K. W. (1996) · Series in machine perception and artificial intelligence · doi:10.1142/2123 · Cited by 23 · see citing works
Learning visual object definitions by observing human activities
Veloso, M., Hundelshausen, F. V., & Rybski, P. E. (2006) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/ichr.2005.1573560 · Cited by 16 · see citing works
Representing actions with kernels
Luo, G., Bergström, N., Ek, C. H., et al. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094567 · Cited by 14 · see citing works
Self-citationConfidence-based cue integration for visual place recognition
Pronobis, A., Caputo, B., Pronobis, A., et al. (2007) · Cited by 8 · see citing works
Baisero, A., Pokorny, F. T., Kragić, D., et al. (2013) · International Conference on Pattern Recognition Applications and Methods · doi:10.5220/0004267300500057 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133 · see citing works
- Authors
- Pieropan, A., Ek, C. H., & Kjellstrom, H.
- Affiliations
- KTH Royal Institute of Technology (SE)
- ORCID
- Ek, Carl Henrik
- Venue
- 2014 IEEE-RAS International Conference on Humanoid Robots
- Publisher
- IEEE
- Published
- 2014-11
- Volume
- 10
- Pages
- 52-58
- OpenAlex
- W2061017280
- Cited by
- 16
- References
- 43
- Citations by year
- 2023: 2 · 2021: 1 · 2019: 4 · 2018: 1 · 2017: 2 · 2016: 4 · 2015: 2
- FWCI
- 3.69
- Type
- conference paper
- Language
- en
- Open access
- green
- urn.kb.se
- Licence
- other-oa
- Locations
- IEEE-RAS International Conference on Humanoid Robots (Humanoids) (publishedVersion)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- Bristol Research (University of Bristol) (submittedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- object affordances, kernel representation, probabilistic models, interaction metrics, scene understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
