Objects2action: Classifying and Localizing Actions without Any Video Example
Open accessAbstract
The goal of this paper is to recognize actions in video without the need for examples. Different from traditional zero-shot approaches we do not demand the design and specification of attribute classifiers and class-to-attribute mappings to allow for transfer from seen classes to unseen classes. Our key contribution is objects2action, a semantic word embedding that is spanned by a skip-gram model of thousands of object categories. Action labels are assigned to an object encoding of unseen video based on a convex combination of action and object affinities. Our semantic embedding has three main characteristics to accommodate for the specifics of actions. First, we propose a mechanism to exploit multiple-word descriptions of actions and objects. Second, we incorporate the automated selection of the most responsive objects per action. And finally, we demonstrate how to extend our zero-shot approach to the spatio-temporal localization of actions in video. Experiments on four action datasets demonstrate the potential of our approach.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Modeling the same st-graph with S-RNN yields superior results to spatio-temporal CRF on CAD-120.
“In this experiment, we show that modeling the same st-graph with S-RNN yields superior results.”
From Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2016) · cites this paper as [29]
S-RNN significantly improves over Koppula et al. on both anticipation and detection.
“S-RNN significantly improves over Koppula et al. on both anticipation [31] and detection [29].”
From Structural-RNN: Deep Learning on Spatio-Temporal Graphs (2016) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Distributed Representations of Sentences and DocumentsPreprint
Le, Q. V., & Mikolov, T. (2014) · International Conference on Machine Learning · doi:10.48550/arxiv.1405.4053 · Cited by 10,307 · see citing works
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · see citing works
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · see citing works
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965 · see citing works
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407 · see citing works
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,309 · see citing works
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206594 · Cited by 1,849 · see citing works
Learning to detect unseen object classes by between-class attribute transfer
Lampert, C. H., Nickisch, H., & Stefan Harmeling, B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvprw.2009.5206594 · Cited by 1,563 · see citing works
Image Classification with the Fisher Vector: Theory and Practice
Sánchez, J., Perronnin, F., Mensink, T., & Verbeek, J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0636-x · Cited by 1,498 · see citing works
Self-citationAction MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition
Rodriguez, M. D., Ahmed, J., & Shah, M. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587727 · Cited by 1,258 · see citing works
Zero-Shot Learning Through Cross-Modal Transfer
Socher, R., Ganjoo, M., Sridhar, H., Bastani, O., Manning, C. D., & Ng, A. Y. (2013) · arXiv · doi:10.48550/arxiv.1301.3666 · Cited by 997 · see citing works
Parikh, D., & Grauman, K. (2011) · International Conference on Computer Vision · doi:10.1109/iccv.2011.6126281 · Cited by 901 · see citing works
Zero-Shot Learning by Convex Combination of Semantic EmbeddingsPreprint
Norouzi, M., Mikolov, T., Samy Bengio, A., Singer, Y., Shlens, J., Frome, A., Corrado, G. S., & Dean, J. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.5650 · Cited by 890 · see citing works
Action bank: A high-level representation of activity in video
Sadanand, S., & Corso, J. J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247806 · Cited by 696 · see citing works
Label-Embedding for Attribute-Based Classification
Akata, Z., Perronnin, F., Harchaoui, Z., & Schmid, C. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.111 · Cited by 639 · see citing works
Recognizing human actions by attributes
Liu, J., Kuipers, B. J., & Savarese, S. (2011) · doi:10.1109/cvpr.2011.5995353 · Cited by 589 · see citing works
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R. J., Darrell, T., & Saenko, K. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.337 · Cited by 500 · see citing works
A discriminative CNN video representation for event detection
Xu, Z., Yang, Y., & Hauptmann, A. G. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298789 · Cited by 438 · see citing works
Better Exploiting Motion for Better Action Recognition
Jain, M., Jeǵou, H., & Bouthémy, P. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.330 · Cited by 385 · see citing works
Self-citationAction Recognition with Stacked Fisher Vectors
Peng, X., Zou, C., Qiao, Y., & Peng, Q. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_38 · Cited by 347 · see citing works
Evaluating knowledge transfer and zero-shot learning in a large-scale setting
Rohrbach, M., Stark, M. M., & Schiele, B. (2011) · doi:10.1109/cvpr.2011.5995627 · Cited by 309 · see citing works
Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions
Elhoseiny, M., Saleh, B., & Elgammal, A. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.321 · Cited by 263 · see citing works
Action Localization with Tubelets from Motion
Jain, M., van Gemert, J. C., Jeǵou, H., Bouthémy, P., & Snoek, C. G. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.100 · Cited by 250 · see citing works
Self-citationThe New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247 · see citing works
Spatiotemporal Deformable Part Models for Action Detection
Tian, Y., Sukthankar, R., & Shah, M. A. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.341 · Cited by 243 · see citing works
Discriminative figure-centric models for joint action localization and recognition
Lan, T., Wang, Y., & Mori, G. (2011) · International Conference on Computer Vision · doi:10.1109/iccv.2011.6126472 · Cited by 231 · see citing works
Cross-dataset action detection
Cao, L., Liu, Z., & Huang, T. S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539875 · Cited by 213 · see citing works
COSTA: Co-Occurrence Statistics for Zero-Shot Classification
Mensink, T., Gavves, E., & Snoek, C. G. M. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.313 · Cited by 194 · see citing works
Self-citationSpatio-temporal Object Detection Proposals
Oneaţă, D., Revaud, J., Verbeek, J. J., & Schmid, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10578-9_48 · Cited by 184 · see citing works
What do 15,000 object categories tell us about classifying and localizing actions?
Jain, M., van Gemert, J. C., & Snoek, C. G. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298599 · Cited by 181 · see citing works
Self-citationAPT: Action localization proposals from dense trajectories
van Gemert, J. C., Jain, M., Gati, E., & Snoek, C. G. M. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.177 · Cited by 143 · see citing works
Self-citationObject Bank: An Object-Level Image Representation for High-Level Visual Recognition
Li, L.-J., Su, H., Lim, Y., & Fei-Fei, L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0660-x · Cited by 130 · see citing works
Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts
Wu, S., Bondugula, S., Florian Luisier, A., Zhuang, X., & Natarajan, P. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.341 · Cited by 113 · see citing works
Evaluating Neural Word Representations in Tensor-Based Compositional Settings
Milajevs, D., Kartsaklis, D., Sadrzadeh, M., & Purver, M. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1079 · Cited by 99 · see citing works
Composite Concept Discovery for Zero-Shot Video Event Detection
Habibian, A., Mensink, T., & Snoek, C. G. M. (2014) · International Conference on Multimedia Retrieval · doi:10.1145/2578726.2578746 · Cited by 72 · see citing works
Self-citationEvaluation of Color Spatio-Temporal Interest Points for Human Action Recognition
Everts, I., van Gemert, J. C., & Gevers, T. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2302677 · Cited by 69 · see citing works
Self-citationYFCC100M: The New Data in Multimedia Research
Thomee, B., Shamma, D. A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv · doi:10.48550/arxiv.1503.01817 · Cited by 69 · see citing works
Event-Driven Semantic Concept Discovery by Exploiting Weakly Tagged Internet Images
Chen, J., Cui, Y., Ye, G., Liu, D., & Chang, S. (2014) · International Conference on Multimedia Retrieval · doi:10.1145/2578726.2578729 · Cited by 65 · see citing works
Zero-Example Event Search using MultiModal Pseudo Relevance Feedback
Jiang, L., Mitamura, T., Yu, S.-I., & Hauptmann, A. G. (2014) · International Conference on Multimedia Retrieval · doi:10.1145/2578726.2578764 · Cited by 61 · see citing works
Boosting VLAD with Supervised Dictionary Learning and High-Order Statistics
Peng, X., Wang, L., Qiao, Y., & Peng, Q. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10578-9_43 · Cited by 61 · see citing works
MediaMill at TRECVID 2014: Searching Concepts, Objects, Instances and Events in Video
Snoek, C. G. M., van de Sande, K. E. A., Fontijne, D., Cappallo, S., van Gemert, J. C., Habibian, A., Mensink, T., Mettes, P., Ran, T., Koelma, D. C., et al. (2014) · UvA-DARE (University of Amsterdam) · Cited by 30 · see citing works
Self-citationSemantic Aware Video Transcription Using Random Forest Classifiers
Sun, C., & Nevatia, R. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10590-1_50 · Cited by 15 · see citing works
Textual Similarity with a Bag-of-Embedded-Words Model
Clinchant, S., & Perronnin, F. (2013) · doi:10.1145/2499178.2499180 · Cited by 11 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [1][1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Labelembedding for attribute-based classification. In CVPR , 2013. 1, 2 -
- [31][31] P. Over, G. Awad, J. Fiscus, and G. Sanders. Trecvid 2013-an introduction to the goals, tasks, data, evaluation mechanisms, and metrics. In TRECVID Workshop , 2013. 2, 7 -
Cited by
Sorted by citations
Structural-RNN: Deep Learning on Spatio-Temporal Graphs
Jain, A., Zamir, A. R., Savarese, S., & Saxena, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.573 · Cited by 1,133 · see citing works
A Survey of Zero-Shot Learning: Settings, Methods, and Applications
Wang, W., Zheng, V. W., Yu, H., & Miao, C. (2019) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3293318 · Cited by 528 · see citing works
Event-based media processing and analysis: A survey of the literature
Tzelepis, C., Ma, Z., Mezaris, V., Ionescu, B., Kompatsiaris, I., Boato, G., Sebe, N., & Yan, S. (2016) · Image and Vision Computing · doi:10.1016/j.imavis.2016.05.005 · Cited by 49 · see citing works
- Authors
- Jain, M., Gemert, J. C. V., Mensink, T., & Snoek, C. G. M.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- University of Amsterdam (NL)
- Venue
- 2015 IEEE International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2015-12
- Pages
- 4588-4596
- OpenAlex
- W2100771357
- Cited by
- 142
- References
- 49
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3 · 2023: 11 · 2022: 11 · 2021: 21 · 2020: 22 · 2019: 14 · 2018: 12 · 2017: 23 · 2016: 20 · 2015: 1
- FWCI
- 7.66
- Type
- conference paper
- Language
- en
- Open access
- green
- dare.uva.nl
- Licence
- other-oa
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- UvA-DARE (University of Amsterdam) (publishedVersion, other-oa)
- arXiv (Cornell University) (submittedVersion)
- UvA-DARE (University of Amsterdam) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Keywords
- zero-shot action recognition, action localization, semantic word embeddings, skip-gram model, object categories, spatiotemporal localization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
