Video Object Grounding Using Semantic Roles in Language Description
Open accessAbstract
We explore the task of Video Object Grounding (VOG), which grounds objects in videos referred to in natural language descriptions. Previous methods apply image grounding based algorithms to address VOG, fail to explore the object relation information and suffer from limited generalization. Here, we investigate the role of object relations in VOG and propose a novel framework VOGNet to encode multi-modal object relations via self-attention with relative position encoding. To evaluate VOGNet, we propose novel contrasting sampling methods to generate more challenging grounding input samples, and construct a new dataset called ActivityNet-SRL (ASRL) based on existing caption and grounding datasets. Experiments on ASRL validate the need of encoding object relations in VOG, and our VOGNet outperforms competitive baselines by a significant margin.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A large amount of image-language pretraining Transformer models are pretrained on large-scale image-text pairs.
“For instance, large amount of imagelanguage pretraining Transformer models are pretrained on large-scale image-text pairs, e.g., VisualBERT [104], VL-BERT [105], ViLBERT [102], LXMERT [103], UNITER [106].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [104]
Common early fusion based MML Transformer models are also known as one-stream architecture, allowing adoption of the merits of BERT due to minimal architectural modification.
“Common early fusion based MML Transformer models [7], [104], [108] are also known as one-stream architecture, allowing the adoption of the merits of BERT due to minimal architectural modification.”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [104]
An intuitive strategy for MML is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector.
“An intuitive strategy is to leverage semantic parts (e.g., objects) pre-extracted by an off-the-shelf detector for MML [103], [104], [105], [106], [112], [204], [310].”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [104]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
(2017) · Cited by 11,625 · see citing works
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S. J., & Zhang, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00636 · Cited by 5,117 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., & Gool, L. V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_2 · Cited by 4,164 · see citing works
The Kinetics Human Action Video DatasetPreprint
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 3,711 · see citing works
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-4009 · Cited by 3,538 · see citing works
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578 · see citing works
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505 · see citing works
The Proposition Bank: An Annotated Corpus of Semantic Roles
Palmer, M. S., Gildea, D., & Kingsbury, P. (2005) · Computational Linguistics · doi:10.1162/0891201053630264 · Cited by 2,317 · see citing works
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · see citing works
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444 · see citing works
A simple neural network module for relational reasoning
Santoro, A., Raposo, D., Barrett, D. G. T., Malinowski, M., Pascanu, R., Battaglia, P., & Lillicrap, T. (2017) · arXiv · doi:10.48550/arxiv.1706.01427 · Cited by 1,281 · see citing works
Dense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,279 · see citing works
AllenNLP: A Deep Semantic Natural Language Processing Platform
Gardner, M., Grus, J., Neumann, M., Tafjord, O., Dasigi, P., Liu, N. F., Peters, M. E., Schmitz, M., & Zettlemoyer, L. (2018) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/w18-2501 · Cited by 1,273 · see citing works
Generation and Comprehension of Unambiguous Object Descriptions
Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A., & Murphy, K. P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.9 · Cited by 1,211 · see citing works
YouTube-8M: A Large-Scale Video Classification BenchmarkPreprint
Abu-El-Haija, S., Kothari, N., Lee, J., Natsev, P., Toderici, G. D., Varadarajan, B., & Vijayanarasimhan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08675 · Cited by 1,135 · see citing works
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., & Gupta, A. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46448-0_31 · Cited by 1,098 · see citing works
Neural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00611 · Cited by 1,095 · see citing works
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · see citing works
(2016) · doi:10.1007/978-3-319-46475-6_5 · Cited by 1,081 · see citing works
Graph R-CNN for Scene Graph Generation
Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_41 · Cited by 889 · see citing works
Fillmore, C. J. (2003) · International Journal of Lexicography · doi:10.1093/ijl/16.3.235 · Cited by 875 · see citing works
Scaling Egocentric Vision: The "Equation missing" Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_44 · Cited by 875 · see citing works
MAttNet: Modular Attention Network for Referring Expression Comprehension
Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., & Berg, T. L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00142 · Cited by 857 · see citing works
BSN: Boundary Sensitive Network for Temporal Action Proposal Generation
Lin, T., Zhao, X., Su, H., Wang, C., & Yang, M. Z. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01225-0_1 · Cited by 795 · see citing works
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 1× · see citing works
“It is a building block for downstream grounded vision+language tasks such as Grounded-VQA [14,31,32,74,77], Grounded-Captioning [35–37 In this work, we address the task of Video Object Grounding (VOG): given a video and its natural language description we aim to localize each referred object.”
MethodFrom Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · see citing works
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 712 · see citing works
Detecting and Recognizing Human-Object Interactions
Gkioxari, G., Girshick, R., Dollár, P., & He, K. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00872 · Cited by 624 · see citing works
Learning to Detect Human-Object Interactions
Chao, Y.-W., Liu, Y., Liu, X., Zeng, H., & Deng, J. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00048 · Cited by 559 · see citing works
TVQA: Localized, Compositional Video Question Answering
Lei, J., Yu, L., Bansal, M., & Berg, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1167 · Cited by 541 · cited 3× · see citing works
“It is a building block for downstream grounded vision+language tasks such as Grounded-VQA [14,31,32,74,77], Grounded-Captioning [35–37 In this work, we address the task of Video Object Grounding (VOG): given a video and its natural language description we aim to localize each referred object.”
MethodScene Graph Generation from Objects, Phrases and Region Captions
Li, Y., Ouyang, W., Zhou, B., Wang, K., & Wang, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.142 · Cited by 518 · see citing works
Grounding of Textual Phrases in Images by Reconstruction
Rohrbach, A., Rohrbach, M., Hu, R., Darrell, T., & Schiele, B. (2016) · Max Planck Digital Library · Cited by 422 · see citing works
Lu, J., Yang, J., Batra, D., & Parikh, D. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00754 · Cited by 412 · see citing works
Linguistically-Informed Self-Attention for Semantic Role Labeling
Strubell, E., Verga, P. W., Andor, D., Weiss, D., & McCallum, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1548 · Cited by 412 · see citing works
(2013) · Iris (University of Trento) · Cited by 360 · see citing works
Simple BERT Models for Relation Extraction and Semantic Role LabelingPreprint
Shi, P., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05255 · Cited by 325 · see citing works
HICO: A Benchmark for Recognizing Human-Object Interactions in Images
Chao, Y.-W., Wang, Z., He, Y., Wang, J., & Deng, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.122 · Cited by 300 · see citing works
Graphical Contrastive Losses for Scene Graph Parsing
Zhang, J., Shih, K. J., Elgammal, A., Tao, A., & Catanzaro, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01180 · Cited by 257 · see citing works
Situation Recognition: Visual Semantic Role Labeling for Image Understanding
Yatskar, M., Zettlemoyer, L., & Farhadi, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.597 · Cited by 253 · see citing works
Motion-Appearance Co-memory Networks for Video Question Answering
Gao, J., Ge, R., Chen, K., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00688 · Cited by 252 · see citing works
Self-citationVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsPreprint
Krishna, R., Zhu, Y., Groth, O., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07332 · Cited by 248 · see citing works
Actor-Centric Relation Network
Sun, C., Shrivastava, A., Vondrick, C., Murphy, K. P., Sukthankar, R., & Schmid, C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_20 · Cited by 235 · see citing works
iCAN: Instance-Centric Attention Network for Human-Object Interaction DetectionPreprint
Gao, C., Zou, Y., & Huang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10437 · Cited by 225 · see citing works
Pixels to Graphs by Associative EmbeddingPreprint
Newell, A., & Deng, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.07365 · Cited by 204 · see citing works
Zhou, L., Kalantidis, Y., Chen, X., Corso, J. J., & Rohrbach, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00674 · Cited by 191 · see citing works
CTAP: Complementary Temporal Action Proposal Generation
Gao, J., Chen, K., & Nevatia, R. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01216-8_5 · Cited by 186 · see citing works
Self-citationVideo Object Segmentation with Language Referring Expressions
Khoreva, A., Rohrbach, A., & Schiele, B. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_8 · Cited by 180 · see citing works
Zero-Shot Grounding of Objects From Natural Language Queries
Sadhu, A., Chen, K., & Nevatia, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00479 · Cited by 161 · see citing works
Self-citationVideo Visual Relation Detection
Shang, X., Ren, T., Guo, J., Zhang, H., & Chua, T. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123380 · Cited by 147 · see citing works
Generation and Comprehension of Unambiguous Object DescriptionsPreprint
Mao, J., Huang, J., Toshev, A., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02283 · Cited by 127 · see citing works
Query-Guided Regression Network with Context Policy for Phrase Grounding
Chen, K., Kovvuri, R., & Nevatia, R. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.95 · Cited by 123 · see citing works
Self-citationConditional Image-Text Embedding Networks
Plummer, B. A., Kordas, P., Kiapour, M. H., Zheng, S., Piramuthu, R., & Lazebnik, S. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01258-8_16 · Cited by 121 · see citing works
Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
Hu, R.-H., Fried, D., Rohrbach, A., Klein, D., Darrell, T., & Saenko, K. (2019) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1655 · Cited by 114 · see citing works
Weakly-Supervised Video Object Grounding from Text by Loss Weighting and Object InteractionPreprint
Zhou, L., Louis, N., & Corso, J. (2018) · British Machine Vision Conference · doi:10.48550/arxiv.1805.02834 · Cited by 111 · see citing works
Krishna, R., Chami, I., Bernstein, M. S., & Fei-Fei, L. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00718 · Cited by 98 · see citing works
Video Relationship Reasoning Using Gated Spatio-Temporal Energy Graph
Tsai, Y.-H. H., Divvala, S., Morency, L., Salakhutdinov, R., & Farhadi, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01067 · Cited by 98 · see citing works
Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video
Chen, Z., Ma, L., Luo, W., & Wong, K.-Y. K. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1183 · Cited by 96 · see citing works
Object Level Visual Reasoning in VideosPreprint
Baradel, F., Neverova, N., Wolf, C., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06157 · Cited by 86 · see citing works
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
Huang, D.-A., Buch, S., Dery, L. M., Garg, A., Fei-Fei, L., & Niebles, J. C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00623 · Cited by 85 · see citing works
Automatic Generation of Grounded Visual Questions
Zhang, S., Qu, L., You, S., Yang, Z., & Zhang, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/592 · Cited by 82 · see citing works
Imagine This! Scripts to Compositions to Videos
Gupta, T., Schwenk, D., Farhadi, A., Hoiem, D., & Kembhavi, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_37 · Cited by 73 · see citing works
Commonly Uncommon: Semantic Sparsity in Situation Recognition
Yatskar, M., Ordóñez, V., Zettlemoyer, L., & Farhadi, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.671 · Cited by 42 · see citing works
Describing Common Human Visual Actions in Images
Ronchi, M. R., & Perona, P. (2015) · British Machine Vision Conference (BMVC) · doi:10.5244/c.29.52 · Cited by 32 · see citing works
TVQA+: Spatio-Temporal Grounding for Video Question AnsweringPreprint
Lei, J., Yu, L., Berg, T. L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11574 · Cited by 25 · see citing works
Neural Baby TalkPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2018) · arXiv (Cornell University) · Cited by 23 · see citing works
Describing Common Human Visual Actions in ImagesPreprint
Ronchi, M. R., & Perona, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.02203 · Cited by 22 · see citing works
Referring RelationshipsPreprint
Krishna, R., Chami, I., Bernstein, M. S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.10362 · Cited by 18 · see citing works
Grounding Semantic Roles in Images
Silberer, C., & Pinkal, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1282 · Cited by 17 · see citing works
Grounded Objects and Interactions for Video CaptioningPreprint
Ma, C., Kadav, A., Melvin, I., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06354 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- [9][9] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV , 2018. 11 -
- [16][16] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In CVPR , 2019. 2 -
- [36][36] Ruotian Luo and Gregory Shakhnarovich. Comprehensionguided referring expressions. CVPR , 2017. 1 -
- [68][68] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV , 2016. 1, 2 -
- [78][78] Bohan Zhuang, Qi Wu, Chunhua Shen, Ian Reid, and Anton van den Hengel. Hcvrd: a benchmark for large-scale humancentered visual relationship detection. In AAAI , 2018. 2
Cited by
Sorted by citations
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072 · see citing works
- Authors
- Sadhu, A., Chen, K., & Nevatia, R.
- Affiliations
- University of Southern California (US)
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 10414-10424
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3014018982
- Cited by
- 36
- References
- 79
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3 · 2023: 8 · 2022: 9 · 2021: 10 · 2020: 1
- FWCI
- 2.16
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Office of Naval Research (N00014-18, N00014)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- thematic roles, object relations, self-attention, relative position encoding, multimodal grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
