The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Attentive Moment Retrieval in Videos
Abstract
In the past few years, language-based video retrieval has attracted a lot of attention. However, as a natural extension, localizing the specific video moments within a video given a description query is seldom explored. Although these two tasks look similar, the latter is more challenging due to two main reasons: 1) The former task only needs to judge whether the query occurs in a video and returns an entire video, but the latter is expected to judge which moment within a video matches the query and accurately returns the start and end points of the moment. Due to the fact that different moments in a video have varying durations and diverse spatial-temporal characteristics, uncovering the underlying moments is highly challenging. 2) As for the key component of relevance estimation, the former usually embeds a video and the query into a common space to compute the relevance score. However, the later task concerns moment localization where not only the features of a specific moment matter, but the context information of the moment also contributes a lot. For example, the query may contain temporal constraint words, such as "first'', therefore need temporal context to properly comprehend them. To address these issues, we develop an Attentive Cross-Modal Retrieval Network. In particular, we design a memory attention mechanism to emphasize the visual features mentioned in the query and simultaneously incorporate their context. In the light of this, we obtain the augmented moment representation. Meanwhile, a cross-modal fusion sub-network learns both the intra-modality and inter-modality dynamics, which can enhance the learning of moment-query representation. We evaluate our method on two datasets: DiDeMo and TACoS. Extensive experiments show the effectiveness of our model as compared to the state-of-the-art methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Neural Collaborative Filtering
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T.-S. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052569 · Cited by 6,652
Self-citationDeep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Samy Bengio, A., Dean, J., Ranzato, M., & Mikolov, T. (2013) · Cited by 2,308
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
Self-citationMSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
Neural Factorization Machines for Sparse Predictive Analytics
He, X., & Chua, T.-S. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080777 · Cited by 1,340
Self-citationTemporal Action Localization in Untrimmed Videos via Multi-stage CNNs
Shou, Z., Wang, D., & Chang, S.-F. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.119 · Cited by 968
Localizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 958
TALL: Temporal Activity Localization via Language Query
Gao, J., Sun, C., Yang, Z., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.563 · Cited by 842
Grounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 777
Visual Translation Embedding Network for Visual Relation Detection
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.331 · Cited by 586
Self-citationNatural Language Object Retrieval
Hu, R., Xu, H., Rohrbach, M., Feng, J., Saenko, K., & Darrell, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.493 · Cited by 532
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Gao, J., Yang, Z., Sun, C., Chen, K., & Nevatia, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.392 · Cited by 498
Grounding Action Descriptions in Videos
Regneri, M., Rohrbach, M., Wetzel, D., Thater, S., Schiele, B., & Pinkal, M. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00207 · Cited by 467
A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461 · cited 1×
“And Singh et al. [30] presented a multi-stream bi-directional RNN network for fine-grained action detection.”
BackgroundVideo Question Answering via Gradually Refined Attention over Appearance and Motion
Xu, D., Zhao, Z., Xiao, J., Wu, F., Zhang, H., He, X., & Zhuang, Y. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123427 · Cited by 429
Self-citationAttention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401
Xu, R., Xiong, C., Chen, W., & Corso, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9512 · Cited by 298
Unsupervised Learning from Narrated Instruction Videos
Alayrac, J.-B., Bojanowski, P., Agrawal, N., Sivic, J., Laptev, I., & Lacoste-Julien, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.495 · Cited by 278
Item Silk Road: Recommending Items from Information Domains to Social Users
Wang, X., He, X., Nie, L., & Chua, T.-S. (2017) · Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3077136.3080771 · Cited by 209
Self-citationTEM: Tree-enhanced Embedding Model for Explainable Recommendation
Wang, X., He, X., Feng, F., Nie, L., & Chua, T.-S. (2018) · Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW '18 · doi:10.1145/3178876.3186066 · Cited by 201
Self-citationScript Data for Attribute-Based Recognition of Composite Activities
Rohrbach, M., Regneri, M., Andriluka, M., Amin, S., Pinkal, M., & Schiele, B. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33718-5_11 · Cited by 185
NeuroStylist: Neural Compatibility Modeling for Clothing Matching
Song, X., Feng, F., Liu, J., Li, Z., Nie, L., & Ma, J. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123314 · Cited by 177
Self-citationLearning Multimodal Attention LSTM Networks for Video Captioning
Xu, J., Yao, T., Zhang, Y., & Mei, T. (2017) · Proceedings of the 25th ACM international conference on Multimedia · doi:10.1145/3123266.3123448 · Cited by 175
Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
Song, J., Gao, L., Guo, Z., Liu, W., Zhang, D., & Shen, H. T. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/381 · Cited by 165
Visual Semantic Search: Retrieving Videos via Complex Textual Queries
Lin, D., Fidler, S., Kong, C., & Urtasun, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.340 · Cited by 158
Item Silk Road: Recommending Items from Information Domains to Social UsersPreprint
Wang, X., He, X., Nie, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03205 · Cited by 149
Self-citationActom sequence models for efficient action detection
Gaidon, A., Harchaoui, Z., & Schmid, C. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995646 · Cited by 145
Video Question Answering via Attribute-Augmented Attention Network Learning
Ye, Y., Zhao, Z., Li, Y., Chen, L., Xiao, J., & Zhuang, Y. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080655 · Cited by 105
Cross-Platform App Recommendation by Jointly Modeling Ratings and Texts
Cao, D., He, X., Nie, L., Wei, X., Hu, X., Wu, S., & Chua, T. (2017) · ACM Transactions on Information Systems · doi:10.1145/3017429 · Cited by 104
Self-citationVideo Question Answering via Hierarchical Spatio-Temporal Attention Networks
Zhao, Z., Yang, Q., Cai, D., He, X., & Zhuang, Y. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/492 · Cited by 104
Self-citationLearning Joint Representations of Videos and Sentences with Web Image Search
Otani, M., Nakashima, Y., Rahtu, E., Heikkilä, J., & Yokoya, N. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46604-0_46 · Cited by 81
Towards Micro-video Understanding by Joint Sequential-Sparse Modeling
Liu, M., Nie, L., Wang, M., & Chen, B. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123341 · Cited by 74
Self-citationVideo Question Answering via Hierarchical Dual-Level Attention Network Learning
Zhao, Z., Lin, J., Jiang, X., Cai, D., He, X., & Zhuang, Y. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123364 · Cited by 44
Self-citationTowards surveillance video search by natural language query
Tellex, S., & Roy, D. K. (2009) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1646396.1646442 · Cited by 42
Learning on Partial-Order Hypergraphs
Feng, F., He, X., Liu, Y., Nie, L., & Chua, T. (2018) · International World Wide Web Conference (WWW) · doi:10.1145/3178876.3186064 · Cited by 41
Self-citationComputational Social Indicators
Feng, F., Nie, L., Wang, X., Hong, R., & Chua, T. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080773 · Cited by 21
Self-citation
Cited by
Yuan, Y., Mei, T., & Zhu, W. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019159 · Cited by 335
Local-Global Video-Text Interactions for Temporal Grounding
Mun, J., Cho, M., & Han, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01082 · Cited by 303
Span-based Localizing Network for Natural Language Video Localization
Zhang, H., Sun, A., Jing, W., & Zhou, J. T. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.585 · Cited by 297
Dense Regression Network for Video Grounding
Zeng, R., Xu, H., Huang, W., Chen, P., Tan, M., & Gan, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01030 · Cited by 286
Read and Attend: Temporal Localisation in Sign Language Videos
Varol, G., Momeni, L., Albanie, S., Afouras, T., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01658 · Cited by 39
“Cross-modal attention has been employed in the literature for various localisation problems such as visual grounding in videos [13, 29, 48, 50] or images [17, 49], keyword spotting in audio [38] or visual speech [30, 41] and audio-visual sound source localisation [3, 22, 37].”
Metadata
- Authors
- Liu, M., Wang, X., Nie, L., He, X., Chen, B., & Chua, T.-S.
- Venue
- The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval
- Publisher
- ACM
- Published
- 2018-06-27
- Pages
- 15-24
- OpenAlex
- W2798354744
- Cited by
- 307
- References
- 43
- FWCI
- 11.00
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.