The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval
Open accessAbstract
Video-text retrieval has been a crucial and fundamental task in multi-modal research. The development of video-text retrieval has been considerably promoted by large-scale multi-modal contrastive pre-training, which primarily focuses on coarse-grained or fine-grained contrast. However, cross-grained contrast, which is the contrast between coarse-grained representations and fine-grained representations, has rarely been explored in prior research. Compared with fine-grained or coarse-grained contrasts, cross-grained contrast calculate the correlation between coarse-grained features and each fine-grained feature, and is able to filter out the unnecessary fine-grained features guided by the coarse-grained feature during similarity calculation, thus improving the accuracy of retrieval. To this end, this paper presents a novel multi-grained contrastive model, namely X-CLIP, for video-text retrieval. However, another challenge lies in the similarity aggregation problem, which aims to aggregate fine-grained and cross-grained similarity matrices to instance-level similarity. To address this challenge, we propose the Attention Over Similarity Matrix (AOSM) module to make the model focus on the contrast between essential frames and words, thus lowering the impact of unnecessary frames and words on retrieval results. With multi-grained contrast and the proposed AOSM module, X-CLIP achieves outstanding performance on five widely-used video-text retrieval datasets, including MSR-VTT (49.3 [email protected]), MSVD (50.4 [email protected]), LSMDC (26.1 [email protected]), DiDeMo (47.8 [email protected]) and ActivityNet (46.2 [email protected]).
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
An Empirical Study of Training Self-Supervised Vision Transformers
Chen, X., Xie, S., & He, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00950 · Cited by 1,648
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,542
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Khattab, O., & Zaharia, M. (2020) · Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3397271.3401075 · Cited by 1,301
VideoBERT: A Joint Model for Video and Language Representation Learning
Sun, C., Myers, A., Vondrick, C., Murphy, K., & Schmid, C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00756 · Cited by 1,172
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974 · cited 1×
“The tokenizer used in the experiment is lower-cased byte pair encoding (BPE) [48] with a 49,152 vocab size.”
MethodLocalizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 958
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
Multi-modal Transformer for Video Retrieval
Gabeur, V., Sun, C., Alahari, K., & Schmid, C. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58548-8_13 · Cited by 615
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., & Liu, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00725 · Cited by 589
End-to-End Learning of Visual Representations From Uncurated Instructional Videos
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Sivic, J., & Zisserman, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00990 · Cited by 580
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.149 · Cited by 488
ActBERT: Learning Global-Local Video-Text Representations
Zhu, L., & Yang, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00877 · Cited by 405
A Joint Sequence Fusion Model for Video Question Answering and Retrieval
Yu, Y., Kim, J., & Kim, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_29 · Cited by 343
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words
Zhang, X., Sun, X., Luo, Y., Ji, J., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01521 · Cited by 275
Self-citationHierarchical Conditional Relation Networks for Video Question Answering
Le, T. M., Le, V., Venkatesh, S., & Tran, T. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00999 · Cited by 260
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval
Mithun, N. C., Li, J., Metze, F., & Roy-Chowdhury, A. K. (2018) · Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval · doi:10.1145/3206025.3206064 · Cited by 250
Dual Encoding for Zero-Example Video Retrieval
Dong, J., Li, X., Xu, C., Ji, S., He, Y., Yang, G., & Wang, X. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00957 · Cited by 236
End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering
Yu, Y., Ko, H., Choi, J., & Kim, G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.347 · Cited by 219
MDMMT: Multidomain Multimodal Transformer for Video Retrieval
Dzabraev, M., Kalashnikov, M., Komkov, S. A., & Petiushko, A. (2021) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw53098.2021.00374 · Cited by 198
T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
Wang, X., Zhu, L., & Yang, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00504 · Cited by 193
Ji, J., Luo, Y., Sun, X., Chen, F., Luo, G., Wu, Y., Gao, Y., & Ji, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16258 · Cited by 183
Self-citationTeachText: CrossModal Generalized Distillation for Text-Video Retrieval
Croitoru, I., Bogolin, S.-V., Leordeanu, M., Jin, H., Zisserman, A., Albanie, S., & Liu, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01138 · Cited by 179
Cross-Modal and Hierarchical Modeling of Video and Text
Zhang, B., Hu, H., & Sha, F. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_23 · Cited by 173
A Straightforward Framework for Video Retrieval Using CLIP
Portillo-Quintero, J. A., Ortíz-Bayliss, J. C., & Terashima‐Marín, H. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77004-4_1 · Cited by 161
Tree-Augmented Cross-Modal Encoding for Complex-Query Video Retrieval
Yang, X., Dong, J., Cao, Y., Wang, X., Wang, M., & Chua, T. (2020) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3397271.3401151 · Cited by 135
The Long-Short Story of Movie Description
Rohrbach, A., Rohrbach, M., & Schiele, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24947-6_17 · Cited by 126
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
Yang, J., Bisk, Y., & Gao, J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01136 · Cited by 115
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
Xu, H., Yan, M., Li, C., Bi, B., Huang, S., Xiao, W., & Huang, F. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.42 · Cited by 84 · cited 1×
“With the success of self-supervised pre-training such as BERT [12] in NLP, vision-language pre-training on large-scale unlabeled cross-modal data has attracted growing attention [23, 31–33, 37, 44, 50, 51, 55, 60].”
BackgroundSelf-citationRelevance-guided Supervision for OpenQA with ColBERT
Khattab, O., Potts, C., & Zaharia, M. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00405 · Cited by 82
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval
Bain, M., Nagrani, A., Varol, G., & Zisserman, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00175 · Cited by 76
Knowing What to Learn: A Metric-Oriented Focal Mechanism for Image Captioning
Ji, J., Ma, Y., Sun, X., Zhou, Y., Wu, Y., & Ji, R. (2022) · IEEE Transactions on Image Processing · doi:10.1109/tip.2022.3183434 · Cited by 59
Self-citationKnowing What it is: Semantic-Enhanced Dual Attention Transformer
Ma, Y., Ji, J., Sun, X., Zhou, Y., Wu, Y., Huang, F., & Ji, R. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3164787 · Cited by 43
Self-citationImproving Video Retrieval by Adaptive Margin
He, F., Wang, Q., Feng, Z., Jiang, W., Lü, Y., Zhu, Y. T., & Tan, X. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462927 · Cited by 27
Cited by
CLIP-Driven Fine-Grained Text-Image Person Re-Identification
Yan, S., Dong, N., Zhang, L., & Tang, J. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3327924 · Cited by 330
Krasilnikov, M., Nikitin, M., & Konushin, A. (2025) · The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences · doi:10.5194/isprs-archives-xlviii-2-w9-2025-169-2025 · Cited by 2
Liu, S., Xu, A., & Hou, Z. (2026) · Sustainability · doi:10.3390/su18031612
Haresamudram, H., Tang, C. I., Suh, S., Lukowicz, P., & Ploetz, T. (2024) · arXiv · Cited by 31
“…through and with Foundational Models Another approach for incorporating knowledge from foundational models is via alignment between data from modalities, e.g., images/videos [120, 150, 194], speech and audio [48, 66, 189, 190], or sensor data [74, 192], with natural language descriptions the data.”
Metadata
- Authors
- Ma, Y., Xu, G., Sun, X., Yan, M., Zhang, J., & Ji, R.
- Venue
- Proceedings of the 30th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2022-10-10
- Pages
- 638-647
- OpenAlex
- W4304014690
- Cited by
- 321
- References
- 44
- FWCI
- 23.27
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.