The DOI matches its Crossref record; 68 of the 69 references it cites were found in a registry or an index.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Open accessAbstract
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models are publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The beginning and end of videos, especially tutorial videos (Miech et al., 2019), often consist of an intro, respectively an outro. These sections are not as related to the content of the videos and contain fewer movements and actions.”
From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Miech et al., 2019
Experimental — accuracy not yet measuredIn the citing paper
“More modalities.In addition to the general modalities of vision, text, and audio, further diverse modalities are emerging, e.g., Pano-AVQA [144] -the first large-scale spatial and audio-visual question answering dataset on 360 • videos, YouTube-360 (YT-360) [145] (360 • videos), AIST++ [146] (a new multimodal dataset of 3D dance motion and music), Artemis [147] (affective language for visual arts)”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [144]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00685 · Cited by 2,346
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,888
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,869
Research works printed but not linked: 4
- radford, 2018, Improving Language Understandingby Generative Pre-Training, Preprint
- 2019, Project webpage
- radford, 2019, Language models are unsupervised multitask learners, Preprint
- damen, 2018, Scaling egocentric vision: The epic-kitchens dataset, ECCV
Cited by
Sorted by citations
Items 201–300 of 990
Deep learning for video-text retrieval: a review
Zhu, C., Jia, Q. F., Chen, W. R., Guo, Y., & Liu, Y. (2023) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-023-00267-8 · Cited by 47
Sharma, D., Dhiman, C., & Kumar, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119773 · Cited by 43
Chrupała, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12967 · Cited by 42
A Survey on Video Moment Localization
Liu, M., Nie, L., Wang, Y., Wang, M., & Rui, Y. (2022) · ACM Computing Surveys · doi:10.1145/3556537 · Cited by 37
End-to-End Learning of Visual Representations from Uncurated Instructional VideosPreprint
Miech, A., Alayrac, J.-B., Smaira, L., Laptev, I., Šivic, J., & Zisserman, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06430 · Cited by 47
Self-citationProcedure-Aware Pretraining for Instructional Video Understanding
Zhou, H., Martín-Martín, R., Kapadia, M., Savarese, S., & Niebles, J. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01033 · Cited by 37
Video Background Music Generation: Dataset, Method and Evaluation
Zhuo, L., Wang, Z., Wang, B., Liao, Y., Bao, C., Peng, S., Han, S., Zhang, A., Fang, F., & Liu, S. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01433 · Cited by 36
Self-supervised Multi-task Procedure Learning from Instructional Videos
Elhamifar, E., & Huynh, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_33 · Cited by 36
Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods
Islam, S., Dash, A., Seum, A., Raj, A. H., Hossain, T., & Shah, F. M. (2021) · SN Computer Science · doi:10.1007/s42979-021-00487-x · Cited by 43
Enhancing Self-supervised Video Representation Learning via Multi-level Feature Optimization
Qian, R., Li, Y., Liu, H., See, J., Ding, S., Liu, X., Li, D., & Lin, W. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00789 · Cited by 35
Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training
Lei, C., Luo, S., Liu, Y., He, W., Wang, J., Wang, G., Tang, H., Miao, C., & Li, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475431 · Cited by 39
Gate-Shift-Fuse for Video Action Recognition
Sudhakaran, S., Escalera, S., & Lanz, O. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3268134 · Cited by 35
CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval
Pei, R., Liu, J., Li, W., Shao, B., Xu, S., Dai, P., Lu, J., & Yan, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01820 · Cited by 43
Egocentric Audio-Visual Object Localization
Huang, C., Tian, Y., Kumar, A., & Xu, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02194 · Cited by 37
Shen, Y., Wang, L., & Elhamifar, E. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01002 · Cited by 35
CiCo: Domain-Aware Sign Language Retrieval via Cross-Lingual Contrastive Learning
Cheng, Y., Wei, F., Bao, J., Chen, D., & Zhang, W. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01823 · Cited by 43
A comprehensive overview of Generative AI (GAI): Technologies, applications, and challenges
Zhang, Z., Zhang, J., Zhang, X., & Mai, W. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129645 · Cited by 45
EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching
Shi, Y., Yang, X., Xu, H., Yuan, C., Li, B., Hu, W., & Zha, Z.-J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01740 · Cited by 38
Multimodal Conversational AI: A Survey of Datasets and Approaches
Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 42
MILES: Visual BERT Pre-training with Injected Language Semantics for Video-Text Retrieval
Ge, Y., Ge, Y., Liu, X., Wang, J., Wu, J., Shan, Y., Qie, X., & Luo, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_40 · Cited by 39
Sign Language Video Retrieval with Free-Form Textual Queries
Duarte, A., Albanie, S., Giró-i-Nieto, X., & Varol, G. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01370 · Cited by 36
X-VARS: Introducing Explainability in Football Refereeing with Multi-Modal Large Language Models
Held, J., Itani, H., Cioppa, A., Giancola, S., Ghanem, B., & Droogenbroeck, M. V. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00332 · Cited by 46
LITA: Language Instructed Temporal-Localization Assistant
Huang, D.-A., Liao, S., Radhakrishnan, S., Yin, H., Molchanov, P. A., Yu, Z., & Kautz, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73039-9_12 · Cited by 89
Transferring Image-CLIP to Video-Text Retrieval via Temporal Relations
Fang, H., Xiong, P., Xu, L., & Luo, W. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3227416 · Cited by 33
i-Code: An Integrative and Composable Multimodal Learning Framework
Yang, Z., Fang, Y., Zhu, C., Pryzant, R., Chen, D., Shi, Y., Yi‐chong, X., Qian, Y., Gao, M., Chen, Y., et al. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i9.26290 · Cited by 45
MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models
Ko, D., Choi, J., Choi, H. K., On, K.-W., Roh, B., & Kim, H. J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01925 · Cited by 38
SSAN: Separable Self-Attention Network for Video Representation Learning
Guo, X., Guo, X., & Lu, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01243 · Cited by 35
Multimodal Video Summarization via Time-Aware Transformers
Shang, X., Yuan, Z., Wang, A., & Wang, C. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475321 · Cited by 31
Drop-DTW: Aligning Common Signal Between Sequences While Dropping OutliersPreprint
Dvornik, N., Hadji, I., Derpanis, K. G., Garg, A., & Jepson, A. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.11996 · Cited by 35
Generative Prompt Model for Weakly Supervised Object Localization
Zhao, Y., Ye, Q., Wu, W., Shen, C., & Wan, F. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00584 · Cited by 35
Learning from Inside: Self-driven Siamese Sampling and Reasoning for Video Question Answering
Yu, W., Zheng, H., Li, M., Ji, L., Wu, L., Xiao, N., & Duan, N. (2021) · neural information processing systems · Cited by 30
Dual Learning with Dynamic Knowledge Distillation for Partially Relevant Video Retrieval
Dong, J., Zhang, M., Zhang, Z., Chen, X., Liu, D., Qu, X., Wang, X., & Liu, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01038 · Cited by 30
Learning Spatiotemporal Features via Video and Text Pair DiscriminationPreprint
Li, T., & Wang, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.05691 · Cited by 43
QVHighlights: Detecting Moments and Highlights in Videos via Natural Language QueriesPreprint
Lei, J., Berg, T. L., & Bansal, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09609 · Cited by 36
Amato, G. M., Bolettieri, P., Carrara, F., Debole, F., Falchi, F., Gennaro, C., Vadicamo, L., & Vairo, C. (2021) · Journal of Imaging · doi:10.3390/jimaging7050076 · Cited by 29
Chadha, A. R., Arora, G., & Kaloty, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07735 · Cited by 36
Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddings
Wray, M., Csurka, G., Larlus, D., & Damen, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00054 · Cited by 28
Joint Multimedia Event Extraction from Video and Article
Chen, B., Lin, X., Thomas, C., Li, M., Yoshida, S., Chum, L., Ji, H., & Chang, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.8 · Cited by 36
M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
Dong, X. M., Zhan, X., Wu, Y., Wei, Y., Kampffmeyer, M., Wei, X.-Y., Lu, M., Wang, Y., & Liang, X. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02057 · Cited by 36
Abstractive Summarization of Spoken and Written Instructions with BERT
Savelieva, A., Au-Yeung, B., & Ramani, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09676 · Cited by 32
Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations
Zhong, Y., Yu, L., Bai, Y., Li, S.-W. D., Yan, X., & Li, Y. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01424 · Cited by 37
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
Lee, S., Chung, J., Yu, Y., Kim, G., Breuel, T. M., Chechik, G., & Song, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01011 · Cited by 34
A comprehensive review of the video-to-text problem
Perez-Martin, J., Bustos, B., Guimarães, S. J. F., Sipiran, I., Pérez, J. E. P., & Said, G. C. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10104-1 · Cited by 36
Learning to Answer Visual Questions From Web Videos
Yang, A., Miech, A., Šivic, J., Laptev, I., & Schmid, C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3173208 · Cited by 36
Self-citationCross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
Wang, Y., Dong, J., Liang, T., Zhang, M., Cai, R., & Wang, X. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548003 · Cited by 32
Test of Time: Instilling Video-Language Models with a Sense of Time
Bagad, P., Tapaswi, M., & Snoek, C. G. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00247 · Cited by 31
Self-citationVideo Understanding as Machine TranslationPreprint
Korbar, B., Petroni, F., Girdhar, R., & Torresani, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07203 · Cited by 30
XKD: Cross-Modal Knowledge Distillation with Domain Alignment for Video Representation Learning
Sarkar, P., & Etemad, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i13.29407 · Cited by 36
CLIP2TV: An Empirical Study on Transformer-based Methods for Video-Text RetrievalPreprint
Gao, Z., Liu, J., Chen, S., Chang, D., Zhang, H., & Yuan, J. (2021) · arXiv (Cornell University) · Cited by 26
Multi-level Alignment Network for Domain Adaptive Cross-modal Retrieval
Dong, J., Long, Z., Mao, X., Lin, C., He, Y., & Ji, S. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.01.114 · Cited by 25
Weakly-Supervised Action Segmentation and Alignment via Transcript-Aware Union-of-Subspaces Learning
Lu, Z., & Elhamifar, E. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00798 · Cited by 25
Masking Modalities for Cross-modal Video Retrieval
Gabeur, V., Nagrani, A., Sun, C., Alahari, K., & Schmid, C. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00217 · Cited by 27
Multimodal neural networks better explain multivoxel patterns in the hippocampus
Choksi, B., Mozafari, M., VanRullen, R., & Reddy, L. (2022) · Neural Networks · doi:10.1016/j.neunet.2022.07.033 · Cited by 28
Alsharid, M., Cai, Y., Sharma, H., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102630 · Cited by 25
P3IV: Probabilistic Procedure Planning from Instructional Videos with Weak Supervision
Zhao, H., Hadji, I., Dvornik, N., Derpanis, K. G., Wildes, R. P., & Jepson, A. D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00295 · Cited by 25
Set-Supervised Action Learning in Procedural Task Videos via Pairwise Order Consistency
Lu, Z., & Elhamifar, E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01928 · Cited by 25
Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos
Souček, T., Alayrac, J.-B., Miech, A., Laptev, I., & Šivic, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01357 · Cited by 25
Self-citationVideo Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
Xu, Y., Cao, H., Xie, L.-H., Li, X.-L., Chen, Z., & Yang, J. (2022) · ACM Computing Surveys · doi:10.1145/3679010 · Cited by 35
Learning to Segment Actions from Observation and Narration
Fried, D., Alayrac, J.-B., Blunsom, P., Dyer, C., Clark, S. C., & Nematzadeh, A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.231 · Cited by 25
Self-citationRetrieval-Augmented Egocentric Video Captioning
Xu, J., Huang, Y., Hou, J., Chen, G., Zhang, Y., Feng, R., & Xie, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01284 · Cited by 42
Feature generation for long-tail classification
Vigneswaran, R., Law, M. T., Balasubramanian, V. N., & Tapaswi, M. (2021) · Indian Conference on Computer Vision, Graphics and Image Processing · doi:10.1145/3490035.3490300 · Cited by 25
Self-citationA Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval
Falcon, A., Serra, G., & Lanz, O. (2022) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3503161.3548365 · Cited by 25
MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration
Hayes, T., Zhang, S., Yin, X. P., Pang, G., Sheng, S., Yang, H. C., Ge, S., Hu, Q., & Parikh, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20074-8_25 · Cited by 28
Duan, H., Zhao, N., Chen, K., & Lin, D.-H. (2022) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr52688.2022.00301 · Cited by 28
Wang, Z., Wu, Y., Narasimhan, K., & Russakovsky, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19781-9_14 · Cited by 26
AutoAD III: The Prequel - Back to the Pixels
Han, T., Bain, M., Nagrani, A., Varol, G., Xie, W., & Zisserman, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01720 · Cited by 31
Learning Video Representations from Textual Web SupervisionPreprint
Stroud, J., Lu, Z., Sun, C., Deng, J., Sukthankar, R., Schmid, C., & Ross, D. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.14937 · Cited by 45
WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-Grained Spatial-Temporal Understanding
Kong, Q., Kawana, Y., Saini, R., Kumar, A., Pan, J., Gu, T., Ozao, Y., Opra, B., Sato, Y., & Kobori, N. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73116-7_1 · Cited by 29
Video-aided Unsupervised Grammar Induction
Zhang, S., Song, L., Jin, L., Xu, K., Yu, D., & Luo, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.119 · Cited by 23
CONQUER: Contextual query-aware ranking for video corpus moment retrieval
HOU, Z., Ngo, C., & Chan, W. K. (2021) · Singapore Management University Institutional Knowledge (InK) (Singapore Management University) · Cited by 23
Ji, K., Liu, J., Hong, W., Zhong, L., Wang, J., Chen, J., & Chu, W. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531960 · Cited by 23
Sports Video Analysis on Large-Scale Data
Wu, D., Zhao, H., Bao, X., & Wildes, R. P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_2 · Cited by 25
TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency
Narasimhan, M., Nagrani, A., Sun, C., Rubinstein, M., Darrell, T., Rohrbach, A., & Schmid, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19830-4_31 · Cited by 23
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models
Pan, J., Lin, Z., Ge, Y., Zhu, X., Zhang, R., Wang, Y., Qiao, Y., & Li, H. (2023) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw60793.2023.00035 · Cited by 35
Building an Open-Vocabulary Video CLIP Model With Better Architectures, Optimization and Data
Wu, Z., Weng, Z., Peng, W., Yang, X., Li, A., Davis, L. S., & Jiang, Y. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3357503 · Cited by 27
Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning
Sun, L., Xu, X., Wu, M., & Xie, W. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681472 · Cited by 42
Özsoy, E., Pellegrini, C., Czempiel, T., Tristram, F., Yuan, K., Bani-Harouni, D., Eck, U., Busam, B., Keicher, M., & Navab, N. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01805 · Cited by 22
VSRNet: End-to-end video segment retrieval with text query
Sun, X. N., Long, X., He, D., Wen, S., & Lian, Z. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108027 · Cited by 22
Exposing the Limits of Video-Text Models through Contrast Sets
Park, J. S., Shen, S., Farhadi, A., Darrell, T., Choi, Y., & Rohrbach, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.261 · Cited by 24
Video-Text Representation Learning via Differentiable Weak Temporal Alignment
Ko, D., Choi, J., Ko, J., Noh, S., On, K.-W., Kim, E., & Kim, H. J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00496 · Cited by 22
StepFormer: Self-Supervised Step Discovery and Localization in Instructional Videos
Dvornik, N., Hadji, I., Zhang, R., Derpanis, K. G., Wildes, R. P., & Jepson, A. D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01817 · Cited by 23
Controllable augmentations for video representation learning
Qian, R., Lin, W., See, J., & Li, D. (2024) · Visual Intelligence · doi:10.1007/s44267-023-00034-7 · Cited by 23
Learning to Predict Activity Progress by Self-Supervised Video Alignment
Donahue, G., & Elhamifar, E. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01766 · Cited by 27
Learning Actionness via Long-Range Temporal Order Verification
Zhukov, D., Alayrac, J.-B., Laptev, I., & Šivic, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58526-6_28 · Cited by 22
Self-citationImproving Video Retrieval by Adaptive Margin
He, F., Wang, Q., Feng, Z., Jiang, W., Lü, Y., Zhu, Y. T., & Tan, X. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3462927 · Cited by 22
Condensed Movies: Story Based Retrieval with Contextual Embeddings
Bain, M., Nagrani, A., Brown, A., & Zisserman, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-69541-5_28 · Cited by 21
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning
Patrick, M., Huang, P.-Y., Misra, I., Metze, F., Vedaldi, A., Asano, Y. M., & Henriques, J. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01039 · Cited by 28
NaQ: Leveraging Narrations as Queries to Supervise Episodic Memory
Ramakrishnan, S. K., Al-Halah, Z., & Grauman, K. L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00647 · Cited by 21
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 28
AViD Dataset: Anonymized Videos from Diverse CountriesPreprint
Piergiovanni, A., & Ryoo, M. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.05515 · Cited by 22
GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval
Wang, Y., Wang, J., Chen, B., Zeng, Z., & Xia, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i6.28389 · Cited by 20
Toward Robust Multimodal Sentiment Analysis using multimodal foundational models
Zhao, X., Poria, S., Li, X., Chen, Y., & Tang, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126974 · Cited by 28
A Benchmark for Structured Procedural Knowledge Extraction from Cooking Videos
Xu, F. F., Ji, L., Shi, B., Du, J., Neubig, G., Bisk, Y., & Duan, N. (2020) · doi:10.18653/v1/2020.nlpbt-1.4 · Cited by 26
Visual Goal-Step Inference using wikiHow
Yang, Y., Panagopoulou, A., Lyu, Q., Zhang, L., Yatskar, M., & Callison-Burch, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.165 · Cited by 39
SViTT: Temporal Learning of Sparse Video-Text Transformers
Li, Y., Min, K., Tripathi, S., & Vasconcelos, N. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01814 · Cited by 20
ProTéGé: Untrimmed Pretraining for Video Temporal Grounding by Video Temporal Grounding
Wang, L., Mittal, G. K., Sajeev, S., Yu, Y., Hall, M., Boddeti, V. N., & Chen, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00636 · Cited by 21
Dual-View Curricular Optimal Transport for Cross-Lingual Cross-Modal Retrieval
Wang, Y., Wang, S., Luo, H., Dong, J., Wang, F., Han, M., Wang, X., & Wang, M. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3365248 · Cited by 25
Structure-Aware Procedural Text Generation From an Image Sequence
Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., Yamakata, Y., & Mori, S. (2020) · IEEE Access · doi:10.1109/access.2020.3043452 · Cited by 21
Progressive Semantic Matching for Video-Text Retrieval
Liu, H., Luo, R., Shang, F., Niu, M., & Liu, Y. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475621 · Cited by 20
RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search
Chen, M., Zhang, K., He, Z., Jing, Y., & Wang, X. S. (2024) · Proceedings of the VLDB Endowment · doi:10.14778/3681954.3681959 · Cited by 36
Metadata
- Authors
- Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Institut national de recherche en sciences et technologies du numérique (FR)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 2630-2640
- OpenAlex
- W2948859046
- Cited by
- 986
- References
- 69
- Citations by year
- 2026: 77 · 2025: 138 · 2024: 170 · 2023: 213 · 2022: 161 · 2021: 155 · 2020: 65 · 2019: 7
- FWCI
- 31.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- European Regional Development Fund (02.1.01/0.0/0.0, CZ.02, CZ.02.1.01, CZ.02.1.01/0.0, CZ.02.1.01/0.0/0.0)
- FP7 Ideas: European Research Council (336845)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video-text retrieval, action localization, domain transfer, large-scale video datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
