The DOI matches its Crossref record; 68 of the 69 references it cites were found in a registry or an index.
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Open accessAbstract
Learning text-video embeddings usually requires a dataset of video clips with manually provided captions. However, such datasets are expensive and time consuming to create and therefore difficult to obtain on a large scale. In this work, we propose instead to learn such embeddings from video data with readily available natural language annotations in the form of automatically transcribed narrations. The contributions of this work are three-fold. First, we introduce HowTo100M: a large-scale dataset of 136 million video clips sourced from 1.22M narrated instructional web videos depicting humans performing and describing over 23k different visual tasks. Our data collection procedure is fast, scalable and does not require any additional manual annotation. Second, we demonstrate that a text-video embedding trained on this data leads to state-of-the-art results for text-to-video retrieval and action localization on instructional video datasets such as YouCook2 or CrossTask. Finally, we show that this embedding transfers well to other domains: fine-tuning on generic Youtube videos (MSR-VTT dataset) and movies (LSMDC dataset) outperforms models trained on these datasets alone. Our dataset, code and models are publicly available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The beginning and end of videos, especially tutorial videos (Miech et al., 2019), often consist of an intro, respectively an outro. These sections are not as related to the content of the videos and contain fewer movements and actions.”
From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Miech et al., 2019
Experimental — accuracy not yet measuredIn the citing paper
“More modalities.In addition to the general modalities of vision, text, and audio, further diverse modalities are emerging, e.g., Pano-AVQA [144] -the first large-scale spatial and audio-visual question answering dataset on 360 • videos, YouTube-360 (YT-360) [145] (360 • videos), AIST++ [146] (a new multimodal dataset of 3D dance motion and music), Artemis [147] (affective language for visual arts)”
From Multimodal learning with transformers: A survey (2023) · cites this paper as [144]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
Hara, K., Kataoka, H., & Satoh, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00685 · Cited by 2,346
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.571 · Cited by 1,888
Image Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,869
Flagged first, then by citations
1 of the 69 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- radford, 2018, Improving Language Understandingby Generative Pre-Training, Preprint
- 2019, Project webpage
- radford, 2019, Language models are unsupervised multitask learners, Preprint
- damen, 2018, Scaling egocentric vision: The epic-kitchens dataset, ECCV
Cited by
Sorted by citations
Items 301–400 of 990
PREGO: Online Mistake Detection in PRocedural EGOcentric Videos
Flaborea, A., di Melendugno, G. M. D., Plini, L., Scofano, L., Matteis, E. D., Furnari, A., Farinella, G. M., & Galasso, F. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01749 · Cited by 31
Mind-the-Gap! Unsupervised Domain Adaptation for Text-Video Retrieval
Chen, Q., Liu, Y., & Albanie, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16192 · Cited by 19
Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment
Wang, W., Zhang, M., Chen, R., Cai, G., Zhou, P., Peng, P., Guo, X., Wu, J., & Sun, X. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/154 · Cited by 19
Visual Consensus Modeling for Video-Text Retrieval
Cao, S., Wang, B., Zhang, W., & Ma, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19891 · Cited by 19
Transferring Knowledge From Text to Video: Zero-Shot Anticipation for Procedural Actions
Sener, F., Saraf, R., & Yao, A. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3218596 · Cited by 20
Tencent Text-Video Retrieval: Hierarchical Cross-Modal Interactions With Multi-Level Representations
Jiang, J., Min, S., Kong, W., Wang, H., Li, Z., & Liu, W. (2022) · IEEE Access · doi:10.1109/access.2022.3227973 · Cited by 26
End-to-End Pre-Training With Hierarchical Matching and Momentum Contrast for Text-Video Retrieval
Shen, W., Song, J., Zhu, X., Li, G., & Shen, H. T. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3275071 · Cited by 19
ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding
Aganian, D., Stephan, B., Eisenbach, M., Stretz, C., & Gross, H. (2023) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra48891.2023.10160633 · Cited by 31
EventTransAct: A Video Transformer-Based Framework for Event-Camera Based Action Recognition
de Blegiers, T., Dave, I. R., Yousaf, A., & Shah, M. A. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341740 · Cited by 20
A Simple Recipe for Contrastively Pre-Training Video-First Encoders Beyond 16 Frames
Papalampidi, P., Koppula, S., Pathak, S., Chiu, J., Heyward, J., Pătrăucean, V., Shen, J., Miech, A., Zisserman, A., & Nematzdeh, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01364 · Cited by 29
Self-citationHowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Shvetsova, N., Kukleva, A., Hong, X., Rupprecht, C., Schiele, B., & Kuehne, H. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72992-8_1 · Cited by 34
Koala: Key Frame-Conditioned Long Video-LLM
Tan, R., Sun, X., Hu, P., Wang, J.-H., Deilamsalehy, H., Plummer, B. A., Russell, B. C., & Saenko, K. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01289 · Cited by 25
Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
Li, S., Du, Y., Torralba, A. B., Šivic, J., & Russell, B. C. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00186 · Cited by 18
Ghoddoosian, R., Sayed, S., & Athitsos, V. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00020 · Cited by 18
Almeida, A., de Villiers, J. P., Freitas, A. D., & Velayudan, M. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.116335 · Cited by 20
A Review of Multi-Modal Learning from the Text-Guided Visual Processing Viewpoint
Ullah, U., Lee, J.-S., An, C.-H., Lee, H., Park, S.-Y., Baek, R., & Choi, H. (2022) · Sensors · doi:10.3390/s22186816 · Cited by 19
Weakly Supervised Video Representation Learning with Unaligned Text for Sequential Videos
Dong, S., Hu, H., Lian, D., Luo, W., Qian, Y., & Gao, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00241 · Cited by 18
PDPP: Projected Diffusion for Procedure Planning in Instructional Videos
Wang, H., Wu, Y., Guo, S., & Wang, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01425 · Cited by 28
Keyword-Aware Relative Spatio-Temporal Graph Networks for Video Question Answering
Cheng, Y., Fan, H., Lin, D., Sun, Y., Kankanhalli, M., & Lim, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3345172 · Cited by 20
CoVR-2: Automatic Data Construction for Composed Video Retrieval
Ventura, L., Yang, A., Schmid, C., & Varol, G. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3463799 · Cited by 31
Motion-Augmented Self-Training for Video Recognition at Smaller Scale
Gavrilyuk, K., Jain, M., Karmanov, I., & Snoek, C. G. M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01026 · Cited by 18
Multi-queue Momentum Contrast for Microvideo-Product Retrieval
Du, Y., Wei, Y., Ji, W., Liu, F., Luo, X., & Nie, L. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570405 · Cited by 18
Video Question Answering with Iterative Video-Text Co-tokenization
Piergiovanni, A., Morton, K., Kuo, W., Ryoo, M. S., & Angelova, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_5 · Cited by 18
Foundation Models for Video Understanding: A SurveyPreprint
Madan, N., Møgelmose, A., Modi, R., Rawat, Y. S., & Moeslund, T. B. (2024) · doi:10.36227/techrxiv.171769139.99464428/v1 · Cited by 24
Semi-Weakly-Supervised Learning of Complex Actions from Instructional Task Videos
Shen, Y., & Elhamifar, E. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00334 · Cited by 17
Learning Video-Conditioned Policies for Unseen Manipulation Tasks
Chane-Sane, E., Schmid, C., & Laptev, I. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10161336 · Cited by 17
Multimodal video retrieval with CLIP: a user study
Alpay, T., Magg, S., Broze, P., & Speck, D. (2023) · Information Retrieval · doi:10.1007/s10791-023-09425-2 · Cited by 22
Pretrained Language Models as Visual Planners for Human Assistance
Patel, D., Eghbal-zadeh, H., Kamra, N., Iuzzolino, M. L., Jain, U., & Desai, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01404 · Cited by 22
Learning Object State Changes in Videos: An Open-World Perspective
Xue, Z., Kumar, A., & Grauman, K. L. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01750 · Cited by 25
Adversarial Multi-Grained Embedding Network for Cross-Modal Text-Video Retrieval
Han, N., Chen, J., Zhang, H., Wang, H., & Chen, H. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3483381 · Cited by 17
Ego4D: Around the World in 3,600 Hours of Egocentric Video
Grauman, K. L., Westbury, A., Byrne, E., Cartillier, V., Chavis, Z. Q., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Kukreja, D., et al. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3381075 · Cited by 20
Hierarchical bi-directional conceptual interaction for text-video retrieval
Han, W., Niu, G., Zhou, M., & Zhang, X. (2024) · Multimedia Systems · doi:10.1007/s00530-024-01525-3 · Cited by 16
MPT: Multi-grained Prompt Tuning for Text-Video Retrieval
Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680839 · Cited by 16
MovieChat+: Question-Aware Sparse Memory for Long Video Question Answering
Song, E., Chai, W., Tian, Y., Hwang, J., Li, X., & Wang, G. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3604614 · Cited by 80
A Survey on Video Temporal Grounding With Multimodal Large Language Model
Wu, J., Liu, W., Liu, Y., Liu, M., Nie, L., Lin, Z., & Chen, C. W. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3615586 · Cited by 26
Learning English with Peppa Pig
Nikolaus, M., Alishahi, A., & Chrupała, G. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00498 · Cited by 20
Stay in Grid: Improving Video Captioning via Fully Grid-Level Representation
Tang, M., Wang, Z., Zeng, Z., Li, X., & Zhou, L. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3232634 · Cited by 17
How Do You Do It? Fine-Grained Action Understanding with Pseudo-Adverbs
Doughty, H., & Snoek, C. G. M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01346 · Cited by 16
Flow Graph to Video Grounding for Weakly-Supervised Multi-step Localization
Dvornik, N., Hadji, I., Pham, H., Bhatt, D., Martínez, B., Fazly, A., & Jepson, A. D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_19 · Cited by 16
PERCEIVER-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
Tang, Z., Cho, J., Lei, J., & Bansal, M. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00439 · Cited by 17
Beyond Instructions: A Taxonomy of Information Types in How-to Videos
Yang, S., Kwak, S., Lee, J., & Kim, J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581126 · Cited by 16
Woo, T., Park, W., Jeong, W., & Park, J. (2023) · Computers & Graphics · doi:10.1016/j.cag.2023.09.013 · Cited by 19
Helping Hands: An Object-Aware Ego-Centric Video Recognition Model
Zhang, C., Gupta, A., & Zisserman, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01278 · Cited by 20
Ump: Unified Modality-Aware Prompt Tuning for Text-Video Retrieval
Zhang, H., Zeng, P., Gao, L., Song, J., & Shen, H. T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3429192 · Cited by 18
Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities
Piergiovanni, A., Noble, I., Kim, D., Ryoo, M. S., Gomes, V., & Angelova, A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02531 · Cited by 17
Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting
Toering, M., Gatopoulos, I., Stol, M., & Hu, V. T. (2022) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv51458.2022.00092 · Cited by 16
Video Description: Datasets & Evaluation Metrics
Rafiq, M. T., Rafiq, G., & Choi, G. S. (2021) · IEEE Access · doi:10.1109/access.2021.3108565 · Cited by 16
Debiased Video-Text Retrieval via Soft Positive Sample Calibration
Zhang, H., Yang, Y., Qi, F., Qian, S., & Xu, C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3248873 · Cited by 16
Accommodating Audio Modality in CLIP for Multimodal Processing
Ruan, L., Hu, A., Song, Y., Zhang, L., Zheng, S., & Jin, Q. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i8.26153 · Cited by 15
Learning to Ground Instructional Articles in Videos through Narrations
Mavroudi, E., Afouras, T., & Torresani, L. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01395 · Cited by 15
Multi-modal Semantic Inconsistency Detection in Social Media News Posts
McCrae, S., Wang, K., & Zakhor, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98355-0_28 · Cited by 17
State-aware Video Procedural Captioning
Nishimura, T., Hashimoto, A., Ushiku, Y., Kameko, H., & Mori, S. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475322 · Cited by 15
Classification-Regression for Chart Comprehension
Levy, M., Ben‐Ari, R., & Lischinski, D. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20059-5_27 · Cited by 15
Weak Supervision and Referring Attention for Temporal-Textual Association LearningPreprint
Fang, Z., Kong, S., Wang, Z., Fowlkes, C. C., & Yang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.11747 · Cited by 15
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 26
About Time: Advances, Challenges, and Outlooks of Action Understanding
Stergiou, A., & Poppe, R. W. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02478-4 · Cited by 17
Towards Efficient Partially Relevant Video Retrieval With Active Moment Discovering
Song, P., Zhang, L., Lan, L., Chen, W., Guo, D., Yang, X., & Wang, M. (2025) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2025.3590937 · Cited by 14
CapERA: Captioning Events in Aerial Videos
Bashmal, L., Bazi, Y., Rahhal, M. M. A., Zuair, M., & Melgani, F. (2023) · Remote Sensing · doi:10.3390/rs15082139 · Cited by 18
Movie101: A New Movie Understanding Benchmark
Yue, Z., Zhang, Q., Hu, A., Zhang, L., Wang, Z., & Jin, Q. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.257 · Cited by 14
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
Lin, X., Tiwari, S., Huang, S., Li, M., Shou, M. Z., Ji, H., & Chang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01426 · Cited by 15
Chop & Learn: Recognizing and Generating Object-State Compositions
Saini, N., Wang, H., Swaminathan, A., Jayasundara, V., He, B., Gupta, K., & Shrivastava, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01852 · Cited by 15
GenHowTo: Learning to Generate Actions and State Transformations from Instructional Videos
Souček, T., Damen, D., Wray, M., Laptev, I., & Šivic, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00627 · Cited by 23
Distilling Vision-Language Models on Millions of Videos
Zhao, Y., Zhao, L., Zhou, X., Wu, J., Chu, C.-T., Miao, H., Schroff, F., Adam, H., Liu, T., Gong, B., et al. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01245 · Cited by 23
Video Question Answering with Procedural Programs
Choudhury, R., Niinuma, K., Kitani, K., & Jeni, L. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72920-1_18 · Cited by 19
Xu, Q., Yang, J., Zhang, F., Pan, J., Chen, K., Shen, Y., Zhou, A. C., & Du, X. (2025) · Proceedings of the ACM on Management of Data · doi:10.1145/3709743 · Cited by 16
Overview of the MedVidQA 2022 Shared Task on Medical Video Question-Answering
Gupta, D. K., & Demner‐Fushman, D. (2022) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2022.bionlp-1.25 · Cited by 14
BiC-Net: Learning Efficient Spatio-temporal Relation for Text-Video Retrieval
Han, N., YaWen, Z., Shi, C., Xiao, G., Chen, H., & Chen, J. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3627103 · Cited by 14
RareAct: A video dataset of unusual interactionsPreprint
Miech, A., Alayrac, J.-B., Ivan Laptev, A., Šivic, J., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01018 · Cited by 14
Self-citationLearning Object Manipulation Skills via Approximate State Estimation from Real VideosPreprint
Petrík, V., Tapaswi, M., Laptev, I., & Šivic, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.06813 · Cited by 14
Self-citationHuman Action Recognition from Various Data Modalities: A ReviewPreprint
Rahmani, H., Bennamoun, M., & Ke, Q. (2021) · doi:10.36227/techrxiv.13708270.v1 · Cited by 14
Temporal validity reassessment: commonsense reasoning about information obsoleteness
Hosokawa, T., Jatowt, A., & Sugiyama, K. (2024) · Discover Computing · doi:10.1007/s10791-024-09433-w · Cited by 13
Text-Video Retrieval With Global-LocalSemantic Consistent Learning
Zhang, H., Zeng, P., Gao, L., Song, J., Duan, Y., Lyu, X., & Shen, H. T. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3574925 · Cited by 15
OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation
Li, H., Xu, M., Zhan, Y., Mu, S., Li, J., Cheng, K., Chen, Y., Chen, T., Ye, M., Wang, J., et al. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00726 · Cited by 42
Characterizing Work-Life for Information Work on Mars
Linder, R., Hunter, C., McLemore, J., Dutta, S., Akbar, F., Grover, T., Breideband, T., Borghouts, J. W., Lu, Y., Mark, G. J., et al. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492859 · Cited by 14
Multimodal Intent Discovery from Livestream Videos
Maharana, A., Tran, Q., Dernoncourt, F., Yoon, S., Bui, T., Chang, W. H., & Bansal, M. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.36 · Cited by 13
Galanopoulos, D., & Mezaris, V. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25069-9_40 · Cited by 13
Language-Guided Audio-Visual Source Separation via Trimodal Consistency
Tan, R., Ray, A., Burns, A., Plummer, B. A., Salamon, J., Nieto, O., Russell, B. C., & Saenko, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01019 · Cited by 13
Promptable Game Models: Text-guided Game Simulation via Masked Diffusion Models
Menapace, W., Siarohin, A., Lathuilière, S., Achlioptas, P., Golyanik, V., Tulyakov, S., & Ricci, E. (2023) · ACM Transactions on Graphics · doi:10.1145/3635705 · Cited by 19
Event-Guided Procedure Planning from Instructional Videos with Text Supervision
Wang, A.-L., Lin, K.-Y., Du, J.-R., Meng, J., & Zheng, W. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01248 · Cited by 18
SlideSpeech: A Large Scale Slide-Enriched Audio-Visual Corpus
Wang, H., Yu, F., Shi, X., Wang, Y., Zhang, S. S., & Li, M. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448079 · Cited by 14
VIT-LENS: Towards Omni-modal Representations
Lei, W., Ge, Y., Yi, K., Zhang, J., Gao, D., Sun, D., Ge, Y., Shan, Y., & Shou, M. Z. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02516 · Cited by 21
FunQA: Towards Surprising Video Comprehension
Xie, B., Zhang, S., Zhou, Z., Li, B., Zhang, Y., Hessel, J., Yang, J., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73232-4_3 · Cited by 13
Learning Text-to-Video Retrieval from Image Captioning
Ventura, L., Schmid, C., & Varol, G. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02202-8 · Cited by 13
Two-level label recovery-based label embedding for multi-label classification with missing labels
Wang, Y., Zheng, W., Cheng, Y., & Zhao, D. (2020) · Applied Soft Computing · doi:10.1016/j.asoc.2020.106868 · Cited by 13
Multimodal Classification of Safety-Report Observations
Paraskevopoulos, G., Pistofidis, P., Banoutsos, G., Georgiou, E., & Katsouros, V. (2022) · Applied Sciences · doi:10.3390/app12125781 · Cited by 13
Cross-Lingual Cross-Modal Consolidation for Effective Multilingual Video Corpus Moment Retrieval
Liu, J., Yu, T., Peng, H., Sun, M., & Li, P. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.142 · Cited by 14
Consistent Intra-Video Contrastive Learning With Asynchronous Long-Term Memory Bank
Chen, Z., Lin, K.-Y., & Zheng, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3207174 · Cited by 13
Stargazer: An Interactive Camera Robot for Capturing How-To Videos Based on Subtle Instructor Cues
Li, J., Sousa, M., Mahadevan, K., Wang, B. K., Aoyagui, P. A., Yu, N., Yang, A., Balakrishnan, R., Tang, A., & Grossman, T. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580896 · Cited by 17
Self-Supervised Object Detection and Retrieval Using Unlabeled Videos
Amrani, E., Ben‐Ari, R., Shapira, I., Hakim, T., & Bronstein, A. M. (2020) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00485 · Cited by 14
Multimodal Pretraining for Dense Video CaptioningPreprint
Huang, G., Pang, B., Zhu, Z., Rivera, C. E., & Soricut, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.11760 · Cited by 44
Speech2Action: Cross-Modal Supervision for Action Recognition
Nagrani, A., Sun, C., Ross, D., Sukthankar, R., Schmid, C., & Zisserman, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01033 · Cited by 12
Learning Interactive Real-World Simulators
Yang, S., Du, Y., Ghasemipour, K., Tompson, J., Kaelbling, L., Schuurmans, D., & Abbeel, P. (2023) · arXiv · doi:10.48550/arxiv.2310.06114 · Cited by 346
“…5], video captioning and question answering data contain rich high-level descriptions but little low-level movement detail [6, 7], human activity data contains rich human action but little mechanical motion [8, 9], and robotics data contains rich robot action but are limited in quantity [10, 11].”
BackgroundVideo-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval
Reddy, A., Martin, A. A., Yang, E., Yates, A., Sanders, K., Murray, K., Kriz, R., de Melo, C. M., Durme, B. V., & Chellappa, R. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01834 · Cited by 19
Zhang, Q., Wang, S., Zhang, T., Ren, G., Zhang, L., Wang, Y., Zheng, B., Li, J., & Zheng, H. (2025) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2025.111227 · Cited by 19
Vid2Coach: Transforming How-To Videos into Task Assistants
Huh, M., Xue, Z., Das, U., Ashutosh, K., Grauman, K. L., & Pavel, A. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747612 · Cited by 12
Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning
He, M., Du, W., Wen, Z., Du, Q., Xie, Y., & Wu, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3225549 · Cited by 12
Sounding Video Generator: A Unified Framework for Text-Guided Sounding Video Generation
Liu, J.-W., Wang, W., Chen, S., Zhu, X., & Liu, J. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3262180 · Cited by 12
Weakly-Supervised Temporal Article Grounding
Chen, L., Niu, Y., Chen, B., Lin, X., Han, G., Thomas, C., Ayyubi, H., Ji, H., & Chang, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.639 · Cited by 12
LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling
Chen, D., Tao, C., Hou, L., Shang, L., Jiang, X., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.545 · Cited by 18
Connecting Vision and Language with Video Localized Narratives
Voigtlaender, P., Changpinyo, S., Pont-Tuset, J., Soricut, R., & Ferrari, V. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00243 · Cited by 12
Metadata
- Authors
- Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Institut national de recherche en sciences et technologies du numérique (FR)
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 2630-2640
- OpenAlex
- W2948859046
- Cited by
- 986
- References
- 69
- Citations by year
- 2026: 77 · 2025: 138 · 2024: 170 · 2023: 213 · 2022: 161 · 2021: 155 · 2020: 65 · 2019: 7
- FWCI
- 31.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- European Regional Development Fund (02.1.01/0.0/0.0, CZ.02, CZ.02.1.01, CZ.02.1.01/0.0, CZ.02.1.01/0.0/0.0)
- FP7 Ideas: European Research Council (336845)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- video-text retrieval, action localization, domain transfer, large-scale video datasets
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
