The DOI matches its Crossref record; 66 of the 66 references it cites were found in a registry or an index.
VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
Open accessAbstract
We present a new large-scale multilingual video description dataset, VATEX1, which contains over 41,250 videos and 825, 000 captions in both English and Chinese. Among the captions, there are over 206,000 English-Chinese parallel translation pairs. Compared to the widely-used MSRVTT dataset [64], VATEX is multilingual, larger, linguistically complex, and more diverse in terms of both video and natural language descriptions. We also introduce two tasks for video-and-language research based on VATEX: (1) Multilingual Video Captioning, aimed at describing a video in various languages with a compact unified captioning model, and (2) Video-guided Machine Translation, to translate a source language description into the target language using the video information as additional spatiotemporal context. Extensive experiments on the VATEX dataset show that, first, the unified multilingual model can not only produce both English and Chinese descriptions for a video more efficiently, but also offer improved performance over the monolingual models. Furthermore, we demonstrate that the spatiotemporal video context can be effectively utilized to align source and target languages and thus assist machine translation. In the end, we discuss the potentials of using VATEXfor other video-and-language research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Bidirectional recurrent neural networks
Schuster, M., & Paliwal, K. K. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650093 · Cited by 10,172 · cited 5×
“We employ the pretrained I3D model [13] for action recognition as the 3D ConvNet to obtain the visual features X , Bidirectional LSTM [50] (bi-LSTM) as the video encoder fenc, and LSTM [26] as the language decoder fdec.”
MethodInfluential(2004) · Cited by 9,718
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Hmdb: a large video database for human motion recognition
Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., & Serre, T. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126543 · Cited by 3,965
2D Human Pose Estimation: New Benchmark and State of the Art Analysis
Andriluka, M., Pishchulin, L., Gehler, P., & Schiele, B. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.471 · Cited by 2,688
The Kinetics Human Action Video Dataset
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.06950 · Cited by 2,593
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
Meteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · Proceedings of the Ninth Workshop on Statistical Machine Translation · doi:10.3115/v1/w14-3348 · Cited by 1,952
MSR-VTT: A Large Video Description Dataset for Bridging Video and Language
Xu, J., Mei, T., Yao, T., & Rui, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.571 · Cited by 1,759
The kinetics human action video dataset
Kay, W. (2024) · TIB Data Manager · doi:10.57702/jtuxnt5m · Cited by 1,646
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Sequence to Sequence -- Video to Text
Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., & Saenko, K. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.515 · Cited by 1,352
Dense-Captioning Events in Videos
Krishna, R., Hata, K., Ren, F., Fei-Fei, L., & Niebles, J. C. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.83 · Cited by 1,251
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
Sigurdsson, G. A., Varol, G., Wang, X., Farhadi, A., Laptev, I., & Gupta, A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_31 · Cited by 1,013
Self-citationLocalizing Moments in Video with Natural Language
Hendricks, L. A., Wang, O., Shechtman, E., Sivic, J., Darrell, T., & Russell, B. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.618 · Cited by 958
Collecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 904
Towards Automatic Learning of Procedures From Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 526
Grounding Action Descriptions in Videos
Regneri, M., Rohrbach, M., Wetzel, D., Thater, S., Schiele, B., & Pinkal, M. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00207 · Cited by 467
A dataset for Movie Description
Rohrbach, A., Rohrbach, M., Tandon, N., & Schiele, B. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298940 · Cited by 435
MAN: Moment Alignment Network for Natural Language Moment Retrieval via Iterative Graph Adjustment
Zhang, D., Dai, X., Wang, X., Wang, Y.-F., & Davis, L. S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00134 · Cited by 331
Self-citationRohrbach, A., Torabi, A., Rohrbach, M., Tandon, N., Pal, C., Larochelle, H., Courville, A., & Schiele, B. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0987-1 · Cited by 300
Das, P., Xu, C., Doell, R. F., & Corso, J. J. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.340 · Cited by 282
Video Captioning via Hierarchical Reinforcement Learning
Wang, X., Chen, W., Wu, J., Wang, Y.-F., & Wang, W. Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00443 · Cited by 262
Self-citationA Shared Task on Multimodal Machine Translation and Crosslingual Image Description
Specia, L., Frank, S., Sima'an, K., & Elliott, D. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2346 · Cited by 227
TGIF: A New Dataset and Benchmark on Animated GIF Description
Li, Y., Song, Y., Cao, L., Tetreault, J., Goldberg, L., Jaimes, A., & Luo, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.502 · Cited by 226
Wang, X., Farhadi, A., & Gupta, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.291 · Cited by 220
Self-citationAre You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 216
Video Description: A Survey of Methods, Datasets, and Evaluation Metrics
Aafaq, N., Mian, A., Liu, W., Gilani, S. Z., & Shah, M. (2020) · ACM Computing Surveys · doi:10.1145/3355390 · Cited by 211
Coherent Multi-sentence Video Description with Variable Level of Detail
Rohrbach, A., Rohrbach, M., Qiu, W., Friedrich, A., Pinkal, M., & Schiele, B. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-11752-2_15 · Cited by 206
Incorporating Global Visual Features into Attention-based Neural Machine Translation.
Calixto, I., & Liu, Q. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1105 · Cited by 200
Towards Automatic Learning of Procedures from Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09788 · Cited by 193
Findings of the Third Shared Task on Multimodal Machine Translation
Barrault, L., Bougares, F., Specia, L., Lala, C., Elliott, D., & Frank, S. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6402 · Cited by 188
How2: A Large-scale Dataset for Multimodal Language Understanding
Sanabria, R., Çağlayan, O., Palaskar, S., Elliott, D., Barrault, L., Specia, L., & Metze, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00347 · Cited by 179
Attention-based Multimodal Neural Machine Translation
Huang, P.-Y., Liu, F., Shiang, S.-R., Oh, J., & Dyer, C. (2016) · Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers · doi:10.18653/v1/w16-2360 · Cited by 178
COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval
Li, X., Xu, C., Wang, X., Lan, W., Jia, Z., Yang, G., & Xu, J. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2896494 · Cited by 163
Self-citationVisual Semantic Search: Retrieving Videos via Complex Textual Queries
Lin, D., Fidler, S., Kong, C., & Urtasun, R. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.340 · Cited by 158
Probing the Need for Visual Context in Multimodal Machine Translation
Çağlayan, O., Madhyastha, P., Specia, L., & Barrault, L. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1422 · Cited by 157
Using Descriptive Video Services to Create a Large Data Source for Video Annotation ResearchPreprint
Torabi, A., Pal, C., Larochelle, H., & Courville, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.01070 · Cited by 149
Reinforced Video Captioning with Entailment Rewards
Pasunuru, R., & Bansal, M. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1103 · Cited by 136
Fluency-Guided Cross-Lingual Image Captioning
Lan, W., Li, X., & Dong, J. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123366 · Cited by 95
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2125 · Cited by 89
Self-citationMultimodal Attention for Neural Machine TranslationPreprint
Çağlayan, O., Barrault, L., & Bougares, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03976 · Cited by 87
Title Generation for User Generated Videos
Zeng, K.-H., Chen, T.-H., Niebles, J. C., & Sun, M. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_38 · Cited by 79
The MeMAD Submission to the WMT18 Multimodal Translation Task
Grönroos, S.-A., Huet, B., Kurimo, M., Laaksonen, J. T., Mérialdo, B., Pham, P., Sjöberg, M., Sulubacak, U., Tiedemann, J., Raphaël Troncy, A., et al. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6439 · Cited by 75
LIUM-CVC Submissions for WMT17 Multimodal Translation Task
Çağlayan, O., Aransa, W., Bardet, A., García-Martínez, M., Bougares, F., Barrault, L., Masana, M., Herranz, L., & van de Weijer, J. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4746 · Cited by 72
A Dataset for Telling the Stories of Social Media Videos
Gella, S., Lewis, M., & Rohrbach, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1117 · Cited by 69
Measuring Lexical Style and Competence: The Type-Token Vocabulary Curve
Youmans, G. (1990) · Cited by 39
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
Wang, X. E., Wu, J., Zhang, D., Su, Y., & Wang, W. Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018965 · Cited by 36
Self-citationShimizu, N., Rong, N., & Miyazaki, T. (2018) · International Conference on Computational Linguistics · Cited by 25
A Human Judgement Corpus and a Metric for Arabic MT Evaluation
Bouamor, H., Alshikhabobakr, H., Mohit, B., & Oflazer, K. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1026 · Cited by 17
Using Artificial Tokens to Control Languages for Multilingual Image Caption GenerationPreprint
Tsutsui, S., & Crandall, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.06275 · Cited by 17
Sheffield MultiMT: Using Object Posterior Predictions for Multimodal Machine Translation
Madhyastha, P., Wang, J., & Specia, L. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4752 · Cited by 17
OSU Multimodal Machine Translation System Report
Ma, M., Li, D., Zhao, K., et al. (2017) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w17-4751 · Cited by 15
Wang, X. E., Wang, Y.-F., & Wang, W. Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05448 · Cited by 13
Self-citation
Cited by
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
“Manually annotated vision and language datasets are costly to obtain and thus relatively small (10k-100k) in scale [3, 15, 69, 122, 129, 139].”
BackgroundHERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.161 · Cited by 408
Self-citationFine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning
Chen, S., Zhao, Y., Jin, Q., & Wu, Q. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01065 · Cited by 335
Object Relational Graph With Teacher-Recommended Learning for Video Captioning
Zhang, Z., Shi, Y., Yuan, C., Li, B., Wang, P., Hu, W., & Zha, Z.-J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01329 · Cited by 326
SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning
Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01742 · Cited by 305
Self-citation
Metadata
- Authors
- Wang, X., Wu, J., Chen, J., Li, L., Wang, Y.-F., & Wang, W. Y.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Volume
- 2019
- Pages
- 4580-4590
- OpenAlex
- W2989322838
- Cited by
- 486
- References
- 66
- FWCI
- 13.75
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.