A Survey of Current Datasets for Vision and Language Research
Open accessAbstract
Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, Margaret Mitchell. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most previous multimodal benchmarks have focused on a single research area such as within affective computing, human multimodal language, language and vision-based question answering, text classification with external multimodal information, and multimodal learning for education.
“Most previous multimodal benchmarks have focused on a single research area such as within affective computing [56], human multimodal language [177], language and vision-based question answering [50, 138], text classification with external multimodal information [60], and multimodal learning for education [65].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460 · see citing works
The Stanford CoreNLP Natural Language Processing Toolkit
Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S. J., & McClosky, D. (2014) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.3115/v1/p14-5010 · Cited by 7,171 · see citing works
Understanding natural language
Winograd, T. A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90002-3 · Cited by 2,803 · see citing works
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Self-citationEvery Picture Tells a Story: Generating Sentences from Images
Farhadi, A., Hejrati, M., Sadeghi, M. A., Young, P., Rashtchian, C., Hockenmaier, J., & Forsyth, D. A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_2 · Cited by 1,240 · see citing works
ReferItGame: Referring to Objects in Photographs of Natural Scenes
Kazemzadeh, S., Ordóñez, V., Matten, M., & Berg, T. L. (2014) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1086 · Cited by 1,082 · see citing works
Long-term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. (2014) · doi:10.21236/ada623249 · Cited by 1,081 · see citing works
Translating Videos to Natural Language Using Deep Recurrent Neural Networks
Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R. J., & Saenko, K. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1173 · Cited by 996 · see citing works
VQA: Visual Question AnsweringPreprint
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 968 · see citing works
Self-citationCollecting Highly Parallel Data for Paraphrase Evaluation
Chen, D. J., & Dolan, W. B. (2011) · Cited by 898 · see citing works
Im2Text: Describing Images Using 1 Million Captioned Photographs
Ordóñez, V., Kulkarni, G. A., & Berg, T. L. (2011) · Cited by 879 · see citing works
Collecting Image Annotations Using Amazon's Mechanical Turk
Rashtchian, C., Young, P., Hodosh, M., & Hockenmaier, J. (2010) · Cited by 541 · see citing works
A model and an hypothesis for language structure
Yngve, V. H. (1960) · Cited by 522 · see citing works
Grounding Action Descriptions in Videos
Regneri, M., Rohrbach, M., Wetzel, D., Thater, S., Schiele, B., & Pinkal, M. (2013) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00207 · Cited by 471 · see citing works
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question AnsweringPreprint
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 437 · see citing works
A Joint Model of Language and Perception for Grounded Attribute Learning
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012) · Maryland Shared Open Access Repository · doi:10.13016/m2ehu6-1m90 · Cited by 248 · see citing works
The New Data and New Challenges in Multimedia Research.Preprint
Thomée, B., Shamma, D. A., Friedland, G. H., Elizalde, B., Ni, K. S., Poland, D., Borth, D., & Li, L.-J. (2015) · arXiv (Cornell University) · Cited by 247 · see citing works
Learning the Visual Interpretation of Sentences
Zitnick, C. L., Parikh, D., & Vanderwende, L. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.211 · Cited by 156 · see citing works
Self-citationWhat’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision
Malmaud, J., Huang, J., Rathod, V., Johnston, N., Rabinovich, A., & Murphy, K. P. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1015 · Cited by 144 · see citing works
Situated Dialogue and Spatial Organization: What, Where… and Why?
Kruijff, G.-J. M., Zender, H., Jensfelt, P., & Christensen, H. I. (2007) · International Journal of Advanced Robotic Systems · doi:10.5772/5701 · Cited by 133 · see citing works
Grounded Language Learning from Video Described with Sentences
Yu, H., & Siskind, J. M. (2013) · Cited by 115 · see citing works
Training a Multilingual Sportscaster: Using Perceptual Context to Learn Language
Chen, D. L., Kim, J., & Mooney, R. J. (2010) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2962 · Cited by 94 · see citing works
A Joint Model of Language and Perception for Grounded Attribute LearningPreprint
Matuszek, C., FitzGerald, N., Zettlemoyer, L., Bo, L., & Fox, D. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.6423 · Cited by 82 · see citing works
Visual Madlibs: Fill in the blank Image Generation and Question AnsweringPreprint
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00278 · Cited by 75 · see citing works
Robust spoken instruction understanding for HRI
Cantrell, R., Scheutz, M. J., Schermerhorn, P., & Wu, X. (2010) · Cited by 59 · see citing works
See No Evil, Say No Evil: Description Generation from Densely Labeled Images
Yatskar, M., Galley, M., Vanderwende, L., & Zettlemoyer, L. (2014) · Joint Conference on Lexical and Computational Semantics · doi:10.3115/v1/s14-1015 · Cited by 54 · see citing works
Self-citationAn AMR parser for English, French, German, Spanish and Japanese and a new AMR-annotated corpus
Vanderwende, L., Menezes, A., & Quirk, C. (2015) · doi:10.3115/v1/n15-3006 · Cited by 44 · see citing works
Self-citationVanderwende, L., Kacmarcik, G., Suzuki, H., & Menezes, A. (2005) · doi:10.3115/1225733.1225738 · Cited by 32 · see citing works
Self-citationRoy, D. K., Hsiao, K.-Y., & Mavridis, N. (2003) · doi:10.3115/1119212.1119222 · Cited by 30 · see citing works
Joint photo stream and blog post summarization and exploration
Kim, G., Moon, S., & Sigal, L. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298927 · Cited by 28 · see citing works
Naim, I., Song, Y. C., Liu, Q., Huang, L., Kautz, H., Luo, J., & Gildea, D. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1017 · Cited by 28 · see citing works
Reporting Bias and Knowledge Extraction
Gordon, J. A. R., & Durme, B. V. (2013) · Cited by 18 · see citing works
Flagged first, then by citations
Research works printed but not linked: 8
- David L. Chen and William B. Dolan. 2011. Col- lecting highly parallel data for paraphrase evalua- tion. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies -Volume 1, HLT '11, pages 190-200, Stroudsburg, PA, USA. Association for Computational Linguistics.
- Jianfu Chen, Polina Kuznetsova, David Warren, and Yejin Choi. 2015. Déjà image-captions: A cor- pus of expressive descriptions in repetition. In Pro- ceedings of the 2015 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 504-514, Denver, Colorado, May-June. Association for Computational Linguistics.
- L. Frazier. 1985. Syntactic complexity. In D. R. Dowty, L. Karttunen, and A. M. Zwicky, editors, Natural Language Parsing: Psychological, Compu- tational, and Theoretical Perspectives, pages 129- 189. Cambridge University Press, Cambridge.
- Mateusz Malinowski and Mario Fritz. 2014. A multi- world approach to question answering about real- world scenes based on uncertain input. In Advances in Neural Information Processing Systems 27, pages 1682-1690.
- Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nicholas Johnston, Andrew Rabinovich, and Kevin Murphy. 2015. Whats cookin? interpreting cook- ing videos using text, speech and vision. In North American Chapter of the Association for Computa- tional Linguistics Human Language Technologies (NAACL HLT 2015), May 31 -June 5, 2015, Denver, Colorado USA.
- Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Question answering about images using visual se- mantic embeddings. In Deep Learning Workshop, ICML 2015.
- Marcus Rohrbach, Sikandar Amin, Mykhaylo An- driluka, and Bernt Schiele. 2012. A database for fine grained activity detection of cooking activities. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, IEEE, June.
- Terry Winograd. 1972. Understanding Natural Lan- guage. Academic Press, New York.
Cited by
Sorted by citations
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776 · see citing works
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
Jang, Y., Song, Y., Yu, Y., Kim, Y., & Kim, G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.149 · Cited by 496 · see citing works
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 487 · see citing works
Visual question answering: A survey of methods and datasets
Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.05.001 · Cited by 442 · see citing works
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Raffaella Bernardi, B., Çakıcı, R., Elliott, D., Erdem, A., Erdem, E., İkizler-Cinbiş, N., Keller, F., Muscat, A., & Plank, B. (2016) · Journal of Artificial Intelligence Research · doi:10.1613/jair.4900 · Cited by 350 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
Self-citationHuang, T.-H., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1147 · Cited by 254 · see citing works
Self-citationMultimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
Visual to Text: Survey of Image and Video Captioning
Li, S., Tao, Z., Li, K., & Fu, Y. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2892755 · Cited by 108 · see citing works
A Survey of Current Datasets for Code-Switching Research
Jose, N., Chakravarthi, B. R., Suryawanshi, S., Sherly, E., & McCrae, J. P. (2020) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs48705.2020.9074205 · Cited by 103 · see citing works
Cross-Lingual Image Caption Generation
Miyazaki, T., & Shimizu, N. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1168 · Cited by 95 · see citing works
Multimodal Pivots for Image Caption Translation
Hitschler, J., Schamoni, S., & Riezler, S. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1227 · Cited by 94 · see citing works
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos
Huang, D.-A., Buch, S., Dery, L. M., Garg, A., Fei-Fei, L., & Niebles, J. C. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00623 · Cited by 85 · see citing works
Crone, D., Bode, S., Murawski, C., & Laham, S. (2018) · PLoS ONE · doi:10.1371/journal.pone.0190954 · Cited by 77 · see citing works
Computer Vision and Natural Language Processing
Wiriyathammabhum, P., Summers-Stay, D. A., Fermuller, C. M., & Aloimonos, Y. (2016) · ACM Computing Surveys · doi:10.1145/3009906 · Cited by 65 · see citing works
Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
Mogadala, A., Kalimuthu, M., & Klakow, D. (2021) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11688 · Cited by 128 · see citing works
Learning Translations via Images with a Massively Multilingual Image Dataset
Hewitt, J., Ippolito, D., Callahan, B., Kriz, R., Wijaya, D. T., & Callison-Burch, C. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1239 · Cited by 42 · see citing works
Automatic description generation from images
BernardiRaffaella, CakiciRuket, ElliottDesmond, ErdemAykut, ErdemErkut, Ikizler-CinbisNazli, KellerFrank, MuscatAdrian, & PlankBarbara (2016) · doi:10.5555/3013558.3013571 · Cited by 41 · see citing works
Measuring the Diversity of Automatic Image Descriptions
van Miltenburg, E., Elliott, D., & Vossen, P. T. J. M. (2018) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 35 · see citing works
Domain Specific Named Entity Recognition Referring to the Real World by Deep Neural Networks
Tomori, S., Ninomiya, T., & Mori, S. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-2039 · Cited by 34 · see citing works
Envisioning Narrative Intelligence: A Creative Visual Storytelling Anthology
Halperin, B. A., & Lukin, S. (2023) · International Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580744 · Cited by 40 · see citing works
“Caption” as a Coherence Relation: Evidence and Implications
Alikhani, M., & Stone, M. (2019) · doi:10.18653/v1/w19-1806 · Cited by 25 · see citing works
Gender Artifacts in Visual Datasets
Meister, N., Zhao, D., Wang, A., Ramaswamy, V., Fong, R., & Russakovsky, O. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00446 · Cited by 26 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“…multimodal benchmarks have focused on a single research area such as within affective computing [56], human multimodal language [177], language and vision-based question answering [50, 138], text classification with external multimodal information [60], and multimodal learning for education [65].”
BackgroundEvolution of automatic visual description techniques-a methodological survey
Bhowmik, A., Kumar, S., & Bhat, N. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10964-3 · Cited by 22 · see citing works
Roynard, X., Deschaud, J., & Goulette, F. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364918767506 · Cited by 21 · see citing works
A Corpus of Images and Text in Online News
Laura Hollink, A., Bedjeti, A., van Harmelen, M., & Elliott, D. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/37jyf8xndpf7 · Cited by 14 · see citing works
Aligning and linking entity mentions in image, text, and knowledge base
Dost, S., Serafini, L., Rospocher, M., Ballan, L., & Sperduti, A. (2022) · Data & Knowledge Engineering · doi:10.1016/j.datak.2021.101975 · Cited by 11 · see citing works
Cross-Dataset Adaptation for Visual Question AnsweringPreprint
Chao, W., Hu, H., & Sha, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.03726 · Cited by 11 · see citing works
The utility of datasets in crowd modelling and analysis: a survey
Waqar, S., Khan, M. U. G., Waseem, M. H., & Qayyum, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13227-x · Cited by 9 · see citing works
A Japanese Chess Commentary Corpus
Mori, S., Richardson, J. T. E., Ushiku, A., Sasada, T., Kameko, H., & Tsuruoka, Y. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/47nofv5mz7rr · Cited by 7 · see citing works
A Dataset and Reranking Method for Multimodal MT of User-Generated Image Captions
Schamoni, S., Hitschler, J., & Riezler, S. (2018) · Conference of the Association for Machine Translation in the Americas · Cited by 7 · see citing works
Pushing the Limits of Radiology with Joint Modeling of Visual and Textual Information
Singh, S. (2018) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/p18-3005 · Cited by 6 · see citing works
Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models
Suglia, A., Konstas, I., & Lemon, O. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15185 · Cited by 8 · see citing works
SCAP: enhancing image captioning through lightweight feature sifting and hierarchical decoding
Zhang, Y., Tong, J., & Liu, H. (2025) · The Visual Computer · doi:10.1007/s00371-025-03824-w · Cited by 5 · see citing works
1 Million Captioned Dutch Newspaper Images
Elliott, D., & Kleppe, M. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2pephtt6o3zw · Cited by 5 · see citing works
Integrating Vision and Language Datasets to Measure Word Concreteness.
Kehat, G., & Pustejovsky, J. D. (2017) · International Joint Conference on Natural Language Processing · Cited by 5 · see citing works
Weakly-supervised video object localization with attentive spatio-temporal correlation
Wang, M., Cui, D., Wu, L., Jian, M., Chen, Y., Wang, D., & Liu, X. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.02.015 · Cited by 5 · see citing works
Crone, D. L., Bode, S., Murawski, C., & Laham, S. M. (2017) · doi:10.31234/osf.io/sja3m · Cited by 7 · see citing works
Moharil, A., Vanschoren, J., Singh, P., & Tamburri, D. A. (2024) · Machine Learning · doi:10.1007/s10994-024-06568-1 · Cited by 4 · see citing works
The role of image representations in vision to language tasks
Madhyastha, P., Wang, J., & Specia, L. (2018) · Natural Language Engineering · doi:10.1017/s1351324918000116 · Cited by 3 · see citing works
On task effects in NLG corpus elicitation: a replication study using mixed effects modeling
van Miltenburg, E., van de Kerkhof, M., Koolen, R., Goudbeek, M. B., & Krahmer, E. (2019) · International Conference on Natural Language Generation · doi:10.18653/v1/w19-8649 · Cited by 3 · see citing works
Optimal Semantic Distance for Negative Example Selection in Grounded Language Acquisition
Pillai, N., Ferraro, F., & Matuszek, C. (2018) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2251fp4n · Cited by 3 · see citing works
Self-citationLocation-Aware Visual Question Generation with Lightweight Models
Suwono, N. C., Chen, J., Hung, T. M., Huang, T.-H., Liao, I.-B., Li, Y., Ku, L., & Sun, S.-H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.88 · Cited by 3 · see citing works
Self-citationMulti-VQG: Generating Engaging Questions for Multiple Images
Yeh, M.-H., Chen, V., Huang, T.-H., & Ku, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.19 · Cited by 4 · see citing works
Self-citationAn Analysis of Action Recognition Datasets for Language and Vision Tasks
Gella, S., & Keller, F. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2011 · Cited by 3 · see citing works
Emerging Trends of Multimodal Research in Vision and Language.Preprint
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2020) · arXiv (Cornell University) · Cited by 3 · see citing works
Automatic Annotation of Text with Pictures
Poots, J., & Ebrahim Bagheri, A. (2018) · IT Professional · doi:10.1109/mitp.2018.011291354 · Cited by 2 · see citing works
SILG: The Multi-environment Symbolic Interactive Language Grounding BenchmarkPreprint
Zhong, V., Hanjie, A. W., Wang, S. I., Narasimhan, K., & Zettlemoyer, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.10661 · Cited by 4 · see citing works
Clustering Examples in Multi-Dataset Benchmarks with Item Response Theory
Rodríguez, P., Htut, P. M., Lalor, J., & Sedoc, J. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.14 · Cited by 1 · see citing works
Cross-validating Image Description Datasets and Evaluation Metrics
Wang, J., & Gaizauskas, R. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2r78gfiifw85 · Cited by 1 · see citing works
Designing PhelkStat: Big Data Analytics for System Event Logs
Salman, M., Welch, B., Raymond, D., Marchany, R. C., & Tront, J. G. (2017) · VTechWorks (Virginia Tech) · Cited by 1 · see citing works
Learning More May Not Be Better: Knowledge Transferability in Vision-and-Language Tasks
Chen, T., García, N., Otani, M., Chu, C., Nakashima, Y., & Nagahara, H. (2024) · Journal of Imaging · doi:10.3390/jimaging10120300 · Cited by 1 · see citing works
Annotation Methodologies for Vision and Language Dataset CreationPreprint
Kehat, G., & Pustejovsky, J. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02769 · Cited by 1 · see citing works
A Dataset and Benchmarks for Multimedia Social AnalysisPreprint
Xue, B., Chan, D. M., & Canny, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.08335 · Cited by 1 · see citing works
e-QRAQ: A Multi-turn Reasoning Dataset and Simulator with ExplanationsPreprint
Rosenbaum, C., Gao, T., & Klinger, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.01776
Evaluation of Automatic Tag Sense Disambiguation Using the MIRFLICKR Image Collection
Kanishcheva, O., Nikolova, I., & Angelova, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-99344-7_6
Visual Understanding and Narration: A Deeper Understanding and Explanation of Visual ScenesPreprint
Lukin, S. M., Bonial, C., & Voss, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00038
Measuring Perceptual and Linguistic Complexity in Multilingual Grounded Language Data
Pillai, N., Matuszek, C., & Ferraro, F. (2021) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.v34i1.128450
Self-citationJapanese Chess Commentary Corpus with Named Entity and Modality Annotation
Kameko, H., Matsuyoshi, S., Richardson, J. T. E., Ushiku, A., Sasada, T., Murawaki, Y., Tsuruoka, Y., & Mori, S. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.847
Image Description using Attention Mechanism
(2019) · International Journal of Recent Technology and Engineering (IJRTE) · doi:10.35940/ijrte.b1555.0982s1119
Image Captioning with External Knowledge
Nikiforova, S. (2023) · doi:10.33540/1709
Objective function in convolutional neural network to generate caption for image - a review
Nursikuwagus, A., Munir, R., & Khodra, M. L. (2023) · AIP conference proceedings · doi:10.1063/5.0129585
Review of Language Models in the Financial Domain
Wagh, A., & Mavle, A. (2024) · Intelligent Systems and Machine Learning Conference (ISML) · doi:10.1109/isml60050.2024.11007354 · Cited by 1 · see citing works
Belqasmi, F., Loucif, S., & Alkhatib, M. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11165875
Hashiguti, S. T., Ferreira, S. R., Maia, H. D. A., Benedito, M. E. D. A., Nardo, J. D. S., Fagundes, I. Z. Z., Amado, G. E., Brito, C. C. D. P., Ângelo, R. D. C., Silva, S. T. D. A. E., et al. (2026) · Trabalhos em Linguística Aplicada · doi:10.1590/01031813v65020268684688
- Authors
- Ferraro, F., Mostafazadeh, N., Huang, T.-H., Vanderwende, L., Devlin, J., Galley, M., & Mitchell, M.
- Affiliations
- Johns Hopkins University (US)
- University of Rochester (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing
- Published
- 2015
- Pages
- 207-213
- OpenAlex
- W1749884237
- Cited by
- 68
- References
- 32
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 4 · 2023: 5 · 2022: 4 · 2021: 8 · 2020: 3 · 2019: 8 · 2018: 13 · 2017: 7 · 2016: 13
- FWCI
- 4.51
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multimodal dataset, visual grounding, image-text dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
