CoChat
Sign Up
Cited by 875Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesGenerative Adversarial Networks and Image Synthesis

Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge

Open access
Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, Erhan, Dumitru·2017·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2016.2587640
Open this paper in CoChat — free accountOpen PDF

Abstract

Automatically describing the content of an image is a fundamental problem in artificial intelligence that connects computer vision and natural language processing. In this paper, we present a generative model based on a deep recurrent architecture that combines recent advances in computer vision and machine translation and that can be used to generate natural sentences describing an image. The model is trained to maximize the likelihood of the target description sentence given the training image. Experiments on several datasets show the accuracy of the model and the fluency of the language it learns solely from image descriptions. Our model is often quite accurate, which we verify both qualitatively and quantitatively. Finally, given the recent surge of interest in this task, a competition was organized in 2015 using the newly released COCO dataset. We describe and analyze the various improvements we applied to our own baseline and show the resulting performance in the competition, which we won ex-aequo with a team from Microsoft Research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2016.2587640

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Performance degradation with larger beam sizes is consistent with previous studies that attribute it to overfitting or discrepancy between optimisation of the metrics and of the learning objective.

    “This is consistent with observations from previous studies [41], which attribute the performance degradation to overfitting or discrepancy between optimisation of the metrics and of the learning objective.”

    From MusCaps: Generating Captions for Music Audio (2021) · cites this paper as [41]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [2][2] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, 'Every picture tells a story: Generating sentences from images,' in ECCV , 2010. -
  2. [3][3] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, 'Baby talk: Understanding and generating simple image descriptions,' in CVPR , 2011. -
  3. [4][4] K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio, 'Learning phrase representations using RNN encoder-decoder for statistical machine translation,' in EMNLP , 2014. -
  4. [5][5] D. Bahdanau, K. Cho, and Y. Bengio, 'Neural machine translation by jointly learning to align and translate,' arXiv:1409.0473 , 2014. - · arXiv
  5. [6][6] I. Sutskever, O. Vinyals, and Q. V. Le, 'Sequence to sequence learning with neural networks,' in NIPS , 2014. -
  6. [7][7] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, 'Overfeat: Integrated recognition, localization and detection using convolutional networks,' arXiv preprint arXiv:1312.6229 , 2013. - · arXiv
  7. [8][8] R. Gerber and H.-H. Nagel, 'Knowledge representation for the generation of quantified natural language descriptions of vehicle traffic in image sequences,' in ICIP . IEEE, 1996. -
  8. [9][9] B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu, 'I2t: Image parsing to text description,' Proceedings of the IEEE , vol. 98, no. 8, 2010. -
  9. [10][10] S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi, 'Composing simple image descriptions using web-scale n-grams,' in Conference on Computational Natural Language Learning , 2011. -
  10. [11][11] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. C. Berg, K. Yamaguchi, T. L. Berg, K. Stratos, and H. D. III, 'Midge: Generating image descriptions from computer vision detections,' in EACL , 2012. -
  11. [12][12] A. Aker and R. Gaizauskas, 'Generating image descriptions using dependency relational patterns,' in ACL , 2010. -
  12. [13][13] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, 'Collective generation of natural image descriptions,' in ACL , 2012. -
  13. [14][14] P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi, 'Treetalk: Composition and compression of trees for image descriptions,' ACL , vol. 2, no. 10, 2014. -
  14. [15][15] D. Elliott and F. Keller, 'Image description using visual dependency representations.' in EMNLP , 2013. -
  15. [16][16] M. Hodosh, P. Young, and J. Hockenmaier, 'Framing image description as a ranking task: Data, models and evaluation metrics.' JAIR , vol. 47, 2013. -
  16. [17][17] Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, 'Improving image-sentence embeddings using large weakly annotated photo collections,' in ECCV , 2014. -
  17. [18][18] V. Ordonez, G. Kulkarni, and T. L. Berg, 'Im2text: Describing images using 1 million captioned photographs,' in NIPS , 2011. -
  18. [19][19] J. Devlin, S. Gupta, R. Girshick, M. Mitchell, and C. L. Zitnick, 'Exploring nearest neighbor approaches for image captioning,' arXiv preprint arXiv:1505.04467 , 2015. - · arXiv
  19. [20][20] M. Kol´ aˇ r, M. Hradiˇ s, and P. Zemˇ c´ ık, 'Technical report: Image captioning with semantically similar images,' arXiv preprint arXiv:1506.03995 , 2015. - · arXiv
  20. [21][21] R. Socher, A. Karpathy, Q. V. Le, C. Manning, and A. Y. Ng, 'Grounded compositional semantics for finding and describing images with sentences,' in ACL , 2014. -
  21. [22][22] A. Karpathy, A. Joulin, and L. Fei-Fei, 'Deep fragment embeddings for bidirectional image sentence mapping,' NIPS , 2014. -
  22. [23][23] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollar, J. Gao, X. He, M. Mitchell, J. Platt, C. L. Zitnick, and G. Zweig, 'From captions to visual concepts and back,' in CVPR , 2015. -
  23. [26][26] R. Kiros and R. Z. R. Salakhutdinov, 'Multimodal neural language models,' in NIPS Deep Learning Workshop , 2013. -
  24. [27][27] J. Mao, W. Xu, Y. Yang, J. Wang, and A. Yuille, 'Explain images with multimodal recurrent neural networks,' in arXiv:1410.1090 , 2014. - · arXiv
  25. [28][28] --, 'Deep captioning with multimodal recurrent neural networks (m-rnn),' ICLR , 2015. -
  26. [29][29] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying visualsemantic embeddings with multimodal neural language models,' in Transactions of the Association for Computational Linguistics , 2015. -
  27. [30][30] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrel, 'Long-term recurrent convolutional networks for visual recognition and description,' in CVPR , 2015. -
  28. [33][33] X. Chen and C. L. Zitnick, 'Mind?s eye: A recurrent visual representation for image caption generation,' Neural computation , vol. 9, no. 8, pp. 1735-1780, 1997. -
  29. [34][34] J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell, 'Language models for image captioning: The quirks and what works,' in ACL , 2015. -
  30. [37][37] A. Graves, 'Generating sequences with recurrent neural networks,' arXiv:1308.0850 , 2013. - · arXiv
  31. [39][39] R. Vedantam, C. L. Zitnick, and D. Parikh, 'CIDEr: Consensusbased image description evaluation,' in arXiv:1411.5726 , 2015. - · arXiv
  32. [40][40] S. Banerjee and A. Lavie, 'Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,' in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization , vol. 29, 2005, pp. 65-72. -
  33. [41][41] C.-Y. Lin, 'Rouge: A package for automatic evaluation of summaries,' in Text summarization branches out: Proceedings of the ACL04 workshop , vol. 8, 2004. -
  34. [42][42] C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, 'Collecting image annotations using amazon's mechanical turk,' in NAACL HLT Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk , 2010, pp. 139-147. -
  35. [43][43] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, 'From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,' in ACL , 2014. -
  36. [44][44] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft coco: Common objects in context,' arXiv:1405.0312 , 2014. - · arXiv
  37. [45][45] W. Zaremba, I. Sutskever, and O. Vinyals, 'Recurrent neural network regularization,' in arXiv:1409.2329 , 2014. - · arXiv
  38. [48][48] S. Bengio, O. Vinyals, N. Jaitly, and N.Shazeer, 'Scheduled sampling for sequence prediction with recurrent neural networks,' in Advances in Neural Information Processing Systems, NIPS , 2015. -
  39. [49][49] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, 'Going deeper with convolutions,' in arXiv:1409.4842 , 2014. - · arXiv
  40. [50][50] L. Breiman, 'Bagging predictors,' Machine Learning , vol. 24, pp. 123-140, 1996. <!-- image --> Oriol Vinyals is a Research Scientist at Google DeepMind, working in Deep Learning. Prior to joining DeepMind, Oriol was part of the Google Brain team. He holds a Ph.D. in EECS from University of California, Berkeley, and a Masters degree from University of California, San Diego. He is a recipient of the 2011 Microsoft Research PhD Fellowship. He was an early adopter of the new deep learning wave at Berkeley, and in his thesis he focused on non-convex optimization and recurrent neural networks. At Google DeepMind he continues working on his areas of interest, which include artificial intelligence, with particular emphasis on machine learning, language, and vision. <!-- image --> Alexander Toshev obtained a Diplom in Computer Science from University of Karlsruhe in 2005, a PhD in Computer and Information Sciences from University of Pennsylvania in 2010. His doctoral work was awarded the 2011 Morris and Dorothy Rubinoff Award by the Engineering Faculty. Since 2011 he has been a research scientist at Google. His research interests lie broadly in computer vision and machine learning, in particular image understanding (object detection, human pose estimation, image seg- mentation) and synergy of text and images. <!-- image --> Samy Bengio (PhD in computer science, University of Montreal, 1993) is a research scientist at Google since 2007. Before that, he was senior researcher in statistical machine learning at IDIAP Research Institute since 1999. His most recent research interests are in machine learning, in particular deep learning, large scale online learning, image ranking and annotation, music and speech processing. He is action editor of the Journal of Machine Learning Research and on the editorial board of the Machine Learn- ing Journal. He was associate editor of the journal of computational statistics, general chair of the Workshops on Machine Learning for Multimodal Interactions (MLMI'2004-2006), programme chair of the International Conference on Learning Representations (ICLR'2015-2016), programme chair of the IEEE Workshop on Neural Networks for Signal Processing (NNSP'2002), chair of BayLearn (2012-2015), and several times on the programme committee of international conferences such as NIPS, ICML, ECML and ICLR. More information can be found on his website: http://bengio.abracadoudou.com. <!-- image --> Dumitru Erhan (PhD in computer science, University of Montreal, 2011) is a software engineer at Google since 2012. Before that, he was scientist at Yahoo! Labs from 2011 to 2012. His research interests span the intersection of deep learning, computer vision and natural language. In particular, he is interested in efficient models for understanding what and where is in image, as well as for answering arbitrary questions about them. More information at http://dumitru.ca.

Cited by

Sorted by citations

Authors
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D.
Affiliations
Google (United States) (US)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2017-04-01
Volume
39
Issue
4
Pages
652-663
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
DOI
10.1109/tpami.2016.2587640
OpenAlex
W2463955103
Cited by
875
References
9
Citations by year
2026: 16 · 2025: 44 · 2024: 74 · 2023: 104 · 2022: 104 · 2021: 105 · 2020: 126 · 2019: 139 · 2018: 96 · 2017: 59 · 2016: 4 · 2015: 4
FWCI
67.56
Type
journal article
Language
en
Open access
Licence
ieeexplore.ieee.org/Xplorehelp/downloads/license-information/OAPA.html (vor)
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
Keywords
image captioning, COCO dataset, generative models, recurrent neural networks, machine translation, image description generation, sentence fluency, likelihood maximization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review