CoChat
Sign Up
PreprintCited by 1,614Multimodal Machine Learning ApplicationsImage Retrieval and Classification TechniquesNatural Language Processing Techniques

Microsoft COCO Captions: Data Collection and Evaluation Server

Open access
Chen, Xinlei, Fang, Hao, Lin, Tsung-Yi, Vedantam, Ramakrishna, Gupta, Saurabh, Dollar, Piotr, Zitnick, C. Lawrence·2015·arXiv
DOI 10.48550/arxiv.1504.00325arXiv 1504.00325
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper we describe the Microsoft COCO Caption dataset and evaluation server. When completed, the dataset will contain over one and a half million captions describing over 330,000 images. For the training and validation images, five independent human generated captions will be provided. To ensure consistency in evaluation of automatic caption generation algorithms, an evaluation server is used. The evaluation server receives candidate captions and scores them using several popular metrics, including BLEU, METEOR, ROUGE and CIDEr. Instructions for using the evaluation server are provided.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1504.00325

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing datasets overwhelmingly focus on high-quality images (e.g., ImageNet and MS COCO) that VLMs are optimized on.

    “in contrast to existing datasets that overwhelmingly focus on high-quality images (e.g., ImageNet [37] and MS COCO [28, 82]) that VLMs are optimized on.”

    From "It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models (2026) · cites this paper as [28]

    The Introduction and Data Collection section (p. 1) identify MS COCO as an image dataset used for captioning, so the paper supports citing it as an example of an existing image dataset. The copy does not establish that its images are high-quality, that existing datasets overwhelmingly favor such images, or that VLMs are optimized on them. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [1][1] K. Barnard and D. Forsyth, 'Learning the semantics of words and pictures,' in ICCV , vol. 2, 2001, pp. 408-415. -
  2. [2][2] K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. Blei, and M. I. Jordan, 'Matching words and pictures,' JMLR , vol. 3, pp. 1107-1135, 2003. -
  3. [3][3] V. Lavrenko, R. Manmatha, and J. Jeon, 'A model for learning the semantics of pictures,' in NIPS , 2003. -
  4. [4][4] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, 'Baby talk: Understanding and generating simple image descriptions,' in CVPR , 2011. -
  5. [5][5] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daum´ e III, 'Midge: Generating image descriptions from computer vision detections,' in EACL , 2012. -
  6. [6][6] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, 'Every picture tells a story: Generating sentences from images,' in ECCV , 2010. -
  7. [7][7] M. Hodosh, P. Young, and J. Hockenmaier, 'Framing image description as a ranking task: Data, models and evaluation metrics.' JAIR , vol. 47, pp. 853-899, 2013. -
  8. [8][8] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and Y. Choi, 'Collective generation of natural image descriptions,' in ACL , 2012. -
  9. [9][9] Y. Yang, C. L. Teo, H. Daum´ e III, and Y. Aloimonos, 'Corpusguided sentence generation of natural images,' in EMNLP , 2011. -
  10. [10][10] A. Gupta, Y. Verma, and C. Jawahar, 'Choosing linguistics over vision to describe images.' in AAAI , 2012. -
  11. [11][11] E. Bruni, G. Boleda, M. Baroni, and N.-K. Tran, 'Distributional semantics in technicolor,' in ACL , 2012. -
  12. [12][12] Y. Feng and M. Lapata, 'Automatic caption generation for news images,' TPAMI , vol. 35, no. 4, pp. 797-812, 2013. -
  13. [13][13] D. Elliott and F. Keller, 'Image description using visual dependency representations.' in EMNLP , 2013, pp. 1292-1302. -
  14. [14][14] A. Karpathy, A. Joulin, and F.-F. Li, 'Deep fragment embeddings for bidirectional image sentence mapping,' in NIPS , 2014. -
  15. [15][15] Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik, 'Improving image-sentence embeddings using large weakly annotated photo collections,' in ECCV , 2014, pp. 529-545. -
  16. [16][16] R. Mason and E. Charniak, 'Nonparametric method for datadriven image captioning,' in ACL , 2014. -
  17. [17][17] P. Kuznetsova, V. Ordonez, T. Berg, and Y. Choi, 'Treetalk: Composition and compression of trees for image descriptions,' TACL , vol. 2, pp. 351-362, 2014. -
  18. [18][18] K. Ramnath, S. Baker, L. Vanderwende, M. El-Saban, S. N. Sinha, A. Kannan, N. Hassan, M. Galley, Y. Yang, D. Ramanan, A. Bergamo, and L. Torresani, 'Autocaption: Automatic caption generation for personal photos,' in WACV , 2014. -
  19. [19][19] A. Lazaridou, E. Bruni, and M. Baroni, 'Is this a wampimuk? cross-modal mapping between distributional semantics and the visual world,' in ACL , 2014. -
  20. [20][20] R. Kiros, R. Salakhutdinov, and R. Zemel, 'Multimodal neural language models,' in ICML , 2014. -
  21. [21][21] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, 'Explain images with multimodal recurrent neural networks,' arXiv preprint arXiv:1410.1090 , 2014. - · arXiv
  22. [24][24] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying visualsemantic embeddings with multimodal neural language models,' arXiv preprint arXiv:1411.2539 , 2014. - · arXiv
  23. [25][25] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, 'Long-term recurrent convolutional networks for visual recognition and description,' arXiv preprint arXiv:1411.4389 , 2014. - · arXiv
  24. [26][26] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Doll´ ar, J. Gao, X. He, M. Mitchell, J. Platt et al. , 'From captions to visual concepts and back,' arXiv preprint arXiv:1411.4952 , 2014. - · arXiv
  25. [27][27] X. Chen and C. L. Zitnick, 'Learning a recurrent visual representation for image caption generation,' arXiv preprint arXiv:1411.5654 , 2014. - · arXiv
  26. [28][28] R. Lebret, P. O. Pinheiro, and R. Collobert, 'Phrase-based image captioning,' arXiv preprint arXiv:1502.03671 , 2015. - · arXiv
  27. [29][29] --, 'Simple image description generator via a linear phrasebased approach,' arXiv preprint arXiv:1412.8419 , 2014. - · arXiv
  28. [30][30] A. Lazaridou, N. T. Pham, and M. Baroni, 'Combining language and vision with a multimodal skip-gram model,' arXiv preprint arXiv:1501.02598 , 2015. - · arXiv
  29. [33][33] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'ImageNet: A Large-Scale Hierarchical Image Database,' in CVPR , 2009. -
  30. [34][34] M. Grubinger, P. Clough, H. M¨ uller, and T. Deselaers, 'The iapr tc12 benchmark: A new evaluation resource for visual information systems,' in LREC Workshop on Language Resources for Contentbased Image Retrieval , 2006. -
  31. [35][35] V. Ordonez, G. Kulkarni, and T. Berg, 'Im2text: Describing images using 1 million captioned photographs.' in NIPS , 2011. -
  32. [36][36] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, 'From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,' TACL , vol. 2, pp. 6778, 2014. -
  33. [37][37] J. Chen, P. Kuznetsova, D. Warren, and Y. Choi, 'D´ ej´ a imagecaptions: A corpus of expressive image descriptions in repetition,' in NAACL , 2015. -
  34. [38][38] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: Common objects in context,' in ECCV , 2014. -
  35. [40][40] C.-Y. Lin, 'Rouge: A package for automatic evaluation of summaries,' in ACL Workshop , 2004. -
  36. [41][41] M. Denkowski and A. Lavie, 'Meteor universal: Language specific translation evaluation for any target language,' in EACL Workshop on Statistical Machine Translation , 2014. -
  37. [42][42] R. Vedantam, C. L. Zitnick, and D. Parikh, 'Cider: Consensus-based image description evaluation,' arXiv preprint arXiv:1411.5726 , 2014. - · arXiv
  38. [43][43] C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky, 'The Stanford CoreNLP natural language processing toolkit,' in Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations , 2014, pp. 55-60. [Online]. Available: http: //www.aclweb.org/anthology/P/P14/P14-5010 -
  39. [45][45] D. Elliott and F. Keller, 'Comparing automatic evaluation measures for image description,' in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics , vol. 2, 2014, pp. 452-457. -
  40. [46][46] C. Callison-Burch, M. Osborne, and P. Koehn, 'Re-evaluation the role of bleu in machine translation research.' in EACL , vol. 6, 2006, pp. 249-256.

Cited by

Sorted by citations

Authors
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1504.00325
arXiv
1504.00325
OpenAlex
W1889081078
Cited by
1,614
References
6
Citations by year
2026: 9 · 2025: 77 · 2024: 196 · 2023: 256 · 2022: 190 · 2021: 285 · 2020: 166 · 2019: 169 · 2018: 112 · 2017: 85 · 2016: 48 · 2015: 19 · 2014: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
image captioning, automatic caption generation, BLEU, meteors, ROUGE, cider

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review