CoChat
Sign Up
PreprintCited by 2,579Human Pose and Action RecognitionAdvanced Neural Network ApplicationsHand Gesture Recognition Systems

Microsoft COCO: Common Objects in Context

Open access
Lin, Tsung-Yi, Maire, Michael, Belongie, Serge, Bourdev, Lubomir, Girshick, Ross, Hays, James, Perona, Pietro, Ramanan, Deva +2 more·2014·arXiv
DOI 10.48550/arxiv.1405.0312arXiv 1405.0312
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in precise object localization. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1405.0312

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Most object detection models use general-purpose datasets where vehicle images are usually taken from a lateral or frontal view instead of a cenital plane.

    “Most object detection models use generalpurpose datasets, such as COCO [12], ImageNet [13], and VOC [14]), where vehicle (i.e., cars) images are usually taken from a lateral or frontal view instead of a cenital plane.”

    From T-YOLO: Tiny Vehicle Detection Based on YOLO and Multi-Scale Convolutional Neural Networks (2023) · cites this paper as [12]

    The Introduction (p. 2) identifies COCO as a dataset for learning object models, and Dataset Statistics (p. 6) says it is designed for object detection. But the paper does not establish that most detection models use such datasets or that COCO car images are usually lateral or frontal rather than overhead; the Discussion (p. 10) instead describes COCO images as having varied viewpoints. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • DETR achieves comparable results to an optimized Faster R-CNN baseline on COCO.

    “Experiments on COCO [24] demonstrate that the DETR approach achieves comparable results to an optimized Faster R-CNN baseline [33].”

    From Transformer Tracking (2021) · cites this paper as [24]

  • The model is trained on the training splits of COCO, TrackingNet, LaSOT, and GOT10k datasets.

    “We train our model on the training splits of COCO [24], TrackingNet [30], LaSOT [14], and GOT10k [19] datasets.”

    From Transformer Tracking (2021) · cites this paper as [24]

  • On the MS-COCO validation set, our model improves upon powerful autoregressive and GAN-based methods for text-to-image generation.

    “For quantitative analysis, we follow prior work and evaluate text-to-image generation on the MS-COCO [51] validation set, where our model improves upon powerful AR [17, 66] and GAN-based [109] methods, cf . Tab. 2.”

    From High-Resolution Image Synthesis with Latent Diffusion Models (2022) · cites this paper as [51]

  • In Deformable-DETR, the encoder accounts for 49% of the GFLOPs but contributes only 11% of the AP.

    “As reported in Lin et al . [19], the encoder accounts for 49% of the GFLOPs but contributes only 11% of the AP in Deformable-DETR.”

    From DETRs Beat YOLOs on Real-time Object Detection (2024) · cites this paper as [19]

    The Algorithmic Analysis section (pp. 8–9) evaluates DPM models on MS COCO and PASCAL VOC, not Deformable-DETR. Searches for the encoder, GFLOPs, and the 49% and 11% figures found no passage in our copy addressing the claimed breakdown. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. [1][1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'ImageNet: A Large-Scale Hierarchical Image Database,' in CVPR , 2009. -
  2. [2][2] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, 'The PASCAL visual object classes (VOC) challenge,' IJCV , vol. 88, no. 2, pp. 303-338, Jun. 2010. -
  3. [3][3] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, 'SUN database: Large-scale scene recognition from abbey to zoo,' in CVPR , 2010. -
  4. [4][4] P. Doll´ ar, C. Wojek, B. Schiele, and P. Perona, 'Pedestrian detection: An evaluation of the state of the art,' PAMI , vol. 34, 2012. -
  5. [5][5] A. Krizhevsky, I. Sutskever, and G. Hinton, 'ImageNet classification with deep convolutional neural networks,' in NIPS , 2012. -
  6. [6][6] R. Girshick, J. Donahue, T. Darrell, and J. Malik, 'Rich feature hierarchies for accurate object detection and semantic segmentation,' in CVPR , 2014. -
  7. [7][7] P. Sermanet, D. Eigen, S. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, 'OverFeat: Integrated recognition, localization and detection using convolutional networks,' in ICLR , April 2014. -
  8. [8][8] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth, 'Describing objects by their attributes,' in CVPR , 2009. -
  9. [9][9] G. Patterson and J. Hays, 'SUN attribute database: Discovering, annotating, and recognizing scene attributes,' in CVPR , 2012. -
  10. [10][10] L. Bourdev and J. Malik, 'Poselets: Body part detectors trained using 3D human pose annotations,' in ICCV , 2009. -
  11. [12][12] S. Palmer, E. Rosch, and P. Chase, 'Canonical perspective and the perception of objects,' Attention and performance IX , vol. 1, p. 4, 1981. -
  12. [13][13] . Hoiem, D, Y. Chodpathumwan, and Q. Dai, 'Diagnosing error in object detectors,' in ECCV , 2012. -
  13. [14][14] G. Brostow, J. Fauqueur, and R. Cipolla, 'Semantic object classes in video: A high-definition ground truth database,' PRL , vol. 30, no. 2, pp. 88-97, 2009. -
  14. [15][15] B. Russell, A. Torralba, K. Murphy, and W. Freeman, 'LabelMe: a database and web-based tool for image annotation,' IJCV , vol. 77, no. 1-3, pp. 157-173, 2008. -
  15. [16][16] S. Bell, P. Upchurch, N. Snavely, and K. Bala, 'OpenSurfaces: A richly annotated catalog of surface appearance,' SIGGRAPH , vol. 32, no. 4, 2013. -
  16. [17][17] V. Ordonez, G. Kulkarni, and T. Berg, 'Im2text: Describing images using 1 million captioned photographs.' in NIPS , 2011. -
  17. [18][18] J. Deng, O. Russakovsky, J. Krause, M. Bernstein, A. Berg, and L. Fei-Fei, 'Scalable multi-label annotation,' in CHI , 2014. -
  18. [19][19] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick, 'Microsoft COCO: Common objects in context,' in ECCV , 2014. -
  19. [20][20] D. Scharstein and R. Szeliski, 'A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,' IJCV , vol. 47, no. 1-3, pp. 7-42, 2002. -
  20. [22][22] L. Fei-Fei, R. Fergus, and P. Perona, 'Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,' in CVPR Workshop of Generative Model Based Vision (WGMBV) , 2004. -
  21. [28][28] A. Torralba, R. Fergus, and W. T. Freeman, '80 million tiny images: A large data set for nonparametric object and scene recognition,' PAMI , vol. 30, no. 11, pp. 1958-1970, 2008. -
  22. [29][29] V. Ordonez, J. Deng, Y. Choi, A. Berg, and T. Berg, 'From large scale image categorization to entry-level categories,' in ICCV , 2013. -
  23. [32][32] E. Hjelm˚ as and B. Low, 'Face detection: A survey,' CVIU , vol. 83, no. 3, pp. 236-274, 2001. -
  24. [34][34] O. Russakovsky, J. Deng, Z. Huang, A. Berg, and L. Fei-Fei, 'Detecting avocados to zucchinis: what have we done, and where are we going?' in ICCV , 2013. -
  25. [35][35] J. Shotton, J. Winn, C. Rother, and A. Criminisi, 'TextonBoost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context,' IJCV , vol. 81, no. 1, pp. 2-23, 2009. -
  26. [36][36] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, 'A comparison and evaluation of multi-view stereo reconstruction algorithms,' in CVPR , 2006. -
  27. [37][37] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, 'Contour detection and hierarchical image segmentation,' PAMI , vol. 33, no. 5, pp. 898-916, 2011. -
  28. [38][38] C. Lampert, H. Nickisch, and S. Harmeling, 'Learning to detect unseen object classes by between-class attribute transfer,' in CVPR , 2009. -
  29. [39][39] G. Heitz and D. Koller, 'Learning spatial context: Using stuff to find things,' in ECCV , 2008. -
  30. [41][41] T. Berg and A. Berg, 'Finding iconic images,' in CVPR , 2009. -
  31. [47][47] T. Brox, L. Bourdev, S. Maji, and J. Malik, 'Object segmentation by alignment of poselet activations to image contours,' in CVPR , 2011. -
  32. [48][48] Y. Yang, S. Hallman, D. Ramanan, and C. Fowlkes, 'Layered object models for image segmentation,' PAMI , vol. 34, no. 9, pp. 1731-1743, 2012. -
  33. [49][49] D. Ramanan, 'Using segmentation to verify object hypotheses,' in CVPR , 2007. -
  34. [50][50] Q. Dai and D. Hoiem, 'Learning to localize detected objects,' in CVPR , 2012. -
  35. [51][51] C. Rashtchian, P. Young, M. Hodosh, and J. Hockenmaier, 'Collecting image annotations using Amazon's Mechanical Turk,' in NAACL Workshop , 2010. <!-- image --> Fig. 11: Icons of 91 categories in the MS COCO dataset grouped by 11 super-categories. We use these icons in our annotation pipeline to help workers quickly reference the indicated object category. <!-- image --> (a) Category Labeling (b) Instance Spotting (d) Segmentation Verification <!-- image --> (c) Instance Segmentation (e) Crowd Labeling <!-- image --> Fig. 12: User interfaces for collecting instance annotations, see text for details. <!-- image --> (a) PASCAL VOC. (b) MS COCO. Fig. 13: Random person instances from PASCAL VOC and MS COCO. At most one instance is sampled per image. TABLE 2: Candidate category list (272). Bold : selected categories (91). Bold ∗ : omitted categories in 2014 release (11). <!-- image --> (a) PASCAL VOC. (b) MS COCO. Fig. 14: Random chair instances from PASCAL VOC and MS COCO. At most one instance is sampled per image. Fig. 15: Examples of borderline segmentations that passed (top) or were rejected (bottom) in the verification stage. <!-- image --> library child's room street lighthouse church dining room park railway office auditorium beach playground restaurant shop river swimming pool kitchen home village forest living room hotel valley gas station bathroom classroom market garden TABLE 3: Scene category list. factory cafeteria harbor farm campus hospital room yard mountain bedroom food court parking lot plaza

Cited by

Sorted by citations

Authors
Lin, T.-Y., Maire, M., Belongie, S., Bourdev, L., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., & Dollár, P.
Affiliations
  • Cornell University (US)
  • Microsoft (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2014
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1405.0312
arXiv
1405.0312
OpenAlex
W2952122856
Cited by
2,579
References
6
Citations by year
2026: 393 · 2025: 451 · 2024: 455 · 2023: 520 · 2022: 234 · 2021: 168 · 2020: 142 · 2019: 102 · 2018: 38 · 2017: 30 · 2016: 24 · 2015: 16 · 2014: 6
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Multidisciplinary University Research Initiative (N00014-10-1-0933)
  • Office of Naval Research (N00014-10-1-0933, N00014)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
Keywords
COCO dataset, Flickr8k, image-text pairs, visual grounding, image captioning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review