CoChat

End-to-End Object Detection with Transformers

Open access
Carion, Nicolas, Massa, Francisco, Synnaeve, Gabriel, Usunier, Nicolas, Kirillov, Alexander, Zagoruyko, Sergey · 2020 · Lecture Notes in Computer Science

The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 14 of the 29 sources its claims cite were checked for support.

Abstract

We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at this https URL.

Overview

CoChat Summary

TL;DR

DETR recasts object detection as direct set prediction using a transformer encoder-decoder and a bipartite matching loss, removing hand-designed components such as anchors and NMS. It matches a tuned Faster R-CNN baseline on COCO while excelling on large objects and extending naturally to panoptic segmentation.

Key findings

  • On COCO val, DETR with ResNet-50 reaches 42.0 AP, comparable to Faster R-CNN-FPN+ at 42.0 AP; DETR improves AP_L by +7.8 but lags on AP_S by -5.5. DETR-DC5-R101 reaches 44.9 AP.
  • Ablations show global encoder self-attention matters: removing all encoder layers drops AP by 3.9 points, with a 6.0 AP drop on large objects. Decoder depth adds +8.2 AP / +9.5 AP_50 from first to last layer, and removing FFN drops 2.3 AP.
  • Loss design matters: GIoU alone loses only 0.7 AP to the combined loss, while ℓ1 without GIoU drops to 35.8 AP versus 40.6 AP for the baseline.
  • For panoptic segmentation, DETR-R101 achieves 45.1 PQ on COCO val and 46 PQ on the test set, outperforming PanopticFPN++ and UPSNet baselines.

Methods

  • DETR uses a CNN backbone, a transformer encoder-decoder, and a shared FFN. It predicts a fixed set of N=100 object queries in parallel, with no anchors or NMS, trained end-to-end via Hungarian bipartite matching between predictions and ground truth.
  • Experiments use COCO 2017 detection and panoptic segmentation: 118k training images and 5k validation images, averaging 7 instances per image and up to 63 instances in training.
  • Training uses AdamW, initial transformer LR 10^-4, backbone LR 10^-5, weight decay 10^-4, ResNet-50/ResNet-101 backbones, scale and random-crop augmentation, and schedules of 300 or 500 epochs; the 500-epoch schedule adds 1.5 AP over the shorter schedule.

Limitations

  • The paper states DETR has lower performance on small objects and requires extra-long training schedules; it also notes new challenges in training, optimization, and small-object performance.
  • The paper states DETR cannot predict more than its fixed number of query slots (100 in experiments); near that limit it saturates, detecting only 30 instances on average when 100 are present.
  • Evident from the text: panoptic results show a mask AP deficit of up to 8 mAP on thing classes compared with baselines, despite competitive PQ, and the model’s behavior on many same-class instances is out-of-distribution.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/978-3-030-58452-8_13

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

4 Valid · 9 Not valid · 5 Not checked · 79 Own results and interpretations

44 more claims

Create a free account to see all 97 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 35
  • Probably not a substantial claim. 16

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

28 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 71 references

Research works printed but not linked: 1

  1. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union. In: CVPR (2019)

Cited by

See all 18,285

Metadata

Authors
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S.
Affiliations
Meta (United States) (US)
Corresponding author
Carion, Nicolas
Venue
Lecture Notes in Computer Science
Published
2020
Pages
213-229
ISSN
  • 0302-9743 (ISSN-L)
  • 1611-3349
OpenAlex
W3030520226
Cited by
18,285
References
66
Citations by year
2026: 2756 · 2025: 4533 · 2024: 4451 · 2023: 3461 · 2022: 2172 · 2021: 855 · 2020: 56
FWCI
504.02
Type
conference paper
Language
en
Open access
Licence
Locations
  • Lecture notes in computer science (publishedVersion, public-domain)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
object detection, DETR, set prediction, bipartite matching, transformer encoder-decoder, object queries, anchor-free detection, panoptic segmentation, COCO, Faster R-CNN comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: End-to-End Object Detection with Transformers. https://app.cochat.ai/papers/10.1007/978-3-030-58452-8_13
The paper
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-030-58452-8_13

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.