End-to-End Object Detection with Transformers
Open accessThe DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index; 14 of the 29 sources its claims cite were checked for support.
Abstract
We present a new method that views object detection as a direct set prediction problem. Our approach streamlines the detection pipeline, effectively removing the need for many hand-designed components like a non-maximum suppression procedure or anchor generation that explicitly encode our prior knowledge about the task. The main ingredients of the new framework, called DEtection TRansformer or DETR, are a set-based global loss that forces unique predictions via bipartite matching, and a transformer encoder-decoder architecture. Given a fixed small set of learned object queries, DETR reasons about the relations of the objects and the global image context to directly output the final set of predictions in parallel. The new model is conceptually simple and does not require a specialized library, unlike many other modern detectors. DETR demonstrates accuracy and run-time performance on par with the well-established and highly-optimized Faster RCNN baseline on the challenging COCO object detection dataset. Moreover, DETR can be easily generalized to produce panoptic segmentation in a unified manner. We show that it significantly outperforms competitive baselines. Training code and pretrained models are available at this https URL.
Overview
CoChat Summary
TL;DR
DETR recasts object detection as direct set prediction using a transformer encoder-decoder and a bipartite matching loss, removing hand-designed components such as anchors and NMS. It matches a tuned Faster R-CNN baseline on COCO while excelling on large objects and extending naturally to panoptic segmentation.
Key findings
- On COCO val, DETR with ResNet-50 reaches 42.0 AP, comparable to Faster R-CNN-FPN+ at 42.0 AP; DETR improves AP_L by +7.8 but lags on AP_S by -5.5. DETR-DC5-R101 reaches 44.9 AP.
- Ablations show global encoder self-attention matters: removing all encoder layers drops AP by 3.9 points, with a 6.0 AP drop on large objects. Decoder depth adds +8.2 AP / +9.5 AP_50 from first to last layer, and removing FFN drops 2.3 AP.
- Loss design matters: GIoU alone loses only 0.7 AP to the combined loss, while ℓ1 without GIoU drops to 35.8 AP versus 40.6 AP for the baseline.
- For panoptic segmentation, DETR-R101 achieves 45.1 PQ on COCO val and 46 PQ on the test set, outperforming PanopticFPN++ and UPSNet baselines.
Methods
- DETR uses a CNN backbone, a transformer encoder-decoder, and a shared FFN. It predicts a fixed set of N=100 object queries in parallel, with no anchors or NMS, trained end-to-end via Hungarian bipartite matching between predictions and ground truth.
- Experiments use COCO 2017 detection and panoptic segmentation: 118k training images and 5k validation images, averaging 7 instances per image and up to 63 instances in training.
- Training uses AdamW, initial transformer LR 10^-4, backbone LR 10^-5, weight decay 10^-4, ResNet-50/ResNet-101 backbones, scale and random-crop augmentation, and schedules of 300 or 500 epochs; the 500-epoch schedule adds 1.5 AP over the shorter schedule.
Limitations
- The paper states DETR has lower performance on small objects and requires extra-long training schedules; it also notes new challenges in training, optimization, and small-object performance.
- The paper states DETR cannot predict more than its fixed number of query slots (100 in experiments); near that limit it saturates, detecting only 30 instances on average when 100 are present.
- Evident from the text: panoptic results show a mask AP deficit of up to 8 mAP on thing classes compared with baselines, despite competitive PQ, and the model’s behavior on many same-class instances is out-of-distribution.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
4 Valid · 9 Not valid · 5 Not checked · 79 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We found helpful to use auxiliary losses [1] in decoder during training, especially to help the model output the correct number of objects of each class.”
What the sources say
- Supportedcites [1] Al-Rfou, R., Choe, D., Constant, N., Guo, M., Jones, L.: Character-level language modeling with deeper self-attention. In: AAAI Conference on Artificial Intelligence (2019) source
The citation identifies the auxiliary-loss technique, not the citing authors’ own finding about object counts. The source’s Auxiliary Losses section (p. 2) describes adding these losses during training, and Character Transformer Model (p. 1) describes its masked self-attention as like that of a transformer decoder. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Their performances are significantly influenced by postprocessing steps to collapse near-duplicate predictions, by the design of the anchor sets and by the heuristics that assign target boxes to anchors [52].”
What the sources say
- Partly supportedcites [52] Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv:1904.07850 (2019) sourceExperimental — accuracy not yet measured
The Introduction (p. 1) says conventional detectors use non-maxima suppression to remove duplicate detections, and Related Work (p. 2) describes anchor designs and overlap thresholds used to assign boxes. Additional Experiments (p. 6) reports missed objects from insufficient anchor placement, supporting an effect on detection performance. The copy does not establish that all three factors significantly influence performance; its measured NMS effect is minor and concerns CenterNet. (LLM, conf=0.78)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Then, we pass fixed sine spatial positional encodings and the output encodings at input once, as in the original transformer [47], and find that this leads to 1.4 AP drop compared to passing the positional encodings directly in attention.”
What the sources say
- Could not verifycites [47] Vinyals, O., Bengio, S., Kudlur, M.: Order matters: sequence to sequence for sets. In: ICLR (2016) sourceExperimental — accuracy not yet measured
Our copy of 'Order Matters' contains no passage on fixed sine positional encodings, on passing encodings once at the input versus inside attention, or on any AP measurement: searches for 'fixed sine positional encodings passed at input once', 'sinusoidal function of position / self-attention', and 'position encoding added to input embeddings object detection average precision ablation' returned only the LSTM/memory attention material (§4.2 Attention Mechanisms, §4.3 Read-Process-Write) and the input-ordering experiments (§4.1, §4.4 sorting, §5.1.2 parsing, §5.2 5-gram). The closest content is §3, which notes a sequence can be turned into a set by pairing each element with its index — a different, non-sinusoidal, non-quantitative point — so the 1.4 AP comparison the sentence attributes here is simply not addressed in this paper. Nothing contrary is stated either, so this is unverifiable rather than disputed. (LLM, conf=0.72)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Recent work [52] demonstrate that the final performance of these systems heavily depends on the exact way these initial guesses are set.”
What the sources say
- Partly supportedcites [52] Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. arXiv:1904.07850 (2019) sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes anchor boxes as potential object locations, and Additional experiments (p. 6) reports that insufficient anchor placement can cause missed objects. This supports the importance of how those initial candidates are set, but the paper does not demonstrate that final system performance heavily depends on their exact configuration. (LLM, conf=0.78)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Closest to our approach are end-to-end set predictions for object detection [43] and instance segmentation [41,30,36,42].”
What the sources say
- Not checked — no full text of the sourcecites [43] Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NeurIPS (2014) source
- Supportedcites [41] Salvador, A., Bellver, M., Baradad, M., Marqués, F., Torres, J., Giró, X.: Recurrent neural networks for semantic instance segmentation. arXiv:1712.00617 (2017) source
The Introduction (p. 2) identifies this work as an end-to-end instance-segmentation method. The Training section (p. 5) says it matches its predicted instances to a set of ground-truth masks without imposing an instance order, supporting its identification here as an end-to-end set-prediction approach for instance segmentation. (LLM, conf=0.91)
- Could not verifycites [30] Parmar, N., et al.: Image transformer. In: ICML (2018) sourceExperimental — accuracy not yet measured
The Introduction (p. 1) describes predicting image pixels, and the Experiments (pp. 5–7) concern image generation and super-resolution. Searches for end-to-end set prediction, object detection, and instance segmentation found no passage in our copy addressing the attributed claim. (LLM, conf=0.77)
- Partly supportedcites [36] Ren, S., He, K., Girshick, R.B., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. PAMI 39 (2015) sourceExperimental — accuracy not yet measured
The Related Work section (p. 3) describes Faster R-CNN as a unified object-detection network, supporting the detection aspect. But the source describes anchor-based box proposals and training (Anchors, p. 3; Loss Function, p. 4), not end-to-end set prediction or instance segmentation—the part of the citing sentence to which this source appears to be attached. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [42] Stewart, R.J., Andriluka, M., Ng, A.Y.: End-to-end people detection in crowded scenes. In: CVPR (2015) source
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
44 more claims
Create a free account to see all 97 claims and every source verdict.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 16
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“and built the first fully end-to-end object detector”
From Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020) · cites this paper as Carion et al. (2020)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“Instead, [46] adopts a specially designed axial-attention [20] which is less scalable to standard computing facilities.”
From Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2021) · cites this paper as [46]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe cited source doesn’t support the claim as written.
In the citing paper
“First, though convolution is completely removed in [46] as in our SETR, their model still follows the conventional FCN design in that spatial resolution of feature maps is reduced progressively.”
From Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2021) · cites this paper as [46]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite its obvious advantages, DETR suffers from several problems: slow training convergence, high computational cost, and hard-to-optimize queries.”
From DETRs Beat YOLOs on Real-time Object Detection (2024) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, their high computational cost prevents them from meeting real-time detection requirements, so the NMS-free architecture does not demonstrate an inference speed advantage.”
From DETRs Beat YOLOs on Real-time Object Detection (2024) · cites this paper as [4]
Experimental — accuracy not yet measured
28 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.2 Ablations · p. 10
“The model has 41.3M parameters, achieves 40.6 and 42.0 AP on short and long schedules respectively, and runs at 28 FPS, similarly to Faster R-CNN-FPN with the same backbone.”
§ 4.1 Comparison with Faster R-CNN · p. 10
“| 6 | 86/23 | 41.3M | 40.6 | 61.6 | 19.9 | 44.3 | 60.2 |”
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
(2016) · Cited by 31,590
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Self-citation
Research works printed but not linked: 1
- Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., Savarese, S.: Generalized intersection over union. In: CVPR (2019)
Cited by
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
“For these four frameworks, we utilize the same settings: multi-scale training [8, 52] (resizing the input such that the shorter side is between 480 and 800 while the longer side is at most 1333), AdamW [40] optimizer (initial learning rate of 0.”
MethodAn Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
“…classification (Bello et al., 2019) or by further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classification (Wu et al., 2020), unsupervised object discovery…”
YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2022) · arXiv · doi:10.48550/arxiv.2207.02696 · Cited by 13,205
YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00721 · Cited by 13,198
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Self-citation
Metadata
- Authors
- Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S.
- Affiliations
- Meta (United States) (US)
- Corresponding author
- Carion, Nicolas
- ORCID
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer International Publishing
- Published
- 2020
- Pages
- 213-229
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W3030520226
- Semantic Scholar
- 962dc29fdc3fbdc5930a10aba114050b82fe5a3e
- Cited by
- 18,285
- References
- 66
- Citations by year
- 2026: 2756 · 2025: 4533 · 2024: 4451 · 2023: 3461 · 2022: 2172 · 2021: 855 · 2020: 56
- FWCI
- 504.02
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Lecture notes in computer science (publishedVersion, public-domain)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (public-domain)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- object detection, DETR, set prediction, bipartite matching, transformer encoder-decoder, object queries, anchor-free detection, panoptic segmentation, COCO, Faster R-CNN comparison
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: End-to-End Object Detection with Transformers. https://app.cochat.ai/papers/10.1007/978-3-030-58452-8_13
- The paper
- Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-End Object Detection with Transformers. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-030-58452-8_13
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.