Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 282

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

End-to-End Temporal Action Detection With Transformer

Open access
Liu, Xiaolong, Wang, Qimeng, Hu, Yao, Tang, Xu, Zhang, Shiwei, Bai, Song, Bai, Xiang·2022·IEEE Transactions on Image Processing
DOI 10.1109/tip.2022.3195321
Open this paper in CoChat — free account

Abstract

Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tip.2022.3195321

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 63

  1. 10.1109/CVPR.2016.119 · DOI · Paper page
  2. 10.1109/CVPR42600.2020.01017 · DOI · Paper page
  3. 10.1109/CVPR.2016.214 · DOI · Paper page
  4. 10.1109/CVPR.2016.211 · DOI · Paper page
  5. 10.1109/ICCV.2017.617 · DOI · Paper page
  6. 10.1109/CVPR.2018.00124 · DOI · Paper page
  7. 10.1145/3123266.3123343 · DOI · Paper page
  8. 10.1109/CVPR.2017.342 · DOI
  9. 10.1109/ICCV.2017.317 · DOI · Paper page
  10. 10.1109/ICCV.2019.00399 · DOI · Paper page
  11. 10.1109/ICCV.2019.00719 · DOI · Paper page
  12. Vaswani, Attention is all you need, 5998
  13. 10.1109/ICCV.2017.610 · DOI · Paper page
  14. 10.1007/978-3-030-01219-9_16 · DOI
  15. 10.1109/ICCV.2017.322 · DOI · Paper page
  16. 10.1109/ICCV.2019.00876 · DOI
  17. 10.1016/j.cviu.2016.10.018 · DOI · Paper page
  18. 10.1109/CVPR.2016.337 · DOI
  19. 10.1109/CVPR.2017.338 · DOI
  20. 10.1109/LSP.2020.3037796 · DOI
  21. 10.1109/CVPR46437.2021.01241 · DOI
  22. 10.1007/978-3-030-01225-0_1 · DOI · Paper page
  23. 10.1007/978-3-030-01216-8_5 · DOI
  24. 10.1109/CVPR.2019.00372 · DOI
  25. 10.1007/978-3-319-46487-9_47 · DOI · Paper page
  26. 10.1109/CVPR.2017.675 · DOI · Paper page
  27. 10.1109/CVPR.2019.00043 · DOI · Paper page
  28. 10.1109/CVPR46437.2021.00333 · DOI · Paper page
  29. 10.1109/TIP.2020.3016486 · DOI
  30. 10.1109/TIP.2020.3044218 · DOI
  31. 10.1007/978-3-030-58548-8_25 · DOI
  32. 10.1109/CVPR.2018.00706 · DOI · Paper page
  33. 10.1007/978-3-030-01225-0_35 · DOI · Paper page
  34. 10.1109/CVPR.2019.00139 · DOI
  35. 10.1007/978-3-030-01270-0_10 · DOI · Paper page
  36. 10.1109/ICCV.2019.00562 · DOI
  37. 10.1109/TIP.2021.3078324 · DOI
  38. 10.1109/TIP.2021.3089361 · DOI
  39. 10.1109/TIP.2019.2922108 · DOI
  40. 10.1609/aaai.v34i07.6760 · DOI
  41. 10.1609/aaai.v35i2.16256 · DOI
  42. 10.1007/978-3-030-58539-6_3 · DOI
  43. 10.1109/ICCV48922.2021.00790 · DOI
  44. 10.1109/ICCV48922.2021.00415 · DOI
  45. 10.1109/CVPR52688.2022.01182 · DOI
  46. 10.1007/s11432-021-3445-y · DOI · Paper page
  47. 10.1109/ICCV.2019.00756 · DOI · Paper page
  48. Wu, 2021, SeqFormer: Sequential transformer for video instance segmentation, arXiv:2112.08275 · arXiv
  49. 10.1109/CVPR42600.2020.00877 · DOI · Paper page
  50. 10.1109/CVPR.2018.00911 · DOI · Paper page
  51. 10.1109/CVPR.2019.00033 · DOI
  52. Nawhal, 2021, Activity graph transformer for temporal action localization, arXiv:2101.08540 · arXiv
  53. 10.1109/ICCV48922.2021.01327 · DOI
  54. Wang, 2021, Temporal action proposal generation with transformers, arXiv:2105.12043 · arXiv
  55. 10.1007/978-3-030-58604-1_8 · DOI
  56. Lei Ba, 2016, Layer normalization, arXiv:1607.06450 · arXiv
  57. 10.1007/978-3-319-46484-8_2 · DOI · Paper page
  58. Loshchilov, Decoupled weight decay regularization, 1
  59. Ghanem, 2017, ActivityNet challenge 2017 summary, arXiv:1710.08011 · arXiv
  60. 10.1007/978-3-030-58598-3_32 · DOI
  61. 10.1109/ICCVW54120.2021.00356 · DOI
  62. 10.1109/CVPR.2017.155 · DOI · Paper page
  63. 10.1109/CVPR52688.2022.01938 · DOI

Cited by

See all 282

Metadata

Authors
Liu, X., Wang, Q., Hu, Y., Tang, X., Zhang, S., Bai, S., & Bai, X.
Venue
IEEE Transactions on Image Processing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022
Volume
31
Pages
5427-5441
ISSN
  • 1057-7149 (ISSN-L)
  • 1941-0042
DOI
10.1109/tip.2022.3195321
OpenAlex
W3173459793
Cited by
282
References
11
FWCI
19.10
Type
journal article
Language
en
Open access
green
Licence
Funders
National Key Research and Development Program of China (2018YFB1004600)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.