The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
HOTR: End-to-End Human-Object Interaction Detection with Transformers
Abstract
Human-Object Interaction (HOI) detection is a task of identifying "a set of interactions" in an image, which involves the i) localization of the subject (i.e., humans) and target (i.e., objects) of interaction, and ii) the classification of the interaction labels. Most existing methods have indirectly addressed this task by detecting human and object instances and individually inferring every pair of the detected instances. In this paper, we present a novel framework, referred by HOTR, which directly predicts a set of 〈human, object, interaction〉 triplets from an image based on a transformer encoder-decoder architecture. Through the set prediction, our method effectively exploits the inherent semantic relationships in an image and does not require time-consuming post-processing which is the main bottleneck of existing methods. Our proposed algorithm achieves the state-of-the-art performance in two HOI detection benchmarks with an inference time under 1 ms after object detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58452-8_13 · Cited by 18,285
End-to-End Object Detection with Transformers
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020) · arXiv · doi:10.48550/arxiv.2005.12872 · Cited by 18,253
(2010) · Cited by 14,134
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Relation Networks for Object Detection
Hu, H., Gu, J., Zhang, Z., Dai, J., & Wei, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00378 · Cited by 1,524
Detecting and Recognizing Human-Object Interactions
Gkioxari, G., Girshick, R., Dollar, P., & He, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00872 · Cited by 602
Learning Human-Object Interactions by Graph Parsing Neural Networks
Qi, S., Wang, W., Jia, B., Shen, J., & Zhu, S.-C. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01240-3_25 · Cited by 572
Learning to Detect Human-Object Interactions
Chao, Y.-W., Liu, Y., Liu, X., Zeng, H., & Deng, J. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00048 · Cited by 520
Gupta, S., & Malik, J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.04474 · Cited by 289
PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection
Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., & Feng, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00056 · Cited by 289
Learning Human-Object Interaction Detection Using Interaction Points
Wang, T., Yang, T., Danelljan, M., Khan, F. S., Zhang, X., & Sun, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00417 · Cited by 252
VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions
Ulutan, O., Iftekhar, A. S. M., & Manjunath, B. S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01363 · Cited by 241
Pose-Aware Multi-Level Feature Network for Human Object Interaction Detection
Wan, B., Zhou, D., Liu, Y., Li, R., & He, X. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00956 · Cited by 240
Transferable Interactiveness Knowledge for Human-Object Interaction Detection
Li, Y.-L., Zhou, S., Huang, X., Xu, L., Ma, Z., Fang, H.-S., Wang, Y., & Lu, C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00370 · Cited by 232
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
Gao, C., Zou, Y., & Huang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.10437 · Cited by 204
PaStaNet: Toward Human Activity Knowledge Engine
Li, Y., Xu, L., Liu, X., Huang, X., Xu, Y., Wang, S., Fang, H.-S., Ma, Z., Chen, M., & Lu, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00046 · Cited by 198
DRG: Dual Relation Graph for Human-Object Interaction Detection
Gao, C., Xu, J., Zou, Y., & Huang, J.-B. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58610-2_41 · Cited by 198
Detailed 2D-3D Joint Representation for Human-Object Interaction
Li, Y., Liu, X., Han, L., Wang, S., Liu, J., Li, J., & Lu, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01018 · Cited by 181
Visual Compositional Learning for Human-Object Interaction Detection
Hou, Z., Peng, X., Qiao, Y., & Tao, D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58555-6_35 · Cited by 176
Deep Contextual Attention for Human-Object Interaction Detection
Wang, T., Anwer, R. M., Khan, M. H., Khan, F. S., Pang, Y., Shao, L., & Laaksonen, J. T. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00579 · Cited by 161
Learning to Detect Human-Object Interactions With Knowledge
Xu, B., Wong, Y., Li, J., Zhao, Q., & Kankanhalli, M. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00212 · Cited by 160
Relation Parsing Neural Network for Human-Object Interaction Detection
Zhou, P., & Chi, M. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00093 · Cited by 153
Gupta, T., Schwing, A. G., & Hoiem, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00977 · Cited by 145
Detecting Unseen Visual Relations Using Analogies
Peyre, J., Šivic, J., Laptev, I., & Schmid, C. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00207 · Cited by 126
Detecting Human-Object Interactions via Functional Generalization
Bansal, A., Rambhatla, S. S., Shrivastava, A., & Chellappa, R. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i07.6616 · Cited by 123
ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detection
Liu, Y., Yuan, J., & Chen, C. W. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413600 · Cited by 121
Detecting Human-Object Interactions with Action Co-occurrence Priors
Kim, D.-J., Sun, X. N., Choi, J., Lin, S. C.-F., & Kweon, I. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58589-1_43 · Cited by 121
Contextual Heterogeneous Graph Network for Human-Object Interaction Detection
Wang, H., Zheng, W., & Yingbiao, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_15 · Cited by 97
Amplifying Key Cues for Human-Object-Interaction Detection
Liu, Y., Chen, Q., & Zisserman, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_15 · Cited by 93
Polysemy Deciphering Network for Human-Object Interaction Detection
Zhong, X., Ding, C., Qu, X., & Tao, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58565-5_5 · Cited by 92
Detecting Visual Relationships Using Box Attention
Kolesnikov, A., Kuznetsova, A., Lampert, C. H., & Ferrari, V. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00217 · Cited by 83
HOI Analysis: Integrating and Decomposing Human-Object InteractionPreprint
Li, Y., Liu, X., Wu, X., Li, Y., & Lu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.16219 · Cited by 83
Cited by
EDTER: Edge Detection with Transformer
Pu, M., Huang, Y., Liu, Y., Guan, Q., & Ling, H. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00146 · Cited by 253
SPViT: Enabling Faster Vision Transformers via Latency-Aware Soft Token Pruning
Kong, Z., Dong, P., Ma, X., Meng, X., Niu, W., Sun, M., Shen, X., Yuan, G., Ren, B., Tang, H., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20083-0_37 · Cited by 169
“Recently, a new trend of leveraging Transformer architecture [80] into the computer vision domain has emerged [38,13,41,21,90,109,33,75].”
Background
Metadata
- Authors
- Kim, B., Lee, J., Kang, J., Kim, E.-S., & Kim, H. J.
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 74-83
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3174480456
- Cited by
- 291
- References
- 33
- FWCI
- 18.28
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.