CoChat
Sign Up
Cited by 406Video Surveillance and Tracking MethodsAdvanced Neural Network ApplicationsAutonomous Vehicle Technology and Safety

Online Multi-object Tracking Using CNN-Based Single Object Tracker with Spatial-Temporal Attention Mechanism

Open access
Chu, Qi, Ouyang, Wanli, Li, Hongsheng, Wang, Xiaogang, Liu, Bin, Yu, Nenghai·2017·2017 IEEE International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2017.518
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we propose a CNN-based framework for online MOT. This framework utilizes the merits of single object trackers in adapting appearance models and searching for target in the next frame. Simply applying single object tracker for MOT will encounter the problem in computational efficiency and drifted results caused by occlusion. Our framework achieves computational efficiency by sharing features and using ROI-Pooling to obtain individual features for each target. Some online learned target-specific CNN layers are used for adapting the appearance model for each target. In the framework, we introduce spatial-temporal attention mechanism (STAM) to handle the drift caused by occlusion and interaction among targets. The visibility map of the target is learned and used for inferring the spatial attention map. The spatial attention map is then applied to weight the features. Besides, the occlusion status can be estimated from the visibility map, which controls the online updating process via weighted loss on training samples with different occlusion statuses in different frames. It can be considered as temporal attention mechanism. The proposed algorithm achieves 34.3% and 46.0% in MOTA on challenging MOT15 and MOT16 benchmark dataset respectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2017.518

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For MOT, attention has only been used to associate a given set of object detections, not tackling the detection and tracking problem jointly.

    “For MOT, attention has only been used to associate a given set of object detections [11,70], not tackling the detection and tracking problem jointly.”

    From TrackFormer: Multi-Object Tracking with Transformers (2022) · cites this paper as [11]

    Section 3.3.1 says the method uses detections from an offline-trained detector, supporting the point that it does not tackle detection and tracking jointly. But its candidate states also include motion-model samples, and Section 3.3.2 uses spatial attention to extract features from those candidates—not only to associate a given set of detections. Page numbers are unavailable in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. B. Babenko, M.-H. Yang, and S. Belongie. Robust object tracking with online multiple instance learning. TPAMI, 33(8):1619-1632, 2011. 1
  2. S.-H. Bae and K.-J. Yoon. Robust online multi-object track- ing based on tracklet confidence and online discriminative appearance learning. In CVPR, 2014. 2, 8
  3. Y. Ban, S. Ba, X. Alameda-Pineda, and R. Horaud. Tracking multiple persons based on a variational bayesian model. In ECCV, 2016. 8
  4. K. Bernardin and R. Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Jour- nal on Image and Video Processing, 2008(1):1-10, 2008.
  5. M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and L. Van Gool. Online multiperson tracking-by-detection from a single, uncalibrated camera. TPAMI, 33(9):1820- 1833, 2011.
  6. W. Brendel, M. Amer, and S. Todorovic. Multiobject track- ing as maximum weight independent set. In CVPR, 2011. 2
  7. W. Choi. Near-online multi-target tracking with aggregated local flow descriptor. In ICCV, 2015. 8
  8. M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg. Learning spatially regularized correlation filters for visual tracking. In ICCV, 2015. 1
  9. A. Dehghan, S. Modiri Assari, and M. Shah. Gmmcp tracker: Globally optimal generalized maximum multi clique prob- lem for multiple object tracking. In CVPR, 2015. 2
  10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 3, 6
  11. C. Dicle, O. I. Camps, and M. Sznaier. The way they move: Tracking multiple targets with similar appearance. In ICCV, 2013. 8
  12. P. Dollár, R. Appel, S. Belongie, and P. Perona. Fast feature pyramids for object detection. TPAMI, 36(8):1532-1545, 2014. 7
  13. R. Girshick. Fast r-cnn. In ICCV, 2015. 1, 2, 3
  14. S. Hamid Rezatofighi, A. Milan, Z. Zhang, Q. Shi, A. Dick, and I. Reid. Joint probabilistic data association revisited. In ICCV, 2015. 8
  15. S. Hare, A. Saffari, and P. H. Torr. Struck: Structured output tracking with kernels. In ICCV, 2011. 1
  16. S. Hong, T. You, S. Kwak, and B. Han. Online tracking by learning discriminative saliency map with convolutional neural network. In ICML, 2015. 1
  17. J. Hong Yoon, C.-R. Lee, M.-H. Yang, and K.-J. Yoon. On- line multi-object tracking via structural constraint event ag- gregation. In CVPR, 2016. 8
  18. W. Hu, X. Li, W. Luo, X. Zhang, S. Maybank, and Z. Zhang. Single and multiple object tracking using log-euclidean rie- mannian subspace and block-division appearance model. TPAMI, 34(12):2420-2440, 2012. 2
  19. C. Huang, B. Wu, and R. Nevatia. Robust object tracking by hierarchical association of detection responses. In ECCV. 2008. 2
  20. H. Izadinia, I. Saleemi, W. Li, and M. Shah. 2t: Multiple people multiple parts tracker. In ECCV, 2012. 2
  21. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Gir- shick, S. Guadarrama, and T. Darrell. Caffe: Convolu- tional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093, 2014. 6 · arXiv
  22. C. Kim, F. Li, A. Ciptadi, and J. M. Rehg. Multiple hypoth- esis tracking revisited. In ICCV, 2015. 8
  23. L. Leal-Taixé, A. Milan, I. Reid, S. Roth, and K. Schindler. Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv preprint arXiv:1504.01942, 2015. 2, 7 · arXiv
  24. Y. Li, C. Huang, and R. Nevatia. Learning to associate: Hybridboosted multi-target tracker for crowded scene. In CVPR, 2009. 7
  25. C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang. Hierarchical convolutional features for visual tracking. In ICCV, 2015. 1
  26. A. Milan, S. Roth, and K. Schindler. Continuous energy minimization for multitarget tracking. TPAMI, 36(1):58-72, 2014. 2, 8
  27. S. Oh, S. Russell, and S. Sastry. Markov chain monte carlo data association for multi-target tracking. IEEE Transactions on Automatic Control, 54(3):481-497, 2009. 2
  28. W. Ouyang, H. Li, X. Zeng, and X. Wang. Learning deep representation with large-scale attributes. In ICCV, 2015. 1
  29. W. Ouyang, X. Wang, X. Zeng, S. Qiu, P. Luo, Y. Tian, H. Li, S. Yang, Z. Wang, C.-C. Loy, et al. Deepid-net: Deformable deep convolutional neural networks for object detection. In CVPR, 2015. 1
  30. W. Ouyang, X. Wang, C. Zhang, and X. Yang. Factors in finetuning deep model for object detection with long-tail dis- tribution. In CVPR, 2016. 2
  31. W. Ouyang, X. Zeng, and X. Wang. Learning mutual visibil- ity relationship for pedestrian detection with a deep model. IJCV, 120(1):14-27, 2016. 2
  32. R. Sanchez-Matilla, F. Poiesi, and A. Cavallaro. Online multi-target tracking with strong and weak detections. In ECCV, 2016. 8
  33. G. Shu, A. Dehghan, O. Oreifej, E. Hand, and M. Shah. Part- based multiple-person tracking with partial occlusion han- dling. In CVPR, 2012. 2
  34. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 6 · arXiv
  35. S. Tang, B. Andres, M. Andriluka, and B. Schiele. Subgraph decomposition for multi-target tracking. In CVPR, 2015. 2
  36. S. Tang, B. Andres, M. Andriluka, and B. Schiele. Multi- person tracking by multicut and deep matching. In ECCV, 2016. 2, 8
  37. S. Tang, M. Andriluka, and B. Schiele. Detection and track- ing of occluded people. IJCV, 110(1):58-69, 2014. 2
  38. L. Wang, W. Ouyang, X. Wang, and H. Lu. Visual tracking with fully convolutional networks. In ICCV, 2015. 1
  39. L. Wang, W. Ouyang, X. Wang, and H. Lu. Stct: Sequen- tially training convolutional networks for visual tracking. In CVPR, 2016. 1
  40. B. Wu and R. Nevatia. Detection and tracking of multi- ple, partially occluded humans by bayesian combination of edgelet based part detectors. IJCV, 75(2):247-266, 2007. 2
  41. Y. Xiang, A. Alahi, and S. Savarese. Learning to track: Online multi-object tracking by decision making. In ICCV, 2015. 2, 8
  42. J. Xing, H. Ai, and S. Lao. Multi-object tracking through occlusions by local tracklets filtering and global tracklets as- sociation with detection responses. In CVPR, 2009. 2
  43. X. Yan, X. Wu, I. A. Kakadiaris, and S. K. Shah. To track or to detect? an ensemble framework for optimal selection. In ECCV, 2012. 2
  44. B. Yang and R. Nevatia. Online learned discriminative part-based appearance models for multi-human tracking. In ECCV. 2
  45. S. Yi, H. Li, and X. Wang. Understanding pedestrian behav- iors from stationary crowd groups. In CVPR, 2015. 1
  46. J. H. Yoon, M.-H. Yang, J. Lim, and K.-J. Yoon. Bayesian multi-object tracking using motion context from multiple ob- jects. In WACV, 2015. 8
  47. A. R. Zamir, A. Dehghan, and M. Shah. Gmcp-tracker: Global multi-object tracking using generalized minimum clique graphs. In ECCV. 2012. 2
  48. C. Zhang, H. Li, X. Wang, and X. Yang. Cross-scene crowd counting via deep convolutional neural networks. In CVPR, 2015. 1
  49. L. Zhang, Y. Li, and R. Nevatia. Global data association for multi-object tracking using network flows. In CVPR, 2008. 2
  50. L. Zhang and L. van der Maaten. Structure preserving object tracking. In CVPR, 2013. 2

Cited by

Sorted by citations

Authors
Chu, Q., Ouyang, W., Li, H., Wang, X., Liu, B., & Yu, N.
Affiliations
  • University of Science and Technology of China (CN)
  • The University of Sydney (AU)
  • Chinese University of Hong Kong (HK)
ORCID
Venue
2017 IEEE International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2017-10
Pages
4846-4855
DOI
10.1109/iccv.2017.518
OpenAlex
W2963481014
Cited by
406
References
8
Citations by year
2026: 6 · 2025: 25 · 2024: 30 · 2023: 49 · 2022: 54 · 2021: 83 · 2020: 69 · 2019: 58 · 2018: 32
FWCI
12.71
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (61371192)
  • Chinese Academy of Sciences (CXJJ-17S044)
  • Chinese University of Hong Kong
  • SenseTime Group
  • Fundamental Research Funds for the Central Universities (WK2100330002)
  • Office of Naval Research (N00014-15-1-2356, N00014)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
Keywords
online multi-object tracking, spatio-temporal attention, occlusion handling, visibility map, ROI pooling, feature sharing, temporal attention, target drift, MOT16

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review