Encoding Color Information for Visual Tracking: Algorithms and Benchmark
Abstract
While color information is known to provide rich discriminative clues for visual inference, most modern visual trackers limit themselves to the grayscale realm. Despite recent efforts to integrate color in tracking, there is a lack of comprehensive understanding of the role color information can play. In this paper, we attack this problem by conducting a systematic study from both the algorithm and benchmark perspectives. On the algorithm side, we comprehensively encode 10 chromatic models into 16 carefully selected state-of-the-art visual trackers. On the benchmark side, we compile a large set of 128 color sequences with ground truth and challenge factor annotations (e.g., occlusion). A thorough evaluation is conducted by running all the color-encoded trackers, together with two recently proposed color trackers. A further validation is conducted on an RGBD tracking benchmark. The results clearly show the benefit of encoding color information for tracking. We also perform detailed analysis on several issues, including the behavior of various combinations between color model and visual tracker, the degree of difficulty of each sequence for tracking, and how different challenge factors affect the tracking performance. We expect the study to provide the guidance, motivation, and benchmark for future work on encoding color in visual tracking.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097 · see citing works
High-Speed Tracking with Kernelized Correlation Filters
Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2345390 · Cited by 5,753 · see citing works
CONDENSATION—Conditional Density Propagation for Visual Tracking
Isard, M., & Blake, A. (1998) · International Journal of Computer Vision · doi:10.1023/a:1008078328650 · Cited by 4,886 · see citing works
Yılmaz, A., Javed, O., & Shah, M. A. (2006) · ACM Computing Surveys · doi:10.1145/1177352.1177355 · Cited by 4,807 · see citing works
Comaniciu, D., Ramesh, V., & Meer, P. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1195991 · Cited by 4,560 · see citing works
Online Object Tracking: A Benchmark
Wu, Y., Lim, J., & Yang, M.-H. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.312 · Cited by 4,138 · see citing works
Kalal, Z., Mikolajczyk, K., & Matas, J. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.239 · Cited by 3,357 · see citing works
Incremental Learning for Robust Visual Tracking
Ross, D. A., Lim, J., Lin, R.-S., & Yang, M.-H. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0075-7 · Cited by 3,061 · see citing works
Exploiting the Circulant Structure of Tracking-by-Detection with Kernels
Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33765-9_50 · Cited by 2,310 · see citing works
Robust object tracking with online multiple instance learning
Babenko, B., Yang, M.-H., & Belongie, S. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.226 · Cited by 2,007 · see citing works
Struck: Structured output tracking with kernels
Hare, S., Saffari, A., & Torr, P. H. S. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126251 · Cited by 1,831 · see citing works
Evaluating Color Descriptors for Object and Scene Recognition
van de Sande, K. E. A., Gevers, T., & Snoek, C. G. M. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.154 · Cited by 1,785 · see citing works
Adaptive Color Attributes for Real-Time Visual Tracking
Danelljan, M., Khan, F. S., Felsberg, M., & van de Weijer, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.143 · Cited by 1,541 · see citing works
Visual Tracking: An Experimental Survey
Smeulders, A. W. M., Chu, D. M., Cucchiara, R., Calderara, S., Dehghan, A., & Shah, M. A. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.230 · Cited by 1,527 · see citing works
Real-Time Compressive Tracking
Zhang, K., Lei Zhang, A., & Yang, M. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33712-3_62 · Cited by 1,442 · see citing works
Robust Fragments-based Tracking using the Integral Histogram
Adam, A., Rivlin, E., & Shimshoni, I. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.256 · Cited by 1,397 · see citing works
Color-Based Probabilistic Tracking
Pérez, P., Hue, C., Vermaak, J., & Gangnet, M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47969-4_44 · Cited by 1,391 · see citing works
Using color to separate reflection components
Shafer, S. A. (1985) · Color Research & Application · doi:10.1002/col.5080100409 · Cited by 1,280 · see citing works
Online selection of discriminative tracking features
Collins, R. T., Liu, Y., & Leordeanu, M. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.205 · Cited by 1,211 · see citing works
Visual tracking via adaptive structural local sparse appearance model
Jia, X., Lu, H., & Yang, M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247880 · Cited by 1,200 · see citing works
Real-Time Tracking via On-line Boosting
Grabner, H., Grabner, M., & Bischof, H. (2006) · Procedings of the British Machine Vision Conference 2006 · doi:10.5244/c.20.6 · Cited by 1,198 · see citing works
Kwon, J., & Lee, K. M. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539821 · Cited by 1,165 · see citing works
MEEM: Robust Tracking via Multiple Experts Using Entropy Minimization
Zhang, J., Ma, S., & Sclaroff, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_13 · Cited by 1,150 · see citing works
Semi-supervised On-Line Boosting for Robust Tracking
Gräbner, H., Leistner, C., & Bischof, H. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_19 · Cited by 1,095 · see citing works
Robust object tracking via sparsity-based collaborative model
Zhong, W., Lu, H., & Yang, M.-H. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247882 · Cited by 990 · see citing works
Real time robust L1 tracker using accelerated proximal gradient approach
Bao, C., Wu, Y., Ling, H., & Ji, H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247881 · Cited by 839 · see citing works
Self-citationRobust Visual Tracking and Vehicle Classification via Sparse Representation
Mei, X., & Ling, H. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.66 · Cited by 837 · see citing works
Self-citationLearning Color Names for Real-World Applications
van de Weijer, J., Schmid, C., Verbeek, J. J., & Larlus, D. (2009) · IEEE Transactions on Image Processing · doi:10.1109/tip.2009.2019809 · Cited by 710 · see citing works
Integral histogram: a fast way to extract histograms in Cartesian spaces
Porikli, F. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.188 · Cited by 701 · see citing works
Robust visual tracking via multi-task sparse learning
Zhang, T., Ghanem, B., Liu, S., & Ahuja, N. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247908 · Cited by 649 · see citing works
Zhang, K., Lei Zhang, A., & Yang, M. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2315808 · Cited by 537 · see citing works
Distribution fields for tracking
Sevilla-Lara, L., & Miller, E. G. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247891 · Cited by 515 · see citing works
Context tracker: Exploring supporters and distracters in unconstrained environments
Dinh, T. B., Vo, N., & Medioni, G. G. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995733 · Cited by 511 · see citing works
Robust tracking using local sparse appearance model and K-selection
Liu, B., Huang, J., Yang, L., & Kulikowsk, C. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995730 · Cited by 482 · see citing works
Transfer Learning Based Visual Tracking with Gaussian Processes Regression
Gao, J. G., Ling, H., Hu, W., & Xing, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10578-9_13 · Cited by 452 · see citing works
Self-citationColoring Local Feature Extraction
van de Weijer, J., & Schmid, C. (2006) · Lecture notes in computer science · doi:10.1007/11744047_26 · Cited by 436 · see citing works
Robust Visual Tracking via Structured Multi-Task Sparse Learning
Zhang, T., Ghanem, B., Liu, S., & Ahuja, N. (2012) · International Journal of Computer Vision · doi:10.1007/s11263-012-0582-z · Cited by 427 · see citing works
Kwon, J., & Lee, K. M. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126369 · Cited by 383 · see citing works
Notice of Removal: The Visual Object Tracking VOT2013 Challenge Results
Kristan, M., Pflugfelder, R. P., Leonardis, A., Matas, J., Porikli, F., Zajc, L. Č., Nebehay, G., Fernandez, G., Vojíř, T., Gatt, A., et al. (2013) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2013.20 · Cited by 310 · see citing works
Performance evaluation of local colour invariants
Burghouts, G. J., & Geusebroek, J. (2008) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2008.07.003 · Cited by 297 · see citing works
Transferring Rich Feature Hierarchies for Robust Visual TrackingPreprint
Wang, N., Li, S., Gupta, A. K., & Yeung, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.04587 · Cited by 293 · see citing works
Oron, S., Bar-Hillel, A., Levi, D., & Avidan, S. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247895 · Cited by 282 · see citing works
Color attributes for object detection
Khan, F. S., Anwer, R. M., van de Weijer, J., Bagdanov, A. D., Vanrell, M., & López, A. M. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248068 · Cited by 277 · see citing works
Tracking Revisited Using RGBD Camera: Unified Benchmark and Baselines
Song, S., & Xiao, J. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.36 · Cited by 275 · see citing works
Yang, M., Wu, Y. N., & Hua, G. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.146 · Cited by 231 · see citing works
Stalder, S., Gräbner, H., & Gool, L. V. (2009) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2009.5457445 · Cited by 215 · see citing works
Modulating Shape Features by Color Attention for Object Recognition
Khan, F. S., van de Weijer, J., & Vanrell, M. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0495-2 · Cited by 125 · see citing works
Coloring Action Recognition in Still Images
Khan, F. S., Anwer, R. M., van de Weijer, J., Bagdanov, A. D., López, A. M., & Felsberg, M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0633-0 · Cited by 114 · see citing works
A Superior Tracking Approach: Building a Strong Tracker through Fusion
Bailer, C., Pagani, A., & Stricker, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10584-0_12 · Cited by 76 · see citing works
Pang, Y., & Ling, H. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.346 · Cited by 72 · see citing works
Self-citationEvaluation of Color STIPs for Human Action Recognition
Everts, I., van Gemert, J. C., & Gevers, T. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.367 · Cited by 71 · see citing works
Online Object Tracking, Learning, and Parsing with And-Or Graphs
Yang, L., Wu, T., & Zhu, S. C. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.443 · Cited by 67 · see citing works
Tracking Using Multilevel Quantizations
Hong, Z., Wang, C., Mei, X., Prokhorov, D. V., & Tao, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10599-4_11 · Cited by 55 · see citing works
Trust-region methods for real-time tracking
Chen, H.-T., & Liu, T.-L. (2002) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2001.937697 · Cited by 46 · see citing works
Nam, H., Hong, S., & Han, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10602-1_8 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ECO: Efficient Convolution Operators for Tracking
Danelljan, M., Bhat, G., Khan, F. S., & Felsberg, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.733 · Cited by 2,805 · see citing works
Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
Danelljan, M., Robinson, A., Shahbaz Khan, F., & Felsberg, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_29 · Cited by 1,853 · see citing works
LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking
Fan, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C., & Ling, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00552 · Cited by 1,742 · see citing works
“Currently, the popular dense benchmarks contain OTB [52, 53], TC-128 [35], VOT [25], NUS-PRO [28], UAV [39], NfS [14] and GOT-10k [22].”
BackgroundInfluentialSelf-citationDetection and tracking meet drones challenge
Zhu, P., Wen, L., Du, D., Bian, X., Fan, H., Hu, Q., & Ling, H. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3119563 · Cited by 1,336 · see citing works
Self-citationLearning Spatio-Temporal Transformer for Visual Tracking
Yan, B., Peng, H., Fu, J., Wang, D., & Lu, H. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01028 · Cited by 1,280 · see citing works
“Table 5: Success score (%) comparisons on the collected large-scale benchmark NOTU and its subsets [24, 58, 33, 42].”
BackgroundMethodDiscriminative Correlation Filter with Channel and Spatial Reliability
Lukezic, A., Vojir, T., Zajc, L. C., Matas, J., & Kristan, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.515 · Cited by 1,153 · see citing works
Learning Spatial-Temporal Regularized Correlation Filters for Visual Tracking
Li, F., Tian, C., Zuo, W., Zhang, L., & Yang, M.-H. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00515 · Cited by 935 · see citing works
Deep visual tracking: Review and experimental comparison
Li, P., Wang, D., Wang, L., & Lu, H. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.11.007 · Cited by 517 · see citing works
LaSOT: A High-quality Large-scale Single Object Tracking Benchmark
Fan, H., Bai, H., Lin, L., Yang, F., Chu, P., Deng, G., Yu, S., Harshit, Huang, M., Liu, J., et al. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01387-y · Cited by 360 · see citing works
Self-citationGradNet: Gradient-Guided Network for Visual Object Tracking
Li, P., Chen, B., Ouyang, W., Wang, D., Yang, X., & Lu, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00626 · Cited by 348 · see citing works
Overview and methods of correlation filter algorithms in object tracking
Liu, S., Liu, D., Srivastava, G., Połap, D., & Woźniak, M. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-020-00161-4 · Cited by 305 · see citing works
End-to-End Representation Learning for Correlation Filter Based Tracking
Valmadre, J., Bertinetto, L., Henriques, J., Vedaldi, A., & Torr, P. H. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.531 · Cited by 274 · see citing works
Siam R-CNN: Visual Tracking by Re-Detection
Voigtlaender, P., Luiten, J., Torr, P. H. S., & Leibe, B. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00661 · Cited by 115 · see citing works
SITUP: Scale Invariant Tracking Using Average Peak-to-Correlation Energy
Ma, H., Acton, S. T., & Lin, Z. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2962694 · Cited by 39 · see citing works
Multiple Cues-Based Robust Visual Object Tracking Method
Khan, B., Jalil, A., Ali, A., Alkhaledi, K., Mehmood, K., Cheema, K. M., Murad, M., Tariq, H., & El-Sherbeeny, A. M. (2022) · Electronics · doi:10.3390/electronics11030345 · Cited by 5 · see citing works
“Comprehensive evaluation and analysis of proposed algorithms with state-of-theart methods on accepted datasets i.e., OTB-50 [33], OTB-100 [34], TC-128 [35], is carried out.”
Influential
- Authors
- Liang, P., Blasch, E., & Ling, H.
- Affiliations
- Meituan (China) (CN)
- United States Air Force Research Laboratory (US)
- Temple University (US)
- Venue
- IEEE Transactions on Image Processing
- Published
- 2015-12
- Volume
- 24
- Issue
- 12
- Pages
- 5630-5644
- ISSN
- 1057-7149 (ISSN-L)
- 1941-0042
- OpenAlex
- W1915785815
- Cited by
- 726
- References
- 55
- Citations by year
- 2026: 16 · 2025: 29 · 2024: 34 · 2023: 61 · 2022: 81 · 2021: 116 · 2020: 127 · 2019: 127 · 2018: 75 · 2017: 40 · 2016: 19 · 2015: 1
- FWCI
- 39.08
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Image Processing (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Natural Science Foundation of China (No. 61528204)
- National Science Foundation (1218156, 1350521)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Keywords
- visual object tracking, color information, color encoding, RGBD tracking, tracking benchmark, occlusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
