Learning by Tracking: Siamese CNN for Robust Target Association
Open accessAbstract
This paper introduces a novel approach to the task of data association within the context of pedestrian tracking, by introducing a two-stage learning scheme to match pairs of detections. First, a Siamese convolutional neural network (CNN) is trained to learn descriptors encoding local spatio-temporal structures between the two input image patches, aggregating pixel values and optical flow information. Second, a set of contextual features derived from the position and size of the compared input patches are combined with the CNN output by means of a gradient boosting classifier to generate the final matching probability. This learning approach is validated by using a linear programming based multi-person tracker showing that even a simple and efficient tracker may outperform much more complex models when fed with our learned matching probabilities. Results on publicly available sequences show that our method meets state-of-the-art standards in multiple people tracking.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
Object Detection with Discriminatively Trained Part-Based Models
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.167 · Cited by 9,853 · see citing works
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951 · see citing works
Friedman, J. H. (2002) · Computational Statistics & Data Analysis · doi:10.1016/s0167-9473(01)00065-2 · Cited by 7,067 · see citing works
DeepFace: Closing the Gap to Human-Level Performance in Face Verification
Taigman, Y., Yang, M., Ranzato, M., & Wolf, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.220 · Cited by 6,542 · see citing works
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036 · see citing works
Two-Frame Motion Estimation Based on Polynomial Expansion
Farnebäck, G. (2003) · Lecture notes in computer science · doi:10.1007/3-540-45103-x_50 · Cited by 3,010 · see citing works
Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics
Bernardin, K., & Stiefelhagen, R. (2008) · EURASIP Journal on Image and Video Processing · doi:10.1155/2008/246309 · Cited by 2,918 · see citing works
Fast Feature Pyramids for Object Detection
Dollár, P., Appel, R., Belongie, S., & Perona, P. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2300479 · Cited by 1,941 · see citing works
You'll never walk alone: Modeling social behavior for multi-target tracking
Pellegrini, S., Ess, A., Schindler, K., & van Gool, L. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459260 · Cited by 1,683 · cited 3× · see citing works
“A different line of attack is to develop sophisticated motion models in order to better predict a tracked person’s location, most notably models that include interactions between nearby people [1, 10, 35, 44, 47, 56].”
BackgroundMethodSelf-citationLearning to compare image patches via convolutional neural networks
Zagoruyko, S., & Nikos Komodakis, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299064 · Cited by 1,540 · see citing works
Multiple object tracking: A literature review
Luo, W., Xing, J., Milan, A., Zhang, X.-Q., Liu, W., Zhao, X., & Kim, T.-K. (2014) · Artificial Intelligence · doi:10.1016/j.artint.2020.103448 · Cited by 1,095 · see citing works
Multiple Object Tracking Using K-Shortest Paths Optimization
Berclaz, J., Fleuret, F., Turetken, E., & Fua, P. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.21 · Cited by 1,015 · see citing works
Global data association for multi-object tracking using network flows
Zhang, L., Li, Y., & Nevatia, R. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587584 · Cited by 945 · see citing works
MOTChallenge 2015: Towards a Benchmark for Multi-Target TrackingPreprint
Leal-Taixé, L., Milan, A., Reid, I. D., Roth, S., & Schindler, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.01942 · Cited by 923 · see citing works
Self-citationComputing the stereo matching cost with a convolutional neural network
Žbontar, J., & LeCun, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298767 · Cited by 816 · see citing works
FlowNet: Learning Optical Flow with Convolutional NetworksPreprint
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 814 · see citing works
Globally-optimal greedy algorithms for tracking a variable number of objects
Pirsiavash, H., Ramanan, D., & Fowlkes, C. C. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995604 · Cited by 790 · see citing works
Rotation-invariant convolutional neural networks for galaxy morphology prediction
Dieleman, S., Willett, K. W., & Dambre, J. (2015) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stv632 · Cited by 781 · see citing works
Learning to Track: Online Multi-object Tracking by Decision Making
Yu, X., Alahi, A., & Savarese, S. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.534 · Cited by 684 · see citing works
Multiple Hypothesis Tracking Revisited
Kim, C., Li, F., Ciptadi, A., & Rehg, J. M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.533 · Cited by 666 · see citing works
Learning to associate: HybridBoosted multi-target tracker for crowded scene
Li, Y., Huang, C., & Nevatia, R. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206735 · Cited by 633 · see citing works
Hough Forests for Object Detection, Tracking, and Action Recognition
Gall, J., Yao, A., Razavi, N., Gool, L. V., & Lempitsky, V. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.70 · Cited by 584 · see citing works
Continuous Energy Minimization for Multitarget Tracking
Milan, A., Roth, S., & Schindler, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.103 · Cited by 556 · see citing works
Self-citationWho are you with and where are you going?
Yamaguchi, K., Berg, A. C., Ortiz, L. E., & Berg, T. L. (2011) · doi:10.1109/cvpr.2011.5995468 · Cited by 502 · see citing works
Bae, S., & Yoon, K. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.159 · Cited by 415 · see citing works
Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor
Choi, W. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.347 · Cited by 409 · see citing works
3D Traffic Scene Understanding From Movable Platforms
Geiger, A., Lauer, M., Wojek, C., Stiller, C., & Urtasun, R. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.185 · Cited by 397 · see citing works
GMCP-Tracker: Global Multi-object Tracking Using Generalized Minimum Clique Graphs
Zamir, A., Dehghan, A., & Shah, M. A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33709-3_25 · Cited by 360 · see citing works
A Unified Framework for Multi-target Tracking and Collective Activity Recognition
Choi, W., & Savarese, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33765-9_16 · Cited by 359 · see citing works
Joint Probabilistic Data Association Revisited
Rezatofighi, S. H., Milan, A., Zhang, Z., Shi, Q., Dick, A., & Reid, I. D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.349 · Cited by 357 · see citing works
Human Tracking Using Convolutional Neural Networks
Fan, J., Xu, W., Wu, Y. N., & Gong, Y. (2010) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2010.2066286 · Cited by 350 · see citing works
Discrete-continuous optimization for multi-target tracking
Andriyenko, A., Schindler, K., & Roth, S. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247893 · Cited by 337 · see citing works
Self-citationA Linear Programming Approach for Multiple Object Tracking
Jiang, H., Fels, S., & Little, J. J. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383180 · Cited by 323 · see citing works
The Way They Move: Tracking Multiple Targets with Similar Appearance
Dicle, C., Camps, O., & Sznaier, M. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.286 · Cited by 312 · see citing works
Transferring Rich Feature Hierarchies for Robust Visual TrackingPreprint
Wang, N., Li, S., Gupta, A. K., & Yeung, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1501.04587 · Cited by 293 · see citing works
An online learned CRF model for multi-target tracking
Yang, B., & Nevatia, R. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247907 · Cited by 293 · see citing works
Learning an Image-Based Motion Context for Multiple People Tracking
Leal-Taixé, L., Fenzi, M., Kuznetsova, A., Rosenhahn, B., & Savarese, S. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.453 · Cited by 281 · see citing works
Bayesian Multi-object Tracking Using Motion Context from Multiple Objects
Yoon, J. H., Yang, M., Lim, J., & Yoon, K. (2015) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2015.12 · Cited by 216 · see citing works
Leal-Taixé, L., Pons‐Moll, G., & Rosenhahn, B. (2011) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2011.6130233 · Cited by 212 · see citing works
Improving multiview face detection with multi-task deep convolutional neural networks
Zhang, C., & Zhang, Z. (2014) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2014.6835990 · Cited by 210 · see citing works
How does person identity recognition help multi-person tracking?
Kuo, C.-H., & Nevatia, R. (2011) · doi:10.1109/cvpr.2011.5995384 · Cited by 207 · see citing works
Robust People Tracking with Global Trajectory Optimization
Berclaz, J., Fleuret, F., & Fua, P. (2006) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2006.258 · Cited by 202 · see citing works
Multi-Target Tracking by Discrete-Continuous Energy Minimization
Milan, A., Schindler, K., & Roth, S. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2505309 · Cited by 193 · see citing works
Self-citationJoint tracking and segmentation of multiple targets
Milan, A., Leal-Taixé, L., Schindler, K., & Reid, I. D. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299178 · Cited by 183 · see citing works
Self-citationLearning to Compare Image Patches via Convolutional Neural NetworksPreprint
Zagoruyko, S., & Nikos Komodakis, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.03641 · Cited by 161 · see citing works
Learning pedestrian dynamics from the real world
Scovanner, P., & Tappen, M. F. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459224 · Cited by 135 · see citing works
Multiple Target Tracking in World Coordinate with Single, Minimally Calibrated Camera
Choi, W., & Savarese, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15561-1_40 · Cited by 132 · see citing works
Multiple Target Tracking Using Spatio-Temporal Markov Chain Monte Carlo Data Association
Yu, Q., Medioni, G., & Cohen, I. (2007) · 2007 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2007.382991 · Cited by 132 · see citing works
Cascaded ensemble of convolutional neural networks and handcrafted features for mitosis detection
Wang, H., Cruz-Roa, Á., Basavanhally, A. N., Gilmore, H. L., Shih, N. N. C., Feldman, M., Tomaszewski, J., Gonzalez, F. A., & Madabhushi, A. (2014) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2043902 · Cited by 109 · see citing works
CNNTracker: Online discriminative object tracking via deep convolutional neural network
Chen, Y., Yang, X., Zhong, B., Pan, S., Chen, D., & Zhang, H. (2015) · Applied Soft Computing · doi:10.1016/j.asoc.2015.06.048 · Cited by 104 · see citing works
Efficient track linking methods for track graphs using network-flow and set-cover techniques
Wu, Z., Kunz, T., & Betke, M. (2011) · doi:10.1109/cvpr.2011.5995515 · Cited by 76 · see citing works
Two-Granularity Tracking: Mediating Trajectory and Detection Graphs for Tracking under Occlusions
Fragkiadaki, K., Zhang, W., Zhang, G., et al. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_40 · Cited by 67 · see citing works
DeepStereo: Learning to Predict New Views from the World's ImageryPreprint
Flynn, J., Neulander, I., Philbin, J. F., & Snavely, N. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06825 · Cited by 62 · see citing works
Learning to Divide and Conquer for Online Multi-target Tracking
Solera, F., Calderara, S., & Cucchiara, R. (2015) · IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.497 · Cited by 52 · see citing works
Alextrac: Affinity learning by exploring temporal reinforcement within association chains
Bewley, A., Ott, L., Ramos, F. T., & Upcroft, B. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487371 · Cited by 36 · see citing works
Learning to Divide and Conquer for Online Multi-Target TrackingPreprint
Solera, F., Calderara, S., & Cucchiara, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.03956 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- S. Chopra, R. Hadsell, and Y. LeCun. Learning a similarity metric discriminatively, with application to face verification. CVPR, 2005. 3
- Z. Khan, T. Balch, and F. Dellaert. Mcmc-based particle filtering for tracking a variable number of interacting targets. TPAMI, 2005. 2
- L. Leal-Taixé, G. Pons-Moll, and B. Rosenhahn. Branch-and-price global optimization for multi-view multi-object tracking. CVPR, 2012. 2
- H. Li, Y. Li, and F. Porikli. Deeptrack: Learning discriminative fea- ture representations online for robust visual tracking. ArXiv e-prints, 2015. 2
- P. M. N. McLaughlin, J. Martinez Del Rincon. Enhancing linear pro- gramming with motion modeling for multi-target tracking. WACV. 7
- S. Wang and C. Fowlkes. Learning optimal parameters for multi- target tracking. BMVC, 2015. 7
Cited by
Sorted by citations
TrackFormer: Multi-Object Tracking with Transformers
Meinhardt, T., Kirillov, A., Leal-Taixe, L., & Feichtenhofer, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00864 · Cited by 1,081 · see citing works
Self-citationTracking Without Bells and Whistles
Bergmann, P., Meinhardt, T., & Leal-Taixe, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00103 · Cited by 1,110 · see citing works
Self-citationDeep learning in video multi-object tracking: A survey
Ciaparrone, G., Luque Sánchez, F., Tabik, S., Troiano, L., Tagliaferri, R., & Herrera, F. (2020) · Neurocomputing · doi:10.1016/j.neucom.2019.11.023 · Cited by 665 · see citing works
Learning a Neural Solver for Multiple Object Tracking
Braso, G., & Leal-Taixe, L. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00628 · Cited by 504 · see citing works
Self-citationQuasi-Dense Similarity Learning for Multiple Object Tracking
Pang, J., Qiu, L., Li, X., Chen, H., Li, Q., Darrell, T., & Yu, F. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00023 · Cited by 480 · see citing works
MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking
Dendorfer, P., Os̆ep, A., Milan, A., Schindler, K., Cremers, D., Reid, I., Roth, S., & Leal-Taixé, L. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01393-0 · Cited by 374 · see citing works
Self-citationEnd-to-End Representation Learning for Correlation Filter Based Tracking
Valmadre, J., Bertinetto, L., Henriques, J., Vedaldi, A., & Torr, P. H. S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.531 · Cited by 274 · see citing works
Kalake, L., Wan, W., & Hou, L. (2021) · IEEE Access · doi:10.1109/access.2021.3060821 · Cited by 68 · see citing works
Deep learning for asteroids autonomous terrain relative navigation
Mancini, P., Cannici, M., & Matteucci, M. (2023) · Advances in Space Research · doi:10.1016/j.asr.2022.04.020 · Cited by 10 · see citing works
- Authors
- Leal-Taixe, L., Canton-Ferrer, C., & Schindler, K.
- Affiliations
- ETH Zurich (CH)
- ORCID
- Schindler, Konrad
- Venue
- 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)
- Publisher
- IEEE
- Published
- 2016-06
- Pages
- 418-425
- OpenAlex
- W2345229200
- Cited by
- 444
- References
- 58
- Citations by year
- 2026: 5 · 2025: 15 · 2024: 27 · 2023: 39 · 2022: 65 · 2021: 82 · 2020: 78 · 2019: 62 · 2018: 48 · 2017: 23
- FWCI
- 15.39
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- pedestrian tracking, data association, Siamese convolutional neural network, target association, optical flow, spatio-temporal features, gradient boosting classifier, multi-person tracking, match probability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
