Cited by 226Video Surveillance and Tracking MethodsAutonomous Vehicle Technology and SafetyAdvanced Neural Network Applications

Learning to Track with Object Permanence

Open access
Tokmakov, Pavel, Li, Jie, Burgard, Wolfram, Gaidon, Adrien·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)

Abstract

Tracking by detection, the dominant approach for online multi-object tracking, alternates between localization and association steps. As a result, it strongly depends on the quality of instantaneous observations, often failing when objects are not fully visible. In contrast, tracking in humans is underlined by the notion of object permanence: once an object is recognized, we are aware of its physical existence and can approximately localize it even under full occlusions. In this work, we introduce an end-to-end trainable approach for joint object detection and tracking that is capable of such reasoning. We build on top of the recent CenterTrack architecture, which takes pairs of frames as input, and extend it to videos of arbitrary length. To this end, we augment the model with a spatio-temporal, recurrent memory module, allowing it to reason about object locations and identities in the current frame using all the previous history. It is, however, not obvious how to train such an approach. We study this question on a new, large-scale, synthetic dataset for multi-object tracking, which provides ground truth annotations for invisible objects, and propose several approaches for supervising tracking behind occlusions. Our model, trained jointly on synthetic and real data, outperforms the state of the art on KITTI and MOT17 datasets thanks to its robustness to occlusions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.01068

Cites

73 of 73 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The projection of non-linear 3D motion into the 2D image domain still poses a challenging problem for many models.

    “However, the projection of non-linear 3D motion [50] into the 2D image domain still poses a challenging problem for many models.”

    From TrackFormer: Multi-Object Tracking with Transformers (2022) · cites this paper as [50]

    Section 3.3.2 says an occluded person's unexpected stop cannot be predicted without observations, supporting the point that non-linear motion challenges tracking. It also describes projecting estimated 3D positions into the camera frame, but does not say that the projection itself is a challenging problem for many models. (LLM, conf=0.82)

    Experimental — accuracy not yet measured
  • Only methods [50,53,54] that follow [66] and pretrain on 6 additional tracking datasets (6M) surpass TrackFormer's performance on MOT17 private detections.

    “Only [50,53,54] which follow [66] and pretrain on 6 additional tracking datasets (6M) surpass our performance.”

    From TrackFormer: Multi-Object Tracking with Transformers (2022) · cites this paper as [50]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 74 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking anal- ysis. In CVPR, 2016. 3, 5
  2. Jiaqi Zou Ke Mei, Chuang Zhu and Shanghang Zhang. In- stance adaptive self-training for unsupervised domain adap- tation. In ECCV, 2020. 3
  3. Linjie Yang, Yuchen Fan, and Ning Xu. Video instance seg- mentation. In ICCV, 2019. 6

Cited by

Sorted by citations

Authors
Tokmakov, P., Li, J., Burgard, W., & Gaidon, A.
Affiliations
  • Toyota Motor Corporation (Switzerland) (CH)
  • University of Freiburg (DE)
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
10840-10849
OpenAlex
W3142617676
Cited by
226
References
74
Citations by year
2026: 14 · 2025: 43 · 2024: 72 · 2023: 48 · 2022: 46 · 2021: 3
FWCI
13.74
Type
conference paper
Language
en
Open access
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • IEEE/CVF International Conference on Computer Vision (ICCV)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
  • Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
multi-target tracking, object permanence, occlusion handling, joint detection and tracking, spatiotemporal memory, end-to-end tracking, KITTI, MOT17

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review