CoChat
Sign Up
PreprintCited by 449Advanced Neural Network ApplicationsVideo Surveillance and Tracking MethodsVisual Attention and Saliency Detection

The DOI matches its DataCite record.

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

Open access
Cheng, Ho Kei, Schwing, Alexander G.·2022·arXiv
DOI 10.48550/arxiv.2207.07115arXiv 2207.07115
Open this paper in CoChat — free accountOpen PDF

Abstract

We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. Prior work on video object segmentation typically only uses one type of feature memory. For videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. In contrast, following the Atkinson-Shiffrin model, we develop an architecture that incorporates multiple independent yet deeply-connected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact thus sustained long-term memory. Crucially, we develop a memory potentiation algorithm that routinely consolidates actively used working memory elements into the long-term memory, which avoids memory explosion and minimizes performance decay for long-term prediction. Combined with a new memory reading mechanism, XMem greatly exceeds state-of-the-art performance on long-video datasets while being on par with state-of-the-art methods (that do not work on long videos) on short-video datasets. Code is available at https://hkchengrex.github.io/XMem

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.07115

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Incorporating long-term memory significantly enhances video understanding by enabling the storage and retrieval of key video information for subsequent tasks.

    “Incorporating long-term memory significantly enhances video understanding by enabling the storage and retrieval of key video information for subsequent tasks [11, 13].”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [11]

    Section 3.3 (pp. 7–8) says XMem stores compact representations of earlier video frames in long-term memory and reads them to support accurate segmentation of long videos. The claim extends this task-specific result to video understanding and subsequent tasks generally. Section 4.3 (p. 13) also reports slightly better segmentation performance without long-term memory when all frames can be retained; its demonstrated benefit is handling long videos efficiently, not an unqualified performance improvement. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • A typical video understanding pipeline driven by long-term memory converts video frames into feature vectors, which can later be retrieved to support various tasks.

    “A typical video understanding pipeline driven by long-term memory converts video frames into feature vectors, which can later be retrieved to support various tasks [11, 12, 13, 14, 15].”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [11]

    Section 3.1 (pp. 4–5) describes extracting features from video frames, storing them in memory, and reading them later to generate segmentation masks. The paper supports that mechanism for video object segmentation, but not the broader claim that the retrieved features support various video-understanding tasks. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Some video understanding systems draw upon the Atkinson-Shiffrin Model to construct hierarchical memory systems.

    “For instance, some video understanding systems [11, 14] draw upon the Atkinson-Shiffrin Model [6] (a highly influential human memory model, detailed in Sec. 3) to construct hierarchical memory systems.”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [11]

  • The Atkinson-Shiffrin model is frequently referenced in AI memory research.

    “Notably, this model and its derivative theories are frequently referenced in AI memory research [61, 11, 14], bridging the fields of human and AI memory studies.”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [11]

  • Some AI long-term memory research is directly inspired by human long-term memory, while other studies align with human long-term memory processing mechanisms without intentional imitation.

    “some AI long-term memory research is directly inspired by human long-term memory [245, 11, 14, 190, 191], while other studies align with human long-term memory processing mechanisms without intentional imitation.”

    From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [11]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 60

  1. [1]1. Atkinson, R.C., Shiffrin, R.M.: Human memory: A proposed system and its control processes. In: Psychology of learning and motivation, vol. 2, pp. 89-195. Elsevier (1968)
  2. [2]2. Bhat, G., Lawin, F.J., Danelljan, M., Robinson, A., Felsberg, M., Van Gool, L., Timofte, R.: Learning what to learn for video object segmentation. In: ECCV (2020)
  3. [3]3. Caelles, S., Maninis, K.K., Pont-Tuset, J., Leal-Taix´ e, L., Cremers, D., Van Gool, L.: One-shot video object segmentation. In: CVPR (2017)
  4. [4]4. Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. In: arXiv:1512.03012 (2015) · arXiv
  5. [5]5. Chen, X., Li, Z., Yuan, Y., Yu, G., Shen, J., Qi, D.: State-aware tracker for realtime video object segmentation. In: CVPR (2020)
  6. [6]6. Chen, Y., Pont-Tuset, J., Montes, A., Van Gool, L.: Blazingly fast video object segmentation with pixel-wise metric learning. In: CVPR (2018)
  7. [7]7. Cheng, H.K., Chung, J., Tai, Y.W., Tang, C.K.: Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In: CVPR (2020)
  8. [8]8. Cheng, H.K., Tai, Y.W., Tang, C.K.: Modular interactive video object segmentation: Interaction-to-mask, propagation and difference-aware fusion. In: CVPR (2021)
  9. [9]9. Cheng, H.K., Tai, Y.W., Tang, C.K.: Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In: NeurIPS (2021)
  10. [10]10. Cheng, J., Tsai, Y.H., Hung, W.C., Wang, S., Yang, M.H.: Fast and accurate online video object segmentation via tracking parts. In: CVPR (2018)
  11. [11]11. Cho, K., Van Merri¨ enboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: Encoder-decoder approaches. In: arXiv (2014)
  12. [12]12. Denninger, M., Sundermeyer, M., Winkelbauer, D., Zidan, Y., Olefir, D., Elbadrawy, M., Lodhi, A., Katam, H.: Blenderproc. In: arXiv:1911.01911 (2019) · arXiv
  13. [13]13. Duarte, K., Rawat, Y.S., Shah, M.: Capsulevos: Semi-supervised video object segmentation using capsule routing. In: ICCV (2019)
  14. [14]14. Duke, B., Ahmed, A., Wolf, C., Aarabi, P., Taylor, G.W.: Sstvos: Sparse spatiotemporal transformers for video object segmentation. In: CVPR (2021)
  15. [16]16. Ge, W., Lu, X., Shen, J.: Video object segmentation using global and instance embedding learning. In: CVPR (2021)
  16. [17]17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  17. [18]18. Hu, L., Zhang, P., Zhang, B., Pan, P., Xu, Y., Jin, R.: Learning position and target consistency for memory-based video object segmentation. In: CVPR (2021)
  18. [19]19. Hu, P., Wang, G., Kong, X., Kuen, J., Tan, Y.P.: Motion-guided cascaded refinement network for video object segmentation. In: CVPR (2018)
  19. [20]20. Hu, Y.T., Huang, J.B., Schwing, A.: Maskrnn: Instance level video object segmentation. In: NIPS (2017)
  20. [21]21. Huang, X., Xu, J., Tai, Y.W., Tang, C.K.: Fast video object segmentation with temporal aggregation network and dynamic template matching. In: CVPR (2020)
  21. [22]22. Jang, W.D., Kim, C.S.: Online video object segmentation via convolutional trident network. In: CVPR (2017)
  22. [23]23. Johnander, J., Danelljan, M., Brissman, E., Khan, F.S., Felsberg, M.: A generative appearance model for end-to-end video object segmentation. In: CVPR (2019)
  23. [24]24. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  24. [25]25. Li, X., Wei, T., Chen, Y.P., Tai, Y.W., Tang, C.K.: Fss-1000: A 1000-class dataset for few-shot segmentation. In: CVPR (2020)
  25. [26]26. Li, X., Loy, C.C.: Video object segmentation with joint re-identification and attention-aware mask propagation. In: ECCV (2018)
  26. [27]27. Li, Y., Shen, Z., Shan, Y.: Fast video object segmentation using the global context module. In: ECCV (2020)
  27. [28]28. Liang, S., Shen, X., Huang, J., Hua, X.S.: Video object segmentation with dynamic memory networks and adaptive object alignment. In: ICCV (2021)
  28. [29]29. Liang, Y., Li, X., Jafari, N., Chen, J.: Video object segmentation with adaptive feature bank and uncertain-region refinement. In: NeurIPS (2020)
  29. [30]30. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. ICLR (2019)
  30. [31]31. Lu, X., Wang, W., Martin, D., Zhou, T., Shen, J., Luc, V.G.: Video object segmentation with episodic graph memory networks. In: ECCV (2020)
  31. [33]33. Mao, Y., Wang, N., Zhou, W., Li, H.: Joint inductive and transductive learning for video object segmentation. In: ICCV (2021)
  32. [34]34. Meinhardt, T., Leal-Taix´ e, L.: Make one-shot video object segmentation efficient again. In: NeurIPS (2020)
  33. [35]35. Oh, S.W., Lee, J.Y., Sunkavalli, K., Joo Kim, S.: Fast video object segmentation by reference-guided mask propagation. In: CVPR (2018)
  34. [36]36. Oh, S.W., Lee, J.Y., Xu, N., Kim, S.J.: Video object segmentation using space-time memory networks. In: ICCV (2019)
  35. [37]37. Park, H., Yoo, J., Jeong, S., Venkatesh, G., Kwak, N.: Learning dynamic network using a reuse gate function in semi-supervised video object segmentation. In: CVPR (2021)
  36. [38]38. Patrick, M., Campbell, D., Asano, Y.M., Metze, I.M.F., Feichtenhofer, C., Vedaldi, A., Henriques, J., et al.: Keeping your eye on the ball: Trajectory attention in video transformers. In: NeurIPS (2021)
  37. [39]39. Perazzi, F., Khoreva, A., Benenson, R., Schiele, B., Sorkine-Hornung, A.: Learning video object segmentation from static images. In: CVPR (2017)
  38. [40]40. Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., SorkineHornung, A.: A benchmark dataset and evaluation methodology for video object segmentation. In: CVPR (2016)
  39. [41]41. Pont-Tuset, J., Perazzi, F., Caelles, S., Arbel´ aez, P., Sorkine-Hornung, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. In: arXiv:1704.00675 (2017) · arXiv
  40. [42]42. Robinson, A., Lawin, F.J., Danelljan, M., Khan, F.S., Felsberg, M.: Learning fast and robust target models for video object segmentation. In: CVPR (2020)
  41. [43]43. Seong, H., Hyun, J., Kim, E.: Kernelized memory network for video object segmentation. In: ECCV (2020)
  42. [44]44. Seong, H., Oh, S.W., Lee, J.Y., Lee, S., Lee, S., Kim, E.: Hierarchical memory matching network for video object segmentation. In: ICCV (2021)
  43. [45]45. Shi, J., Yan, Q., Xu, L., Jia, J.: Hierarchical image saliency detection on extended cssd. In: TPAMI (2015)
  44. [47]47. Ventura, C., Bellver, M., Girbau, A., Salvador, A., Marques, F., Giro-i Nieto, X.: Rvos: End-to-end recurrent network for video object segmentation. In: CVPR (2019)
  45. [48]48. Voigtlaender, P., Chai, Y., Schroff, F., Adam, H., Leibe, B., Chen, L.C.: Feelvos: Fast end-to-end embedding learning for video object segmentation. In: CVPR (2019)
  46. [49]49. Voigtlaender, P., Leibe, B.: Online adaptation of convolutional neural networks for video object segmentation. In: BMVC (2017)
  47. [50]50. Wang, H., Jiang, X., Ren, H., Hu, Y., Bai, S.: Swiftnet: Real-time video object segmentation. In: CVPR (2021)
  48. [51]51. Wang, L., Lu, H., Wang, Y., Feng, M., Wang, D., Yin, B., Ruan, X.: Learning to detect salient objects with image-level supervision. In: CVPR (2017)
  49. [52]52. Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.: Fast online object tracking and segmentation: A unifying approach. In: CVPR (2019)
  50. [53]53. Wang, Z., Xu, J., Liu, L., Zhu, F., Shao, L.: Ranet: Ranking attention network for fast video object segmentation. In: ICCV (2019)
  51. [54]54. Xie, H., Yao, H., Zhou, S., Zhang, S., Sun, W.: Efficient regional memory network for video object segmentation. In: CVPR (2021)
  52. [55]55. Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., Singh, V.: Nystr¨ omformer: A nystr¨ om-based algorithm for approximating self-attention. In: AAAI (2021)
  53. [56]56. Xu, K., Wen, L., Li, G., Bo, L., Huang, Q.: Spatiotemporal cnn for video object segmentation. In: CVPR (2019)
  54. [57]57. Xu, N., Yang, L., Fan, Y., Yue, D., Liang, Y., Yang, J., Huang, T.: Youtube-vos: A large-scale video object segmentation benchmark. In: ECCV (2018)
  55. [58]58. Xu, X., Wang, J., Li, X., Lu, Y.: Reliable propagation-correction modulation for video object segmentation. In: AAAI (2022)
  56. [59]59. Yang, Z., Wei, Y., Yang, Y.: Collaborative video object segmentation by foreground-background integration. In: ECCV (2020)
  57. [60]60. Yang, Z., Wei, Y., Yang, Y.: Associating objects with transformers for video object segmentation. In: NeurIPS (2021)
  58. [62]62. Zeng, Y., Zhang, P., Zhang, J., Lin, Z., Lu, H.: Towards high-resolution salient object detection. In: ICCV (2019)
  59. [63]63. Zhang, L., Lin, Z., Zhang, J., Lu, H., He, Y.: Fast video object segmentation via dynamic targeting network. In: ICCV (2019)
  60. [64]64. Zhang, Y., Wu, Z., Peng, H., Lin, S.: A transductive approach for video object segmentation. In: CVPR (2020)

Cited by

Sorted by citations

Metadata

Authors
Cheng, H. K., & Schwing, A. G.
Affiliations
University of Illinois Urbana-Champaign (US)
Corresponding author
Cheng, Ho Kei
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
Pages
640-658
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2207.07115
arXiv
2207.07115
OpenAlex
W4312396403
Cited by
449
References
0
Citations by year
2026: 86 · 2025: 151 · 2024: 145 · 2023: 66 · 2022: 1
FWCI
32.90
Type
conference paper
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
video object segmentation, sensory memory, working memory, long-term memory, memory consolidation, unified memory architecture

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review