CoChat
Sign Up
Cited by 386Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningVideo Surveillance and Tracking Methods

Efficient Video Object Segmentation via Network Modulation

Open access
Yang, Linjie, Wang, Yanran, Xiong, Xuehan, Yang, Jianchao, Katsaggelos, Aggelos K.·2018·2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
DOI 10.1109/cvpr.2018.00680
Open this paper in CoChat — free accountOpen PDF

Abstract

Video object segmentation targets segmenting a specific object throughout a video sequence when given only an annotated first frame. Recent deep learning based approaches find it effective to fine-tune a general-purpose segmentation model on the annotated frame using hundreds of iterations of gradient descent. Despite the high accuracy that these methods achieve, the fine-tuning process is inefficient and fails to meet the requirements of real world applications. We propose a novel approach that uses a single forward pass to adapt the segmentation model to the appearance of a specific object. Specifically, a second meta neural network named modulator is trained to manipulate the intermediate layers of the segmentation network given limited visual and spatial information of the target object. The experiments show that our approach is 70× faster than fine-tuning approaches and achieves similar accuracy. Our model and code have been released at https://github.com/linjieyangsc/video_seg.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr.2018.00680

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Adopting a memory architecture to store object information from past frames is indispensable for accurately tracking objects in video.

    “To address the above challenges, adopting a memory architecture to store the object information from past frames is indispensable for accurately tracking objects in video [5,7,10,29,40,52,56,57,64,66,68].”

    From SAM2LONG: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree (2025) · cites this paper as [64]

    Section 3.4 (p. 4) says a rough position and size from the previous frame are enough to infer the object mask with the current RGB image, and warns that relying too heavily on the previous mask can propagate errors. Section 3 (p. 2) describes a visual and spatial modulation approach, not an indispensable memory architecture storing object information from past frames. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 33

  1. [2][2] S. Caelles, K.-K. Maninis, J. Pont-Tuset, L. Leal-Taix´ e, D. Cremers, and L. Van Gool. One-shot video object segmentation. In CVPR , 2017. 1, 2, 4, 5, 6, 7 -
  2. [3][3] Y. Chen, M. W. Hoffman, S. G. Colmenarejo, M. Denil, T. P. Lillicrap, M. Botvinick, and N. de Freitas. Learning to learn without gradient descent by gradient descent. In ICML , 2016. 2 -
  3. [4][4] J. Cheng, Y.-H. Tsai, S. Wang, and M.-H. Yang. Segflow: Joint learning for video object segmentation and optical flow. In IEEE International Conference on Computer Vision (ICCV) , 2017. 1, 2, 5, 6 -
  4. [6][6] H. de Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. CoRR , abs/1707.00683, 2017. 3 - · arXiv
  5. [7][7] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009. 4 -
  6. [9][9] Q. Fan, F. Zhong, D. Lischinski, D. Cohen-Or, and B. Chen. Jumpcut:non-successive mask transfer and interpolation for video cutout. In ACM Trans. Graph., 34(6) , 2015. 2 -
  7. [10][10] C. Finn, P. Abbeel, and S. Levine. Model-agnostic metalearning for fast adaptation of deep networks. In ICML , 2017. 2 -
  8. [11][11] G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens. exploring the structure of a real-time, arbitrary neural artistic stylization network. In BMVC , 2017. 2 -
  9. [13][13] B. Hariharan and R. Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV , 2017. 2 -
  10. [14][14] X. Huang and S. J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. CoRR , abs/1703.06868, 2017. 2, 3 - · arXiv
  11. [15][15] E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. CVPR , 2017. 1 -
  12. [16][16] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In NIPS , pages 2017-2025, 2015. 2 -
  13. [17][17] S. D. Jain and K. Grauman. Supervoxel-consistent foreground propagation in video. In ECCV , 2014. 2, 5 -
  14. [18][18] V. Jampani, R. Gadde, and P. V. Gehler. Video propagation networks. In CVPR , 2017. 2, 5, 6 -
  15. [19][19] P. Kr¨ ahenb¨ uhl and V. Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In NIPS , pages 109-117, 2011. 2, 5 -
  16. [20][20] F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg. Video segmentation by tracking many figure-ground segments. In ICCV , 2013. 1 -
  17. [21][21] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ ar, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 4, 5 -
  18. [22][22] N. M¨ arki, F. Perazzi, O. Wang, and A. Sorkine-Hornung. Bilateral space video segmentation. In CVPR , 2016. 1, 2, 5, 6 -
  19. [23][23] F. Perazzi, A. Khoreva, R. Benenson, B. Schiele, and A.Sorkine-Hornung. Learning video object segmentation from static images. In CVPR , 2017. 1, 2, 4, 5, 6, 7 -
  20. [24][24] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In CVPR , 2016. 1, 5 -
  21. [25][25] F. Perazzi, O. Wang, M. Gross, and A. Sorkine-Hornung. Fully connected object proposals for video segmentation. In ICCV , 2015. 2 -
  22. [26][26] E. Perez, H. de Vries, F. Strub, V. Dumoulin, and A. C. Courville. Learning visual reasoning without strong priors. CoRR , abs/1707.03017, 2017. 2, 3 - · arXiv
  23. [27][27] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbel´ aez, A. SorkineHornung, and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.00675 , 2017. 4, 5 - · arXiv
  24. [28][28] S. Ravi and H. Larochelle. Optimization as a model for fewshot learning. ICLR , 2017. 2 -
  25. [29][29] J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid. Epicflow: Edge-preserving interpolation of correspondences for optical flow. In CVPR , 2015. 1 -
  26. [30][30] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory-augmented neural networks. In ICML , 2016. 2 -
  27. [31][31] E. Shelhamer, J. Long, and T. Darrell. Fully convolutional networks for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence , 39(4):640-651, 2017. 1 -
  28. [32][32] J. Shin Yoon, F. Rameau, J. Kim, S. Lee, S. Shin, and I. So Kweon. Pixel-level matching for video object segmentation using convolutional neural networks. In CVPR , 2017. 1, 2, 5, 6 -
  29. [33][33] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. 3, 4 - · arXiv
  30. [34][34] P. Tokmakov, K. Alahari, and C. Schmid. Learning video object segmentation with visual memory. In ICCV , 2017. 2, 5, 6 -
  31. [35][35] Y.-H. Tsai, M.-H. Yang, and M. J. Black. Video segmentation via object flow. In CVPR , 2016. 1, 2, 5, 6 -
  32. [36][36] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In NIPS , 2016. 2 -
  33. [37][37] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In ECCV , 2014. 8

Cited by

Sorted by citations

Authors
Yang, L., Wang, Y., Xiong, X., Yang, J., & Katsaggelos, A. K.
Affiliations
  • Snap (United States) (US)
  • Northwestern University (US)
  • Google (United States) (US)
ORCID
Venue
2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Publisher
IEEE
Published
2018-06
Pages
6499-6507
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr.2018.00680
OpenAlex
W2962825871
Cited by
386
References
2
Citations by year
2026: 3 · 2025: 23 · 2024: 32 · 2023: 46 · 2022: 70 · 2021: 73 · 2020: 75 · 2019: 57 · 2018: 7
FWCI
15.97
Type
conference paper
Language
en
Open access
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
Keywords
video object segmentation, network modulation, fine-tuning efficiency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review