The DOI matches its Crossref record; 264 of the 264 references it cites were found in a registry or an index.
A Survey on Deep Learning Technique for Video Segmentation
Open accessAbstract
Video segmentation-partitioning video frames into multiple segments or objects-plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to creating virtual background in video conferencing. Recently, with the renaissance of connectionism in computer vision, there has been an influx of deep learning based approaches for video segmentation that have delivered compelling performance. In this survey, we comprehensively review two basic lines of research - generic object segmentation (of unknown categories) in videos, and video semantic segmentation - by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. We also offer a detailed overview of representative literature on both methods and datasets. We further benchmark the reviewed methods on several well-known datasets. Finally, we point out open issues in this field, and suggest opportunities for further research. We also provide a public website to continuously track developments in this fast advancing field: https://github.com/tfzhou/VS-Survey.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–264 of 264
Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Network
Ye, L., Rochan, M., Liu, Z., Zhang, X., & Wang, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3054384 · Cited by 112
Wang, H., Deng, C. D., Yan, J., & Tao, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00404 · Cited by 109
Motion-Guided Cascaded Refinement Network for Video Object Segmentation
Hu, P., Wang, G., Kong, X., Kuen, J., & Tan, Y. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00152 · Cited by 108
Video Scene Parsing with Predictive Feature Learning
Jin, X., Li, X., Xiao, H., Shen, X., Lin, Z., Yang, J., Chen, Y., Dong, J., Liu, L., Jie, Z., et al. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.595 · Cited by 107
AGSS-VOS: Attention Guided Single-Shot Video Object Segmentation
Lin, H., Qi, X., & Jia, J. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00405 · Cited by 107
Memory Aggregation Networks for Efficient Interactive Video Object Segmentation
Miao, J., Wei, Y., & Yang, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01038 · Cited by 107
Reinforcement Cutting-Agent Learning for Video Object Segmentation
Han, J., Yang, L., Zhang, D., Chang, X., & Liang, X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00946 · Cited by 106
Siam, M., Jiang, C., Lu, S., Petrich, L., Gamal, M., Elhoseiny, M., & Jägersand, M. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794254 · Cited by 103
Fast Template Matching and Update for Video Object Tracking and Segmentation
Sun, M., Xiao, J., Lim, E. G., Zhang, B., & Zhao, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01080 · Cited by 101
CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing
Duarte, K., Rawat, Y. S., & Shah, M. A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00857 · Cited by 99
Deep Spatio-Temporal Random Fields for Efficient Video Segmentation
Chandra, S., Couprie, C., & Kokkinos, I. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00929 · Cited by 84
DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation
Zeng, X., Liao, R., Gu, L., Xiong, Y., Fidler, S., & Urtasun, R. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00403 · Cited by 83
Fast Video Object Segmentation With Temporal Aggregation Network and Dynamic Template Matching
Huang, X., Xu, J., Tai, Y., & Tang, C. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00890 · Cited by 83
Learning Multi-Object Tracking and Segmentation From Automatic Annotations
Porzi, L., Hofinger, M., Ruiz, I., Serrat, J., Bulo, S. R., & Kontschieder, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00688 · Cited by 82
Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow
Ding, M., Wang, Z., Zhou, B., Shi, J., Lu, Z., & Luo, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6699 · Cited by 81
CompFeat: Comprehensive Feature Aggregation for Video Instance Segmentation
Fu, Y. L., Yang, L., Liu, D., Huang, T. S., & Shi, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i2.16225 · Cited by 80
Fast User-Guided Video Object Segmentation by Interaction-And-Propagation Networks
Oh, S. W., Lee, J., Xu, N., & Kim, S. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00539 · Cited by 77
Spatio-Temporal Person Retrieval via Natural Language Queries
Yamaguchi, M., Saito, K., Ushiku, Y., & Harada, T. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.162 · Cited by 77
UnOVOST: Unsupervised Offline Video Object Segmentation and Tracking
Luiten, J., Zulfikar, I. E., & Leibe, B. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093285 · Cited by 76
Fast Video Object Segmentation via Dynamic Targeting Network
Zhang, L., Lin, Z., Zhang, J., Lu, H., & He, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00568 · Cited by 74
Video Instance Segmentation Tracking With a Modified VAE Architecture
Lin, C.-C., Hung, Y., Feris, R. S., & He, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01316 · Cited by 73
Multiclass semantic video segmentation with object-level active inference
Liu, B., & He, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299057 · Cited by 72
Semi-Supervised Video Segmentation Using Tree Structured Graphical Models
Badrinarayanan, V., Budvytis, I., & Cipolla, R. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.54 · Cited by 72
Joint Optical Flow and Temporally Consistent Semantic Segmentation
Hur, J., & Roth, S. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46604-0_12 · Cited by 71
Learning Discriminative Feature with CRF for Unsupervised Video Object Segmentation
Zhen, M., Li, S., Zhou, L., Shang, J., Feng, H., Fang, T., & Quan, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58583-9_27 · Cited by 68
Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation
Hui, T., Huang, S., Liu, S., Ding, Z., Li, G., Wang, W., Han, J., & Wang, F. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00417 · Cited by 66
Self-citationLocality-Aware Inter-and Intra-Video Reconstruction for Self-Supervised Correspondence Learning
Li, L., Zhou, T., Wang, W., Yang, L., Li, J., & Yang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00852 · Cited by 66
Self-citationVideo Object Segmentation by Learning Location-Sensitive Embeddings
Ci, H., Wang, C., & Wang, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_31 · Cited by 64
DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping*
Yang, Y.-C., Lai, B., & Soatto, S. (2021) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr46437.2021.00285 · Cited by 64
Video Object Segmentation Using Global and Instance Embedding Learning
Wen-bin, G., Lu, X., & Shen, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01656 · Cited by 60
Visual-Textual Capsule Routing for Text-Based Video Segmentation
McIntosh, B., Duarte, K., Rawat, Y. S., & Shah, M. A. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00996 · Cited by 57
Interactive Video Object Segmentation in the WildPreprint
Benard, A., & Gygli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.00269 · Cited by 56
MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation
Xu, S., Liu, D., Bao, L., Liu, W., & Zhou, P. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00040 · Cited by 56
Learning Dynamic Network Using a Reuse Gate Function in Semi-supervised Video Object Segmentation
Park, H., Yoo, J., Jeong, S., Venkatesh, G., & Kwak, N. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00830 · Cited by 55
Griffin, B., & Corso, J. J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00912 · Cited by 52
Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation
Zhou, T., Li, J., Li, X., & Shao, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00691 · Cited by 52
Self-citationWeakly Supervised Instance Segmentation for Videos with Temporal Mask Consistency
Liu, Q., Ramanathan, V., Mahajan, D. K., Yuille, A., & Yang, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01375 · Cited by 52
Weakly Supervised Multiclass Video Segmentation
Liu, X., Tao, D., Song, M., Ruan, Y., Chen, C., & Bu, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.15 · Cited by 51
Budget-Aware Deep Semantic Video Segmentation
Mahasseni, B., Todorović, S., & Fern, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.224 · Cited by 51
F2Net: Learning to Focus on the Foreground for Unsupervised Video Object Segmentation
Liu, D., Yu, D., Wang, C., & Zhou, P. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16308 · Cited by 50
Context Modulated Dynamic Networks for Actor and Action Video Segmentation with Language Queries
Wang, H., Deng, C. D., Ma, F., & Yang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6895 · Cited by 48
Online Meta Adaptation for Fast Video Object Segmentation
Xiao, H., Kang, B., Liu, Y., Zhang, M., & Feng, J. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2890659 · Cited by 47
Learning to Associate Every Segment for Video Panoptic Segmentation
Woo, S., Kim, D., Lee, J., & Kweon, I. S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00273 · Cited by 47
Saleh, F. S., Aliakbarian, M. S., Salzmann, M., Petersson, L., & Álvarez, J. M. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.232 · Cited by 46
Interactive Video Object Segmentation Using Global and Local Transfer Modules
Heo, Y., Koh, Y. J., & Kim, C. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58520-4_18 · Cited by 45
Roto++: accelerating professional rotoscoping using shape manifolds
Li, W., Viola, F., Starck, J., Brostow, G., & Campbell, N. D. F. (2016) · ACM Transactions on Graphics · doi:10.1145/2897824.2925973 · Cited by 44
Towards a Mathematical Understanding of the Difficulty in Learning with Feedforward Neural Networks
Shen, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00091 · Cited by 44
Unsupervised Video Object Segmentation with Joint Hotspot Tracking
Zhang, L., Zhang, J., Lin, Z., Měch, R., Lu, H., & He, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58568-6_29 · Cited by 43
Make One-Shot Video Object Segmentation Efficient AgainPreprint
Meinhardt, T., & Leal-Taixé, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.01866 · Cited by 43
Discontinuity-aware video object cutout
Zhong, F., Qin, X., Peng, Q., & Meng, X. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366194 · Cited by 39
Polar Relative Positional Encoding for Video-Language Segmentation
Ning, K., Xie, L., Wu, F. Q., & Tian, Q. C. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/132 · Cited by 38
Learning to Track Instances without Video Annotations
Fu, Y. L., Liu, S., Iqbal, U., Mello, S. D., Shi, H., & Kautz, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00857 · Cited by 37
Delving into the Cyclic Mechanism in Semi-supervised Video Object SegmentationPreprint
Li, Y., Xu, N., Peng, J., See, J., & Lin, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12176 · Cited by 35
Guided Interactive Video Object Segmentation Using Reliability-Based Attention Maps
Heo, Y., Koh, Y. J., & Kim, C. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00724 · Cited by 34
Efficient Video Segmentation Using Parametric Graph Partitioning
Yu, C.-P., Le, H., Zelinsky, G. J., & Samaras, D. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.361 · Cited by 33
Coarse-to-Fine Semantic Video Segmentation Using Supervoxel Trees
Jain, A., Chatterjee, S., & Vidal, R. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.234 · Cited by 28
ScribbleBox: Interactive Annotation Framework for Video Object Segmentation
Chen, B., Ling, H., Zeng, X., Gao, J., Xu, Z., & Fidler, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58601-0_18 · Cited by 27
Learning to Recommend Frame for Interactive Video Object Segmentation in the Wild
Yin, Z., Zheng, J., Luo, W., Qian, S., Zhang, H., & Gao, S. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01519 · Cited by 26
Occluded Video Instance SegmentationPreprint
Qi, J. (2024) · arXiv (Cornell University) · doi:10.57702/dx5t2axm · Cited by 24
Query-Memory Re-Aggregation for Weakly-supervised Video Object Segmentation
Lin, F., Xie, H., Li, Y., & Zhang, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i3.16300 · Cited by 23
Memory Selection Network for Video Propagation
Wu, R., Lin, H., Qi, X., & Jia, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_11 · Cited by 22
Learning a Spatio-Temporal Embedding for Video Instance SegmentationPreprint
Hu, A., Kendall, A., & Cipolla, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.08969 · Cited by 20
The 2017 DAVIS Challenge on Video Object Segmentation
Pont-Tuset, J., Perazzi, F., Caelles, S., Arbeláez, P., Sorkine-Hornung, A., & Van Gool, L. (2017) · arXiv · doi:10.48550/arxiv.1704.00675 · Cited by 13
The Shape-Time Random Field for Semantic Video Labeling
Kae, A., Marlin, B. M., & Miller, E. G. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.42 · Cited by 10
Cited by
VideoMAC: Video Masked Autoencoders Meet ConvNets
Pei, G., Chen, T., Jiang, X., Liu, H., Sun, Z., & Yao, Y. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02145 · Cited by 23
“Recently, researchers have embarked on exploring MVM approaches [16, 19, 65, 66] to comprehend unsupervised video representations, exhibiting effectiveness across a multitude of downstream tasks [4, 25, 31, 32, 40–44, 60, 61, 71, 74, 75].”
Background
Metadata
- Authors
- Zhou, T., Porikli, F., Crandall, D. J., Van Gool, L., & Wang, W.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2023-06-01
- Volume
- 45
- Issue
- 6
- Pages
- 7099-7122
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3182236906
- Cited by
- 255
- References
- 264
- FWCI
- 17.89
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.