The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
FusionSeg: Learning to Combine Motion and Appearance for Fully Automatic Segmentation of Generic Objects in Videos
Abstract
We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for training. Through experiments on three challenging video segmentation benchmarks, our method substantially improves the state-of-the-art for segmenting generic (unseen) objects. Code and pre-trained models are available on the project website.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 1×
“Category-specific semantic segmentation State-of-theart semantic segmentation techniques for images rely on fully convolutional deep learning architectures that are endto-end trainable [33, 59, 28, 5].”
BackgroundA Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 2×
“In particular, we convert the raw optical flow to a 3-channel (RGB) color-coded optical flow image [3].”
MethodBeyond short snippets: Deep networks for video classification
Ng, J. Y.-H., Hausknecht, M., Vijayanarasimhan, S., Vinyals, O., Monga, R., & Toderici, G. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299101 · Cited by 2,062
A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
Perazzi, F., Pont-Tuset, J., McWilliams, B., Van Gool, L., Gross, M., & Sorkine-Hornung, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.85 · Cited by 1,989
Hierarchical Convolutional Features for Visual Tracking
Ma, C., Huang, J.-B., Yang, X., & Yang, M.-H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.352 · Cited by 1,776
Semantic object classes in video: A high-definition ground truth database
Brostow, G. J., Fauqueur, J., & Cipolla, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.04.005 · Cited by 1,397
Large-scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G. D., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · Cited by 1,266
Visual Tracking with Fully Convolutional Networks
Wang, L., Ouyang, W., Wang, X., & Lu, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.357 · Cited by 983
Beyond pixels: exploring new representations and applications for motion analysis
Freeman, W. T., Adelson, E. H., & Liu, C. (2009) · Cited by 805
Efficient hierarchical graph-based video segmentation
Grundmann, M., Kwatra, V., Han, M., & Essa, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539893 · Cited by 628
Saliency Detection via Absorbing Markov Chain
Jiang, B., Zhang, L., Lu, H., Yang, C., & Yang, M.-H. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.209 · Cited by 602
FlowNet: Learning Optical Flow with Convolutional Networks
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 562
Fast Object Segmentation in Unconstrained Video
Papazoglou, A., & Ferrari, V. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.223 · Cited by 533
Video Segmentation by Tracking Many Figure-Ground Segments
Li, F., Kim, T., Humayun, A., Tsai, D., & Rehg, J. M. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.273 · Cited by 512
Learning object class detectors from weakly annotated video
Prest, A., Leistner, C., Civera, J., Schmid, C., & Ferrari, V. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248065 · Cited by 443
Key-segments for video object segmentation
Lee, Y. J., Kim, J., & Grauman, K. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126471 · Cited by 443
Self-citationVideo SnapCut: robust video object cutout using localized classifiers
Bai, X., Wang, J., Simons, D., & Sapiro, G. (2009) · ACM Transactions on Graphics · doi:10.1145/1531326.1531376 · Cited by 367
Wang, J., Bhat, P., Colburn, R. A., Agrawala, M., & Cohen, M. F. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073233 · Cited by 321
Video Segmentation via Object Flow
Tsai, Y.-H., Yang, M.-H., & Black, M. J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.423 · Cited by 318
Zhang, D., Javed, O., & Shah, M. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.87 · Cited by 309
Li, Y., Sun, J., & Shum, H.-Y. (2005) · ACM Transactions on Graphics · doi:10.1145/1073204.1073234 · Cited by 286
Video Segmentation by Non-Local Consensus voting.
Faktor, A., & Irani, M. (2014) · Cited by 235
Supervoxel-Consistent Foreground Propagation in Video
Jain, S. D., & Grauman, K. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_43 · Cited by 210
Self-citationBilateral Space Video Segmentation
Marki, N., Perazzi, F., Wang, O., & Sorkine-Hornung, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.87 · Cited by 208
Learning to segment moving objects in videos
Fragkiadaki, K., Arbelaez, P., Felsen, P., & Malik, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299035 · Cited by 193
Hough-based tracking of non-rigid objects
Godec, M., Roth, P. M., & Bischof, H. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126228 · Cited by 193
Fast action proposals for human action detection and search
Yu, G., & Yuan, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298735 · Cited by 192
Motion Coherent Tracking with Multi-label MRF optimization
Tsai, D., Flagg, M., & Rehg, J. (2010) · Procedings of the British Machine Vision Conference 2010 · doi:10.5244/c.24.56 · Cited by 183
Spatio-temporal Object Detection Proposals
Oneata, D., Revaud, J., Verbeek, J., & Schmid, C. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10578-9_48 · Cited by 181
Maximum weight cliques with mutex constraints for video object segmentation
Ma, T., & Latecki, L. J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247735 · Cited by 175
Fully Connected Object Proposals for Video Segmentation
Perazzi, F., Wang, O., Gross, M., & Sorkine-Hornung, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.369 · Cited by 173
LIVEcut: Learning-based interactive video segmentation by evaluation of multiple propagated cues
Price, B. L., Morse, B. S., & Cohen, S. D. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459293 · Cited by 150
Tracking as Repeated Figure/Ground Segmentation
Ren, X., & Malik, J. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383177 · Cited by 148
Label propagation in video sequences
Badrinarayanan, V., Galasso, F., & Cipolla, R. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540054 · Cited by 141
Li, H., Li, Y., & Porikli, F. (2014) · British Machine Vision Conference (BMVC) · doi:10.5244/c.28.56 · Cited by 140
Occlusion boundary detection and figure/ground assignment from optical flow
Sundberg, P., Brox, T., Maire, M., Arbeláez, P., & Malik, J. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995364 · Cited by 136
Discriminative Segment Annotation in Weakly Labeled Video
Tang, K., Sukthankar, R., Yagnik, J., & Fei-Fei, L. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.321 · Cited by 125
Video Segmentation with Just a Few Strokes
Nagaraja, N. S., Schmidt, F. R., & Brox, T. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.370 · Cited by 120
Active Frame Selection for Label Propagation in Videos
Vijayanarasimhan, S., & Grauman, K. L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33715-4_36 · Cited by 109
Self-citationTrack and Segment: An Iterative Unsupervised Approach for Video Object Proposals
Xiao, F., & Lee, Y. J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.107 · Cited by 109
JOTS: Joint Online Tracking and Segmentation
Wen, L., Du, D., Lei, Z., Li, S. Z., & Yang, M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298835 · Cited by 103
Video Segmentation with Superpixels
Galasso, F., Cipolla, R., & Schiele, B. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-37331-2_57 · Cited by 100
Combining Self Training and Active Learning for Video Segmentation
Fathi, A., Balcan, M.-F., Ren, X., & Rehg, J. M. (2011) · British Machine Vision Conference (BMVC) · doi:10.5244/c.25.78 · Cited by 92
Learning to Find Object Boundaries Using Motion Cues
Stein, A. N., Hoiem, D., & Hebert, M. (2007) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2007.4408841 · Cited by 63
Robust video segment proposals with painless occlusion handling
Wu, Z., Li, F., Sukthankar, R., & Rehg, J. M. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299047 · Cited by 23
Cited by
Anabranch network for camouflaged object segmentation
Le, T.-N., Nguyen, T. V., Nie, Z., Tran, M.-T., & Sugimoto, A. (2019) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2019.04.006 · Cited by 764
YouTube-VOS: Sequence-to-Sequence Video Object Segmentation
Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., & Huang, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01228-1_36 · Cited by 598
See More, Know More: Unsupervised Video Object Segmentation With Co-Attention Siamese Networks
Lu, X., Wang, W., Ma, C., Shen, J., Shao, L., & Porikli, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00374 · Cited by 521
Yang, L., Fan, Y., & Xu, N. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00529 · Cited by 470
Video Object Segmentation without Temporal Information
Maninis, K.-K., Caelles, S., Chen, Y., Pont-Tuset, J., Leal-Taixe, L., Cremers, D., & Van Gool, L. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2838670 · Cited by 354
Metadata
- Authors
- Jain, S. D., Xiong, B., & Grauman, K.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 2117-2126
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2963983744
- Cited by
- 361
- References
- 59
- FWCI
- 15.45
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.