The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index.
Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
Open accessAbstract
Deep learning approaches to optical flow estimation have seen rapid progress over the recent years. One common trait of many networks is that they refine an initial flow estimate either through multiple stages or across the levels of a coarse-to-fine representation. While leading to more accurate results, the downside of this is an increased number of parameters. Taking inspiration from both classical energy minimization approaches as well as residual networks, we propose an iterative residual refinement (IRR) scheme based on weight sharing that can be combined with several backbone networks. It reduces the number of parameters, improves the accuracy, or even achieves both. Moreover, we show that integrating occlusion prediction and bi-directional flow estimation into our IRR scheme can further boost the accuracy. Our full network achieves state-of-the-art results for both optical flow and occlusion estimation across several standard datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Enhanced Deep Residual Networks for Single Image Super-Resolution
Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.151 · Cited by 7,736
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
Region-based convolutional networks for accurate object detection and segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2437384 · Cited by 2,912
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
Mayer, N., Ilg, E., Hausser, P., Fischer, P., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.438 · Cited by 2,488
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields
Black, M. J., & Anandan, P. (1996) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0006 · Cited by 1,570
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214
LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
Hui, T.-W., Tang, X., & Loy, C. C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00936 · Cited by 817
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298720 · Cited by 697
DeMoN: Depth and Motion Network for Learning Monocular Stereo
Ummenhofer, B., Zhou, H., Uhrig, J., Mayer, N., Ilg, E., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.596 · Cited by 692
Deep Feature Flow for Video Recognition
Zhu, X., Xiong, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.441 · Cited by 653
Sun, D., Roth, S., & Black, M. J. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-013-0644-x · Cited by 614
Self-citationUnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
Meister, S., Hur, J., & Roth, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12276 · Cited by 520
Self-citationCascade Residual Learning: A Two-Stage Convolutional Neural Network for Stereo Matching
Pang, J., Sun, W., Ren, J. S., Yang, C., & Yan, Q. (2017) · 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) · doi:10.1109/iccvw.2017.108 · Cited by 491
Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models
Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., & Sivic, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.487 · Cited by 473
SegFlow: Joint Learning for Video Object Segmentation and Optical Flow
Cheng, J., Tsai, Y.-H., Wang, S., & Yang, M.-H. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.81 · Cited by 446
Symmetric Stereo Matching for Occlusion Handling
Sun, J., Li, Y., Kang, S. B., & Shum, H.-Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.337 · Cited by 415
Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness
Yu, J. J., Harley, A. W., & Derpanis, K. G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_1 · Cited by 368
Learning for Disparity Estimation Through Feature Constancy
Liang, Z., Feng, Y., Guo, Y., Liu, H., Chen, W., Qiao, L., Zhou, L., & Zhang, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00297 · Cited by 354
Occlusion Aware Unsupervised Learning of Optical Flow
Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., & Xu, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00513 · Cited by 320
Zhu, Y., & Newsam, S. (2017) · 2017 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2017.8296389 · Cited by 314
Unsupervised deep learning for optical flow estimation
Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., & Zha, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10723 · Cited by 296
Coherent Online Video Style Transfer
Chen, D., Liao, J., Yuan, L., Yu, N., & Hua, G. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.126 · Cited by 281
Bailer, C., Taetz, B., & Stricker, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.457 · Cited by 233
Bilateral Filtering-Based Optical Flow Estimation with Occlusion Detection
Xiao, J., Cheng, H., Sawhney, H., Rao, C., & Isnardi, M. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_17 · Cited by 229
Accurate Optical Flow via Direct Cost Volume Processing
Xu, J., Ranftl, R., & Koltun, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.615 · Cited by 213
Semantic Video Segmentation by Gated Recurrent Flow Propagation
Nilsson, D., & Sminchisescu, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00713 · Cited by 204
Semantic Video CNNs Through Representation Warping
Gadde, R., Jampani, V., & Gehler, P. V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.477 · Cited by 198
Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2894353 · Cited by 195
Unsupervised Learning of Multi-Frame Optical Flow with Occlusions
Janai, J., Güney, F., Ranjan, A., Black, M., & Geiger, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01270-0_42 · Cited by 193
Ilg, E., Saikia, T., Keuper, M., & Brox, T. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01258-8_38 · Cited by 185
Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids
Chen, Q., & Koltun, V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.509 · Cited by 165
Detect, Replace, Refine: Deep Structured Prediction for Pixel Wise Labeling
Gidaris, S., & Komodakis, N. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.760 · Cited by 147
Segmentation-Aware Convolutional Networks Using Local Attention Masks
Harley, A. W., Derpanis, K. G., & Kokkinos, I. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.539 · Cited by 142 · cited 1×
“To address this, bilateral filters or local attention maps [21, 24] have been proposed as viable solutions.”
BackgroundSymmetrical Dense Optical Flow Estimation with Occlusions Detection
Álvarez, L., Rachid Deriche, B., Papadopoulo, T., & Sánchez, J. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0041-4 · Cited by 140
CNN-Based Patch Matching for Optical Flow with Thresholded Hinge Embedding Loss
Bailer, C., Varanasi, K., & Stricker, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.290 · Cited by 127
Exploiting Semantic Information and Deep Matching for Optical Flow
Bai, M., Luo, W., Kundu, K., & Urtasun, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46466-4_10 · Cited by 125
Optical Flow in Mostly Rigid Scenes
Wulff, J., Sevilla-Lara, L., & Black, M. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.731 · Cited by 121
Unsupervised convolutional neural networks for motion estimation
Ahmadi, A., & Patras, I. (2016) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2016.7532634 · Cited by 120
PatchBatch: A Batch Augmented Loss for Optical Flow
Gadot, D., & Wolf, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.459 · Cited by 102
A Fusion Approach for Multi-Frame Optical Flow Estimation
Ren, Z., Gallo, O., Sun, D., Yang, M., Sudderth, E. B., & Kautz, J. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00225 · Cited by 102
Fast Guided Global Interpolation for Depth and Motion
Li, Y., Min, D., Do, M. N., & Lu, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46487-9_44 · Cited by 98
MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimation
Hur, J., & Roth, S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.42 · Cited by 95
Self-citationSemi-Supervised Learning for Optical Flow with Generative Adversarial Networks
Lai, W., Huang, J., & Yang, M. (2017) · neural information processing systems · Cited by 89
SPM-BP: Sped-Up PatchMatch Belief Propagation for Continuous MRFs
Li, Y., Min, D., Brown, M. S., Do, M. N., & Lu, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.456 · Cited by 79
Robust Interpolation of Correspondences for Large Displacement Optical Flow
Hu, Y., Li, Y., & Song, R. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.509 · Cited by 71
Local Layering for Joint Motion Estimation and Occlusion Detection
Sun, D., Liu, C., & Pfister, H. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.144 · Cited by 69
Joint motion estimation and segmentation of complex scenes with label costs and occlusion modeling
Unger, M., Werlberger, M., Pock, T., & Bischof, H. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247887 · Cited by 54
Güney, F., & Geiger, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54190-7_13 · Cited by 54
InterpoNet, a Brain Inspired Neural Network for Optical Flow Dense Interpolation
Zweig, S., & Wolf, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.674 · Cited by 49
A TV-L1 Optical Flow Method with Occlusion Detection
Ballester, C., Garrido, L., Lazcano, V. A., & Caselles, V. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-32717-9_4 · Cited by 47
Continual Occlusion and Optical Flow Estimation
Neoral, M., Šochman, J., & Matas, J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-20870-7_10 · Cited by 40
Guided Optical Flow LearningPreprint
Zhu, Y., Lan, Z., Newsam, S. D., & Hauptmann, A. G. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.02295 · Cited by 36
S2F: Slow-to-Fast Interpolator Flow
Yang, Y., & Soatto, S. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.401 · Cited by 25
FlowFields++: Accurate Optical Flow Correspondences Meet Robust Interpolation
Schuster, R., Bailer, C., Wasenmüller, O., & Stricker, D. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451182 · Cited by 20
Structure-from-Motion-Aware PatchMatch for Adaptive Optical Flow Estimation
Maurer, D., Marniok, N., Goldlüecke, B., & Bruhn, A. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_35 · Cited by 8
Cited by
(2020) · doi:10.1007/978-3-030-58536-5_24 · Cited by 2,824
GMFlow: Learning Optical Flow via Global Matching
Xu, H., Zhang, J., Cai, J., Rezatofighi, H., & Tao, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00795 · Cited by 418
Learning to Estimate Hidden Motions with Global Motion Aggregation
Jiang, S., Campbell, D., Lu, Y., Li, H., & Hartley, R. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00963 · Cited by 365
ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement
Alaluf, Y., Patashnik, O., & Cohen-Or, D. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00664 · Cited by 302
Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data
Kwon, H., Wang, B., Abowd, G. D., & Plötz, T. (2021) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3478096 · Cited by 48
Metadata
- Authors
- Hur, J., & Roth, S.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 5747-5756
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2939996168
- Cited by
- 288
- References
- 63
- FWCI
- 13.16
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.