The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind Them
Open accessAbstract
The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that “classical” flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. One key implementation detail is the median filtering of intermediate flow fields during optimization. While this improves the robustness of classical methods it actually leads to higher energy solutions, meaning that these methods are not optimizing the original objective function. To understand the principles behind this phenomenon, we derive a new objective function that formalizes the median filtering heuristic. This objective function includes a non-local smoothness term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that can better preserve motion details. To take advantage of the trend towards video in wide-screen format, we further introduce an asymmetric pyramid downsampling scheme that enables the estimation of longer range horizontal motions. The methods are evaluated on the Middlebury, MPI Sintel, and KITTI datasets using the same parameter settings.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Stochastic relaxation, gibbs distributions, and the bayesian restoration of images.
Geman, S., & Geman, D. (1984) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1984.4767596 · Cited by 17,949
Numerical recipes in C: the art of scientific computing
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5638 · Cited by 17,933
Nonlinear total variation based noise removal algorithms
Rudin, L. I., Osher, S., & Fatemi, E. (1992) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(92)90242-f · Cited by 15,704
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
(1981) · Cited by 11,963
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312
A Non-Local Algorithm for Image Denoising
Buades, A., Coll, B., & Morel, J.-M. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.38 · Cited by 7,616 · cited 5×
“success of the non-local regularization [21] in image restoration and stereo.”
BackgroundMethodInfluentialHorn, B. K., & Schunck, B. G. (1981) · SPIE Proceedings · doi:10.1117/12.965761 · Cited by 7,394
Marr, D. (1982) · Cumulative Index of Computer Aided Architectural Design · Cited by 5,733
Barlow, H. B. (1983) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(83)90030-5 · Cited by 5,424
Numerical Recipes in C: The Art of Scientific Computing.
Birchenhall, C., Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · The Economic Journal · doi:10.2307/2234664 · Cited by 4,355
Performance of optical flow techniques
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994) · International Journal of Computer Vision · doi:10.1007/bf01420984 · Cited by 4,305
Computer vision: algorithms and applications
(2011) · Choice Reviews Online · doi:10.5860/choice.48-5140 · Cited by 3,911
(1986) · Cited by 3,616
Cubic convolution interpolation for digital image processing
Keys, R. (1981) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1981.1163711 · Cited by 3,255
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 1×
“The field of optical flow estimation is making steady progress as evidenced by the increasing accuracy of current meth-ods on the Middlebury optical flow benchmark (Baker et al. 2007).”
BackgroundSelf-citationBlake, A., & Zisserman, A. (1987) · The MIT Press eBooks · doi:10.7551/mitpress/7132.001.0001 · Cited by 2,105
Crimins, F. (2003) · Applied Biochemistry and Biotechnology · doi:10.1007/s12010-003-0001-6 · Cited by 2,000
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
Self-citationA Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 5×
“Note in practice the weight λC on the coupling term is usually small or is steadily increased from small values (Wedel et al. 2008b; Zach et al. 2007).”
BackgroundMethodInfluentialThe Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields
Black, M. J., & Anandan, P. (1996) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0006 · Cited by 1,570
Self-citationNonlocal Operators with Applications to Image Processing
Gilboa, G., & Osher, S. (2009) · Multiscale Modeling & Simulation · doi:10.1137/070698592 · Cited by 1,423
Secrets of optical flow estimation and their principles
Sun, D., Roth, S., & Black, M. J. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539939 · Cited by 1,389
Self-citationLarge Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295
Adaptive support-weight approach for correspondence search
Yoon, K.-J., & Kweon, I. S. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.70 · Cited by 1,194
Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods
Bruhn, A., Weickert, J., & Schnörr, C. (2005) · International Journal of Computer Vision · doi:10.1023/b:visi.0000045324.43199.43 · Cited by 1,160
Hierarchical model-based motion estimation
Bergen, J. R., Anandan, P., Hanna, K. J., & Hingorani, R. (1992) · Lecture Notes in Computer Science · doi:10.1007/3-540-55426-2_27 · Cited by 1,154
Representing moving images with layers
Wang, J. Y. A., & Adelson, E. H. (1994) · IEEE Transactions on Image Processing · doi:10.1109/83.334981 · Cited by 1,090
Lucas/Kanade meets Horn/Schunck : combining local and global optic flow methods
Weickert, J., Bruhn, A., & Schnörr, C. (2003) · doi:10.22028/d291-26234 · Cited by 963
Nagel, H.-H., & Enkelmann, W. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767833 · Cited by 786
A Database and Evaluation Methodology for Optical Flow
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408903 · Cited by 618
Self-citationMotion Detail Preserving Optical Flow Estimation
Xu, L., Jia, J., & Matsushita, Y. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.236 · Cited by 493
An Improved Algorithm for TV-L 1 Optical Flow
Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03061-1_2 · Cited by 423
Particle Video: Long-Range Motion Estimation Using Point Trajectories
Sand, P., & Teller, S. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0136-6 · Cited by 365
Anisotropic Huber-L1 Optical Flow
Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.108 · Cited by 354
Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88690-7_7 · Cited by 230
Self-citationBilateral Filtering-Based Optical Flow Estimation with Occlusion Detection
Xiao, J., Cheng, H., Sawhney, H., Rao, C., & Isnardi, M. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_17 · Cited by 229
Motion estimation with non-local total variation regularization
Werlberger, M., Pock, T., & Bischof, H. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539945 · Cited by 216
Zimmer, H., Bruhn, A., & Weickert, J. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-011-0422-6 · Cited by 215
Fusion Moves for Markov Random Field Optimization
Lempitsky, V., Rother, C., Roth, S., & Blake, A. (2010) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2009.143 · Cited by 196
Self-citationBlack, M. J., & Jepson, A. D. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.541407 · Cited by 191
Self-citationHuman-assisted motion annotation
Liu, C., Freeman, W. T., Adelson, E. H., & Weiss, Y. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587845 · Cited by 186
Structure- and motion-adaptive regularization for high accuracy optic flow
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459375 · Cited by 177
Large Displacement Optical Flow from Nearest Neighbor Fields
Chen, Z., Jin, H., Lin, Z., Cohen, S. D., & Wu, Y. N. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.316 · Cited by 176
FusionFlow: Discrete-continuous optimization for optical flow estimation
Lempitsky, V., Roth, S., & Rother, C. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587751 · Cited by 145 · cited 2×
“…al. 2010; Zimmer et al. 2009) to reduce the influence of lighting changes, incremental warping (Bergen et al. 1992), warping with bicubic interpolation (Lempitsky et al. 2008; Wedel et al. 2008b), temporal averaging of image derivatives (Horn 1986; Wedel et al. 2008b),…”
BackgroundSelf-citationBab‐Hadiashar, A., & Suter, D. (1998) · International Journal of Computer Vision · doi:10.1023/a:1008090730467 · Cited by 142
Towards ultimate motion estimation: combining highest accuracy with real-time performance
Bruhn, A., & Weickert, J. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.240 · Cited by 138
Large displacement optical flow computation withoutwarping
Steinbrücker, F., Pock, T., & Cremers, D. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459364 · Cited by 137
A generative perspective on MRFs in low-level vision
Schmidt, U., Gao, Q., & Roth, S. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5539844 · Cited by 132
Self-citationConsistent segmentation for optical flow estimation
Zitnick, C. W., Jojić, N., & Kang, S. B. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.61 · Cited by 130
Over-Parameterized Variational Optical Flow
Nir, T., Bruckstein, A. M., & Kimmel, R. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0051-2 · Cited by 118
Layered image motion with explicit occlusions, temporal consistency, and depth ordering
Sun, D., Sudderth, E. B., & Black, M. J. (2010) · Cited by 116
Self-citationZimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H.-P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03641-5_16 · Cited by 107 · cited 6×
“mation to deal with large motions [13,17], texture decomposition [65,67] or high-order filter constancy [8,17,29,39, 77] to reduce the influence of lighting changes, incremental”
BackgroundMethodInfluentialA Fully-Connected Layered Model of Foreground and Background Flow
Sun, D., Wulff, J., Sudderth, E. B., Pfister, H., & Black, M. J. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.317 · Cited by 104
Self-citationRegularization of discontinuous flow fields
Shulman, D., & Herve, J.-Y. (2003) · IEEE Workshop on Visual Motion · doi:10.1109/wvm.1989.47097 · Cited by 95
Layered segmentation and optical flow estimation over time
Sun, D., Sudderth, E. B., & Black, M. J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247873 · Cited by 90
Self-citationA new median formula with applications to PDE based denoising
Li, Y., Li, Y., Osher, S., et al. (2009) · Communications in Mathematical Sciences · doi:10.4310/cms.2009.v7.n3.a11 · Cited by 87
A Segmentation Based Variational Model for Accurate Optical Flow Estimation
Xu, L., Chen, J., & Jia, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_51 · Cited by 79
Optical flow estimation on coarse-to-fine region-trees using discrete optimization
Lei, C., & Yang, Y.-H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459253 · Cited by 67 · cited 1×
“…(Nagel and Enkelmann 1986; Sun et al. 2008; Wedel et al. 2009; Zimmer et al. 2011, 2009), rigidity constraints (Wedel et al. 2008a, 2009), an over-parameterized smoothness term (Nir et al. 2008), or image segmentation (Black and Jepson 1996; Lei and Yang 2009; Xu et al. 2008; Zitnick et al. 2005).”
BackgroundLearning to find occlusion regions
Humayun, A., Aodha, O. M., & Brostow, G. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995517 · Cited by 62
Efficient Nonlocal Regularization for Optical Flow
Krähenbühl, P., & Koltun, V. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33718-5_26 · Cited by 60
Duality TV-L1 flow with fundamental matrix prior
Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008) · 2008 23rd International Conference Image and Vision Computing New Zealand · doi:10.1109/ivcnz.2008.4762119 · Cited by 58 · cited 17×
“We find that one algorithmic choice produces the most significant improvements: applying a median filter to intermediate flow values during incremental estimation and warping (Wedel et al. 2008b, 2009).”
BackgroundMethodResultInfluentialEnabling warping on stereoscopic images
Niu, Y., Feng, W., & Liu, F. (2012) · ACM Transactions on Graphics · doi:10.1145/2366145.2366202 · Cited by 53
Segmenting video into classes of algorithm-suitability
Aodha, O. M., Brostow, G., & Pollefeys, M. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540099 · Cited by 51
Local grouping for optical flow
Ren, X. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587536 · Cited by 51
Optical flow estimation using learned sparse model
Jia, K., Wang, X., & Tang, X. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126522 · Cited by 43
A polar representation of motion and implications for optical flow
Adato, Y., Zickler, T. E., & Ben‐Shahar, O. (2011) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2011.5995419 · Cited by 19
Decomposing and regularizing sparse/non-sparse components for motion field estimation
Chen, Z., Wang, J., & Wu, Y. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247874 · Cited by 17
Residual Images Remove Illumination Artifacts!
Vaudrey, T., & Klette, R. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03798-6_48 · Cited by 15
Low level vision via switchable Markov random fields
Lin, D., & Fisher, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247957 · Cited by 7
Cited by
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
Self-citationObject scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270
Secrets of optical flow estimation and their principles
Sun, D., Roth, S., & Black, M. J. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539939 · Cited by 1,389
Self-citationOptical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Self-citationStereo Matching by Training a Convolutional Neural Network to Compare Image Patches
Žbontar, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1510.05970 · Cited by 931
Metadata
- Authors
- Sun, D., Roth, S., & Black, M. J.
- Venue
- International Journal of Computer Vision
- Published
- 2013-09-03
- Volume
- 106
- Issue
- 2
- Pages
- 115-137
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2155302366
- Cited by
- 614
- References
- 72
- FWCI
- 53.90
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.