The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Secrets of optical flow estimation and their principles
Abstract
The accuracy of optical flow estimation algorithms has been improving steadily as evidenced by results on the Middlebury optical flow benchmark. The typical formulation, however, has changed little since the work of Horn and Schunck. We attempt to uncover what has made recent advances possible through a thorough analysis of how the objective function, the optimization method, and modern implementation practices influence accuracy. We discover that “classical” flow formulations perform surprisingly well when combined with modern optimization and implementation techniques. Moreover, we find that while median filtering of intermediate flow fields during optimization is a key to recent performance gains, it leads to higher energy solutions. To understand the principles behind this phenomenon, we derive a new objective that formalizes the median filtering heuristic. This objective includes a nonlocal term that robustly integrates flow estimates over large spatial neighborhoods. By modifying this new term to include information about flow and image boundaries we develop a method that ranks at the top of the Middlebury benchmark.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Numerical recipes in C: the art of scientific computing
(1993) · Choice Reviews Online · doi:10.5860/choice.30-5638 · Cited by 17,933
Nonlinear total variation based noise removal algorithms
Rudin, L. I., Osher, S., & Fatemi, E. (1992) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(92)90242-f · Cited by 15,704
(1981) · Cited by 11,963
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312
A Non-Local Algorithm for Image Denoising
Buades, A., Coll, B., & Morel, J.-M. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.38 · Cited by 7,616
Horn, B. K., & Schunck, B. G. (1981) · SPIE Proceedings · doi:10.1117/12.965761 · Cited by 7,394
Numerical Recipes in C: The Art of Scientific Computing.
Birchenhall, C., Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · The Economic Journal · doi:10.2307/2234664 · Cited by 4,355
Performance of optical flow techniques
Barron, J. L., Fleet, D. J., & Beauchemin, S. S. (1994) · International Journal of Computer Vision · doi:10.1007/bf01420984 · Cited by 4,305
(1986) · Cited by 3,616
Cubic convolution interpolation for digital image processing
Keys, R. (1981) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1981.1163711 · Cited by 3,255
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205
Self-citationBlake, A., & Zisserman, A. (1987) · The MIT Press eBooks · doi:10.7551/mitpress/7132.001.0001 · Cited by 2,105
Crimins, F. (2003) · Applied Biochemistry and Biotechnology · doi:10.1007/s12010-003-0001-6 · Cited by 2,000
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658
The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields
Black, M. J., & Anandan, P. (1996) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0006 · Cited by 1,570
Self-citationNonlocal Operators with Applications to Image Processing
Gilboa, G., & Osher, S. (2009) · Multiscale Modeling & Simulation · doi:10.1137/070698592 · Cited by 1,423
Adaptive support-weight approach for correspondence search
Yoon, K.-J., & Kweon, I. S. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.70 · Cited by 1,194
Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods
Bruhn, A., Weickert, J., & Schnörr, C. (2005) · International Journal of Computer Vision · doi:10.1023/b:visi.0000045324.43199.43 · Cited by 1,160
Hierarchical model-based motion estimation
Bergen, J. R., Anandan, P., Hanna, K. J., & Hingorani, R. (1992) · Lecture Notes in Computer Science · doi:10.1007/3-540-55426-2_27 · Cited by 1,154
Lucas/Kanade meets Horn/Schunck : combining local and global optic flow methods
Weickert, J., Bruhn, A., & Schnörr, C. (2003) · doi:10.22028/d291-26234 · Cited by 963
Nagel, H.-H., & Enkelmann, W. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767833 · Cited by 786
A Database and Evaluation Methodology for Optical Flow
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408903 · Cited by 618
Self-citationSun, D., Roth, S., & Black, M. J. (2014) · International Journal of Computer Vision · doi:10.1007/s11263-013-0644-x · Cited by 614
Self-citationAn Improved Algorithm for TV-L 1 Optical Flow
Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03061-1_2 · Cited by 423
Particle Video: Long-Range Motion Estimation Using Point Trajectories
Sand, P., & Teller, S. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-008-0136-6 · Cited by 365
Anisotropic Huber-L1 Optical Flow
Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009) · Procedings of the British Machine Vision Conference 2009 · doi:10.5244/c.23.108 · Cited by 354
Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88690-7_7 · Cited by 230
Self-citationBilateral Filtering-Based Optical Flow Estimation with Occlusion Detection
Xiao, J., Cheng, H., Sawhney, H., Rao, C., & Isnardi, M. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_17 · Cited by 229
Motion estimation with non-local total variation regularization
Werlberger, M., Pock, T., & Bischof, H. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539945 · Cited by 216
Black, M. J., & Jepson, A. D. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.541407 · Cited by 191
Self-citationStructure- and motion-adaptive regularization for high accuracy optic flow
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459375 · Cited by 177
FusionFlow: Discrete-continuous optimization for optical flow estimation
Lempitsky, V., Roth, S., & Rother, C. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587751 · Cited by 145
Self-citationBab‐Hadiashar, A., & Suter, D. (1998) · International Journal of Computer Vision · doi:10.1023/a:1008090730467 · Cited by 142
Consistent segmentation for optical flow estimation
Zitnick, C. W., Jojić, N., & Kang, S. B. (2005) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2005.61 · Cited by 130
Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H.-P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03641-5_16 · Cited by 107 · cited 4×
“Significant ones use oriented smoothness [25, 31, 33, 40], rigidity constraints [32, 33], or image segmentation [9, 21, 41, 37].”
BackgroundInfluentialA new median formula with applications to PDE based denoising
Li, Y., Li, Y., Osher, S., et al. (2009) · Communications in Mathematical Sciences · doi:10.4310/cms.2009.v7.n3.a11 · Cited by 87
Optical flow estimation on coarse-to-fine region-trees using discrete optimization
Lei, C., & Yang, Y.-H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459253 · Cited by 67
A Segmentation Based Variational Model for Accurate Optical Flow Estimation
Xu, L., Chen, J., & Jia, J. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88682-2_51 · Cited by 66
Duality TV-L1 flow with fundamental matrix prior
Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008) · 2008 23rd International Conference Image and Vision Computing New Zealand · doi:10.1109/ivcnz.2008.4762119 · Cited by 58
Segmenting video into classes of algorithm-suitability
Aodha, O. M., Brostow, G., & Pollefeys, M. (2010) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2010.5540099 · Cited by 51
Local grouping for optical flow
Ren, X. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587536 · Cited by 51
Research works printed but not linked: 4
- wedel, 2008, An improved algorithm for TV-L1 optical flow, Dagstuhl Motion Work-shop
- glaer, 1983, Scene matching by hierarchical correlation, CVPR, 432
- hsiao, 2003, A new convex edgepreserving median prior with applications to tomography, IEEE TMI, 22, 580
- adelson, 1984, Pyramid methods in image processing, RCA Engineer, 29, 33
Cited by
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205
Self-citationVoxelMorph: A Learning Framework for Deformable Medical Image Registration
Balakrishnan, G., Zhao, A., Sabuncu, M. R., Guttag, J., & Dalca, A. V. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2897538 · Cited by 2,174
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
Self-citationDeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.175 · Cited by 997
Metadata
- Authors
- Sun, D., Roth, S., & Black, M. J.
- Venue
- 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2010-06
- Pages
- 2432-2439
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2033959528
- Cited by
- 1,389
- References
- 42
- FWCI
- 73.95
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.