Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Abstract
Optical flow estimation is classically marked by the requirement of dense sampling in time. While coarse-to-fine warping schemes have somehow relaxed this constraint, there is an inherent dependency between the scale of structures and the velocity that can be estimated. This particularly renders the estimation of detailed human motion problematic, as small body parts can move very fast. In this paper, we present a way to approach this problem by integrating rich descriptors into the variational optical flow setting. This way we can estimate a dense optical flow field with almost the same high accuracy as known from variational optical flow, while reaching out to new domains of motion analysis where the requirement of dense sampling in time is no longer satisfied.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Classical optical flow estimation methods are sensitive to displacement magnitude; variational methods have advantages for small displacements, while large displacements require extra treatment and combinatorial techniques.
“Classical optical flow estimation methods are sensitive to the displacement magnitude. Variational methods (Horn and Schunck, 1981) have clear advantages in case of small displacements, whereas large displacements require extra treatment and the use of additional combinatorial techniques (Brox et al, 2004; Brox and Malik, 2011).”
From What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation? (2018) · cites this paper as Brox and Malik, 2011
Cites
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
(1981) · Cited by 11,963 · see citing works
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312 · see citing works
Horn, B. K. P., & Schunck, B. G. (1981) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.965761 · Cited by 7,443 · see citing works
Learning realistic human actions from movies
Ivan Laptev, A., Marszałek, M., Schmid, C., & Rozenfeld, B. (2008) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2008.4587756 · Cited by 3,481 · see citing works
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634 · see citing works
Self-citationA Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · see citing works
Blake, A., & Zisserman, A. (1987) · The MIT Press eBooks · doi:10.7551/mitpress/7132.001.0001 · Cited by 2,093 · see citing works
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · see citing works
The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields
Black, M. J., & Anandan, P. (1996) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0006 · Cited by 1,571 · see citing works
A new point matching algorithm for non-rigid registration
Chui, H., & Rangarajan, A. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00009-2 · Cited by 1,560 · see citing works
Sigal, L., Bălan, A. O., & Black, M. J. (2009) · International Journal of Computer Vision · doi:10.1007/s11263-009-0273-6 · Cited by 1,202 · see citing works
Shape Matching and Object Recognition Using Low Distortion Correspondences
Berg, A. C., Berg, T. L., & Malik, J. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2005.320 · Cited by 850 · see citing works
Self-citationA Database and Evaluation Methodology for Optical Flow
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408903 · Cited by 618 · see citing works
SIFT Flow: Dense Correspondence across Different Scenes
Liu, C., Yuen, J., Torralba, A. B., Šivic, J., & Freeman, W. T. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-88690-7_3 · Cited by 553 · see citing works
From contours to regions: An empirical evaluation
Arbeláez, P., Maire, M., Fowlkes, C. C., & Malik, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206707 · Cited by 447 · see citing works
Self-citationNearest-Neighbor Methods in Learning and Vision
(2006) · The MIT Press eBooks · doi:10.7551/mitpress/4908.001.0001 · Cited by 442 · see citing works
An Improved Algorithm for TV-L 1 Optical Flow
Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-03061-1_2 · Cited by 423 · see citing works
Highly Accurate Optic Flow Computation with Theoretically Justified Warping
Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006) · International Journal of Computer Vision · doi:10.1007/s11263-005-3960-y · Cited by 409 · see citing works
Self-citationHumanEva: Synchronized video and Motion Capture Dataset for Evaluation of Articulated Human Motion
Sigal, L., Bălan, A. O., & Black, M. J. (2006) · Cited by 357 · see citing works
Nonparametric scene parsing: Label transfer via dense scene alignment
Liu, C., Yuen, J., & Torralba, A. B. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206536 · Cited by 313 · see citing works
Sun, D., Roth, S., Lewis, J. P., & Black, M. J. (2008) · European Conference on Computer Vision · doi:10.1007/978-3-540-88690-7_7 · Cited by 303 · see citing works
Reliable Estimation of Dense Optical Flow Fields with Large Displacements
Álvarez, L., Weickert, J., & Sánchez, J. (2000) · International Journal of Computer Vision · doi:10.1023/a:1008170101536 · Cited by 300 · see citing works
Dense estimation and object-based segmentation of the optical flow with robust techniques
Mémin, É., & Pérez, P. (1998) · IEEE Transactions on Image Processing · doi:10.1109/83.668027 · Cited by 291 · see citing works
Large displacement optical flow
Brox, T., Bregler, C., & Malik, J. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206697 · Cited by 281 · see citing works
Self-citationGeometric blur for template matching
Berg, A. C., & Malik, J. (2005) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2001.990529 · Cited by 280 · see citing works
Self-citationRobust dynamic motion estimation over time
Black, M. J., & Anandan, P. (2002) · doi:10.1109/cvpr.1991.139705 · Cited by 242 · see citing works
Robust computation of optical flow in a multi-scale differential framework
Weber, J. W., & Malik, J. (1995) · International Journal of Computer Vision · doi:10.1007/bf01421489 · Cited by 228 · see citing works
Self-citationFusionFlow: Discrete-continuous optimization for optical flow estimation
Lempitsky, V., Roth, S., & Rother, C. (2008) · 2008 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2008.4587751 · Cited by 145 · see citing works
Large displacement optical flow computation withoutwarping
Steinbrücker, F., Pock, T., & Cremers, D. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459364 · Cited by 137 · see citing works
Towards ultimate motion estimation: combining highest accuracy with real-time performance
Bruhn, A., & Weickert, J. (2005) · doi:10.1109/iccv.2005.240 · Cited by 128 · see citing works
Illumination-Robust Variational Optical Flow with Photometric Invariants
Mileva, Y. M., Bruhn, A., & Weickert, J. (2007) · Springer eBooks · doi:10.1007/978-3-540-74936-3_16 · Cited by 107 · see citing works
Zimmer, H., Bruhn, A., Weickert, J., Valgaerts, L., Salgado, A., Rosenhahn, B., & Seidel, H.-P. (2009) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-03641-5_16 · Cited by 107 · see citing works
Layered Estimation of Atmospheric Mesoscale Dynamics From Satellite Imagery
Héas, P., Mémin, É., Papadakis, N., & Szantaï, A. (2007) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2007.906156 · Cited by 70 · see citing works
Nonlinear Variational Method for Optical Flow Computation
Cohen, I. (2006) · Cited by 69 · see citing works
Optical flow estimation on coarse-to-fine region-trees using discrete optimization
Lei, C., & Yang, Y.-H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459253 · Cited by 67 · see citing works
Efficient MRF Deformation Model for Non-Rigid Image Matching
Shekhovtsov, A., Kovtun, I., & Hlaváč, V. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383205 · Cited by 58 · see citing works
Discrete-Continuous Optimization for Optical Flow Estimation
Roth, S., Lempitsky, V., & Rother, C. (2009) · Statistical and Geometrical Approaches to Visual Motion Analysis · doi:10.1007/978-3-642-03061-1_1 · Cited by 27 · see citing works
A Feature-Based Approach for Determining Dense Long Range Correspondences
Wills, J., & Belongie, S. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24672-5_14 · Cited by 17 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Learning Spatiotemporal Features with 3D Convolutional Networks
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.510 · Cited by 9,593 · see citing works
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · see citing works
Self-citationObject scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270 · see citing works
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842 · see citing works
Dense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669 · see citing works
Flow-Guided Feature Aggregation for Video Object Detection
Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.52 · Cited by 729 · see citing works
“The recent focus is on large displacements [3], and combinatorial matching (e.”
BackgroundUnsupervised deep learning for optical flow estimation
Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., & Zha, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10723 · Cited by 296 · see citing works
What Makes Good Synthetic Training Data for Learning Disparity and Optical Flow Estimation?
Mayer, N., Ilg, E., Fischer, P., Hazirbas, C., Cremers, D., Dosovitskiy, A., & Brox, T. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1082-6 · Cited by 191 · see citing works
Self-citationSegmentation-Aware Convolutional Networks Using Local Attention Masks
Harley, A. W., Derpanis, K. G., & Kokkinos, I. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.539 · Cited by 142 · see citing works
Optical Flow Estimation with Deep Learning, a Survey on Recent Advances
Savian, S., Elahi, M., & Tillo, T. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-32583-1_12 · Cited by 29 · see citing works
- Authors
- Brox, T., & Malik, J.
- Affiliations
- University of California, Berkeley (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2011-03
- Volume
- 33
- Issue
- 3
- Pages
- 500-513
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2131747574
- Cited by
- 1,295
- References
- 40
- Citations by year
- 2026: 18 · 2025: 34 · 2024: 41 · 2023: 61 · 2022: 59 · 2021: 80 · 2020: 98 · 2019: 124 · 2018: 138 · 2017: 121 · 2016: 136 · 2015: 121 · 2014: 99 · 2013: 84 · 2012: 50
- FWCI
- 60.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Deutscher Akademischer Austauschdienst
- Multidisciplinary University Research Initiative
- Office of Naval Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- optical flow, descriptor matching, variational optical flow, dense optical flow, human motion analysis
- MeSH terms
- Algorithms, Artificial Intelligence, Computer Simulation, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Models, Theoretical, Motion, Pattern Recognition, Automated, Sensitivity and Specificity, Signal Processing, Computer-Assisted, Subtraction Technique, Video Recording, Information Storage and Retrieval, Phantoms, Imaging, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
