Unsupervised deep learning for optical flow estimation
Open accessAbstract
Recent work has shown that optical flow estimation can be formulated as a supervised learning problem. Moreover, convolutional networks have been successfully applied to this task. However, supervised flow learning is obfuscated by the shortage of labeled training data. As a consequence, existing methods have to turn to large synthetic datasets for easily computer generated ground truth. In this work, we explore if a deep network for flow estimation can be trained without supervision. Using image warping by the estimated flow, we devise a simple yet effective unsupervised method for learning optical flow, by directly minimizing photometric consistency. We demonstrate that a flow network can be trained from end-to-end using our unsupervised scheme. In some cases, our results come tantalizingly close to the performance of methods trained with full supervision.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The method outperforms recent methods that explicitly compute optical flow and motion masks.
“We also outperform recent methods ( [38,51]) that explicitly compute optical flow as well as motion masks.”
From Digging Into Self-Supervised Monocular Depth Estimation (2019) · cites this paper as [51]
Our models generate higher quality outputs and do not produce holes in the depth maps or border artifacts that can be seen in many existing baselines.
“We can see that our models generate higher quality outputs and do not produce 'holes' in the depth maps or border artifacts that can be seen in many existing baselines e.g.[76,51,15,73].”
From Digging Into Self-Supervised Monocular Depth Estimation (2019) · cites this paper as [51]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
A performance evaluation of local descriptors
Mikolajczyk, K., & Schmid, C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.188 · Cited by 6,723 · see citing works
Caffe: Convolutional Architecture for Fast Feature EmbeddingPreprint
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1408.5093 · Cited by 4,748 · see citing works
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · see citing works
Recurrent Models of Visual AttentionPreprint
Mnih, V., Heess, N., Graves, A., & Kavukcuoglu, K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.6247 · Cited by 3,286 · see citing works
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture notes in computer science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,634 · see citing works
Fully Convolutional Networks for Semantic SegmentationPreprint
Long, J., Shelhamer, E., & Darrell, T. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.4038 · Cited by 2,490 · see citing works
Object scene flow for autonomous vehicles
Menze, M., & Geiger, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298925 · Cited by 2,270 · see citing works
Unsupervised Learning of Video Representations using LSTMs
Srivastava, N., Mansimov, E., & Salakhutdinov, R. (2015) · arXiv · doi:10.48550/arxiv.1502.04681 · Cited by 1,861 · see citing works
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842 · see citing works
Learning to compare image patches via convolutional neural networks
Zagoruyko, S., & Nikos Komodakis, A. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7299064 · Cited by 1,540 · see citing works
Hinton, G. E., Krizhevsky, A., & Wang, S. D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-21735-7_6 · Cited by 1,360 · see citing works
Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295 · see citing works
Mayer, N., Ilg, E., Cremers, D., Dosovitskiy, A., & Brox, T. (2016) · Cited by 1,214 · see citing works
GradientBased Learning Applied to Document Recognition
Haykin, S., & Kosko, B. (2009) · Institute of Electrical and Electronics Engineers eBooks · doi:10.1109/9780470544976.ch9 · Cited by 1,134 · see citing works
DeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.175 · Cited by 1,019 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
A performance evaluation of local descriptors
Mikolajczyk, K., & Schmid, C. (2003) · 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. · doi:10.1109/cvpr.2003.1211478 · Cited by 955 · see citing works
Computing the stereo matching cost with a convolutional neural network
Žbontar, J., & LeCun, Y. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298767 · Cited by 816 · see citing works
FlowNet: Learning Optical Flow with Convolutional NetworksPreprint
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 814 · see citing works
Multiple Object Recognition with Visual Attention
Ba, J., Mnih, V., & Kavukcuoglu, K. (2014) · arXiv · doi:10.48550/arxiv.1412.7755 · Cited by 808 · see citing works
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298720 · Cited by 757 · see citing works
Sun, D., Roth, S., & Black, M. J. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0644-x · Cited by 698 · see citing works
The Generalized PatchMatch Correspondence Algorithm
Barnes, C., Shechtman, E., Goldman, D. B., & Finkelstein, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15558-1_3 · Cited by 538 · see citing works
Motion Detail Preserving Optical Flow Estimation
Li, X., Jia, J., & Matsushita, Y. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.236 · Cited by 499 · see citing works
Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models
Aubry, M., Maturana, D., Efros, A. A., Russell, B. C., & Šivic, J. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.487 · Cited by 471 · see citing works
van Hateren, J. H., & Ruderman, D. (1998) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.1998.0577 · Cited by 407 · see citing works
Learning Dense Correspondence via 3D-Guided Cycle Consistency
Zhou, T., Krähenbühl, P., Aubry, M., Qi-xing Huang, A., & Efros, A. A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.20 · Cited by 351 · see citing works
DeepMatching: Hierarchical Deformable Dense Matching
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0908-3 · Cited by 274 · see citing works
Hollingworth, A. (2004) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/0096-1523.30.3.519 · Cited by 259 · see citing works
Renée Baillargeon, B. (2004) · Current Directions in Psychological Science · doi:10.1111/j.0963-7214.2004.00281.x · Cited by 228 · see citing works
Full Flow: Optical Flow Estimation By Global Optimization over Regular Grids
Chen, Q., & Koltun, V. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.509 · Cited by 166 · see citing works
Discovering Hidden Factors of Variation in Deep NetworksPreprint
Cheung, B., Livezey, J. A., Bansal, A. K., & Olshausen, B. A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6583 · Cited by 152 · see citing works
Towards ultimate motion estimation: combining highest accuracy with real-time performance
Bruhn, A., & Weickert, J. (2005) · doi:10.1109/iccv.2005.240 · Cited by 128 · see citing works
Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow
Bao, L., Yang, Q., & Jin, H. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2359374 · Cited by 109 · see citing works
Simple-Cell-Like Receptive Fields Maximize Temporal Coherence in Natural Video
Hurri, J., & Hyvärinen, A. (2003) · Neural Computation · doi:10.1162/089976603321192121 · Cited by 99 · see citing works
Unsupervised Learning of Edges
Li, Y., Paluri, M., Rehg, J. M., & Dollár, P. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.179 · Cited by 94 · see citing works
Fast Edge-Preserving PatchMatch for Large Displacement Optical Flow
Bao, L., Yang, Q., & Jin, H. (2014) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2014.452 · Cited by 90 · see citing works
Learning to Match Aerial Images with Deep Attentive Architectures
Altwaijry, H., Trulls, E., Hays, J. H., Fua, P., & Belongie, S. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.385 · Cited by 85 · see citing works
A Tube-and-Droplet-Based Approach for Representing and Analyzing Motion Trajectories
Lin, W., Zhou, Y., Xu, H., et al. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2608884 · Cited by 58 · see citing works
Unsupervised Feature Learning from Temporal DataPreprint
Goroshin, R., Bruna, J., Tompson, J., Eigen, D., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.02518 · Cited by 29 · see citing works
Full Flow: Optical Flow Estimation By Global Optimization over Regular GridsPreprint
Chen, Q., & Koltun, V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.03513 · Cited by 19 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Jaderberg, M.; Simonyan, K.; Zisserman, A.; and Kavukcuoglu, K. 2015. Spatial transformer networks. In NIPS .
- Liu, C.; Yuen, J.; and Torralba, A. 2011. Sift flow: Dense correspondence across scenes and its applications. TPAMI .
- Wang, X., and Gupta, A. 2015. Unsupervised learning of visual representations using videos. In ICCV .
Cited by
Sorted by citations
Digging Into Self-Supervised Monocular Depth Estimation
Godard, C., Aodha, O. M., Firman, M., & Brostow, G. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00393 · Cited by 2,561 · see citing works
Optical Flow Estimation with Deep Learning, a Survey on Recent Advances
Savian, S., Elahi, M., & Tillo, T. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-32583-1_12 · Cited by 29 · see citing works
- Authors
- Ren, Z., Yan, J., Ni, B., Liu, B., Yang, X., & Zha, H.
- Affiliations
- Shanghai Jiao Tong University (CN)
- East China Normal University (CN)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2017-02-12
- Volume
- 31
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2604909019
- Cited by
- 296
- References
- 49
- Citations by year
- 2026: 7 · 2025: 9 · 2024: 22 · 2023: 29 · 2022: 37 · 2021: 48 · 2020: 56 · 2019: 43 · 2018: 40 · 2017: 5
- FWCI
- 11.51
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Science Foundation (61521062, 1639792, 1620345)
- National Natural Science Foundation of China (B07022, 6152106 2, 15JC1401700, 61527804, 61602176, 61672231, IIS-1639792)
- China Postdoctoral Science Foundation (2016M590337, B07022)
- Science and Technology Commission of Shanghai Municipality (B07022, 15JC1401700, 14XD1402100)
- National Key Research and Development Program of China (61521062, 2016YFB1001003)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- optical flow, unsupervised learning, photometric consistency, image warping, synthetic training data, end-to-end learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
