LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
Open accessAbstract
FlowNet2 [14], the state-of-the-art convolutional neural network (CNN) for optical flow estimation, requires over 160M parameters to achieve accurate flow estimation. In this paper we present an alternative network that attains performance on par with FlowNet2 on the challenging Sintel final pass and KITTI benchmarks, while being 30 times smaller in the model size and 1.36 times faster in the running speed. This is made possible by drilling down to architectural details that might have been missed in the current frameworks: (1) We present a more effective flow inference approach at each pyramid level through a lightweight cascaded network. It not only improves flow estimation accuracy through early correction, but also permits seamless incorporation of descriptor matching in our network. (2) We present a novel flow regularization layer to ameliorate the issue of outliers and vague flow boundaries by using a feature-driven local convolution. (3) Our network owns an effective structure for pyramidal feature extraction and embraces feature warping rather than image warping as practiced in FlowNet2. Our code and trained models are available at github.com/twhui/LiteFlowNet.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LiteFlowNet trained on the synthetic Scene Flow dataset can generalize well in real-world scenarios, though sometimes artifacts are present in some regions.
“LiteFlowNet shows good generalization ability. LiteFlowNet trained on a synthetic dataset (Scene Flow(Dosovitskiy et al. 2015)) can generalize well in realworld scenarios, though sometimes artifacts present in some regions.”
From DF-VO: What Should Be Learnt for Visual Odometry? (2021) · cites this paper as Hui et al. 2018
The Training Details in §4 say LiteFlowNet was trained on a Chairs → Things3D schedule. The KITTI results say it generalized well to real-world data *after fine-tuning on KITTI*, not solely after synthetic-data training. The paper discusses artifacts, but the cited examples concern other conditions, including variants with components disabled, rather than establishing the claimed artifact qualification for that generalization result. (LLM, conf=0.87)
Experimental — accuracy not yet measuredLiteFlowNet is fast, lightweight, and accurate.
“In this work, we choose LiteFlowNet(Hui et al. 2018) as our backbone network for optical flow prediction since LiteFlowNet is fast, lightweight, and accurate.”
From DF-VO: What Should Be Learnt for Visual Odometry? (2021) · cites this paper as Hui et al. 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · see citing works
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · see citing works
A Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842 · see citing works
Flagged first, then by citations
Research works printed but not linked: 32
- C. Bailer, B. Taetz, and D. Stricker. Flow Fields: Dense correspondence fields for highly accurate large displacement optical flow estimation. ICCV, pages 4015-4023, 2015. 3, 7
- C. Bailer, K. Varanasi, and D. Stricker. CNN-based patch matching for optical flow with thresholded hinge embedding loss. CVPR, pages 3250-3259, 2017. 3, 7
- C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Gold- man. PatchMatch: A randomized correspondence algorithm for structural image editing. SIGGRAGH, pages 83-97, 2009. 2
- M. J. Black, Y. Yacoobt, A. D. Jepsont, and D. J. Fleets. Learning parameterized models of image motion. CVPR, pages 674-679, 1997. 2, 3
- T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High ac- curacy optical flow estimation based on a theory for warping. ECCV, pages 25-36, 2004. 1, 2, 3
- T. Brox and J. Mailk. Large displacement optical flow: De- scriptor matching in variational motion estimation. PAMI, 33(3):500-513, 2011. 1, 2, 7
- A. Geiger, P. Lenz, and R. Urtasun. Are we ready for au- tonomous driving? CVPR, pages 3354-3361, 2012. 6
- F. Gney and A. Geiger. Deep discrete flow. ACCV, pages 207-224, 2016. 3, 7
- T.-W. Hui and R. Chung. Determining motion directly from normal flows upon the use of a spherical eye platform. CVPR, pages 2267-2274, 2013. 1
- E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox. FlowNet2.0: Evolution of optical flow estimation with deep networks. CVPR, pages 2462-2470, 2017. 1, 2, 3, 4, 6, 7, 8
- M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial transformer networks. NIPS, pages 2017-2025, 2015. 3, 4
- J. Lu, H. Yang, D. Min, and M. N. Do. PatchMatch Fil- ter: Efficient edge-aware filtering meets randomized search. CVPR, pages 1854-1861, 2013. 2
- N. Mayer, E. Ilg, P. Husser, P. Fischer, D. Cremers, A. Doso- vitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estima- tion. CVPR, pages 4040-4048, 2016. 6, 7
- M. Menze and A. Geiger. Object scene flow for autonomous vehicles. CVPR, pages 3061-3070, 2015. 6
- T. Nir, A. M. Bruckstein, and R. Kimmel. Over- parameterized variational optical flow. IJCV, 76(2):205-216, 2008.
- N. Papenberg, A. Bruhn, T. Brox, S. Didas, and J. Weick- ert. Highly accurate optic flow computation with theoreti- cally justified warping. IJCV, 67(2):141-158, 2006. 1, 3
- A. Ranjan and M. J. Black. Optical flow estimation using a spatial pyramid network. CVPR, pages 4161-4170, 2017. 1, 2, 3, 6, 7
- J. Revaud, P. Weinzaepfel, Z. Harchaoui, and C. Schmid. EpicFlow: Edge-preserving interpolation of correspon- dences for optical flow. CVPR, pages 1164-1172, 2015. 1, 2, 6, 7
- D. Rosenbaum, D. Zoran, and Y. Weiss. Learning the local statistics of optical flow. NIPS, pages 2373-2381, 2013. 3
- S. Roth and M. Black. On the spatial statistics of optical flow. ICCV, pages 42-49, 2005. 3
- S. Roth and M. J. Black. Fields of experts: A framework for learning image priors. CVPR, pages 860-867, 2005. 3
- D. Sun, S. Roth, and M. J. Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. IJCV, 106(2):115-137, 2014. 1, 6, 7
- D. Sun, S. Roth, J. Lewis, and M. J. Black. Learning optical flow. ECCV, pages 83-97, 2008. 2, 3
- Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. DeepFace: Closing the gap to human-level performance in face verifica- tion. CVPR, pages 1701-1708, 2014. 2, 5
- D. Tschumperlé and R. Deriche. Vector-valued image reg- ularization with PDEs: A common framework for different applications. PAMI, 27(4):506-517, 2005. 5
- P. Weinzaepfel, J. Revaud, Z. Harchaoui, and C. Schmid. DeepFlow: Large displacement optical flow with deep matching. ICCV, pages 500-513, 2013. 2, 3, 7
- M. Werlberger, W. Trobin, T. Pock, A. Wedel, D. Cremers, and H. Bischof. Anisotropic Huber-L 1 optical flow. BMVC, 2009. 2, 5
- J. Wulff and M. J. Black. Efficient sparse-to-dense optical flow estimation using a learned basis and layers. CVPR, pages 120-130, 2015. 3, 7
- J. Xu, R. Ranftl, and V. Koltun. Accurate optical flow via direct cost volume processings. CVPR, pages 1289-1297, 2017. 6, 7
- S. Zagoruyko and N. Komodakis. Learning to compare im- age patches via convolutional neural networks. CVPR, pages 4353-4361, 2015. 3
- H. Zimmer, A. Bruhn, and J. Weickert. Optic flow in har- mony. IJCV, 93(3):368-388, 2011. 1, 2, 5
- S. Zweig and L. Wolf. InterpoNet, A brain inspired neural network for optical flow dense interpolation. CVPR, pages 4563-4572, 2017.
Cited by
Sorted by citations
Visual Odometry Revisited: What Should Be Learnt?
Zhan, H., Weerasekera, C. S., Bian, J.-W., & Reid, I. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197374 · Cited by 201 · see citing works
Enhanced Surveillance Video Compression with Dual Reference Frames Generation
Zhao, L., Wang, S., Wang, S., Ye, Y., Ma, S., & Gao, W. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3073114 · Cited by 43 · see citing works
DF-VO: What Should Be Learnt for Visual Odometry?
Zhan, H., Weerasekera, C. S., Bian, J.-W., Garg, R., & Reid, I. (2021) · arXiv · doi:10.48550/arxiv.2103.00933 · Cited by 31 · see citing works
Haq, E. U., Huarong, X., Xuhui, C., Wanqing, Z., Jianping, F., & Abid, F. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-019-08167-y · Cited by 12 · see citing works
- Authors
- Hui, T.-W., Tang, X., & Loy, C. C.
- Affiliations
- Chinese University of Hong Kong (HK)
- Venue
- 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
- Publisher
- IEEE
- Published
- 2018-06
- Pages
- 8981-8989
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2964156315
- Cited by
- 817
- References
- 4
- Citations by year
- 2026: 31 · 2025: 81 · 2024: 106 · 2023: 131 · 2022: 130 · 2021: 144 · 2020: 118 · 2019: 68 · 2018: 8
- FWCI
- 25.47
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by-nc-sa
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- Chinese University of Hong Kong
- SenseTime Group
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Surveillance and Tracking Methods
- Keywords
- optical flow, lightweight convolutional neural network, feature warping, descriptor matching, model compression
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
