The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Video Enhancement with Task-Oriented Flow
Open accessAbstract
Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Horn, B. K. P., & Schunck, B. G. (1981) · Artificial Intelligence · doi:10.1016/0004-3702(81)90024-2 · Cited by 10,312
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,319
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 11×
“For example, in temporal frame interpolation, most algorithms first estimate how pixels move between input frames (frame 1 and 3), and then move pixels to the estimated location in the output frame (frame 2) (Baker et al 2011).”
InfluentialA Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
High-quality video view interpolation using a layered representation
Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., & Szeliski, R. (2004) · ACM Transactions on Graphics · doi:10.1145/1015706.1015766 · Cited by 1,385
Fast, Accurate, and Lightweight Super-Resolution with Cascading Residual Network
Ahn, N., Kang, B., & Sohn, K.-A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_16 · Cited by 1,365
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., & Shi, W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.304 · Cited by 763
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
Flow-Guided Feature Aggregation for Video Object Detection
Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.52 · Cited by 729 · cited 2×
“…(Liu et al 2017), light-field interpolation (Wang et al 2017), novel view synthesis (Zhou et al 2016), eye gaze manipulation (Ganin et al 2016), object detection (Zhu et al 2017), denoising (Wen et al 2017), and super-resolution (Caballero et al 2017; Tao et al 2017; Makansi et al 2017).”
Video Frame Interpolation via Adaptive Separable Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.37 · Cited by 725
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298720 · Cited by 697
Super-resolution: a comprehensive survey
Nasrollahi, K., & Moeslund, T. B. (2014) · Machine Vision and Applications · doi:10.1007/s00138-014-0623-4 · Cited by 684
Video Super-Resolution With Convolutional Neural Networks
Kappeler, A., Yoo, S., Dai, Q., & Katsaggelos, A. K. (2016) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2016.2532323 · Cited by 647
Jo, Y., Oh, S. W., Kang, J., & Kim, S. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00340 · Cited by 590
View Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Frame-Recurrent Video Super-Resolution
Sajjadi, M. S. M., Vemulapalli, R., & Brown, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00693 · Cited by 568
FlowNet: Learning Optical Flow with Convolutional Networks
Fischer, P., Dosovitskiy, A., Ilg, E., Haeusser, P., Hazırbaş, C., Golkov, V., van der Smagt, P., Cremers, D., & Brox, T. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.06852 · Cited by 562
Detail-Revealing Deep Video Super-Resolution
Tao, X., Gao, H., Liao, R., Wang, J., & Jia, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.479 · Cited by 556
On Bayesian Adaptive Video Super Resolution
Liu, C., & Sun, D. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.127 · Cited by 526
Video Frame Interpolation via Adaptive Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.244 · Cited by 499
Context-Aware Synthesis for Video Frame Interpolation
Niklaus, S., & Liu, F. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00183 · Cited by 411
To Learn Image Super-Resolution, Use a GAN to Learn How to Do Image Degradation First
Bulat, A., Yang, J., & Tzimiropoulos, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01231-1_12 · Cited by 407
Burst Denoising with Kernel Prediction Networks
Mildenhall, B., Barron, J. T., Chen, J., Sharlet, D., Ng, R., & Carroll, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00265 · Cited by 404
Maggioni, M., Boracchi, G., Foi, A., & Egiazarian, K. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2199324 · Cited by 388
Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness
Yu, J. J., Harley, A. W., & Derpanis, K. G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_1 · Cited by 368
Deep multi-scale video prediction beyond mean square error
Mathieu, M., Couprie, C., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05440 · Cited by 339
Dense estimation and object-based segmentation of the optical flow with robust techniques
Memin, E., & Perez, P. (1998) · IEEE Transactions on Image Processing · doi:10.1109/83.668027 · Cited by 292
Large displacement optical flow
Brox, T., Bregler, C., & Malik, J. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206697 · Cited by 277
Integrated Imaging and Vision Techniques for Industrial Inspection: Advances and Applications
(2015) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-1-4471-6741-9 · Cited by 270
A Bayesian approach to adaptive video super resolution
Liu, C., & Sun, D. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995614 · Cited by 246
Bidirectional recurrent convolutional networks for multi-frame super-resolution
Huang, Y., Wang, W., & Wang, L. (2015) · neural information processing systems · Cited by 222
Multi-frame Quality Enhancement for Compressed Video
Yang, R., Xu, M., Wang, Z., & Li, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00697 · Cited by 220
Accurate Optical Flow via Direct Cost Volume Processing
Xu, J., Ranftl, R., & Koltun, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.615 · Cited by 213
Video Rain Streak Removal by Multiscale Convolutional Sparse Coding
Li, M., Xie, Q., Zhao, Q., Wei, W., Gu, S., Tao, J., & Meng, D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00695 · Cited by 209
Structure- and motion-adaptive regularization for high accuracy optic flow
Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459375 · Cited by 177
A High-Quality Video Denoising Algorithm Based on Reliable Motion Estimation
Liu, C., & Freeman, W. T. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15558-1_51 · Cited by 173
Self-citationBurst Image Deblurring Using Permutation Invariant Convolutional Neural Networks
Aittala, M., & Durand, F. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01237-3_45 · Cited by 126
Light field video capture using a learning-based hybrid imaging system
Wang, T.-C., Zhu, J.-Y., Kalantari, N. K., Efros, A. A., & Ramamoorthi, R. (2017) · ACM Transactions on Graphics · doi:10.1145/3072959.3073614 · Cited by 120
DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation
Ganin, Y., Kononenko, D., Sungatullina, D., & Lempitsky, V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_20 · Cited by 116
Deep Kalman Filtering Network for Video Compression Artifact Reduction
Lu, G. S., Ouyang, W., Xu, D., Zhang, X., Gao, Z., & Sun, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_35 · Cited by 109
Video Denoising Based on a Spatiotemporal Gaussian Scale Mixture Model
Varghese, G., & Wang, Z. (2010) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2010.2051366 · Cited by 102
Optical Flow Guided TV-L1 Video Interpolation and Restoration
Werlberger, M., Pock, T., Unger, M., & Bischof, H. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23094-3_20 · Cited by 91
Multi-Level Video Frame Interpolation: Exploiting the Interaction Among Different Levels
Yu, Z., Li, H., Wang, Z., Hu, Z., & Chen, C. W. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2242631 · Cited by 59
End-to-End Learning of Video Super-Resolution with Motion Compensation
Makansi, O., Ilg, E., & Brox, T. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66709-6_17 · Cited by 59
Wen, B., Li, Y., Pfister, L., & Bresler, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.35 · Cited by 49
Fractal Analysis for Reduced Reference Image Quality Assessment
Xu, Y., Liu, D., Quan, Y., & Callet, P. L. (2015) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2413298 · Cited by 44
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video InterpolationPreprint
Jiang, H., Sun, D., Jampani, V., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00080 · Cited by 44
Cited by
Artificial intelligence in the creative industries: a review
Anantrasirichai, N., & Bull, D. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10039-7 · Cited by 793
Low-Light Image and Video Enhancement Using Deep Learning: A Survey
Li, C., Guo, C., Han, L., Jiang, J., Cheng, M.-M., Gu, J., & Loy, C. C. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3126387 · Cited by 657
TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution
Tian, Y., Zhang, Y., Fu, Y., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00342 · Cited by 634
Basicvsr++: improving video super-resolution with enhanced propagation and alignment
Chan, K. C. K., Zhou, S., Xu, X., & Loy, C. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00588 · Cited by 604
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
Chan, K. C. K., Wang, X., Yu, K., Dong, C., & Loy, C. C. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00491 · Cited by 604
Metadata
- Authors
- Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T.
- Venue
- International Journal of Computer Vision
- Published
- 2019-02-12
- Volume
- 127
- Issue
- 8
- Pages
- 1106-1125
- ISSN
- 0920-5691 (ISSN-L)
- 1573-1405
- OpenAlex
- W2769654144
- Cited by
- 1,393
- References
- 49
- FWCI
- 53.74
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.