Deep Video Frame Interpolation Using Cyclic Frame Generation
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Video frame interpolation algorithms predict intermediate frames to produce videos with higher frame rates and smooth view transitions given two consecutive frames as inputs. We propose that: synthesized frames are more reliable if they can be used to reconstruct the input frames with high quality. Based on this idea, we introduce a new loss term, the cycle consistency loss. The cycle consistency loss can better utilize the training data to not only enhance the interpolation results, but also maintain the performance better with less training data. It can be integrated into any frame interpolation network and trained in an end-to-end manner. In addition to the cycle consistency loss, we propose two extensions: motion linearity loss and edge-guided training. The motion linearity loss approximates the motion between two input frames to be linear and regularizes the training. By applying edge-guided training, we further improve results by integrating edge information into training. Both qualitative and quantitative experiments demonstrate that our model outperforms the state-of-the-art methods. The source codes of the proposed method and more experimental results will be available at https://github.com/alex04072000/CyclicGen.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 1×
“Convolutional neural networks (CNNs) (Krizhevsky, Sutskever, and Hinton 2012) have been employed in many modern methods for video frame interpolation (Niklaus, Mai, and Liu 2017a; 2017b; Liu et al. 2017; Long et al. 2016).”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The optimization is performed by Adam optimizer (Kingma and Ba 2015).”
MethodA Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004 · cited 2×
“To compute the edge map of an image, we tried several algorithms, including (Canny 1986; Kanopoulos, Vasanthavada, and Baker 1988; Marr and Hildreth 1980; Xie and Tu 2015).”
MethodUnpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602 · cited 2×
“…from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard, Mac Aodha, and Brostow 2017), and image-to-image translation (Zhu et al. 2017).”
BackgroundLabaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
Back-translation for cross-cultural research.
Brislin, R. W. (1970) · Journal of Cross-Cultural Psychology · doi:10.1177/135910457000100301 · Cited by 14,091 · cited 1×
“For language translation, it has been shown effective to improve translations by using back translation and reconciliation (Brislin 1970).”
MethodMarr, D., & Hildreth, E. (1980) · Proceedings of the Royal Society of London. Series B. Biological Sciences · doi:10.1098/rspb.1980.0020 · Cited by 6,510 · cited 1×
“To compute the edge map of an image, we tried several algorithms, including (Canny 1986; Kanopoulos, Vas-anthavada, and Baker 1988; Marr and Hildreth 1980; Xie and Tu 2015).”
MethodUcf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575 · cited 7×
“UCF101 For UCF101, we compute both PSNR and SSIM using the motion masks provided by (Liu et al. 2017).”
MethodInfluentialFlownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · cited 1×
“CNNs have been shown effective in optical flow estimation such as (Bailer, Taetz, and Stricker 2015; Dosovitskiy et al. 2015; Gadot and Wolf 2016; G¨uney and Geiger 2016; Teney and Hebert 2016; Tran et al. 2016; Weinzaepfel et al. 2013).”
MethodUnsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · cited 2×
“…from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard, Mac Aodha, and Brostow 2017), and image-to-image translation (Zhu et al. 2017).”
BackgroundHolistically-Nested Edge Detection
Xie, S., & Tu, Z. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.164 · Cited by 3,096
A Computational Approach to Edge Detection
CANNY, J. (1987) · Readings in Computer Vision · doi:10.1016/b978-0-08-051581-6.50024-6 · Cited by 2,275
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 6×
“Conventional methods (Baker et al. 2011; Werlberger et al. 2011; Yu et al. 2013) for frame interpolation usually estimate dense motion correspondences between consecutive frames via stereo matching or optical flow prediction, and can synthesize intermediate frames based on the estimated…”
MethodInfluentialDesign of an image edge detection filter using the Sobel operator
Kanopoulos, N., Vasanthavada, N., & Baker, R. L. (1988) · IEEE Journal of Solid-State Circuits · doi:10.1109/4.996 · Cited by 1,212
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned-Miller, E., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00938 · Cited by 1,053
DeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.175 · Cited by 997
Generating Videos with Scene Dynamics
Vondrick, C., Pirsiavash, H., & Torralba, A. B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02612 · Cited by 785
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
Self-citationVideo Frame Interpolation via Adaptive Separable Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.37 · Cited by 725
A Database and Evaluation Methodology for Optical Flow
Baker, S., Roth, S., Scharstein, D., Black, M. J., Lewis, J. P., & Szeliski, R. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408903 · Cited by 618
Video Frame Interpolation via Adaptive Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.244 · Cited by 499
Motion Detail Preserving Optical Flow Estimation
Xu, L., Jia, J., & Matsushita, Y. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.236 · Cited by 493
Dense Point Trajectories by GPU-Accelerated Large Displacement Optical Flow
Sundaram, N., Brox, T., & Keutzer, K. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_32 · Cited by 422
Context-Aware Synthesis for Video Frame Interpolation
Niklaus, S., & Liu, F. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00183 · Cited by 411
Learning Dense Correspondence via 3D-Guided Cycle Consistency
Zhou, T., Krahenbuhl, P., Aubry, M., Huang, Q., & Efros, A. A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.20 · Cited by 350 · cited 2×
“…from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard, Mac Aodha, and Brostow 2017), and image-to-image translation (Zhu et al. 2017).”
BackgroundDual Motion GAN for Future-Flow Embedded Video Prediction
Liang, X., Lee, L., Dai, W., & Xing, E. P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.194 · Cited by 328
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
Xue, T., Wu, J., Bouman, K. L., & Freeman, W. T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.02586 · Cited by 320 · cited 1×
“Some frame synthesis methods leverage CNNs to directly generate images (Goodfellow et al. 2014) and videos (Von-drick, Pirsiavash, and Torralba 2016; Xue et al. 2016).”
MethodBailer, C., Taetz, B., & Stricker, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.457 · Cited by 233
Disambiguating visual relations using loop constraints
Zach, C., Klopschitz, M., & Pollefeys, M. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539801 · Cited by 229 · cited 1×
“Higher-order cycle consistency has been exploited in diverse vision tasks, such as structure from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth…”
MethodLearning Image Matching by Simply Watching Video
Long, G., Kneip, L., Alvarez, J. M., Li, H., Zhang, X., & Yu, Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_26 · Cited by 188 · cited 1×
“Convolutional neural networks (CNNs) (Krizhevsky, Sutskever, and Hinton 2012) have been employed in many modern methods for video frame interpolation (Niklaus, Mai, and Liu 2017a; 2017b; Liu et al. 2017; Long et al. 2016).”
MethodConsistent Shape Maps via Semidefinite Programming
Huang, Q., & Guibas, L. (2013) · Computer Graphics Forum · doi:10.1111/cgf.12184 · Cited by 170 · cited 1×
“…consistency has been exploited in diverse vision tasks, such as structure from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard,…”
BackgroundConsistent shape maps via semidefinite programming
Qi-xing Huang, A., & Guibas, L. J. (2013) · doi:10.5555/2600289.2600314 · Cited by 149
FlowWeb: Joint image set alignment by weaving consistent, pixel-wise correspondences
Zhou, T., Lee, Y. J., Yu, S. X., & Efros, A. A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298723 · Cited by 149 · cited 1×
“…from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard, Mac Aodha, and Brostow 2017), and image-to-image translation (Zhu et al. 2017).”
BackgroundImage Co-segmentation via Consistent Functional Maps
Wang, F., Huang, Q., & Guibas, L. J. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.110 · Cited by 127 · cited 1×
“…vision tasks, such as structure from motion (Zach, Klopschitz, and Pollefeys 2010), 3D shape matching (Huang and Guibas 2013), co-segmentation (Wang, Huang, and Guibas 2013), dense semantic alignment (Zhou et al. 2015; 2016), depth estimation (Godard, Mac Aodha, and Brostow 2017), and…”
Deep End2End Voxel2Voxel Prediction
Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.57 · Cited by 107
PatchBatch: A Batch Augmented Loss for Optical Flow
Gadot, D., & Wolf, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.459 · Cited by 102 · cited 1×
“CNNs have been shown effective in optical flow estimation such as (Bailer, Taetz, and Stricker 2015; Dosovitskiy et al. 2015; Gadot and Wolf 2016; G¨uney and Geiger 2016; Teney and Hebert 2016; Tran et al. 2016; Weinzaepfel et al. 2013).”
MethodOptical Flow Guided TV-L1 Video Interpolation and Restoration
Werlberger, M., Pock, T., Unger, M., & Bischof, H. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23094-3_20 · Cited by 92 · cited 1×
“Conventional methods (Baker et al. 2011; Werlberger et al. 2011; Yu et al. 2013) for frame interpolation usually estimate dense motion correspondences between consecutive frames via stereo matching or optical flow prediction, and can synthesize intermediate frames based on the estimated…”
MethodMulti-Level Video Frame Interpolation: Exploiting the Interaction Among Different Levels
Yu, Z., Li, H., Wang, Z., Hu, Z., & Chen, C. W. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2242631 · Cited by 59 · cited 1×
“Conventional methods (Baker et al. 2011; Werlberger et al. 2011; Yu et al. 2013) for frame interpolation usually estimate dense motion correspondences between consecutive frames via stereo matching or optical flow prediction, and can synthesize intermediate frames based on the estimated…”
MethodLearning to Extract Motion from Videos in Convolutional Neural Networks
Teney, D., & Hebert, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-54193-8_26 · Cited by 57 · cited 1×
“CNNs have been shown effective in optical flow estimation such as (Bailer, Taetz, and Stricker 2015; Dosovitskiy et al. 2015; Gadot and Wolf 2016; G¨uney and Geiger 2016; Teney and Hebert 2016; Tran et al. 2016; Weinzaepfel et al. 2013).”
MethodSuper SloMo: High Quality Estimation of Multiple Intermediate Frames for Video InterpolationPreprint
Jiang, H., Sun, D., Jampani, V., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00080 · Cited by 44 · cited 3×
“The compared methods include (1) synthesis-based methods which directly generate interpolated frames: CtxSyn (Niklaus and Liu 2018), SepConv (Niklaus, Mai, and Liu 2017b), and SuperSlomo (Jiang et al. 2018); and flow-based methods which utilize optical flows for frame interpolation: MDP-Flow2 (Xu, Jia, and Matsushita…”
MethodLearning to Extract Motion from Videos in Convolutional Neural NetworksPreprint
Teney, D., & Hebert, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07532 · Cited by 34 · cited 1×
“CNNs have been shown effective in optical flow estimation such as (Bailer, Taetz, and Stricker 2015; Dosovitskiy et al. 2015; Gadot and Wolf 2016; G¨uney and Geiger 2016; Teney and Hebert 2016; Tran et al. 2016; Weinzaepfel et al. 2013).”
Method
Cited by
Softmax Splatting for Video Frame Interpolation
Niklaus, S., & Liu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00548 · Cited by 381
Channel Attention Is All You Need for Video Frame Interpolation
Choi, M., Kim, H., Han, B., Xu, N., & Lee, K. M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6693 · Cited by 328
Real-Time Intermediate Flow Estimation for Video Frame Interpolation
Huang, Z., Zhang, T., Heng, W., Shi, B., & Zhou, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19781-9_36 · Cited by 326
AdaCoF: Adaptive Collaboration of Flows for Video Frame Interpolation
Lee, H., Kim, T., Chung, T.-Y., Pak, D., Ban, Y., & Lee, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00536 · Cited by 270
Metadata
- Authors
- Liu, Y.-L., Liao, Y.-T., Lin, Y.-Y., & Chuang, Y.-Y.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2019-07-17
- Volume
- 33
- Issue
- 01
- Pages
- 8794-8802
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2904829482
- Cited by
- 161
- References
- 42
- FWCI
- 9.45
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.