The DOI matches its Crossref record; 50 of the 50 references it cites were found in a registry or an index.
MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement
Open accessAbstract
Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., super-resolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Self-citationVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
Self-citation(2010) · International Conference on Machine Learning · Cited by 13,995
Overview of the H.264/AVC video coding standard
Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.815165 · Cited by 8,184
Enhanced Deep Residual Networks for Single Image Super-Resolution
Lim, B., Son, S., Kim, H., Nah, S., & Lee, K. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.151 · Cited by 7,736
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
High Accuracy Optical Flow Estimation Based on a Theory for Warping
Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24673-2_3 · Cited by 2,570
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 4×
“(3) We demonstrate that the proposed method performs favorably against the state-of-the-art frame interpolation algorithms on several benchmark datasets, including the Middlebury [28], UCF101 [29], and Vimeo90K [9] datasets.”
MethodInfluentialA Naturalistic Open Source Movie for Optical Flow Evaluation
Butler, D. J., Wulff, J., Stanley, G. B., & Black, M. J. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33783-3_44 · Cited by 1,842
Video Enhancement with Task-Oriented Flow
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-01144-2 · Cited by 1,393
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
Brox, T., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.143 · Cited by 1,295
Video Enhancement with Task-Oriented FlowPreprint
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1,190
DeepFlow: Large Displacement Optical Flow with Deep Matching
Weinzaepfel, P., Revaud, J., Harchaoui, Z., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.175 · Cited by 997
Two deterministic half-quadratic regularization algorithms for computed imaging
Charbonnier, P., Blanc-Feraud, L., Aubert, G., & Barlaud, M. (2002) · Proceedings of 1st International Conference on Image Processing · doi:10.1109/icip.1994.413553 · Cited by 879
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned-Miller, E., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00938 · Cited by 829
Self-citationVideo Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
Video Frame Interpolation via Adaptive Separable Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.37 · Cited by 725
EpicFlow: Edge-preserving interpolation of correspondences for optical flow
Revaud, J., Weinzaepfel, P., Harchaoui, Z., & Schmid, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298720 · Cited by 697
Deep Stereo: Learning to Predict New Views from the World's Imagery
Flynn, J., Neulander, I., Philbin, J., & Snavely, N. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.595 · Cited by 574
Video Frame Interpolation via Adaptive Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.244 · Cited by 499
Motion Detail Preserving Optical Flow Estimation
Xu, L., Jia, J., & Matsushita, Y. (2012) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.236 · Cited by 493
Context-Aware Synthesis for Video Frame Interpolation
Niklaus, S., & Liu, F. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00183 · Cited by 411
Maggioni, M., Boracchi, G., Foi, A., & Egiazarian, K. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2199324 · Cited by 388
Phase-based frame interpolation for video
Meyer, S., Wang, O., Zimmer, H., Grosse, M., & Sorkine-Hornung, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298747 · Cited by 294
Bayesian estimation of motion vector fields
Konrad, J., & Dubois, E. (1992) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.161350 · Cited by 284
Video Super-Resolution via Deep Draft-Ensemble Learning
Liao, R., Tao, X., Li, R., Ma, Z., & Jia, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.68 · Cited by 258
A Bayesian approach to adaptive video super resolution
Liu, C., & Sun, D. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995614 · Cited by 246
Accurate Optical Flow via Direct Cost Volume Processing
Xu, J., Ranftl, R., & Koltun, V. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.615 · Cited by 213
Learning Image Matching by Simply Watching Video
Long, G., Kneip, L., Alvarez, J. M., Li, H., Zhang, X., & Yu, Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_26 · Cited by 188
Self-citationLarge Displacement Optical Flow from Nearest Neighbor Fields
Chen, Z., Jin, H., Lin, Z., Cohen, S. D., & Wu, Y. N. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.316 · Cited by 176
A fast hierarchical motion vector estimation algorithm using mean pyramid
Nam, K. M., Kim, J.-S., Park, R., & Shim, Y. S. (1995) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/76.465087 · Cited by 146
Enabling Adaptive High-Frame-Rate Video Streaming in Mobile Cloud Gaming Applications
Wu, J., Yuen, C., Cheung, N. M., Chen, J., & Chen, C. W. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2441412 · Cited by 107
Frame Rate Up-Conversion Using Trilateral Filtering
Wang, C., Zhang, L., He, Y., et al. (2010) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2010.2046057 · Cited by 97
Motion-Compensated Frame Rate Up-Conversion—Part II: New Algorithms for Frame Interpolation
Wang, D., Vincent, A., Blanchfield, P., et al. (2010) · IEEE Transactions on Broadcasting · doi:10.1109/tbc.2010.2043895 · Cited by 96
Semi-Supervised Learning for Optical Flow with Generative Adversarial Networks
Lai, W., Huang, J., & Yang, M. (2017) · neural information processing systems · Cited by 89
Self-citationNew Frame Rate Up-Conversion Algorithms With Low Computational Complexity
Kim, U. S., & Sunwoo, M. H. (2014) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2278142 · Cited by 78
High-Order Model and Dynamic Filtering for Frame Rate Up-Conversion
Bao, W., Zhang, X., Chen, L., et al. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2825100 · Cited by 52
Self-citationFrame Interpolation with Multi-Scale Deep Loss Functions and Generative Adversarial NetworksPreprint
van Amersfoort, J., Shi, W., Acosta, A., Massa, F., Totz, J., Wang, Z., & Caballero, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06045 · Cited by 49
Frame Rate Up Conversion Based on Variational Image Fusion
Lee, W. H., Choi, K., & Ra, J. B. (2013) · IEEE Transactions on Image Processing · doi:10.1109/tip.2013.2288139 · Cited by 37
Modeling and Optimization of High Frame Rate Video Transmission Over Wireless Networks
Wu, J., Yuen, C., Cheung, N. M., Chen, J., & Chen, C. W. (2015) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2015.2509063 · Cited by 37
Frame Rate Upconversion Using Optical Flow and Patch-Based Reconstruction
Kaviani, H. R., & Shirani, S. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2469120 · Cited by 33
On handling of occlusion for frame rate up-conversion using video in-painting
Biswas, M., & Namboodiri, V. (2010) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2010.5652571 · Cited by 4
Cited by
Depth-Aware Video Frame Interpolation
Bao, W., Lai, W.-S., Ma, C., Zhang, X., Gao, Z., & Yang, M.-H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00382 · Cited by 554
Self-citationReal-world single image super-resolution: A brief review
Chen, H., He, X., Qing, L., Wu, Y., Ren, C., Sheriff, R. E., & Zhu, C. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.09.005 · Cited by 405
Softmax Splatting for Video Frame Interpolation
Niklaus, S., & Liu, F. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00548 · Cited by 381
Channel Attention Is All You Need for Video Frame Interpolation
Choi, M., Kim, H., Han, B., Xu, N., & Lee, K. M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6693 · Cited by 328
Real-Time Intermediate Flow Estimation for Video Frame Interpolation
Huang, Z., Zhang, T., Heng, W., Shi, B., & Zhou, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19781-9_36 · Cited by 326
Metadata
- Authors
- Bao, W., Lai, W.-S., Zhang, X., Gao, Z., & Yang, M.-H.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021-03-01
- Volume
- 43
- Issue
- 3
- Pages
- 933-948
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2973673960
- Cited by
- 362
- References
- 50
- FWCI
- 22.12
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.