The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Softmax Splatting for Video Frame Interpolation
Open accessAbstract
Differentiable image sampling in the form of backward warping has seen broad adoption in tasks like depth estimation and optical flow prediction. In contrast, how to perform forward warping has seen less attention, partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way. We propose softmax splatting to address this paradigm shift and show its effectiveness on the application of frame interpolation. Specifically, given two input frames, we forward-warp the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. We then use a synthesis network to predict the interpolation result from the warped representations. Our softmax splatting allows us to not only interpolate frames at an arbitrary time but also to fine tune the feature pyramid and the optical flow. We show that our synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Zhang, R., Isola, P., Efros, A. A., Shechtman, E., & Wang, O. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00068 · Cited by 14,623
Ucf101: A dataset of 101 human actions classes from videos in the wild
Soomro, K., Zamir, A. R., & Shah, M. (2012) · arXiv · doi:10.48550/arxiv.1212.0402 · Cited by 4,575
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Unsupervised Monocular Depth Estimation with Left-Right Consistency
Godard, C., Aodha, O. M., & Brostow, G. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.699 · Cited by 3,270 · cited 1×
“(cid:1) Prominent examples where this approach has been used include unsupervised depth estimation [13, 31, 54], unsupervised optical flow prediction [32, 47, 52], optical flow prediction [18, 42, 45], novel view synthesis [8, 27, 55], video frame interpolation [3, 22, 28, 29], and video enhancement…”
BackgroundFlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., & Brox, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.179 · Cited by 3,152
Unsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910
PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
Sun, D., Yang, X., Liu, M.-Y., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00931 · Cited by 2,648
A Database and Evaluation Methodology for Optical Flow
Baker, S., Scharstein, D., Lewis, J. P., Roth, S., Black, M. J., & Szeliski, R. (2011) · International Journal of Computer Vision · doi:10.1007/s11263-010-0390-2 · Cited by 2,205 · cited 6×
“We thus use brightness constancy as a measure of occlusion [2], which can be obtained via backward warping ←− ω as follows.”
MethodInfluentialDeconvolution and Checkerboard Artifacts
Odena, A., Dumoulin, V., & Olah, C. (2016) · Distill · doi:10.23915/distill.00003 · Cited by 1,640
Video Enhancement with Task-Oriented Flow
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-01144-2 · Cited by 1,393
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Video Enhancement with Task-Oriented FlowPreprint
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1,190
The Perception-Distortion Tradeoff
Blau, Y., & Michaeli, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00652 · Cited by 955
Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation
Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned-Miller, E., & Kautz, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00938 · Cited by 829
LiteFlowNet: A Lightweight Convolutional Neural Network for Optical Flow Estimation
Hui, T.-W., Tang, X., & Loy, C. C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00936 · Cited by 817
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints
Mahjourian, R., Wicke, M., & Angelova, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00594 · Cited by 775
Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation
Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., & Shi, W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.304 · Cited by 763
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743
Learning-based view synthesis for light field cameras
Kalantari, N. K., Wang, T.-C., & Ramamoorthi, R. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2980251 · Cited by 726
Video Frame Interpolation via Adaptive Separable Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.37 · Cited by 725
Self-citationView Synthesis by Appearance Flow
Zhou, T., Tulsiani, S., Sun, W., Malik, J., & Efros, A. A. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_18 · Cited by 579
Detail-Revealing Deep Video Super-Resolution
Tao, X., Gao, H., Liao, R., Wang, J., & Jia, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.479 · Cited by 556
Depth-Aware Video Frame Interpolation
Bao, W., Lai, W.-S., Ma, C., Zhang, X., Gao, Z., & Yang, M.-H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00382 · Cited by 554
UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
Meister, S., Hur, J., & Roth, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12276 · Cited by 520
Video Frame Interpolation via Adaptive Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.244 · Cited by 499
Self-citationContext-Aware Synthesis for Video Frame Interpolation
Niklaus, S., & Liu, F. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00183 · Cited by 411
Self-citationLearning Blind Video Temporal Consistency
Lai, W.-S., Huang, J.-B., Wang, O., Shechtman, E., Yumer, E., & Yang, M.-H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01267-0_11 · Cited by 405
Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness
Yu, J. J., Harley, A. W., & Derpanis, K. G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_1 · Cited by 368
Bao, W., Lai, W.-S., Zhang, X., Gao, Z., & Yang, M.-H. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2941941 · Cited by 362
Occlusion Aware Unsupervised Learning of Optical Flow
Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., & Xu, W. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00513 · Cited by 320
Video Compression Through Image Interpolation
Wu, C.-Y., Singhal, N., & Krähenbühl, P. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01237-3_26 · Cited by 316
Phase-based frame interpolation for video
Meyer, S., Wang, O., Zimmer, H., Grosse, M., & Sorkine-Hornung, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298747 · Cited by 294
Residual Conv-Deconv Grid Network for Semantic Segmentation
Fourure, D., Emonet, R., Fromont, E., Muselet, D., Tremeau, A., & Wolf, C. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.181 · Cited by 223
Real-Time Neural Style Transfer for Videos
Huang, H., Wang, H., Luo, W., Ma, L., Jiang, W., Zhu, X., Li, Z., & Liu, W. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.745 · Cited by 217
Semantic Video CNNs Through Representation Warping
Gadde, R., Jampani, V., & Gehler, P. V. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.477 · Cited by 198
Learning Image Matching by Simply Watching Video
Long, G., Kneip, L., Alvarez, J. M., Li, H., Zhang, X., & Yu, Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46466-4_26 · Cited by 188
PhaseNet for Video Frame Interpolation
Meyer, S., Djelouah, A., McWilliams, B., Sorkine-Hornung, A., Gross, M., & Schroers, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00059 · Cited by 184
Temporally coherent completion of dynamic video
Huang, J., Kang, S. B., Ahuja, N., et al. (2016) · ACM Transactions on Graphics · doi:10.1145/2980179.2982398 · Cited by 167
Deep Video Frame Interpolation Using Cyclic Frame Generation
Liu, Y.-L., Liao, Y.-T., Lin, Y.-Y., & Chuang, Y.-Y. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018794 · Cited by 161
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesPreprint
Nguyen-Phuoc, T., Li, C., Balaban, S., & Yang, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.06575 · Cited by 106
Unsupervised Video Interpolation Using Cycle Consistency
Reda, F. A., Sun, D., Dündar, A., et al. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00098 · Cited by 101
Learning to Synthesize Motion Blur
Brooks, T., & Barron, J. T. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00700 · Cited by 94
Slow Flow: Exploiting High-Speed Cameras for Accurate and Diverse Optical Flow Reference Data
Janai, J., Güney, F., Wulff, J., Black, M. J., & Geiger, A. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.154 · Cited by 91
Zhu, S. C., Yuille, A., & Mumford, D. B. (2000) · International Journal of Computer Vision · doi:10.1023/a:1026531017760 · Cited by 86
Geometry-Aware Deep Network for Single-Image Novel View Synthesis
Liu, M., He, X., & Salzmann, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00485 · Cited by 76
Temporally coherent local tone mapping of HDR video
Aydın, T. O., Stefanoski, N., Croci, S., Groß, M. H., & Smolić, A. (2014) · ACM Transactions on Graphics · doi:10.1145/2661229.2661268 · Cited by 63
Motion Compensated Frame Interpolation with a Symmetric Optical Flow Constraint
Rakêt, L. L., Roholm, L., Bruhn, A., & Weickert, J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33179-4_43 · Cited by 49
Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation
Wulff, J., & Black, M. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-12939-2_39 · Cited by 13
Depth-Assisted Full Resolution Network for Single Image-Based View Synthesis
Cun, X., Xu, F., Pun, C., & Gao, H. (2018) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2018.2884188 · Cited by 12
Cited by
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes
Li, Z., Niklaus, S., Snavely, N., & Wang, O. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00643 · Cited by 674
Self-citationBasicvsr++: improving video super-resolution with enhanced propagation and alignment
Chan, K. C. K., Zhou, S., Xu, X., & Loy, C. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00588 · Cited by 604
Real-Time Intermediate Flow Estimation for Video Frame Interpolation
Huang, Z., Zhang, T., Heng, W., Shi, B., & Zhou, S. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19781-9_36 · Cited by 326
Geometry-Free View Synthesis: Transformers and no 3D Priors
Rombach, R., Esser, P., & Ommer, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01409 · Cited by 75
MemFlow: Optical Flow Estimation and Prediction with Memory
Dong, Q., & Fu, Y. (2024) · arXiv · doi:10.48550/arxiv.2404.04808 · Cited by 49
Metadata
- Authors
- Niklaus, S., & Liu, F.
- Venue
- 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2020-06
- Pages
- 5436-5445
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3034921716
- Cited by
- 381
- References
- 54
- FWCI
- 18.33
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.