Enhanced Bi-Prediction With Convolutional Neural Network for High-Efficiency Video Coding
The DOI matches its Crossref record; 47 of the 48 references it cites were found in a registry or an index.
Abstract
In this paper, we propose an enhanced bi-prediction scheme based on the convolutional neural network (CNN) to improve the rate-distortion performance in video compression. In contrast to the traditional bi-prediction strategy which computes the linear superposition as the predictive signals with pixel-to-pixel correspondence, the proposed scheme employs CNN to directly infer the predictive signals in a data-driven manner. As such, the predicted blocks are fused in a nonlinear fashion to improve the coding performance. Moreover, the patch-to-patch inference strategy with CNN also improves the prediction accuracy since the patch-level information for the prediction of each individual pixel can be exploited. The proposed enhanced bi-prediction scheme is further incorporated into the high-efficiency video coding standard, and the experimental results exhibit a significant performance improvement under different coding configurations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 2×
“Moreover, the skip connection [19] is introduced into our network to avoid the long-term memory [20] of the input features.”
BackgroundSelf-citationAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“The Adam algorithm [44] is used to update the coefficients W (i) and b(i) in the network with the minimization of the loss function.”
MethodDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“Unit (ReLU) [13] and Parametric ReLU (PReLU) [14], which was also analyzed in [15] and [16].”
BackgroundSelf-citation(2010) · Cited by 14,134
(2010) · International Conference on Machine Learning · Cited by 13,995
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Overview of the High Efficiency Video Coding (HEVC) Standard
Sullivan, G. J., Ohm, J.-R., Han, W.-J., & Wiegand, T. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2221191 · Cited by 8,263 · cited 9×
“264/AVC [1] and HEVC [2], performs motion-compensation from two reference frame lists.”
MethodInfluentialOverview of the H.264/AVC video coding standard
Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.815165 · Cited by 8,184 · cited 1×
“B I-PREDICTION, which serves as an indispensable module in the state-of-the-art video coding standards such as H.264/AVC [1] and HEVC [2], performs motion-compensation from two reference frame lists.”
BackgroundAccurate Image Super-Resolution Using Very Deep Convolutional Networks
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.182 · Cited by 7,870 · cited 4×
“[20], [27] deepened the network architecture to enhance the non-linear representation ability, which achieves better performance in SR problem.”
BackgroundInfluentialLearning a Deep Convolutional Network for Image Super-Resolution
Dong, C., Loy, C. C., He, K., & Tang, X. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10593-2_13 · Cited by 5,494 · cited 1×
“[26] first introduced the deep learning into super-resolution problem.”
BackgroundFlownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313 · cited 2×
“[17] proposed the FlowNet aiming at estimating the optical flow of video sequences.”
BackgroundMethodDeeply-Recursive Convolutional Network for Image Super-Resolution
Kim, J., Lee, J. K., & Lee, K. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.181 · Cited by 3,025 · cited 1×
“[20], [27] deepened the network architecture to enhance the non-linear representation ability, which achieves better performance in SR problem.”
BackgroundImage denoising: Can plain neural networks compete with BM3D?
Burger, H. C., Schuler, C. J., & Harmeling, S. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247952 · Cited by 1,325 · cited 1×
“[23] proposed a multi-layer perceptron where all layers are fully connected for natural image denoising.”
BackgroundCompression Artifacts Reduction by a Deep Convolutional Network
Dong, C., Deng, Y., Loy, C. C., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.73 · Cited by 883 · cited 1×
“Moreover, other image restoration problems such as compression artifact removal and de-blurring have also been studied in [30] and [31], respectively, and significant quality”
Background(2008) · Cited by 838
Video Frame Synthesis Using Deep Voxel Flow
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., & Agarwala, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.478 · Cited by 743 · cited 1×
“[33] proposed an end-to-end video interpolation method which converts the optical flow into the”
MethodVideo Frame Interpolation via Adaptive Separable Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.37 · Cited by 725
Video Frame Interpolation via Adaptive Convolution
Niklaus, S., Mai, L., & Liu, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.244 · Cited by 499
Restoring an Image Taken through a Window Covered with Dirt or Rain
Eigen, D., Krishnan, D., & Fergus, R. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.84 · Cited by 454 · cited 1×
“[25] restore the images which suffer from some specific noisy patterns, and the image quality has been significantly improved.”
BackgroundUnderstanding convolutional neural networks with a mathematical model
Kuo, C.-C. J. (2016) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2016.11.003 · Cited by 414 · cited 2×
“Firstly, the CNN based fusion introduces the non-linear synthesis philosophy, which can well rectify correlations according the analysis in [15]”
BackgroundMethodA Convolutional Neural Network Approach for Post-Processing in HEVC Intra Coding
Dai, Y., Liu, D., & Wu, F. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-51811-4_3 · Cited by 332 · cited 1×
“Regarding the post-processing at the decoder side, the excellent signal reconstruction ability is taken advantage of in [35] and [36] to improve the quality of the decoded frames.”
BackgroundImage Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections
Mao, X.-J., Shen, C., & Yang, Y.-B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.08921 · Cited by 323 · cited 1×
“[28] applied the auto-encoder structure to establish the mapping from the degraded image and the original image.”
MethodA Machine Learning Approach for Non-blind Image Deconvolution
Schuler, C. J., Burger, H. C., Harmeling, S., & Scholkopf, B. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.142 · Cited by 319 · cited 1×
“Moreover, other image restoration problems such as compression artifact removal and de-blurring have also been studied in [30] and [31], respectively, and significant quality”
BackgroundThe Efficiency of Motion-Compensating Prediction for Hybrid Coding of Video Sequences
Girod, B. (1987) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.1987.1146632 · Cited by 228 · cited 1×
“[3] proposed the concept of B pictures in 1990, and subsequently theoretical analyses regarding the application of bi-prediction in video coding are provided in [4]–[6].”
BackgroundEfficiency analysis of multihypothesis motion-compensated prediction for video coding
Girod, B. (2000) · IEEE Transactions on Image Processing · doi:10.1109/83.821595 · Cited by 209
The SJTU 4K video sequence dataset
Song, L., Tang, X., Zhang, W., Yang, X., & Xia, P. (2013) · 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2013.6603201 · Cited by 193 · cited 1×
“The sequences for training include CampfireParty , Fountains , Marathon , Runners , RushHour , TallBuildings , TrafficFlow and Wood [48].”
MethodGeneralized B pictures and the draft H.264/AVC video-compression standard
Flierl, M., & Girod, B. (2003) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.814963 · Cited by 171 · cited 1×
“[3] proposed the concept of B pictures in 1990, and subsequently theoretical analyses regarding the application of bi-prediction in video coding are provided in [4]–[6].”
BackgroundCNN-based in-loop filtering for coding efficiency improvement
Park, W.-S., & Kim, M. (2016) · 2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP) · doi:10.1109/ivmspw.2016.7528223 · Cited by 159 · cited 1×
“Regarding the post-processing at the decoder side, the excellent signal reconstruction ability is taken advantage of in [35] and [36] to improve the quality of the decoded frames.”
BackgroundImage Restoration: From Sparse and Low-Rank Priors to Deep Priors [Lecture Notes]
Zhang, L., & Zuo, W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2717489 · Cited by 115
An Efficient Four-Parameter Affine Motion Model for Video Coding
Li, L., Li, H., Liu, D., Zhu Li, B., Yang, H., Lin, S., Chen, H., & Wu, F. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2699919 · Cited by 114
Convolutional Neural Network-Based Fractional-Pixel Motion Compensation
Yan, N., Liu, D., Li, H., Li, B., Li, L., & Wu, F. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2018.2816932 · Cited by 87 · cited 1×
“[41], [42] enhanced the quality of interpolated pixels with CNN based half-pixel interpolation.”
Affine multipicture motion-compensated prediction
Wiegand, T., Steinbach, E. G., & Girod, B. (2005) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.841690 · Cited by 76 · cited 1×
“Such strong assumption may not coincide with the actual motion of objects, which also motivates the affine based motion compensation schemes in [10]–[12].”
BackgroundSpatial-temporal residue network based in-loop filter for video coding
Jia, C., Wang, S., Zhang, X., Wang, S., & Ma, S. (2017) · IEEE International Conference on Visual Communications and Image Processing (VCIP) · doi:10.1109/vcip.2017.8305149 · Cited by 76 · cited 1×
“[37] further utilized the temporal and spatial correlations to enhance the reconstruction quality.”
MethodSelf-citationThe CNN as a Guided Multilayer RECOS Transform [Lecture Notes]
Kuo, C. ‐. J. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2671158 · Cited by 72 · cited 2×
“One distinct property of deep learning is that the non-linear activation is enabled with various activation functions such as the Rectified Linear Unit (ReLU) [13] and Parametric ReLU (PReLU) [14], which was also analyzed in [15], [16].”
MethodConvolutional Neural Networks Based Intra Prediction for HEVC
Cui, W., Zhang, T., Zhang, S., Jiang, F., Zuo, W., Wan, Z., & Zhao, D. (2017) · DCC · doi:10.1109/dcc.2017.53 · Cited by 72 · cited 1×
“[39] introduced the CNN into intra prediction to improve prediction accuracy.”
MethodGlobal brightness-variation compensation for video coding
Kamikura, K., Watanabe, H., Jozawa, H., Kotera, H., & Ichinose, S. (1998) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/76.736731 · Cited by 61 · cited 1×
“[7] proposed a global brightness-variation compensation scheme that compensates for global brightness variation caused by fade in/out, camera-iris adjustment, etc.”
BackgroundControl-Point Representation and Differential Coding Affine-Motion Compensation
Huang, H., Woods, J. W., Zhao, Y., & Bai, H. (2013) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2013.2254977 · Cited by 59
Video Frame Interpolation via Adaptive ConvolutionPreprint
Niklaus, S., Mai, L., & Liu, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07514 · Cited by 59 · cited 2×
“[18] presented a video frame interpolation method using the convolutional neural network (CNN) which estimates the motion vector of each pixel according to the patch surrounding this pixel.”
BackgroundMethodA convolutional neural network approach for half-pel interpolation in video coding
Yan, N., Liu, D., Li, H., & Wu, F. (2017) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2017.8050458 · Cited by 53 · cited 1×
“[41], [42] enhanced the quality of interpolated pixels with CNN based half-pixel interpolation.”
Localized Weighted Prediction for Video Coding
Yin, P., Tourapis, A. M., & Boyce, J. M. (2005) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2005.1465598 · Cited by 31
CNN-Based Bi-Directional Motion Compensation for High Efficiency Video Coding
Zhao, Z., Wang, S., Wang, S., Zhang, X., Ma, S., & Yang, J. (2018) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas.2018.8351189 · Cited by 31 · cited 2×
“In [43], CNN is applied to improve the bi-prediction in the RA configuration.”
MethodSelf-citationVideo coding with motion-compensated interpolation for CD-ROM applications
Puri, A., Aravind, R., Haskell, B. G., & Leonardi, R. (1990) · Signal Processing Image Communication · doi:10.1016/0923-5965(90)90016-b · Cited by 28 · cited 1×
“[3] proposed the concept of B pictures in 1990, and subsequently theoretical analyses regarding the application of bi-prediction in video coding are provided in [4]–[6].”
BackgroundCNN-based transform index prediction in multiple transforms framework to assist entropy coding
Puri, S., Lasserre, S., & Callet, P. L. (2017) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco.2017.8081317 · Cited by 27 · cited 1×
“[38] proposed a CNN network to predict the index of the chosen transform kernel per residual block, such that the signaling overhead can be saved.”
BackgroundA neural network approach to transform image coding
Chua, L. O., & Lin, T. (1988) · International Journal of Circuit Theory and Applications · doi:10.1002/cta.4490160304 · Cited by 24 · cited 3×
“The first work that incorporates the neural network into image coding dated back to 1988 when Chua and Lin [32] proposed a learning-based transformation to improve the coding performance.”
BackgroundGeneralized bi-prediction method for future video coding
Chen, C.-C., Xiu, X., He, Y., & Ye, Y. (2016) · Picture Coding Symposium · doi:10.1109/pcs.2016.7906342 · Cited by 15 · cited 1×
“[9] extended the motion of weighted bi-prediction to prediction unit level to further improve the coding performance.”
BackgroundDeep neural network based single pixel prediction for unified video coding
Li, H., & Trocan, M. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.07.037 · Cited by 13 · cited 1×
“Li and Trocan [40] showed that deep learning had the ability to generate better prediction in intra coding.”
BackgroundContext-adaptive pixel based prediction for intra frame encoding
Zhang, Y., Zhang, L., Ma, S., Zhao, D., & Gao, W. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5495264 · Cited by 7 · cited 1×
“As such, it is closely relevant with the prediction techniques of video coding, and many successful coding techniques benefit from image restoration [21].”
BackgroundSelf-citation
Research works printed but not linked: 2
- 2018, Trained Models in the Caffe Platform
- bossen, 2011, Common Test Conditions and Software Reference Configurations
Cited by
Artificial intelligence in the creative industries: a review
Anantrasirichai, N., & Bull, D. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10039-7 · Cited by 793
“They have been applied to optimise a range of coding tools including intra prediction [219, 222], motion estimation [297], transforms [221], quantization [298], entropy coding [220] and loop filtering [299].”
MethodImage and Video Compression With Neural Networks: A Review
Ma, S., Zhang, X., Jia, C., Zhao, Z., Wang, S., & Wang, S. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2910119 · Cited by 459
“In addition, considering the limitation of the traditional bidirectional prediction using simple average of two prediction hypothesises, we further improve its efficiency by utilizing a six-layer CNN with 13×13 receptive field size to infer the inter prediction block in a nonlinear fashion [91], [92], which achieves 2.”
MethodSelf-citationBVI-DVC: A Training Database for Deep Video Compression
Ma, D., Zhang, F., & Bull, D. R. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3108943 · Cited by 163
“Image and video compression based on deep neural networks has become a popular research topic offering evident enhancements over conventional coding tools for: intra prediction [22, 23], motion estimation [24, 25], transforms [26, 27], quantisation [28], entropy coding [29, 30], post-processing [31, 32] and loop filte…”
BackgroundDeep Learning-Based Video Coding: A Review and a Case Study
Liu, D., Li, Y., Lin, J., Li, H., & Wu, F. (2021) · ACM Computing Surveys · doi:10.1145/3368405 · Cited by 192
“[148] propose a method to enhance the prediction quality.”
MethodMachine learning based video coding optimizations: A survey
Zhang, Y., Kwong, S., & Wang, S. (2019) · Information Sciences · doi:10.1016/j.ins.2019.07.096 · Cited by 88
Self-citation
Metadata
- Authors
- Zhao, Z., Wang, S., Wang, S., Zhang, X., Ma, S., & Yang, J.
- Affiliations
- Peking University (CN)
- City University of Hong Kong (HK)
- University of Southern California (US)
- Venue
- IEEE Transactions on Circuits and Systems for Video Technology
- Published
- 2019-11
- Volume
- 29
- Issue
- 11
- Pages
- 3291-3301
- ISSN
- 1051-8215 (ISSN-L)
- 1558-2205
- OpenAlex
- W2896406755
- Cited by
- 75
- References
- 48
- Citations by year
- 2025: 9 · 2024: 9 · 2023: 10 · 2022: 7 · 2021: 17 · 2020: 14 · 2019: 9
- FWCI
- 7.16
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Circuits and Systems for Video Technology (publishedVersion)
- Funders
- National Natural Science Foundation of China (61632001, 61421062, 61520106004, 61571017)
- National Basic Research Program of China (2015CB351800)
- Hong Kong RGC Early Career Scheme (9048122 (CityU 21211018))
- City University of Hong Kong (7200539/CS)
- Peking University
- Microsoft
- Topics
- Physical Sciences › Computer Science › Signal Processing › Video Coding and Compression Technologies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Data Compression Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- HEVC, convolutional neural networks, video compression, rate-distortion performance, motion prediction, data-driven prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.