The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Wireless Deep Video Semantic Transmission
Open accessAbstract
In this paper, we design a new class of high-efficiency deep joint source-channel coding methods to achieve end-to-end video transmission over wireless channels. The proposed methods exploit nonlinear transform and conditional coding architecture to adaptively extract semantic features across video frames, and transmit semantic feature domain representations over wireless channels via deep joint source-channel coding. Our framework is collected under the name deep video semantic transmission (DVST). In particular, benefiting from the strong temporal prior provided by the feature domain context, the learned nonlinear transform function becomes temporally adaptive, resulting in a richer and more accurate entropy model guiding the transmission of current frame. Accordingly, a novel rate adaptive transmission mechanism is developed to customize deep joint source-channel coding for video sources. It learns to allocate the limited channel bandwidth within and among video frames to maximize the overall transmission performance. The whole DVST design is formulated as an optimization problem whose goal is to minimize the end-to-end transmission rate-distortion performance under perceptual quality metrics or machine vision task performance metrics. Across standard video source test sequences and various communication scenarios, experiments show that our DVST can generally surpass traditional wireless video coded transmission schemes. The proposed DVST framework can well support future semantic communications due to its video content-aware and machine vision task integration abilities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., & Guo, B. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00986 · Cited by 35,534
Learning Multiple Layers of Features from Tiny Images
Krizhevsky, A. (2024) · doi:10.57702/zp44cu3g · Cited by 23,395
Overview of the High Efficiency Video Coding (HEVC) Standard
Sullivan, G. J., Ohm, J.-R., Han, W.-J., & Wiegand, T. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2221191 · Cited by 8,263
Overview of the H.264/AVC video coding standard
Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2003.815165 · Cited by 8,184
Multiscale structural similarity for image quality assessment
Wang, Z., Simoncelli, E. P., & Bovik, A. C. (2004) · The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 · doi:10.1109/acssc.2003.1292216 · Cited by 6,253 · cited 1×
“For the reconstruction task, we optimize DVST in terms of the mean squared error (MSE) for the peak-signal-to-noise ratio (PSNR) metric, or multiscale structural similarity [39] (MS-SSIM) for perceptual quality.”
MethodCalcuation of average PSNR differences between RD-curves
Bjøntegaard, G. (2001) · Medical Entomology and Zoology · Cited by 6,090
Deep Learning Enabled Semantic Communication Systems
Xie, H., Qin, Z., Li, G. Y., & Juang, B.-H. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3071210 · Cited by 1,602
Deep Joint Source-Channel Coding for Wireless Image Transmission
Bourtsoulatze, E., Burth Kurka, D., & Gunduz, D. (2019) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2019.2919300 · Cited by 1,476
Semantic object classes in video: A high-definition ground truth database
Brostow, G. J., Fauqueur, J., & Cipolla, R. (2009) · Pattern Recognition Letters · doi:10.1016/j.patrec.2008.04.005 · Cited by 1,397
Video Enhancement with Task-Oriented Flow
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-01144-2 · Cited by 1,393
Optical Flow Estimation Using a Spatial Pyramid Network
Ranjan, A., & Black, M. J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.291 · Cited by 1,344
Variational image compression with a scale hyperprior
Ballé, J., Minnen, D. C., Singh, S., Hwang, S. J., & Johnston, N. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01436 · Cited by 1,318
End-to-end Optimized Image Compression
Ballé, J., Laparra, V., & Simoncelli, E. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01704 · Cited by 1,275
Video Enhancement with Task-Oriented FlowPreprint
Xue, T., Chen, B., Wu, J., Wei, D., & Freeman, W. T. (2019) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 1,190
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050 · cited 1×
“Herein, we take semantic segmentation as an example visual analytic task, employ HyperSeg [47] as a powerful analytics model to generate segmentation prediction, and evaluate the capacity of our DVST on the popular CamVid benchmark [48], [49].”
MethodDVC: An End-To-End Deep Video Compression Framework
Lu, G., Ouyang, W., Xu, D., Zhang, X., Cai, C., & Gao, Z. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01126 · Cited by 739
Training Deep Neural Networks on Noisy Labels with Bootstrapping
Reed, S., Lee, H., Anguelov, D., Szegedy, C., Erhan, D., & Rabinovich, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.6596 · Cited by 667
DeepJSCC-f: Deep Joint Source-Channel Coding of Images With Feedback
Kurka, D. B., & Gunduz, D. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2987203 · Cited by 470
Rissanen, J., & Langdon, G. (1981) · IEEE Transactions on Information Theory · doi:10.1109/tit.1981.1056282 · Cited by 454
Deep Learning for Joint Source-Channel Coding of Text
Farsad, N., Rao, M., & Goldsmith, A. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461983 · Cited by 438
Design of Low-Density Parity Check Codes for 5G New Radio
Richardson, T., & Kudekar, S. (2018) · IEEE Communications Magazine · doi:10.1109/mcom.2018.1700839 · Cited by 433
Nonlinear Transform Source-Channel Coding for Semantic Communications
Dai, J., Wang, S., Tan, K., Si, Z., Qin, X., Niu, K., & Zhang, P. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3180802 · Cited by 364
Self-citationZhang, P., Xu, W., Gao, H., Niu, K., Xu, X., Qin, X., Yuan, C., Qin, Z., Zhao, H., Wei, J., et al. (2022) · Engineering · doi:10.1016/j.eng.2021.11.003 · Cited by 311
Self-citationDensity Modeling of Images using a Generalized Normalization Transformation
Ballé, J., Laparra, V., & Simoncelli, E. P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.06281 · Cited by 308
Video Coding for Machines: A Paradigm of Collaborative Compression and Intelligent Analytics
Duan, L., Liu, J., Yang, W., Huang, T., & Gao, W. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3016485 · Cited by 260
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation
Nirkin, Y., Wolf, L., & Hassner, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00405 · Cited by 254
Semantic Communications: Principles and Challenges
Qin, Z., Tao, X., Lü, J., Tong, W., & Li, G. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.01389 · Cited by 254
Ballé, J., Chou, P. A., Minnen, D., Singh, S., Johnston, N., Agustsson, E., Hwang, S. J., & Toderici, G. (2021) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.3034501 · Cited by 231
An End-to-End Learning Framework for Video Compression
Lu, G., Zhang, X., Ouyang, W., Chen, L., Gao, Z., & Xu, D. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2988453 · Cited by 224
DeepWiVe: Deep-Learning-Aided Wireless Video Transmission
Tung, T.-Y., & Gunduz, D. (2022) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2022.3191354 · Cited by 211
High-Fidelity Generative Image Compression
Mentzer, F., Toderici, G. D., Tschannen, M., & Agustsson, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09965 · Cited by 206
Bandwidth-Agile Image Transmission With Deep Joint Source-Channel Coding
Kurka, D. B., & Gunduz, D. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3090048 · Cited by 188
A Paradigm Shift toward Semantic Communications
Niu, K., Dai, J., Yao, S., Wang, S., Si, Z., Qin, X., & Zhang, P. (2022) · IEEE Communications Magazine · doi:10.1109/mcom.001.2200099 · Cited by 138
Self-citationJoint source-channel turbo decoding of entropy-coded sources
Guyader, A., Fabre, É., Guillemot, C. M., & Robert, M. (2001) · IEEE Journal on Selected Areas in Communications · doi:10.1109/49.947033 · Cited by 116
Communication Beyond Transmitting Bits: Semantics-Guided Source and Channel Coding
Dai, J., Zhang, P., Niu, K., Wang, S., Si, Z., & Qin, X. (2022) · IEEE Wireless Communications · doi:10.1109/mwc.017.2100705 · Cited by 115
Self-citationEfficient Nonlinear Transforms for Lossy Image Compression
Ballé, J. (2018) · Picture Coding Symposium · doi:10.1109/pcs.2018.8456272 · Cited by 78
Semantics-Native Communication with Contextual ReasoningPreprint
Seo, H., Park, J., Bennis, M., & Debbah, M. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05681 · Cited by 38
Wireless Image Retrieval at the Edge
Jankowski, M., Gunduz, D., & Mikolajczyk, K. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036955 · Cited by 3 · cited 3×
“As one modern version, recent deep learning methods for realizing JSCC have stimulated significant interest in both artificial intelligence (AI) and wireless communication communities [15]–[19].”
BackgroundMethod
Cited by
Semantic Communication: A Survey on Research Landscape, Challenges, and Future Directions
Getu, T. M., Kaddoum, G., & Bennis, M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3520707 · Cited by 110
Metadata
- Authors
- Wang, S., Dai, J., Liang, Z., Niu, K., Si, Z., Dong, C., Qin, X., & Zhang, P.
- Venue
- IEEE Journal on Selected Areas in Communications
- Published
- 2023-01
- Volume
- 41
- Issue
- 1
- Pages
- 214-229
- ISSN
- 0733-8716 (ISSN-L)
- 1558-0008
- OpenAlex
- W4313350178
- Cited by
- 263
- References
- 40
- FWCI
- 22.94
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.