The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Toward Semantic Communications: Deep Learning-Based Image Semantic Coding
Abstract
Semantic communications has received growing interest since it can remarkably reduce the amount of data to be transmitted without missing critical information. Most existing works explore the semantic encoding and transmission for text and apply techniques in Natural Language Processing (NLP) to interpret the meaning of the text. In this paper, we conceive the semantic communications for image data that is much more richer in semantics and bandwidth sensitive. We propose an reinforcement learning based adaptive semantic coding (RL-ASC) approach that encodes images beyond pixel level. Firstly, we define the semantic concept of image data that includes the category, spatial arrangement, and visual feature as the representation unit, and propose a convolutional semantic encoder to extract semantic concepts. Secondly, we propose the image reconstruction criterion that evolves from the traditional pixel similarity to semantic similarity and perceptual performance. Thirdly, we design a novel RL-based semantic bit allocation model, whose reward is the increase in rate-semantic-perceptual performance after encoding a certain semantic concept with adaptive quantization level. Thus, the task-related information is preserved and reconstructed properly while less important data is discarded. Finally, we propose the Generative Adversarial Nets (GANs) based semantic decoder that fuses both locally and globally features via an attention module. Experimental results demonstrate that the proposed RL-ASC is noise robust and could reconstruct visually pleasant and semantic consistent image in low bit rate condition.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.322 · Cited by 29,137
MobileNetV2: Inverted Residuals and Linear Bottlenecks
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00474 · Cited by 28,755
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 1×
“1) Datasets: We train and evaluate the RL-ASC model based on the scene parsing and instance segmentation of the Cityscapes dataset [51].”
MethodPerceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365
Calcuation of average PSNR differences between RD-curves
Bjøntegaard, G. (2001) · Medical Entomology and Zoology · Cited by 6,090
The JPEG still picture compression standard
Wallace, G. K. (1991) · Communications of the ACM · doi:10.1145/103085.103089 · Cited by 3,784
JPEG2000: Image Compression Fundamentals, Standards and Practice
Taubman, D. S. (2002) · Journal of Electronic Imaging · doi:10.1117/1.1469618 · Cited by 3,159
Semantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974
UnitBox: An Advanced Object Detection Network
Yu, J., Jiang, Y., Wang, Z., Cao, Z., & Huang, T. (2016) · Proceedings of the 24th ACM international conference on Multimedia · doi:10.1145/2964284.2967274 · Cited by 1,731
Deep Learning Enabled Semantic Communication Systems
Xie, H., Qin, Z., Li, G. Y., & Juang, B.-H. (2021) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2021.3071210 · Cited by 1,602
Deep Joint Source-Channel Coding for Wireless Image Transmission
Bourtsoulatze, E., Burth Kurka, D., & Gunduz, D. (2019) · IEEE Transactions on Cognitive Communications and Networking · doi:10.1109/tccn.2019.2919300 · Cited by 1,476
End-to-end Optimized Image Compression
Ballé, J., Laparra, V., & Simoncelli, E. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01704 · Cited by 1,275
Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules
Cheng, Z., Sun, H., Takeuchi, M., & Katto, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00796 · Cited by 1,161
The Perception-Distortion Tradeoff
Blau, Y., & Michaeli, T. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00652 · Cited by 955
Full Resolution Image Compression with Recurrent Neural Networks
Toderici, G., Vincent, D., Johnston, N., Hwang, S. J., Minnen, D., Shor, J., & Covell, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.577 · Cited by 886
Understanding how image quality affects deep neural networks
Dodge, S., & Karam, L. (2016) · 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2016.7498955 · Cited by 835
Generative Adversarial Networks for Extreme Learned Image Compression
Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., & Van Gool, L. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00031 · Cited by 575
Wireless Image Transmission Using Deep Source Channel Coding With Attention Modules
Xu, J., Ai, B., Chen, W., Yang, A., Sun, P., & Rodrigues, M. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3082521 · Cited by 511
A Lite Distributed Semantic Communication System for Internet of Things
Xie, H., & Qin, Z. (2021) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2020.3036968 · Cited by 497
Conditional Probability Models for Deep Image Compression
Mentzer, F., Agustsson, E., Tschannen, M., Timofte, R., & Gool, L. V. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00462 · Cited by 490
Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., & Yan, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00495 · Cited by 466
Deep Learning for Joint Source-Channel Coding of Text
Farsad, N., Rao, M., & Goldsmith, A. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461983 · Cited by 438
Johnston, N., Vincent, D., Minnen, D., Covell, M., Singh, S., Chinen, T., Jin Hwang, S., Shor, J., & Toderici, G. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00461 · Cited by 366
Semantic Communications: Principles and Challenges
Qin, Z., Tao, X., Lü, J., Tong, W., & Li, G. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.01389 · Cited by 254
Self-citationContext-adaptive Entropy Model for End-to-end Optimized Image Compression
Lee, J., Cho, S.-H., & Beack, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10452 · Cited by 249
Strengths and weaknesses of deep learning models for face recognition against image degradations
Grm, K., Štruc, V., Artiges, A., Caron, M., & Ekenel, H. K. (2018) · IET Biometrics · doi:10.1049/iet-bmt.2017.0083 · Cited by 216
High-Fidelity Generative Image Compression
Mentzer, F., Toderici, G. D., Tschannen, M., & Agustsson, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09965 · Cited by 206
Towards Image Understanding from Deep Compression without Decoding
Torfason, R., Mentzer, F., Agustsson, E., Tschannen, M., Timofte, R., & Gool, L. V. (2018) · Lirias · doi:10.48550/arxiv.1803.06131 · Cited by 199
Santurkar, S., Budden, D., & Shavit, N. (2018) · 2018 Picture Coding Symposium (PCS) · doi:10.1109/pcs.2018.8456298 · Cited by 190
Bandwidth-Agile Image Transmission With Deep Joint Source-Channel Coding
Kurka, D. B., & Gunduz, D. (2021) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2021.3090048 · Cited by 188
Variable-Rate Deep Image Compression through Spatially-Adaptive Feature Transform
Song, M., Choi, J., & Han, B. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00238 · Cited by 181
Coarse-to-Fine Hyper-Prior Modeling for Learned Image Compression
Hu, Y., Yang, W., & Liu, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6736 · Cited by 132
DSSLIC: Deep Semantic Segmentation-based Layered Image Compression
Akbari, M. A., Liang, J., & Han, J. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683541 · Cited by 119
Semantic Communications for Speech Signals
Weng, Z., Qin, Z., & Li, G. Y. (2021) · International Conference on Communications · doi:10.1109/icc42927.2021.9500590 · Cited by 115
Reinforcement Learning-powered Semantic Communication via Semantic SimilarityPreprint
Lu, K., Li, R., Chen, X., Zhao, Z., & Zhang, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.12121 · Cited by 74
Semantic Structured Image Coding Framework for Multiple Intelligent Applications
Sun, S., He, T., & Chen, Z. (2020) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2020.3042517 · Cited by 58
Task-Driven Semantic Coding via Reinforcement Learning
Li, X., Shi, J., & Chen, Z. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3091909 · Cited by 56
Learning based Facial Image Compression with semantic fidelity metric
Chen, Z., & He, T. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.086 · Cited by 34
On The Classification-Distortion-Perception TradeoffPreprint
Liu, D., Zhang, H., & Xiong, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08816 · Cited by 33
Li, B., Ye, L., Liang, J., Wang, Y., & Han, J. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.05.047 · Cited by 33
Beyond Coding: Detection-driven Image Compression with Semantically Structured Bit-stream
He, T., Sun, S., Guo, Z., et al. (2019) · Picture Coding Symposium · doi:10.1109/pcs48520.2019.8954525 · Cited by 20
Cited by
Deep Learning Enabled Semantic Communications With Speech Recognition and Synthesis
Weng, Z., Qin, Z., Tao, X., Pan, C., Liu, G., & Li, G. Y. (2023) · IEEE Transactions on Wireless Communications · doi:10.1109/twc.2023.3240969 · Cited by 286
Self-citationSemantic Communication: A Survey on Research Landscape, Challenges, and Future Directions
Getu, T. M., Kaddoum, G., & Bennis, M. (2024) · Proceedings of the IEEE · doi:10.1109/jproc.2024.3520707 · Cited by 110
Metadata
- Authors
- Huang, D., Gao, F., Tao, X., Du, Q., & Lu, J.
- Venue
- IEEE Journal on Selected Areas in Communications
- Published
- 2023-01
- Volume
- 41
- Issue
- 1
- Pages
- 55-71
- ISSN
- 0733-8716 (ISSN-L)
- 1558-0008
- OpenAlex
- W4313350220
- Cited by
- 331
- References
- 46
- FWCI
- 20.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.