CoChat
Sign Up
PreprintCited by 198Advanced Data Compression TechniquesImage and Video Quality AssessmentVideo Coding and Compression Technologies

Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff

Open access
Blau, Yochai, Michaeli, Tomer·2019·arXiv
DOI 10.48550/arxiv.1901.07821arXiv 1901.07821
Open this paper in CoChat — free accountOpen PDF

Abstract

Lossy compression algorithms are typically designed and analyzed through the lens of Shannon's rate-distortion theory, where the goal is to achieve the lowest possible distortion (e.g., low MSE or high SSIM) at any given bit rate. However, in recent years, it has become increasingly accepted that "low distortion" is not a synonym for "high perceptual quality", and in fact optimization of one often comes at the expense of the other. In light of this understanding, it is natural to seek for a generalization of rate-distortion theory which takes perceptual quality into account. In this paper, we adopt the mathematical definition of perceptual quality recently proposed by Blau & Michaeli (2018), and use it to study the three-way tradeoff between rate, distortion, and perception. We show that restricting the perceptual quality to be high, generally leads to an elevation of the rate-distortion curve, thus necessitating a sacrifice in either rate or distortion. We prove several fundamental properties of this triple-tradeoff, calculate it in closed form for a Bernoulli source, and illustrate it visually on a toy MNIST example.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1901.07821

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Agustsson, E., Mentzer, F., Tschannen, M., Cavigelli, L., Timofte, R., Benini, L., and Gool, L. V. Soft-to-hard vector quantization for end-to-end learning compressible representations. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2017.
  2. Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R., and Van Gool, L. Generative adversarial networks for extreme learned image compression. arXiv preprint arXiv:1804.02958 , 2018. · arXiv
  3. Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., and Murphy, K. Fixing a broken elbo. In Proceedings of the International Conference on Machine Learning (ICML) , 2018.
  4. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning (ICML) , 2017.
  5. Ball´ e, J., Laparra, V., and Simoncelli, E. P. End-to-end optimization of nonlinear transform codes for perceptual quality. In Proceedings of the Picture Coding Symposium (PCS) , 2016.
  6. Ball´ e, J., Laparra, V., and Simoncelli, E. P. End-to-end optimized image compression. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.
  7. Ball´ e, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with a scale hyperprior. In Proceedings of the International Conference on Learning Representations (ICLR) , 2018.
  8. Blau, Y. and Michaeli, T. The perception-distortion tradeoff. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  9. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., and ZelnikManor, L. The 2018 PIRM challenge on perceptual image super-resolution. In Proceedings of the European Conference on Computer Vision Workshops (ECCVW) , 2018.
  10. Cover, T. M. and Thomas, J. A. Elements of information theory . John Wiley & Sons, 2012.
  11. Csisz´ ar, I., Shields, P . C., et al. Information theory and statistics: A tutorial. Foundations and Trends R © in Communications and Information Theory , 1(4):417-528, 2004.
  12. Denton, E. L., Chintala, S., Szlam, A., and Fergus, R. Deep generative image models using a laplacian pyramid of adversarial networks. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2015.
  13. Galteri, L., Seidenari, L., Bertini, M., and Del Bimbo, A. Deep generative adversarial compression artifact removal. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.
  14. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2014.
  15. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2017.
  16. Isola, P., Zhu, J.-Y ., Zhou, T., and Efros, A. A. Image-toimage translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
  17. Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV) , 2016.
  18. Johnston, N., Vincent, D., Minnen, D., Covell, M., Singh, S., Chinen, T., Hwang, S. J., Shor, J., and Toderici, G. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  19. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradientbased learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.
  20. Li, M., Zuo, W., Gu, S., Zhao, D., and Zhang, D. Learning convolutional networks for content-weighted image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  21. Mathieu, M., Couprie, C., and LeCun, Y. Deep multi-scale video prediction beyond mean square error. In Proceedings of the International Conference on Learning Representation (ICLR) , 2016.
  22. Matsumoto, R. Introducing the perception-distortion tradeoff into the rate-distortion theory of general information sources. IEICE Communications Express , 7(11):427-431, 2018a.
  23. Mechrez, R., Talmi, I., Shama, F., and Zelnik-Manor, L. Maintaining natural image statistics with the contextual loss. In Proceedings of the Asian Conference on Computer Vision (ACCV) , 2018a.
  24. Mechrez, R., Talmi, I., and Zelnik-Manor, L. The contextual loss for image transformation with non-aligned data. In Proceedings of the European Conference on Computer Vision (ECCV) , 2018b.
  25. Mentzer, F., Agustsson, E., Tschannen, M., Timofte, R., and Van Gool, L. Conditional probability models for deep image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  26. Minnen, D., Ball´ e, J., and Toderici, G. D. Joint autoregressive and hierarchical priors for learned image compression. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2018.
  27. Mittal, A., Soundararajan, R., and Bovik, A. C. Making a completely blind image quality analyzer. IEEE Signal Processing Letters , 20(3):209-212, 2013.
  28. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., and Efros, A. A. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.
  29. Rippel, O. and Bourdev, L. Real-time adaptive image compression. In Proceedings of the International Conference on Machine Learning (ICML) , 2017.
  30. Rix, A. W., Beerends, J. G., Hollier, M. P., and Hekstra, A. P. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing (ICASSP) , 2001.
  31. Rott Shaham, T. and Michaeli, T. Deformation aware image compression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  32. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., and Chen, X. Improved techniques for training gans. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2016.
  33. Santurkar, S., Budden, D., and Shavit, N. Generative compression. In Proceedings of the Picture Coding Symposium (PCS) , 2018.
  34. Shama, F., Mechrez, R., Shoshan, A., and ZelnikManor, L. Adversarial feedback loop. arXiv preprint arXiv:1811.08126 , 2018. · arXiv
  35. Shannon, C. E. Coding theorems for a discrete source with a fidelity criterion. IRE Nat. Conv. Rec , 4(142-163):1, 1959.
  36. Shoshan, A., Mechrez, R., and Zelnik-Manor, L. DynamicNet: Tuning the objective without re-training. arXiv preprint arXiv:1811.08760 , 2018. · arXiv
  37. Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014. · arXiv
  38. Toderici, G., O'Malley, S. M., Hwang, S. J., Vincent, D., Minnen, D., Baluja, S., Covell, M., and Sukthankar, R. Variable rate image compression with recurrent neural networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2016.
  39. Toderici, G., Vincent, D., Johnston, N., Jin Hwang, S., Minnen, D., Shor, J., and Covell, M. Full resolution image compression with recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.
  40. Tschannen, M., Agustsson, E., and Lucic, M. Deep generative models for distribution-preserving lossy compression. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2018.
  41. Van Erven, T. and Harremos, P. R´ enyi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory , 60(7):3797-3820, 2014.
  42. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Loy, C. C., Qiao, Y., and Tang, X. ESRGAN: Enhanced superresolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision Workshops (ECCVW) , 2018.
  43. Wang, Z. and Bovik, A. C. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE Signal Processing Magazine , 26(1):98-117, 2009.
  44. Wang, Z. and Simoncelli, E. P. Reduced-reference image quality assessment using a wavelet-domain natural image statistic model. In Human Vision and Electronic Imaging X , volume 5666, pp. 149-160, 2005.
  45. Zhang, R., Isola, P., and Efros, A. A. Colorful image colorization. In Proceedings of the European Conference on Computer Vision (ECCV) , 2016.
  46. Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018.
  47. Zhu, J.-Y., Zhang, R., Pathak, D., Darrell, T., Efros, A. A., Wang, O., and Shechtman, E. Toward multimodal imageto-image translation. In Proceedings of the Conference on Neural Information Processing Systems (NIPS) , 2017b.

Cited by

Sorted by citations

Authors
Blau, Y., & Michaeli, T.
Affiliations
Technion – Israel Institute of Technology (IL)
ORCID
Michaeli, Tomer
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
675-685
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1901.07821
arXiv
1901.07821
OpenAlex
W2913277358
Cited by
198
References
6
Citations by year
2026: 4 · 2025: 22 · 2024: 52 · 2023: 47 · 2022: 31 · 2021: 24 · 2020: 14 · 2019: 2 · 2018: 1 · 2012: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Data Compression Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
  • Physical Sciences › Computer Science › Signal Processing › Video Coding and Compression Technologies
Keywords
lossy compression, rate-distortion theory, perceptual quality, rate-distortion curve, Bernoulli sources, image compression

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review