CoChat
Sign Up
PreprintCited by 2Machine Learning and Data ClassificationAdvanced Data Compression TechniquesDistributed Sensor Networks and Detection Algorithms

The DOI matches its DataCite record; the references it cites are not checked yet.

Studying the Interplay between Information Loss and Operation Loss in Representations for Classification

Open access
Silva, Jorge F., Tobar, Felipe, Vicuña, Mario, Cordova, Felipe·2021·arXiv
DOI 10.48550/arxiv.2112.15238arXiv 2112.15238
Open this paper in CoChat — free accountOpen PDF

Abstract

Information-theoretic measures have been widely adopted in the design of features for learning and decision problems. Inspired by this, we look at the relationship between i) a weak form of information loss in the Shannon sense and ii) the operation loss in the minimum probability of error (MPE) sense when considering a family of lossy continuous representations (features) of a continuous observation. We present several results that shed light on this interplay. Our first result offers a lower bound on a weak form of information loss as a function of its respective operation loss when adopting a discrete lossy representation (quantization) instead of the original raw observation. From this, our main result shows that a specific form of vanishing information loss (a weak notion of asymptotic informational sufficiency) implies a vanishing MPE loss (or asymptotic operational sufficiency) when considering a general family of lossy continuous representations. Our theoretical findings support the observation that the selection of feature representations that attempt to capture informational sufficiency is appropriate for learning, but this selection is a rather conservative design principle if the intended goal is achieving MPE in classification. Supporting this last point, and under some structural conditions, we show that it is possible to adopt an alternative notion of informational sufficiency (strictly weaker than pure sufficiency in the mutual information sense) to achieve operational sufficiency in learning.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2112.15238

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 37

  1. [1][1] A. Achille and S. Soatto. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research , 19:1-34, 2018. -
  2. [2][2] A. Achille and S. Soatto. Information dropout: Learning optimal representations through noisy computations. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(12):2897 - 2905, January 2018. -
  3. [3][3] A. Alemi, I. Fisher, J. Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR , pages 24-26, April 2017. -
  4. [5][5] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35:1798-1828, 2013. -
  5. [6][6] A. Berlinet and Igor Vajda. On asymtotic sufficiency and optimality of quantizations. Journal of Statistical Planning and Inference , 136:4217-4238, 2005. -
  6. [7][7] B. Bloem-Reddy and Y. W. Teh. Probabilistic symmetry and invariant neural networks. Journal of Machine Learning Research , 21:1-61, 2020. -
  7. [8][8] O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research , 2:499-526, 2002. -
  8. [10][10] Leo Breiman. Probability . Addison-Wesley, 1968. -
  9. [11][11] T. M. Cover and J. A. Thomas. Elements of Information Theory . Wiley Interscience, New York, second edition, 2006. -
  10. [13][13] Georges A. Darbellay and Igor Vajda. Estimation of the information by an adaptive partition of the observation space. IEEE Transactions on Information Theory , 45(4):1315-1321, 1999. -
  11. [14][14] L. Devroye, L. Györfi, and G. Lugosi. A Probabilistic Theory of Pattern Recognition . New York: Springer-Verlag, 1996. -
  12. [15][15] Y. Dubois, B. Bloem-Reddy, K. Ullrich, and C. J. Maddison. Lossy compression for losless prediction. In at ICLR 2021 neural compression workshop , pages 1-26, 2021. -
  13. [16][16] M. L. Eaton. Group invariance in applicartions in statistics. In Regional Conference Series in Probability and Statistics Volume 1 . Instituto of Mathematical Statistics and American Statistical Asociation, 1989. -
  14. [17][17] M. Feder and N. Merhav. Relationship between entropy and error probability. IEEE Transactions on Information Theory , 40(1):259-266, January 1994. -
  15. [19][19] Mauricio Gonzales and Jorge F Silva. Data-driven representations for testing independence: A connection with mutual information estimation. In IEEE International Symposium on Information Theory . IEEE, 2020. -
  16. [20][20] R. Gray and Lee D. Davisson. Introduction to Statistical Signal Processing . Cambridge Univ Press, 2004. -
  17. [21][21] R. M. Gray. Entropy and Information Theory . Springer - Verlag, New York, 1990. -
  18. [23][23] Robert M. Gray. Probability, Random Processes, and Ergodic Properties . Springer, second edition edition, 2009. -
  19. [24][24] Siu-Wai Ho and Sergio Verdú. On the interplay between conditional entropy and error probability. IEEE Transactions on Information Theory , 56(12):5930-5942, December 2010. -
  20. [25][25] F. Liese, Domingo Morales, and Igor Vajda. Asymptotically sufficient partition and quantization. IEEE Transactions on Information Theory , 52(12):5599-5606, 2006. -
  21. [26][26] G. Lugosi and Andrew B. Nobel. Consistency of data-driven histogram methods for density estimation and classification. The Annals of Statistics , 24(2):687-706, 1996. -
  22. [27][27] Andrew B. Nobel. Histogram regression estimation using data-dependent partitions. The Annals of Statistics , 24(3):1084-1105, 1996. -
  23. [28][28] S. Prasad. Bayesian error-based sequeneces of statistical information bounds. IEEE Transactions on Information Theory , 61(9):5052-5062, September 2015. -
  24. [29][29] J. Rotman. An Introduction to the Theory of Groups , volume 148 of Graduate Texts in Mathematics . Springer - Verlag, New York, 4th edition, 1995. -
  25. [30][30] S. Shalev-Shwartz, O. Shamir, and N. Srebro. Learnability, stability and uniform convergence. Journal of Machine Learning Research , 11:2635-2670, 2010. -
  26. [31][31] J. F. Silva and S. Narayanan. Universal consistency of data-driven partitions for divergence estimation. In IEEE International Symposium on Information Theory . IEEE, 2007. -
  27. [32][32] Jorge F. Silva and Shrikanth Narayanan. Information divergence estimation based on data-dependent partitions. Journal of Statistical Planning and Inference , 140(11):3180 - 3198, November 2010. -
  28. [33][33] Jorge F. Silva and Shrikanth Narayanan. Non-product data-dependent partitions for mutual information estimation: Strong consistency and applications. IEEE Transactions on Signal Processing , 58(7):3497-3511, July 2010. -
  29. [34][34] Jorge F Silva and Shrikanth Narayanan. Complexity-regularized tree-structured partition for mutual information estimation. IEEE Transactions on Information Theory , 58(3):1940 - 1952, 2012. -
  30. [35][35] D. J. Strouse and D.J. Schwab. The deterministic information bottleneck. Mass. Inst. Tech. Neural Computing , 26(1611-1630), 2017. -
  31. [36][36] M. Tegmark and T. Wu. Pareto-optimal data compression for binary classifiaction tasks. Entropy , 22(7):1-27, 2019. -
  32. [37][37] N. Tishby, F. Pereira, and W. Bialek. The information bottleneck method. In Proceedings of the Thirty-seventh Annual Allerton Conference on Communication, Control, and Computing , pages 368-377, September 1999. -
  33. [38][38] Igor Vajda. On convergence of information contained in quantized observations. IEEE Transactions on Information Theory , 48(8):2163-2172, 2002. -
  34. [40][40] M. Vera, P. Piantanida, and L.R. Vega. The role of information complexity and randomization in representation learning. https://arxiv.org/abs/1802.05355, 2018. - · arXiv
  35. [41][41] H. Xu and S. Mannor. Robustness and generalization. Machine Learning , 86(391-423), 2012. -
  36. [42][42] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets. In Advances in Neural Information Processing System 30 , pages 3391-3401. -
  37. [43][43] A. Zaidi, I. Estella-Aguerri, and S. Shamai. On the information bottleneck problems: Models, connections, applications and information theoretic views. Entropy , 22(151):1-36, 2020.

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Silva, J. F., Tobar, F., Vicuña, M., & Cordova, F.
Affiliations
University of Chile (CL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2112.15238
arXiv
2112.15238
OpenAlex
W4226238762
Cited by
1
References
0
Citations by year
2025: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Data Compression Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
Keywords
information loss, operational losses, minimum error probability, feature selection, quantization, continuous monitoring, mutual information, classification, lower bounds

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review