CoChat
Sign Up
PreprintCited by 480Stochastic Gradient Optimization TechniquesGaussian Processes and Bayesian InferenceNeural Networks and Applications

On Variational Bounds of Mutual Information

Open access
Poole, Ben, Ozair, Sherjil, Oord, Aaron van den, Alemi, Alexander A., Tucker, George·2019·arXiv
DOI 10.48550/arxiv.1905.06922arXiv 1905.06922
Open this paper in CoChat — free accountOpen PDF

Abstract

Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and scalable objectives, recent work has turned to variational bounds parameterized by neural networks, but the relationships and tradeoffs between these bounds remains unclear. In this work, we unify these recent developments in a single framework. We find that the existing variational lower bounds degrade when the MI is large, exhibiting either high bias or high variance. To address this problem, we introduce a continuum of lower bounds that encompasses previous bounds and flexibly trades off bias and variance. On high-dimensional, controlled problems, we empirically characterize the bias and variance of the bounds and their gradients and demonstrate the effectiveness of our new bounds for estimation and representation learning.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1905.06922

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • If we can find such an fθ, then the resulting classifier is approximately equal to the density ratio between μπ(s̃ | s, a) and the state marginal p(s̃).

    “If we can find such an fθ, then the resulting classifier is approximately equal to the density ratio between µπ(s̃ | s, a) and the state marginal p(s̃) (Poole et al., 2019; Zheng et al., 2023):”

    From Predictive Representations: Building Blocks of Intelligence (2024) · cites this paper as Poole et al., 2019

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 45

  1. Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410 , 2016. · arXiv
  2. Barber, D. and Agakov, F. The im algorithm: A variational approach to information maximization. In NIPS , pp. 201208. MIT Press, 2003.
  3. Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Hjelm, D., and Courville, A. Mutual information neural estimation. In International Conference on Machine Learning , pp. 530-539, 2018.
  4. Bell, A. J. and Sejnowski, T. J. An informationmaximization approach to blind separation and blind deconvolution. Neural computation , 7(6):1129-1159, 1995.
  5. Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, 2017.
  6. Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disentangling in β -vae. arXiv preprint arXiv:1804.03599 , 2018. · arXiv
  7. Chen, T. Q., Li, X., Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational autoencoders. arXiv preprint arXiv:1802.04942 , 2018. · arXiv
  8. Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1422-1430, 2015.
  9. Donsker, M. D. and Varadhan, S. S. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics , 36 (2):183-212, 1983.
  10. Dosovitskiy, A., Springenberg, J. T., Riedmiller, M., and Brox, T. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems , pp. 766-774, 2014.
  11. Foster, A., Jankowiak, M., Bingham, E., Teh, Y. W., Rainforth, T., and Goodman, N. Variational optimal experiment design: Efficient automation of adaptive experiments. In NeurIPS Bayesian Deep Learning Workshop , 2018.
  12. Gabri´ e, M., Manoel, A., Luneau, C., Barbier, J., Macris, N., Krzakala, F., and Zdeborov´ a, L. Entropy and mutual information in models of deep neural networks. arXiv preprint arXiv:1805.09785 , 2018. · arXiv
  13. Gao, S., Ver Steeg, G., and Galstyan, A. Efficient estimation of mutual information for strongly dependent variables. In Artificial Intelligence and Statistics , pp. 277-286, 2015.
  14. Higgins, I., Matthey, L., Glorot, X., Pal, A., Uria, B., Blundell, C., Mohamed, S., and Lerchner, A. Early visual concept learning with unsupervised deep learning. arXiv preprint arXiv:1606.05579 , 2016. · arXiv
  15. Higgins, I., Amos, D., Pfau, D., Racani` ere, S., Matthey, L., Rezende, D. J., and Lerchner, A. Towards a definition of disentangled representations. CoRR , abs/1812.02230, 2018. · arXiv
  16. Hjelm, R. D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018. · arXiv
  17. Hu, W., Miyato, T., Tokui, S., Matsumoto, E., and Sugiyama, M. Learning discrete representations via information maximizing self-augmented training. arXiv preprint arXiv:1702.08720 , 2017. · arXiv
  18. Kim, H. and Mnih, A. Disentangling by factorising. arXiv preprint arXiv:1802.05983 , 2018. · arXiv
  19. Kingma, D. P. and Welling, M. Auto-encoding variational bayes. nternational Conference on Learning Representations , 2013.
  20. Kolchinsky, A., Tracey, B. D., and Wolpert, D. H. Nonlinear information bottleneck. arXiv preprint arXiv:1705.02436 , 2017. · arXiv
  21. Kraskov, A., St¨ ogbauer, H., and Grassberger, P. Estimating mutual information. Physical review E , 69(6):066138, 2004.
  22. Krause, A., Perona, P., and Gomes, R. G. Discriminative clustering by regularized information maximization. In Advances in neural information processing systems , pp. 775-783, 2010.
  23. Kumar, A., Sattigeri, P., and Balakrishnan, A. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848 , 2017. · arXiv
  24. Locatello, F., Bauer, S., Lucic, M., Gelly, S., Sch¨ olkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811.12359 , 2018. · arXiv
  25. Ma, Z. and Collins, M. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812 , 2018. · arXiv
  26. Mescheder, L., Nowozin, S., and Geiger, A. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. arXiv preprint arXiv:1701.04722 , 2017. · arXiv
  27. Mnih, A. and Teh, Y. W. A fast and simple algorithm for training neural probabilistic language models. arXiv preprint arXiv:1206.6426 , 2012. · arXiv
  28. Moyer, D., Gao, S., Brekelmans, R., Galstyan, A., and Ver Steeg, G. Invariant representations without adversarial training. In Advances in Neural Information Processing Systems , pp. 9102-9111, 2018.
  29. Nemenman, I., Bialek, W., and van Steveninck, R. d. R. Entropy and information in neural spike trains: Progress on the sampling problem. Physical Review E , 69(5): 056111, 2004.
  30. Nguyen, X., Wainwright, M. J., and Jordan, M. I. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory , 56(11):5847-5861, 2010.
  31. Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision , pp. 69-84. Springer, 2016.
  32. Palmer, S. E., Marre, O., Berry, M. J., and Bialek, W. Predictive information in a sensory population. Proceedings of the National Academy of Sciences , 112(22):6908-6913, 2015.
  33. Paninski, L. Estimation of entropy and mutual information. Neural computation , 15(6):1191-1253, 2003.
  34. Peng, X. B., Kanazawa, A., Toyer, S., Abbeel, P., and Levine, S. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. arXiv preprint arXiv:1810.00821 , 2018. · arXiv
  35. Poole, B., Alemi, A. A., Sohl-Dickstein, J., and Angelova, A. Improved generator objectives for gans. arXiv preprint arXiv:1612.02780 , 2016. · arXiv
  36. Rainforth, T., Cornish, R., Yang, H., and Warrington, A. On nesting monte carlo estimators. In International Conference on Machine Learning , pp. 4264-4273, 2018.
  37. Reshef, D. N., Reshef, Y. A., Finucane, H. K., Grossman, S. R., McVean, G., Turnbaugh, P. J., Lander, E. S., Mitzenmacher, M., and Sabeti, P. C. Detecting novel associations in large data sets. science , 334(6062):1518-1524, 2011.
  38. Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning , pp. 1278-1286, 2014.
  39. Ryan, E. G., Drovandi, C. C., McGree, J. M., and Pettitt, A. N. A review of modern computational algorithms for bayesian optimal design. International Statistical Review , 84(1):128-154, 2016.
  40. Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., and Cox, D. D. On the information bottleneck theory of deep learning. In International Conference on Learning Representations , 2018.
  41. Tishby, N. and Zaslavsky, N. Deep learning and the information bottleneck principle. In Information Theory Workshop (ITW), 2015 IEEE , pp. 1-5. IEEE, 2015.
  42. Tishby, N., Pereira, F. C., and Bialek, W. The information bottleneck method. arXiv preprint physics/0004057 , 2000. · arXiv
  43. Tomczak, J. and Welling, M. Vae with a vampprior. In Storkey, A. and Perez-Cruz, F. (eds.), Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics , volume 84 of Proceedings of Machine Learning Research , pp. 1214-1223, Playa Blanca, Lanzarote, Canary Islands, 09-11 Apr 2018. PMLR.
  44. van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelcnn decoders. In Advances in Neural Information Processing Systems , pp. 4790-4798, 2016.
  45. van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018. · arXiv

Cited by

Sorted by citations

Authors
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G.
Affiliations
  • Google (United States) (US)
  • Université de Montréal (CA)
ORCID
Alemi, Alexander A.
Venue
arXiv
Publisher
arXiv
Published
2019
Pages
5171-5180
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1905.06922
arXiv
1905.06922
OpenAlex
W2946006146
Cited by
480
References
1
Citations by year
2026: 6 · 2025: 40 · 2024: 77 · 2023: 62 · 2022: 67 · 2021: 134 · 2020: 72 · 2019: 21 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
mutual information, variational bounds, mutual information estimation, mutual information maximization, bias-variance trade-off, representation learning, gradient estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review