CoChat
Sign Up
Cited by 84Distributed Sensor Networks and Detection AlgorithmsStochastic Gradient Optimization TechniquesNeural Networks and Applications

The DOI matches its Crossref record, with a small difference in title or year; 13 of the 13 references it cites were found in a registry or an index.

Distributed Variational Representation Learning

Open access
Aguerri, Inaki Estella, Zaidi, Abdellatif·2021·IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI 10.1109/tpami.2019.2928806
Open this paper in CoChat — free accountOpen PDF

Abstract

The problem of distributed representation learning is one in which multiple sources of information X1,..., XKare processed separately so as to learn as much information as possible about some ground truth Y. We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, K encoders, K ≥ 2, compress their observations X1,..., XKseparately in a manner such that, collectively, the produced representations preserve as much information as possible about Y. We study both discrete memoryless (DM) and memoryless vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff between complexity (or rate) and relevance (or information) for a class of memoryless sources (the observations X1,..., XKbeing conditionally independent given Y). For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We also provide two algorithms that allow to compute this bound: i) a Blahut-Arimoto type iterative algorithm which enables to compute optimal complexity-relevance encoding mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2019.2928806

Cites

13 of 13 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 13 references

Research works printed but not linked: 57

  1. glorot, 2010, Understanding the difficulty of training deep feedforward neural networks, Proc Int Conf Artif Intell Statist, 249
  2. papamakarios, 2017, Masked autoregressive flow for density estimation, ArXiv e-prints
  3. 10.1109/18.104312 · DOI
  4. 10.1109/TIT.2014.2358692 · DOI
  5. saxe, 2018, On the information bottleneck theory of deep learning, Proc Int Conf Learn Representations
  6. 10.1109/TIT.2014.2358692 · DOI
  7. 10.1109/ITA.2018.8503149 · DOI
  8. 10.1109/TIT.2016.2617862 · DOI
  9. 10.1109/TPAMI.2017.2784440 · DOI · Paper page
  10. chechik, 2005, Information bottleneck for Gaussian variables, J Mach Learn Res, 6, 165
  11. 10.1109/TIT.1975.1055437 · DOI
  12. 10.1109/TIT.1962.1057738 · DOI
  13. rezende, 0, Stochastic backpropagation and approximate inference in deep generative models, Proc 31st Int Conf Mach Learn, 32, 22
  14. 10.1109/TIT.1972.1054855 · DOI · Paper page
  15. 10.1017/CBO9781139030687 · DOI
  16. 10.1109/ISIT.2007.4557285 · DOI
  17. 10.1073/pnas.0507432102 · DOI
  18. 10.1145/345508.345578 · DOI
  19. 10.1109/ISIT.2014.6875354 · DOI
  20. 10.1002/0471200611 · DOI
  21. 10.1137/120891009 · DOI
  22. aguerri, 2018, Distributed information bottleneck method for discrete and Gaussian sources, Proc IEEE Int Zurich Seminar Commun, 154
  23. 10.1016/j.neunet.2019.05.003 · DOI
  24. 10.1145/3219166.3219194 · DOI
  25. yu, 2018, Understanding convolutional neural network training with information theory, arXiv preprint arXiv 1804 06324
  26. kingma, 2013, Auto-encoding variational bayes, arXiv preprint arXiv 1312 6114
  27. ugur, 2017, A generalization of Blahut-Arimoto algorithm to computing rate-distortion regions of multiterminal source coding under logarithmic loss, Proc IEEE Inf Theory Workshop, 349
  28. dempster, 1977, Maximum likelihood from incomplete data via the EM algorithm, J Royal Statistical Society Series B, 39, 1, 10.1111/j.2517-6161.1977.tb01600.x · DOI · Paper page
  29. 10.1145/279943.279962 · DOI
  30. 10.1109/TIT.2013.2288257 · DOI
  31. vapnik, 2013, The Nature of Statistical Learning Theory
  32. jia, 2010, Factorized latent spaces with structured sparsity, Proc Int Conf Neural Inf Process, 982
  33. gönen, 2011, Multiple kernel learning algorithms, J Mach Learn Res, 12, 2211
  34. kumar, 2011, A co-training approach for multi-view spectral clustering, Proc 28th Int Conf Mach Learn, 393
  35. dhillon, 2011, Multi-view learning of word embeddings via CCA, Proc Int Conf Neural Inf Process, 199
  36. 10.1109/TMI.2003.815867 · DOI
  37. 10.1023/A:1007958904918 · DOI
  38. 10.1109/ISIT.2016.7541465 · DOI
  39. 10.1017/CBO9780511546921 · DOI
  40. lehmann, 2006, Theory of Point Estimation
  41. bousquet, 2002, Stability and generalization, J Mach Learn Res, 2, 499
  42. shalev-shwartz, 2010, Learnability, stability and uniform convergence, J Mach Learn Res, 11, 2635
  43. xu, 2017, Information-theoretic analysis of generalization capability of learning algorithms, Proc Int Conf Neural Inf Process, 2521
  44. 10.1007/978-1-4419-1570-2 · DOI
  45. 10.1109/TIT.2015.2462848 · DOI
  46. principe, 2000, Neural and Adaptive Systems Fundamentals through Simulations, 672
  47. 10.1109/2.36 · DOI
  48. 10.1109/ISIT.2018.8437786 · DOI
  49. 10.1109/TIT.1968.1054142 · DOI
  50. quinlan, 2014, C4 5 Programs for Machine Learning
  51. olsen, 2008, On the impact of entropy estimation on transcriptional regulatory network inference based on mutual information, EURASIP J Bioinf Syst Biol, 2009
  52. 10.1109/TIT.2015.2412945 · DOI
  53. 10.1162/089976603321780272 · DOI · Paper page
  54. 10.1109/ITW.2018.8613433 · DOI
  55. 10.1109/TIT.2008.928951 · DOI
  56. 10.1109/TPAMI.2013.50 · DOI · Paper page
  57. 10.1109/TIT.2019.2897564 · DOI

Cited by

Sorted by citations

Metadata

Authors
Aguerri, I. E., & Zaidi, A.
Affiliations
Paris-Est Sup (FR)
ORCID
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2021-01-01
Volume
43
Issue
1
Pages
120-138
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
DOI
10.1109/tpami.2019.2928806
OpenAlex
W2849120009
Cited by
84
References
13
Citations by year
2026: 11 · 2025: 15 · 2024: 10 · 2023: 8 · 2022: 19 · 2021: 11
FWCI
5.34
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Distributed Sensor Networks and Detection Algorithms
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
Keywords
information bottleneck, information-theoretic compression, variational bounds, evidence lower bound, Blahut-Arimoto algorithm, variational inference, single-letter characterization, discrete memoryless sources

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review