CoChat
Sign Up
Cited by 104Neural Networks and ApplicationsQuantum Computing Algorithms and ArchitectureQuantum many-body systems

Information Perspective to Probabilistic Modeling: Boltzmann Machines versus Born Machines

Open access
Cheng, Song, Chen, Jing, Wang, Lei·2018·Entropy
DOI 10.3390/e20080583PMC PMC7513111
Open this paper in CoChat — free accountOpen PDF

Abstract

We compare and contrast the statistical physics and quantum physics inspired approaches for unsupervised generative modeling of classical data. The two approaches represent probabilities of observed data using energy-based models and quantum states, respectively. Classical and quantum information patterns of the target datasets therefore provide principled guidelines for structural design and learning in these two approaches. Taking the Restricted Boltzmann Machines (RBM) as an example, we analyze the information theoretical bounds of the two approaches. We also estimate the classical mutual information of the standard MNIST datasets and the quantum Rényi entropy of corresponding Matrix Product States (MPS) representations. Both information measures are much smaller compared to their theoretical upper bound and exhibit similar patterns, which imply a common inductive bias of low information complexity. By comparing the performance of RBM with various architectures on the standard MNIST datasets, we found that the RBM with local sparse connection exhibit high learning efficiency, which supports the application of tensor network states in machine learning problems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/e20080583

Cites

72 of 72 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 22

  1. [4]Goodfellow I. Bengio Y. Courville A. Deep Learning MIT Press Cambridge, MA, USA 2016
  2. [17]Miles Stoudenmire E. Schwab D.J. Supervised learning with quantum-inspired tensor networks arXiv 2016 1605.05775
  3. [19]Levine Y. Yakira D. Cohen N. Shashua A. Deep learning and quantum entanglement: fundamental connections with implications to network design arXiv 2017 1704.01552
  4. [21]Liu D. Ran S.J. Wittek P. Peng C. García R.B. Su G. Lewenstein M. Machine learning by two-dimensional hierarchical tensor networks: A quantum information theoretic perspective on deep architectures arXiv 2017 1710.04833
  5. [22]Zhang Y.H. Entanglement entropy of target functions for image classification and convolutional neural network arXiv 2017 1710.05520
  6. [23]Pestun V. Vlassopoulos Y. Tensor network language model arXiv 2017 1710.10248
  7. [24]Gao X. Zhang Z. Duan L. An efficient quantum algorithm for generative machine learning arXiv 2017 1711.02038
  8. [25]Huang Y. Provably efficient neural network representation for image classification arXiv 2017 1711.04606
  9. [33]Koller D. Friedman N. Probabilistic Graphical Models, Principles and Techniques MIT Press Cambridge, MA, USA 2009
  10. [34]MacKay D.J. Information Theory, Inference and Learning Algorithms Cambridge University Press Cambridge, UK 2003
  11. [36]Zeng B. Chen X. Zhou D.L. Wen X.G. Quantum information meets quantum matter–from quantum entanglement to topological phase in many-body systems arXiv 2015 1508.02595
  12. [40]Alemi A.A. Fischer I. Dillon J.V. Murphy K. Deep variational information bottleneck arXiv 2016 1612.00410
  13. [55]Morningstar A. Melko R.G. Deep learning the ising model near criticality arXiv 2017 1708.04622
  14. [62]Cover T.M. Thomas J.A. Elements of Information Theory John Wiley & Sons Hoboken, NJ, USA 2012
  15. [63]Preskill J. Quantum Shannon theory arXiv 2016 1604.07450
  16. [67]Peng K.H. Zhang H. Mutual information-based RBM neural networks Proceedings of the 2016 23rd International Conference on Pattern Recognition (ICPR) Cancun, Mexico 4–8 December 2016 2458 2463
  17. [68]Koch-Janusz M. Ringel Z. Mutual information, neural networks and the renormalization group arXiv 2017 1704.06279
  18. [69]Salakhutdinov R. Hinton G.E. Deep Boltzmann machines Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS’09) Clearwater Beach, FL, USA 16–18 April 2009 Volume 5 448
  19. [87]Farhi E. Harrow A.W. Quantum supremacy through the quantum approximate optimization algorithm arXiv 2016 1602.07674
  20. [88]Amin M.H. Andriyash E. Rolfe J. Kulchytskyy B. Melko R. Quantum Boltzmann machine arXiv 2016 1601.02036
  21. [89]Kieferova M. Wiebe N. Tomography and generative data modeling via quantum Boltzmann training arXiv 2016 1612.05204
  22. [90]Benedetti M. Realpe-Gómez J. Biswas R. Perdomo-Ortiz A. Quantum-assisted learning of graphical models with arbitrary pairwise connectivity arXiv 2016 1609.02542

Cited by

Sorted by citations

Related

Authors
Cheng, S., Chen, J., & Wang, L.
Affiliations
Chinese Academy of Sciences (CN)
Corresponding author
Wang, Lei
ORCID
Venue
Entropy
Publisher
MDPI AG
Published
2018-08-07
Volume
20
Issue
8
Pages
583
ISSN
1099-4300 (ISSN-L)
DOI
10.3390/e20080583
PMC
PMC7513111
OpenAlex
W2771527763
Cited by
104
References
72
Citations by year
2026: 3 · 2025: 10 · 2024: 15 · 2023: 11 · 2022: 18 · 2021: 22 · 2020: 8 · 2019: 12 · 2018: 3 · 2014: 1
FWCI
6.25
Type
journal article
Language
en
Open access
Licence
Locations
  • Entropy (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
  • National R&D Program of China (2017YFA0302901)
  • National Natural Science Foundation of China (No. 11190024, No. 11474331, No. 11774398)
  • the Ministry of Science and Technology of China (2016YFA0300603)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Quantum Computing Algorithms and Architecture
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Quantum many-body systems
Keywords
Boltzmann machine, restricted Boltzmann machine, matrix product states, quantum machine learning, energy-based models, mutual information, Rényi entropy, information-theoretic limits, inductive bias, tensor networks, MNIST, sparse connections

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review