The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Deep learning and the information bottleneck principle
Open accessAbstract
Deep Neural Networks (DNNs) are analyzed via the theoretical framework of the information bottleneck (IB) principle. We first show that any DNN can be quantified by the mutual information between the layers and the input and output variables. Using this representation we can calculate the optimal information theoretic limits of the DNN and obtain finite sample generalization bounds. The advantage of getting closer to the theoretical limit is quantifiable both by the generalization bound and by the network's simplicity. We argue that both the optimal architecture, number of layers and features/connections at each layer, are related to the bifurcation points of the information bottleneck tradeoff, namely, relevant compression of the input layer with respect to the output layer. The hierarchical representations at the layered network naturally correspond to the structural phase transitions along the information curve. We believe that this new insight can lead to new optimality bounds and deep learning algorithms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Representation Learning: A Review and New Perspectives
Bengio, Y., Courville, A., & Vincent, P. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.50 · Cited by 13,001
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 9,239
The information bottleneck method
Tishby, N., Pereira, F. C., & Bialek, W. (2000) · arXiv · doi:10.48550/arxiv.physics/0004057 · Cited by 2,626 · cited 2×
“As was shown in [6], the optimal solutions for the IB variational problem satisfy the following self-consistent equations for some value of β , where Z ( x ; β ) is the normalization factor, also known as the partition function.”
MethodSelf-citationRose, K. (1998) · Proceedings of the IEEE · doi:10.1109/5.726788 · Cited by 876
Statistical mechanics and phase transitions in clustering
Rose, K., Gurewitz, E., & Fox, G. C. (1990) · Physical Review Letters · doi:10.1103/physrevlett.65.945 · Cited by 458
An exact mapping between the Variational Renormalization Group and Deep LearningPreprint
Mehta, P., & Schwab, D. J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.3831 · Cited by 451
Learning and generalization with the information bottleneck
Shamir, O., Sabato, S., & Tishby, N. (2010) · Theoretical Computer Science · doi:10.1016/j.tcs.2010.04.006 · Cited by 165
Self-citationAn Information Theoretic Tradeoff between Complexity and Accuracy
Gilad-Bachrach, R., Navot, A., & Tishby, N. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45167-9_43 · Cited by 99
Self-citation
Cited by
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
Wang, C.-Y., Yeh, I.-H., & Mark Liao, H.-Y. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72751-1_1 · Cited by 4,823
A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Machine Learning in Drug Discovery: A Review
Dara, S., Dhamercherla, S., Jadav, S. S., Babu, C. M., & Ahsan, M. J. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10058-4 · Cited by 792
Neural Sign Language Translation
Camgoz, N. C., Hadfield, S., Koller, O., Ney, H., & Bowden, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00812 · Cited by 784
Barlow Twins: Self-Supervised Learning via Redundancy Reduction
Žbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03230 · Cited by 751
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T. Z., Kumar, V., Levine, S., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2304.13705 · Cited by 673
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Zhao, T., Kumar, V., Levine, S., & Finn, C. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.016 · Cited by 674
Han, W., Chen, H., & Poria, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.723 · Cited by 570
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2021) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 557
On the information bottleneck theory of deep learning*
Saxe, A. M., Bansal, Y., Dapello, J., Advani, M., Kolchinsky, A., Tracey, B. D., & Cox, D. D. (2019) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ab3985 · Cited by 493
On Variational Bounds of Mutual Information
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. (2019) · arXiv · doi:10.48550/arxiv.1905.06922 · Cited by 480
“Mutual information estimators are used in computational neuroscience (Palmer et al., 2015), Bayesian optimal experimental design (Ryan et al., 2016; Foster et al., 2018), understanding neural networks (Tishby et al., 2000; Tishby & Zaslavsky, 2015; Gabri´e et al., 2018), and more.”
MethodDiscovering Physical Concepts with Neural Networks
Iten, R., Metger, T., Wilming, H., del Rio, L., & Renner, R. (2020) · Physical Review Letters · doi:10.1103/physrevlett.124.010508 · Cited by 458
A Survey on Data Augmentation for Text Classification
Bayer, M., Kaufhold, M.-A., & Reuter, C. (2023) · ACM Computing Surveys · doi:10.1145/3544558 · Cited by 425
Relay Backpropagation for Effective Learning of Deep Convolutional Neural Networks
Shen, L., Lin, Z., & Huang, Q. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46478-7_29 · Cited by 421
Deep Semi-Supervised Anomaly Detection
Ruff, L., Vandermeulen, R. A., Görnitz, N., Binder, A., Müller, E., Müller, K.-R., & Kloft, M. (2019) · arXiv · doi:10.48550/arxiv.1906.02694 · Cited by 388
Information dropout: learning optimal representations through noisy computation
Achille, A., & Soatto, S. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2784440 · Cited by 356
InfoVAE: Information Maximizing Variational Autoencoders
Zhao, S., Song, J., & Ermon, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02262 · Cited by 328
Deep multi-view learning methods: A review
Yan, X., Hu, S., Mao, Y., Ye, Y., & Yu, H. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.090 · Cited by 322
Deep Variational Information Bottleneck
Alemi, A. A., Fischer, I., Dillon, J. V., & Murphy, K. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.00410 · Cited by 321
Normalizing Flows for Probabilistic Modeling and Inference
Papamakarios, G., Nalisnick, E., Rezende, D. J., Mohamed, S., & Lakshminarayanan, B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02762 · Cited by 318
TensorFlow Quantum: A Software Framework for Quantum Machine Learning
Broughton, M., Verdon, G., McCourt, T., Martinez, A. J., Yoo, J. H., Isakov, S. V., Massey, P., Halavati, R., Niu, M. Y., Zlokapa, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2003.02989 · Cited by 308
Sapkota, R., Flores-Calero, M., Qureshi, R., Badgujar, C., Nepal, U., Poulose, A., Zeno, P., Vaddevolu, U. B. P., Khan, S., Shoman, M., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11253-3 · Cited by 308
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
Matsubara, Y., Levorato, M., & Restuccia, F. (2023) · ACM Computing Surveys · doi:10.1145/3527155 · Cited by 304
Data Uncertainty Learning in Face Recognition
Chang, J., Lan, Z., Cheng, C., & Wei, Y. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00575 · Cited by 320
BS-Nets: An End-to-End Framework for Band Selection of Hyperspectral Image
Cai, Y., Liu, X., & Cai, Z. (2020) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2019.2951433 · Cited by 278
Emergence of Invariance and Disentanglement in Deep Representations
Achille, A., & Soatto, S. (2018) · 2018 Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2018.8503149 · Cited by 267
Artificial intelligence in drug design
Zhong, F., Xing, J., Li, X., Liu, X., Fu, Z., Xiong, Z., Lu, D., Wu, X., Zhao, J., Tan, X., et al. (2018) · Science China Life Sciences · doi:10.1007/s11427-018-9342-2 · Cited by 259
What is Semantic Communication? A View on Conveying Meaning in the Era of Machine Intelligence
Lan, Q., Wen, D., Zhang, Z., Zeng, Q., Chen, X., Popovski, P., & Huang, K. (2021) · Journal of Communications and Information Networks · doi:10.23919/jcin.2021.9663101 · Cited by 255
Identifying behavioral structure from deep variational embeddings of animal motion
Luxem, K., Mocellin, P., Fuhrmann, F., Kürsch, J., Miller, S. R., Palop, J. J., Remy, S., & Bauer, P. (2022) · Communications Biology · doi:10.1038/s42003-022-04080-7 · Cited by 218
Artificial Intelligence and Biosensors in Healthcare and Its Clinical Relevance: A Review
(2023) · doi:10.1109/access.2023.3285596 · Cited by 205
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193
“Several methods utilise information-theoretic regularisation techniques, building on the information bottleneck (Tishby & Zaslavsky, 2015).”
MethodCommunication-Computation Trade-off in Resource-Constrained Edge Inference
Shao, J., & Zhang, J. (2020) · IEEE Communications Magazine · doi:10.1109/mcom.001.2000373 · Cited by 162
Explainable Artificial Intelligence for Tabular Data: A Survey
Sahakyan, M., Aung, Z., & Rahwan, T. (2021) · IEEE Access · doi:10.1109/access.2021.3116481 · Cited by 135
“Tishby and Zaslavsky [108] noted that neural networks with certain properties could be studied in the information plane — the plane of mutual information values that each layer preserves on the input and output variables.”
Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?
Ravichander, A., Belinkov, Y., & Hovy, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.295 · Cited by 132
Pixel-Based Image Encryption Without Key Management for Privacy-Preserving Deep Neural Networks
Sirichotedumrong, W., Kinoshita, Y., & Kiya, H. (2019) · IEEE Access · doi:10.1109/access.2019.2959017 · Cited by 105
TSViz: Demystification of Deep Learning Models for Time-Series Analysis
Siddiqui, S. A., Mercier, D., Munir, M., Dengel, A., & Ahmed, S. (2019) · IEEE Access · doi:10.1109/access.2019.2912823 · Cited by 95
“Information bottleneck theory [10] has also gained popularity as a method for explaining the generalization and learning capabilities of deep learning based models.”
MethodGlobal Model Interpretation Via Recursive Partitioning
Yang, C., Rangarajan, A., & Ranka, S. (2018) · 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS) · doi:10.1109/hpcc/smartcity/dss.2018.00256 · Cited by 93
Wireless Network Intelligence at the Edge
Park, J., Samarakoon, S., Bennis, M., & Debbah, M. (2019) · Proceedings of the IEEE · doi:10.1109/jproc.2019.2941458 · Cited by 92
Deep Learning Under Privileged Information Using Heteroscedastic Dropout
Lambert, J., Sener, O., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00926 · Cited by 93
“Although we use IB [35], Gaussian dropout [33] and the re-parametrization trick [21], we are the first to our knowledge to apply any of these methods to the LUPI problem.”
Removing Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Gat, I., Schwartz, I., Schwing, A., & Hazan, T. (2020) · arXiv · doi:10.48550/arxiv.2010.10802 · Cited by 63
“More broadly, other components of information theory have been studied in deep nets, for example, the information bottleneck criteria [35, 36].”
BackgroundA Survey of Machine Unlearning
Nguyen, T. T., Huynh, T. T., Ren, Z., Nguyen, P. L., Liew, A. W.-C., Yin, H., & Nguyen, Q. V. H. (2022) · arXiv · doi:10.48550/arxiv.2209.02299 · Cited by 193
Averaging Is Probably Not the Optimum Way of Aggregating Parameters in Federated Learning
Xiao, P., Cheng, S., Stankovic, V., & Vukobratovic, D. (2020) · Entropy · doi:10.3390/e22030314 · Cited by 54
“In [29], they extend the approach of [28] demonstrating the effectiveness of the Information Plane visualization of DNNs.”
BackgroundThe partial information decomposition of generative neural network models
Tax, T., Mediano, P., & Shanahan, M. (2017) · Entropy · doi:10.3390/e19090474 · Cited by 55
Ye, R., Gao, Q., Qian, Y., Sun, J., & Li, T. (2024) · Agronomy · doi:10.3390/agronomy14051034 · Cited by 53
Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
Ma, X., Guo, J., Zhang, R., Fan, Y., & Cheng, X. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531772 · Cited by 35
A theoretical framework for deep transfer learning
Galanti, T., Wolf, L., & Hazan, T. (2016) · Information and Inference · doi:10.1093/imaiai/iaw008 · Cited by 29
Zhou, C., Kadambi, P., Mattina, M., & Whatmough, P. N. (2020) · arXiv · doi:10.48550/arxiv.2001.04974 · Cited by 27
Large language models at work in China’s labor market
Chen, Q., Ge, J., Xie, H., Xu, X., & Yang, Y. (2025) · China Economic Review · doi:10.1016/j.chieco.2025.102413 · Cited by 47
Eysenbach, B., Salakhutdinov, R., & Levine, S. (2021) · arXiv · doi:10.48550/arxiv.2109.03214 · Cited by 15
Witkowski, O., & Ikegami, T. (2019) · Artificial Life · doi:10.1162/artl_a_00288 · Cited by 14
Neural networks memorise personal information from one sample
Hartley, J., Sanchez, P. P., Haider, F., & Tsaftaris, S. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-48034-3 · Cited by 10
“This is because under the information bottleneck (IB) principle, information learnt from the other samples in the training dataset is sufficient to reduce the uncertainty of the label distribution 36,37 .”
BackgroundIntelligence plays dice: Stochasticity is essential for machine learning
Sabuncu, M. R. (2020) · arXiv · doi:10.48550/arxiv.2008.07496 · Cited by 3
“Another promising direction is Tishby et al.’s Information Bottleneck Principle (IBP) [62], which studies the tradeoff between prediction accuracy and information compression to provide generalizable insights into the behavior of deep learning models.”
BackgroundOpening the Black Box of Deep Neural Networks via Information
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv · doi:10.48550/arxiv.1703.00810
Self-citation
Metadata
- Authors
- Tishby, N., & Zaslavsky, N.
- Venue
- 2015 IEEE Information Theory Workshop (ITW)
- Publisher
- IEEE
- Published
- 2015-04
- Pages
- 1-5
- OpenAlex
- W2096516049
- Cited by
- 1,409
- References
- 10
- FWCI
- 18.37
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.