Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Chao, A., Gotelli, N. J., Hsieh, T. C., Sander, E. L., Ma, K. H., Colwell, R. K., & Ellison, A. M. (2014) · Ecological Monographs · doi:10.1890/13-0133.1 · Cited by 4,256
“Asymptotic estimators for Hill numbers with q ¼ 1 are closely related to the well-known entropy estimation (for reviews, see Paninski 2003, Chao et al. 2013).”
Opening the Black Box of Deep Neural Networks via Information
Shwartz-Ziv, R., & Tishby, N. (2017) · arXiv · doi:10.48550/arxiv.1703.00810
“Good overviews on methods for mutual information estimation can be found in Paninski (2003) and Kraskov et al. (2004).”
MethodA comprehensive survey on sentiment analysis: Approaches, challenges and trends
Birjali, M., Kasri, M., & Beni-Hssane, A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107134 · Cited by 921
Conditional Likelihood Maximisation: A Unifying Framework for Mutual Information Feature Selection
Brown, G., Pocock, A., Zhao, M., & Luján, M. (2012) · Research Explorer (The University of Manchester) · Cited by 1,359
Mutual Information Neural Estimation.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, D. (2018) · International Conference on Machine Learning · Cited by 920
Causality detection based on information-theoretic approaches in time series analysis
HLAVACKOVASCHINDLER, K., PALUS, M., VEJMELKA, M., & BHATTACHARYA, J. (2007) · Physics Reports · doi:10.1016/j.physrep.2006.12.004 · Cited by 800
“Unfortunately, the variance of the corresponding estimator turns out to be very large [167].”
BackgroundMethodInfluentialMINE: Mutual Information Neural Estimation
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, R. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.04062 · Cited by 405
“Despite being a pivotal quantity across data science, mutual information has historically been difficult to compute (Paninski, 2003).”
BackgroundEquitability, mutual information, and the maximal information coefficient
Kinney, J. B., & Atwal, G. S. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1309933111 · Cited by 667
“Estimates of mutual information that rely on this simple binning procedure are commonly called “naive” estimates (22).”
BackgroundThe communicative function of ambiguity in language
Piantadosi, S. T., Tily, H., & Gibson, E. (2012) · Cognition · doi:10.1016/j.cognition.2011.10.004 · Cited by 628
“Figure 1b also shows that lower negative log probability (higher frequency) phonological forms tend to be mapped to many more word meanings than higher negative log probability (lower frequency) phonological forms, across all languages.”
Psychological entropy: a framework for understanding uncertainty-related anxiety
Hirsh, J. B., Mar, R. A., & Peterson, J. B. (2012) · Psychological Review · doi:10.1037/a0026767 · Cited by 606
Gene Regulatory Network Inference from Single-Cell Data Using Multivariate Information Measures
Chan, T. E., Stumpf, M. P. H., & Babtie, A. C. (2017) · Cell Systems · doi:10.1016/j.cels.2017.08.014 · Cited by 578
Information Transduction Capacity of Noisy Biochemical Signaling Networks
Cheong, R., Rhee, A., Wang, C. J., Nemenman, I., & Levchenko, A. (2011) · Science · doi:10.1126/science.1204553 · Cited by 547
Meyer, P. E., Lafitte, F., & Bontempi, G. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-461 · Cited by 510
“The empirical estimator (also called "plug-in", "maximum likelihood" or "naïve", see [23]) is the entropy of the empirical distribution.”
BackgroundSynergy, Redundancy, and Independence in Population Codes
Schneidman, E., Bialek, W., & Berry, M. J. (2003) · The Journal of Neuroscience · doi:10.1523/jneurosci.23-37-11539.2003 · Cited by 497
“Many authors have explored strategies for estimating spike train entropies (Treves and Panzeri, 1995; Strong et al., 1998; Victor, 2002; Nemenman et al., 2003; Paninski, 2003), and there is continuing interest in finding improved strategies.”
BackgroundHow Efficiency Shapes Human Language
Gibson, E., Futrell, R., Piantadosi, S. P., Dautriche, I., Mahowald, K., Bergen, L., & Levy, R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.02.003 · Cited by 485
“However, it is difficult to get reliable estimates of these information-theoretic quantities without very large datasets [162, 163].”
On Variational Bounds of Mutual Information
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. (2019) · arXiv · doi:10.48550/arxiv.1905.06922 · Cited by 480
“Copyright 2019 by the author(s). samples but not the underlying distributions (Paninski, 2003; McAllester & Stratos, 2018).”
BackgroundThe Structure of Multi-Neuron Firing Patterns in Primate Retina
Shlens, J., Field, G. D., Gauthier, J. L., Grivich, M. I., Petrusca, D., Sher, A., Litke, A. M., & Chichilnisky, E. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.1282-06.2006 · Cited by 468
“This estimation problem is intimately related to estimating the entropy of a distribution, as well as applications in universal source coding, and is the subject of intense investigation (Nemenman et al., 2002; Paninski, 2003; Orlitsky et al., 2004).”
Comparison of co-expression measures: mutual information, correlation, and model based indices
Song, L., Langfelder, P., & Horvath, S. (2012) · BMC Bioinformatics · doi:10.1186/1471-2105-13-328 · Cited by 460
Modeling Single-Neuron Dynamics and Computations: A Balance of Detail and Abstraction
Herz, A. V. M., Gollisch, T., Machens, C. K., & Jaeger, D. (2006) · Science · doi:10.1126/science.1127240 · Cited by 451
The physics of brain network structure, function and control
Lynn, C. W., & Bassett, D. S. (2019) · Nature Reviews Physics · doi:10.1038/s42254-019-0040-8 · Cited by 400
Kullback-Leibler divergence estimation of continuous distributions
Perez-Cruz, F. (2008) · 2008 IEEE International Symposium on Information Theory · doi:10.1109/isit.2008.4595271 · Cited by 400
“Other methods concentrate on estimating the divergence for discrete random variables [19], [4], but we will not discuss them further as they lie outside the scope of this paper.”
BackgroundMethodDynamic Population Coding of Category Information in Inferior Temporal and Prefrontal Cortex
Meyers, E. M., Freedman, D. J., Kreiman, G., Miller, E. K., & Poggio, T. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90248.2008 · Cited by 419
“Although there exists potentially more accurate methods for estimating mutual information (Paninski 2003; Shlens et al. 2007), because our results do not depend critically on the exact MI values, we preferred the simplicity of this method.”
JIDT: An Information-Theoretic Toolkit for Studying the Dynamics of Complex Systems
Lizier, J. T. (2014) · Frontiers in Robotics and AI · doi:10.3389/frobt.2014.00011 · Cited by 390
Composable Security Proof for Continuous-Variable Quantum Key Distribution with Coherent States
Leverrier, A. (2015) · Physical Review Letters · doi:10.1103/physrevlett.114.070501 · Cited by 382
Ince, R. A. A., Giordano, B. L., Kayser, C., Rousselet, G. A., Gross, J., & Schyns, P. G. (2017) · Human Brain Mapping · doi:10.1002/hbm.23471 · Cited by 366
“A wide variety of approaches have been proposed to address this problem [Paninski, 2003; Panzeri et al., 2007].”
BackgroundInformation Processing in the Primate Retina: Circuitry and Coding
Field, G. D., & Chichilnisky, E. J. (2007) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.30.051606.094252 · Cited by 365
Cross-Modal Contrastive Learning for Text-to-Image Generation
Zhang, H., Koh, J. Y., Baldridge, J., Lee, H., & Yang, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00089 · Cited by 343
“As directly maximizing the mutual information is challenging [37, 3, 50], the InfoNCE loss [36] was proposed to maximize a lower bound of the mutual information I(v1; v2).”
BackgroundCorrecting for the sampling bias problem in spike train information measures
Panzeri, S., Senatore, R., Montemurro, M. A., & Petersen, R. S. (2007) · Journal of Neurophysiology · doi:10.1152/jn.00559.2007 · Cited by 378
“Rigorous studies of the trade-off between bias and variance are reported in Paninski (2003) and Nemenman et al. (2004).”
InfluentialText feature extraction based on deep learning: a review
Liang, H., Sun, X., Sun, Y., & Gao, Y. (2017) · EURASIP Journal on Wireless Communications and Networking · doi:10.1186/s13638-017-0993-1 · Cited by 373
“Mutual information, originally a concept in information theory, is applied to represent relationships between information and the statistical measurement of correlation of two random variables [13, 14].”
BackgroundMethodOf passwords and people: measuring the effect of password-composition policies
Komanduri, S., Shay, R., Kelley, P. G., Mazurek, M. L., Bauer, L., Christin, N., Cranor, L. F., & Egelman, S. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979321 · Cited by 345
“Traditional trigram methods, when applied to passwords, require thousands of samples to calculate an accurate estimate [11].”
BackgroundMethodDistinct contribution of adult-born hippocampal granule cells to context encoding
Danielson, N. B., Kaifosh, P., Zaremba, J. D., Lovett-Barron, M., Tsai, J., Denny, C. A., Balough, E. M., Goldberg, A. R., Drew, L. J., Hen, R., et al. (2016) · Neuron · doi:10.1016/j.neuron.2016.02.019 · Cited by 352
“, 1993), based on the method of sieves, is a highly biased estimator in the setting of sparse firing rates, as were observed in our data (Paninski, 2003), therefore it was not possible to quantify absolute spatial information rates in these cells.”
Kernel Methods for Measuring Independence
Gretton, A., Herbrich, R., Smola, A. J., Bousquet, O., & Schölkopf, B. (2005) · ANU Open Research (Australian National University) · doi:10.5555/1046920.1194914 · Cited by 332
Vision-Language Pre-Training with Triple Contrastive Learning
Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., & Huang, J. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01522 · Cited by 309
“However, the MI estimation of high dimensional random variables is quite difficult and intractable [31], especially for deep neural networks.”
BackgroundAnalyzing Neural Responses to Natural Signals: Maximally Informative Dimensions
Sharpee, T., Rust, N. C., & Bialek, W. (2004) · Neural Computation · doi:10.1162/089976604322742010 · Cited by 298
“The true value I spike can be found either by subtracting an expected bias value, which is of the order of ∼ 1 / ( P (spike) N 2 ln 2) (Treves & Panzeri, 1995; Panzeri & Treves, 1996; Pola et al., 2002; Paninski, 2003b), or by extrapolating to N → ∞ (Brenner et al., 2000b; Strong et al., 1998).”
BackgroundMethodInfluentialAdvances in Quantitative Electroencephalogram Analysis Methods
Thakor, N. V., & Tong, S. (2004) · Annual Review of Biomedical Engineering · doi:10.1146/annurev.bioeng.5.040202.121601 · Cited by 305
Daub, C. O., Steuer, R., Selbig, J., & Kloska, S. (2004) · BMC Bioinformatics · doi:10.1186/1471-2105-5-118 · Cited by 286
“Recently, an entropy estimator was suggested [21] and showed in an extensive comparison to other commonly used estimators to be superior.”
ResultA Comprehensive Evaluation of Mutual Information Analysis Using a Fair Evaluation Framework
Whitnall, C., & Oswald, E. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-22792-9_18 · Cited by 283
“Unfortunately, estimators for MI do not ‘behave nicely’ as do other statistics (such as the correlation coefficient—see below); in fact, there are no universal rates of convergence [14], so that whatever estimator we pick, we can always find a distribution for which the error vanishes arbitrarily slowly.”
BackgroundInformation-Theoretic Feature Selection in Microarray Data Using Variable Complementarity
Meyer, P. E., Schretter, C., & Bontempi, G. (2008) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2008.923858 · Cited by 280
“This entropy estimation method is known to be biased [31].”
MethodQuinn, C. J., Coleman, T. P., Kiyavash, N., & Hatsopoulos, N. G. (2011) · Journal of Computational Neuroscience · doi:10.1007/s10827-010-0247-2 · Cited by 273
“2009), mutual information (Paninski 2003), and coherence (Salvador et al.”
Information-Theoretic Inference of Large Transcriptional Regulatory Networks
Meyer, P. E., Kontos, K., Lafitte, F., & Bontempi, G. (2007) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1155/2007/79879 · Cited by 273
“Several estimators of mutual information have been proposed in literature [5, 6, 20, 21].”
BackgroundMethodA Neural Basis for Motor Primitives in the Spinal Cord
Hart, C. B., & Giszter, S. F. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.5894-08.2010 · Cited by 263
“Mutual information (MI) is a measure of how much of the Shannon information is held in common between two or more channels, or neurons (Moddemeijer, 1989, 1997, 1999; Paninski, 2003).”
BackgroundMethodPredicting the birth of a spoken word
Roy, B. C., Frank, M. C., DeCamp, P., Miller, M., & Roy, D. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1419773112 · Cited by 250
“Paninski L (2003) Estimation of entropy and mutual information.”
BackgroundValiant, G., & Valiant, P. (2011) · Proceedings of the forty-third annual ACM symposium on Theory of computing · doi:10.1145/1993636.1993727 · Cited by 233
“In a different direction, Paninski gave a simple though non–constructive proof of the existence of a sublinear sample estimator for additively approximating the entropy to within a constant; the proof is via a direct application of the Stone-Weierstrass theorem to the set of Poisson functions [33, 34].”
BackgroundMethodA Tutorial for Information Theory in Neuroscience
Timme, N. M., & Lapish, C. (2018) · eneuro · doi:10.1523/eneuro.0052-18.2018 · Cited by 242
Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective
Timme, N., Alford, W., Flecker, B., & Beggs, J. M. (2014) · Journal of Computational Neuroscience · doi:10.1007/s10827-013-0458-4 · Cited by 226
“…for some, but not all, of the information measures discussed herein (Treves and Panzeri 1995; Panzeri and Treves 1996; Strong et al. 1997; Paninski 2003; Schneidman et al. 2003a; Nemenman et al. 2004; Kennel et al. 2005; Averbeck et al. 2006; HlaváčkováSchindler et al. 2007; Panzeri…”
BackgroundSynergy, Redundancy, and Independence in Population Codes, Revisited
Latham, P. E., & Nirenberg, S. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.5319-04.2005 · Cited by 232
“This is because one does not actually have to calculate ⌬I [a difficult estimation problem, especially for population codes (Paninski, 2003, 2004; Nemenman et al., 2004)], but instead one can build decoders that do and do not take some aspect of correlations into account.”
BackgroundChen, S., & Mar, J. C. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2217-z · Cited by 235
“We used the R package minet, which calculates MI by equal-width binning for discretization and empirical entropy estimation as described in [33].”
MethodA toolbox for the fast information analysis of multiple-site LFP, EEG and spike train recordings
Magri, C., Whittingstall, K., Singh, V., Logothetis, N. K., & Panzeri, S. (2009) · BMC Neuroscience · doi:10.1186/1471-2202-10-81 · Cited by 226
“Consequently, entropy estimates are lower than their true values [34].”
BackgroundInformation Processing in Living Systems
Tkačik, G., & Bialek, W. (2016) · Annual Review of Condensed Matter Physics · doi:10.1146/annurev-conmatphys-031214-014803 · Cited by 215
“[14] L Paninski, Estimation of entropy and mutual information.”
BackgroundEcohydrologic process networks: 1. Identification
Ruddell, B. L., & Kumar, P. (2009) · Water Resources Research · doi:10.1029/2008wr007279 · Cited by 217
“A good mathematical review is provided by Paninski [2003] .”
MethodMinimax Estimation of Functionals of Discrete Distributions
Jiao, J., Venkat, K., Han, Y., & Weissman, T. (2015) · IEEE Transactions on Information Theory · doi:10.1109/tit.2015.2412945 · Cited by 194
“We thank Adel Javanmard for discussions related to constructing confidence intervals for high dimensional statistical problems, and Liam Paninski for interesting discussions related to Paninski [20].”
BackgroundMethodInfluentialPractical Measures of Integrated Information for Time-Series Data
Barrett, A. B., & Seth, A. K. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001052 · Cited by 225
“Again, future work could investigate the applicability of more advanced techniques [53,54] that avoid this step.”
BackgroundCausal Network Inference by Optimal Causation Entropy
Sun, J., Taylor, D., & Bollt, E. M. (2015) · SIAM Journal on Applied Dynamical Systems · doi:10.1137/140956166 · Cited by 218
“While there are several techniques for estimating entropies for general multivariate data, such estimations are increasingly inaccurate for small sample sizes and high-dimensional random variables [52].”
MethodMutual information: a measure of dependency for nonlinear time series
Dionisio, A., Menezes, R., & Mendes, D. A. (2004) · Physica A: Statistical Mechanics and its Applications · doi:10.1016/j.physa.2004.06.144 · Cited by 213
Information and efficiency in the nervous system-a synthesis.
Sengupta, B., Stemmler, M. B., & Friston, K. J. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003157 · Cited by 212
“Although, sophisticated machine learning tools try to estimate mutual information from limited data [53–55], the numerics of mutual information are fraught with difficulties.”
BackgroundAdjusting for Chance Clustering Comparison Measures
Romano, S., Vinh, N. X., Bailey, J., & Verspoor, K. M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.01286 · Cited by 192
Minimax Rates of Entropy Estimation on Large Alphabets via Best Polynomial Approximation
Wu, Y., & Yang, P. (2016) · IEEE Transactions on Information Theory · doi:10.1109/tit.2016.2548468 · Cited by 173
“For (12) we need to choose a pair of priors, which, in this case, are distributions on the probability simplex M k , is to ensure the entropy values are separated.”
BackgroundMethodInfluentialFixing a Broken ELBOPreprint
Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., & Murphy, K. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00464 · Cited by 496
“Computing mutual information in high dimensions is problematic (Paninski, 2003; Zheng & Benjamini, 2016), thus we turn to recently developed tools in variational inference to approximate it.”
BackgroundHeiny, A. T., Miotto, O., Srinivasan, K. N., Khan, A. M., Zhang, G. L., Brusic, V., Tan, T. W., & August, J. T. (2007) · PLoS ONE · doi:10.1371/journal.pone.0001190 · Cited by 196
“It has been shown that, for an alignment of N sequences, alignment size bias is proportional to 1/N [70].”
BackgroundEntropy and information in neural spike trains: Progress on the sampling problem
Nemenman, I., Bialek, W., & de Ruyter van Steveninck, R. (2004) · Physical Review E · doi:10.1103/physreve.69.056111 · Cited by 196
Precision of Spike Trains in Primate Retinal Ganglion Cells
Uzzell, V. J., & Chichilnisky, E. J. (2004) · Journal of Neurophysiology · doi:10.1152/jn.01171.2003 · Cited by 192
“Two other models have been proposed to reproduce the trial-to-trial variability of visual system neurons: the leaky integrate and fire model (Pillow et al. 2003; Reich et al. 1997) and a more elaborate model involving nonlinear feedback and two noise sources (Keat et al. 2001).”
MethodVoita, E., Sennrich, R., & Titov, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1448 · Cited by 127
“Even though the information-theoretic view provides insights into processes shaping the representations, direct MI estimation from finite samples for densities on multi-dimensional spaces is challenging (Paninski, 2003).”
BackgroundRNA Interference and its Role in Cancer Therapy
Mansoori, B., Shotorbani, S. S., & Baradaran, B. (2014) · PubMed · doi:10.5681/apb.2014.046 · Cited by 180
Verified Uncertainty Calibration
Kumar, A., Liang, P., & Ma, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10155 · Cited by 131
“This is analogous to the difficulty of measuring the mutual information between two continuous signals [27].”
BackgroundLossy‐Context Surprisal: An Information‐Theoretic Model of Memory Effects in Sentence Processing
Futrell, R., Gibson, E., & Levy, R. P. (2020) · Cognitive Science · doi:10.1111/cogs.12814 · Cited by 160
Mutual information is copula entropy
Ma, J., & Sun, Z. (2011) · Tsinghua Science and Technology · doi:10.1016/s1007-0214(11)70008-6 · Cited by 171
Information theory: a tutorial introduction
(2016) · Choice Reviews Online · doi:10.5860/choice.195473 · Cited by 183
The Information Bottleneck Problem and its Applications in Machine Learning
Goldfeld, Z., & Polyanskiy, Y. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991561 · Cited by 165
“However, mutual information estimation from highdimensional data is a notoriously difficult [75].”
BackgroundDivergence Estimation of Continuous Distributions Based on Data-Dependent Partitions
Wang, Q., Kulkarni, S. R., & Verdu, S. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.853314 · Cited by 180
Deep Brain Stimulation Reduces Neuronal Entropy in the MPTP-Primate Model of Parkinson's Disease
Dorval, A. D., Russo, G. S., Hashimoto, T., Xu, W., Grill, W. M., & Vitek, J. L. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90763.2008 · Cited by 172
“Estimates were verified (data not shown) for qualitative similarity with results using the MillerMadow bias correction (Miller 1955; Paninski 2003) and the Nemenman-Shafee-Bialek estimate (Nemenman et al. 2002).”
BackgroundMethodChao, A., Wang, Y. T., & Jost, L. (2013) · Methods in Ecology and Evolution · doi:10.1111/2041-210x.12108 · Cited by 169
“Paninski (2003) proposed a ‘best upper bound’ estimator which minimizes an upper bound on the mean squared error.”
BackgroundLearning and generalization with the information bottleneck
Shamir, O., Sabato, S., & Tishby, N. (2010) · Theoretical Computer Science · doi:10.1016/j.tcs.2010.04.006 · Cited by 165
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171
“MI can be defined for continuous O and S but is more expensive to estimate, especially in high dimensions [39], [40].”
BackgroundFairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
“It can be defined for continuous O and S but is more expensive to estimate, especially in high dimensions [32, 35].”
BackgroundMaximizing the information content of experiments in systems biology
Liepe, J., Filippi, S., Komorowski, M., & Stumpf, M. P. H. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002888 · Cited by 170
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
BackgroundNatural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C.-N., Ho, S.-W., & Tan, C.-W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 134
“Analytical results of estimators for entropy and mutual information can be found in [73].”
BackgroundCavagnaro, D. R., Myung, J. I., Pitt, M. A., & Kujala, J. V. (2009) · Neural Computation · doi:10.1162/neco.2009.02-09-959 · Cited by 212
Estimating the Entropy Rate of Spike Trains via Lempel-Ziv Complexity
Amigó, J. M., Szczepański, J., Wajnryb, E., & Sanchez-Vives, M. V. (2004) · Neural Computation · doi:10.1162/089976604322860677 · Cited by 160
“(See Shannon, 1948; Cover & Thomas, 1991, for the general theory of information and Rieke, Warland, de Ruyter van Steveninck, & Bialek, 1998; Borst and Theunissen, 1999; Paninski, 2003 for more elaborated and practical discussions in relation to computational neuroscience.)”
BackgroundUnderstanding autoencoders with information theoretic concepts
Yu, S., & Príncipe, J. C. (2019) · Neural Networks · doi:10.1016/j.neunet.2019.05.003 · Cited by 183
Wavelet Feature Selection for Image Classification
Huang, K., & Aviyente, S. (2008) · IEEE Transactions on Image Processing · doi:10.1109/tip.2008.2001050 · Cited by 162
“Usually, the ranges of and are partitioned into and intervals, respectively [39], [ 40 ].”
BackgroundEncoding Stimulus Information by Spike Numbers and Mean Response Time in Primary Auditory Cortex
Nelken, I., Chechik, G., Mrsic‐Flogel, T. D., King, A. J., & Schnupp, J. W. H. (2005) · Journal of Computational Neuroscience · doi:10.1007/s10827-005-1739-3 · Cited by 159
“This approach is similar to the method of sieves used by Paninski (2003) in his theoretical analysis of entropy and MI estimation from experimentallymeasured count matrices.”
BackgroundMethodInfluentialConvergence properties of three spike-triggered analysis techniques
Paninski, L. (2003) · Network Computation in Neural Systems · doi:10.1088/0954-898x_14_3_304 · Cited by 172
Self-citationZhao, W., Serpedin, E., & Dougherty, E. R. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl364 · Cited by 149
“Recent progress in estimating information theoretic quantities has led to a number of good estimators in this regard (Beirlant et al ., 1997; Paninski, 2003, 2004; Treves and Panzeri, 1995).”
Networks in financial markets based on the mutual information rate
Fiedor, P. (2014) · Physical Review E · doi:10.1103/physreve.89.052801 · Cited by 168
Encoding uncertainty in the hippocampus
Harrison, L. M., Duggins, A., & Friston, K. J. (2006) · Neural Networks · doi:10.1016/j.neunet.2005.11.002 · Cited by 155
“Recent work has shown that there are intelligent Bayesian priors that dramatically reduce the bias in the calculation of entropy from small data-sets (Nemenman & Bialek, 2002; Paninski, 2003).”
BackgroundOn directed information theory and Granger causality graphs
Amblard, P., & Michel, O. (2010) · Journal of Computational Neuroscience · doi:10.1007/s10827-010-0231-x · Cited by 170
“Phys Rev E 046211:1–6 [38] Paninski L (2003) Estimation of entropy and mutual information.”
BackgroundEstimating Entropy on $m$ Bins Given Fewer Than $m$ Samples
Paninski, L. (2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.833360 · Cited by 148
“It is worth noting that the above proofs can be easily strengthened to almost sure convergence, given a suitably chosen probability space; see [2] for an example of such a probability space, along with the exponential tail probability inequalities (derived, again via McDiarmid) sufficient for the application of the Bo…”
BackgroundMethodInfluentialSelf-citationInformation theory applications for biological sequence analysis
Vinga, S. (2013) · Briefings in Bioinformatics · doi:10.1093/bib/bbt068 · Cited by 147
“Alternative methods based on thorough descriptions of the statistical and convergence properties of different estimators [70] might also support future applications in DNA and proteins.”
MethodSantos, S. D. S., Takahashi, D. Y., Nakata, A., & Fujita, A. (2013) · Briefings in Bioinformatics · doi:10.1093/bib/bbt051 · Cited by 161
“As an analytical statistical test for MI is in general not available [15], the significance of the MI should be tested by assuming strong constraints on the data or by using a permutation procedure.”
MethodThe Relation between Granger Causality and Directed Information Theory: A Review
Amblard, P., & Michel, O. (2012) · Entropy · doi:10.3390/e15010113 · Cited by 166
“We refer to [47], [10], [68], [46], [27] for inform ation on the different ways to estimate information measures.”
BackgroundGao, X., Gao, F., Tao, D., & Li, X. (2013) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2013.2271356 · Cited by 137
Neural Coding of Natural Stimuli: Information at Sub-Millisecond Resolution
Nemenman, I., Lewen, G. D., Bialek, W., & de Ruyter van Steveninck, R. R. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000025 · Cited by 142
Estimating Information Flow in Deep Neural NetworksPreprint
Goldfeld, Z., van den Berg, E., Greenewald, K. H., Melnyk, I., Nguyen, N. H., Kingsbury, B. E. D., & Polyanskiy, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.05728 · Cited by 141
“But, these benefits come at a price: mutual information is often impossible to compute analytically, and its estimation from samples is difficult (Paninski, 2003).”
BackgroundRobust Sensitivity Analysis for Stochastic Systems
Lam, H. (2016) · Mathematics of Operations Research · doi:10.1287/moor.2015.0776 · Cited by 198
“[4] for a review of older works, Paninski [45], and more recently Liu et al.”
MethodTo Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review
Shwartz-Ziv, R., & LeCun, Y. (2024) · Entropy · doi:10.3390/e26030252 · Cited by 195
“This problem has been extensively studied [143,144], revealing the inefficiency of solutions for large dimensions and the limited scalability of known approximations with respect to the sample size and dimension.”
BackgroundBagging Statistical Network Inference from Large-Scale Gene Expression Data
Simoes, R. D. M., & Emmert‐Streib, F. (2012) · PLoS ONE · doi:10.1371/journal.pone.0033624 · Cited by 154
“Mutual Information is frequently estimated from the marginal and joint entropy H of two discretized random variables X and Y [42], In our study, we use four MI estimators based on continuous data and four MI estimators based on discretized data.”
BackgroundMethodQuantifying aerosol mixing state with entropy and diversity measures
Riemer, N., & West, M. (2013) · Atmospheric chemistry and physics · doi:10.5194/acp-13-11423-2013 · Cited by 146
“(2010); Leinster and Cobbold (2012); Feoli (2012); Scheiner (2012), and techniques for measuring diversity (Chao and Shen, 2003; Schmera and Podani, 2013; Gotelli and Chao, 2013), despite the well-known difficulties in estimating entropy in an unbiased fashion (Har5 ris, 1975; Paninski, 2003).”
BackgroundEfficient multivariate entropy estimation via $k$-nearest neighbour distances
Berrett, T. B., Samworth, R. J., & Yuan, M. (2018) · The Annals of Statistics · doi:10.1214/18-aos1688 · Cited by 182
“Paninski and Yajima (2008), for example, show that a smaller bandwidth than would be required for a consistent density estimator can still yield a consistent entropy estimator.”
BackgroundMethodSimple Unsupervised Graph Representation Learning
Mo, Y., Peng, L., Xu, J., Shi, X., & Zhu, X. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i7.20748 · Cited by 199
“In the literature, considering the computation cost of MI maximization (Paninski 2003; Belghazi et al. 2018), the MI maximization in the SUGRL is transferred to contrastive learning, which involves the definitions of anchor embeddings, positive and negative embeddings, as well as the contrastive…”
Pillow, J. W., Ahmadian, Y., & Paninski, L. (2010) · Neural Computation · doi:10.1162/neco_a_00058 · Cited by 133
“Finally, the idea that we might use model-based approaches to computing the information I[x; r] (a problem which is otherwise quite difficult when considered nonparametrically (Paninski, 2003)) has appeared in several earlier works (Harris et al.”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.