Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 101–200 of 1,004
Bipartite Graph Embedding via Mutual Information Maximization
Cao, J., Lin, X., Guo, S., Liu, L., Liu, T., & Wang, B. (2021) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3437963.3441783 · Cited by 162
“een inputs and corresponding latent embeddings provides a desirable paradigm for the unsupervised learning [35]. However, estimating MI is generally intractable in highdimensional continuous settings [25]. MINE [1] derives a lower bound of MI and works by training a discriminator to distinguish samples coming from the…”
BackgroundDoes My Device Leak Information? An a priori Statistical Power Analysis of Leakage Detection Tests
Mather, L., Oswald, E., Bandenburg, J., & Wójcik, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-42033-7_25 · Cited by 142
“There are no unbiased estimators, and it has been proven that there is no estimator that does not perform di erently depending on the underlying structure of the data [22].”
BackgroundAuditory Cortical Detection and Discrimination Correlates with Communicative Significance
Liu, R. C., & Schreiner, C. E. (2007) · PLoS Biology · doi:10.1371/journal.pbio.0050173 · Cited by 130
“It should be pointed out that when probability distributions must be inferred from finite trials, all information estimates are subject to bias [52].”
BackgroundMethodTesting the Efficiency of Sensory Coding with Optimal Stimulus Ensembles
Machens, C. K., Gollisch, T., Kolesnikova, O., & Herz, A. V. M. (2005) · Neuron · doi:10.1016/j.neuron.2005.06.015 · Cited by 146
“In the future, however, it may be possible to include biascorrection techniques (Strong et al., 1998; Paninski, 2003; Hsu et al., 2004) within the online analysis.”
BackgroundThe Power of Linear Estimators
Valiant, G. J., & Valiant, P. (2011) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2011.81 · Cited by 129
“Nevertheless, in [21], [22], Paninski proved the existence of a sublinear-sample estimator; the proof is non-constructive, via a direct application of the Stone-Weierstrass theorem to the set of Poisson functions.”
BackgroundMethodInfluentialA Mutual Information Maximization Perspective of Language Representation Learning
Kong, L., d’Autume, C. D. M., Wang, L., Yu, L., Dai, Z., & Yogatama, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08350 · Cited by 101
“Maximizing mutual information directly is generally intractable when the function f consists of modern encoders such as neural networks (Paninski, 2003), so we need to resort to a lower bound on I(A,B).”
BackgroundEstimating the Entropy of Binary Time Series: Methodology, Some Theory and a Simulation Study
Gao, Y., Kontoyiannis, I., & Bienenstock, E. (2008) · Entropy · doi:10.3390/entropy-e10020071 · Cited by 148
Motor control by precisely timed spike patterns
Srivastava, K. H., Holmes, C. M., Vellema, M., Pack, A. R., Elemans, C. P. H., Nemenman, I. M., & Sober, S. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611734114 · Cited by 127
“2A) using an estimator of mutual information (16) that we developed specifically for this purpose.”
BackgroundMethodHow Efficiency Shapes Human Language
Gibson, E., Futrell, R., Piantadosi, S. T., Dautriche, I., Mahowald, K., Bergen, L., & Levy, R. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.09.005 · Cited by 105
The quantity and quality of information in hydrologic models
Nearing, G., & Gupta, H. V. (2015) · Water Resources Research · doi:10.1002/2014wr015895 · Cited by 134
“Equation (12) was estimated at each evaluation time step using the method of sieves [Paninski, 2003] with bin width according to Scott’s rule [Scott, 2004].”
MethodMulti-Source Causal Feature Selection
Yu, K., Liu, L., Li, J., Ding, W., & Le, T. D. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2908373 · Cited by 125
“(5), it is difficult to calculate P (C|S) especially for a large sized S [17].”
BackgroundTrust Evaluation in Online Social Networks Using Generalized Network Flow
Jiang, W., Wu, J., Li, F., Wang, G., & Zheng, H. (2015) · IEEE Transactions on Computers · doi:10.1109/tc.2015.2435785 · Cited by 132
“(2) Taking the minimum. tðs; dÞ ¼ minftðs; v1Þ; tðvi; viþ1Þ; . . . ; tðvn; dÞg ¼ 1.”
Maximum Likelihood Estimation of Functionals of Discrete Distributions
Jiao, J., Venkat, K., Han, Y., & Weissman, T. (2017) · IEEE Transactions on Information Theory · doi:10.1109/tit.2017.2733537 · Cited by 137
“We gain one of our key insights into the bias of the MLE by relating it to the approximation error induced by the Bernstein polynomial approximation of the function f , which was first observed in Paninski [14].”
InfluentialFeature Selection via Mutual Information: New Theoretical Insights
Beraha, M., Metelli, A. M., Papini, M., Tirinzoni, A., & Restelli, M. (2019) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2019.8852410 · Cited by 123
“Estimating the MI can be reduced to the estimation of several entropies [21]; numerous methods have been employed in feature selection, either based on nearest neighbors approaches [22] or on histograms [15].”
MethodSuperlinear Population Encoding of Dynamic Hand Trajectory in Primary Motor Cortex
Paninski, L., Shoham, S., Fellows, M. R., Hatsopoulos, N. G., & Donoghue, J. P. (2004) · The Journal of Neuroscience · doi:10.1523/jneurosci.0919-04.2004 · Cited by 121
“5 ) ; the median ratio of information contained in k ជ ⅐ w ជ relative to the information in k ជ 2 ⅐ w ជ , with k ជ 2 being the next most informative filter orthogonal to k ជ , was Ն10 ( for additional details , see Paninski , 2003a ) .”
BackgroundMethodInfluentialSelf-citationMiotto, O., Heiny, A. T., Tan, T. W., August, J. T., & Brusić, V. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-s1-s18 · Cited by 129
“Previous studies [24] indicate that smaller set sizes introduce an error which is inversely proportional to the sequence count.”
BackgroundInformation theory in neuroscience
Dimitrov, A. G., Lazar, A. A., & Victor, J. D. (2011) · Journal of Computational Neuroscience · doi:10.1007/s10827-011-0314-3 · Cited by 118
“This diversity proved important, as Paninski (2003) proved an inconsistency theorem showing that most common estimation techniques can encounter conditions that lead to arbitrary poor estimates.”
BackgroundCellular noise and information transmission
Levchenko, A., & Nemenman, I. M. (2014) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2014.05.002 · Cited by 134
“Further, since information measures all dependences in data, it requires a lot larger data sets to be estimated reliably compared to simpler, linear measures of dependency [62–69], which makes experiments even harder.”
Novelli, L., Wollstadt, P., Mediano, P. A. M., Wibral, M., & Lizier, J. T. (2019) · Network Neuroscience · doi:10.1162/netn_a_00092 · Cited by 130
“Information-theoretic estimators suffer from the “curse of dimensionality” for large sets of variables (Paninski, 2003; Roulston, 1999);”
BackgroundResolving the Generalized Bas-Relief Ambiguity by Entropy Minimization
Alldrin, N., Mallick, S. P., & Kriegman, D. J. (2007) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2007.383208 · Cited by 115
“Given these facts, it seems finding the minima 3 See [12] for a bias corrected estimator. will be very difficult.”
BackgroundApproaches to Information-Theoretic Analysis of Neural Activity
Victor, J. D. (2006) · Biological Theory · doi:10.1162/biot.2006.1.3.302 · Cited by 132
“These include Paninski's estimator (Paninski 2003), which is provably the least biased of all polynomial estimators, the " KT " (Krichevsky and Trofimov 1981) and " SG " (Schurmann and Grassberger 1996) estimators, which are based on single Dirichlet priors (Wolpert and Wolf 1995), the " NSB "…”
BackgroundMaximum-Entropy Adversarial Data Augmentation for Improved Generalization and RobustnessPreprint
Zhao, L., Liu, T., Peng, X., & Metaxas, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.08001 · Cited by 204
“However, mutual information is shown to be intractable in the literature [7, 47, 53], and therefore directly optimizing this objective is challenging.”
BackgroundGolino, H. F., Moulder, R. G., Jr., Shi, D., Christensen, A. P., Garrido, L. E., Nieto, M. D., Nesselroade, J. R., Sadana, R., Thiyagarajan, J. A., & Boker, S. M. (2020) · Multivariate Behavioral Research · doi:10.1080/00273171.2020.1779642 · Cited by 128
“In terms of estimation, several estimators are available for entropy (especially empirical entropy or Shannon entropy; Paninski, 2003).”
BackgroundOrdinal symbolic analysis and its application to biomedical recordings
Amigó, J. M., Keller, K., & Unakafova, V. A. (2014) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2014.0091 · Cited by 111
“Note in this connection that algebraic correction terms to finite size effects can be found in [68,69].”
BackgroundBias analysis in entropy estimation
Schürmann, T. (2004) · Journal of Physics A Mathematical and General · doi:10.1088/0305-4470/37/27/l02 · Cited by 111
Fundamental trade-offs between information flow in single cells and cellular populations
Suderman, R., Bachman, J. A., Smith, A., Sorger, P. K., & Deeds, E. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1615660114 · Cited by 110
Wei, H., & Billings, S. A. (2008) · International Journal of Modelling Identification and Control · doi:10.1504/ijmic.2008.020543 · Cited by 117
Benchmarking NLDAS-2 Soil Moisture and Evapotranspiration to Separate Uncertainty Contributions
Nearing, G., Mocko, D. M., Peters‐Lidard, C. D., Kumar, S. V., & Xia, Y. (2016) · Journal of Hydrometeorology · doi:10.1175/jhm-d-15-0063.1 · Cited by 110
“There are many ways to numerically estimate entropy and mutual information (Paninski 2003), and here we used maximum likelihood estimators.”
MethodConservation and Variability of Dengue Virus Proteins: Implications for Vaccine Design
Khan, A. M., Miotto, O., Nascimento, E. J. M., Srinivasan, K. N., Heiny, A. T., Zhang, G., Marques, E. T. A., Tan, T. W., Brusić, V., Salmon, J., et al. (2008) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0000272 · Cited by 115
“For an alignment of N sequences, alignment size bias is proportional to 1/N [29].”
BackgroundNonstationary Frequency Analysis of Annual Maximum Rainfall Using Climate Covariates
Vasiliades, L., Γαλιατσάτου, Π., & Loukas, A. G. (2014) · Water Resources Management · doi:10.1007/s11269-014-0761-5 · Cited by 110
Entropy estimates of small data sets
Bonachela, J. A., Haye Hinrichsen, B., & Muñoz, M. A. (2008) · Journal of Physics A Mathematical and Theoretical · doi:10.1088/1751-8113/41/20/202001 · Cited by 106
Extraction of Network Topology From Multi-Electrode Recordings: Is there a Small-World Effect?
Gerhard, F., Pipa, G., Lima, B., Neuenschwander, S., & Gerstner, W. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00004 · Cited by 114
“It has been shown that this type of neuron model is equivalent to a spike response model with a particular noise term using the escape-rate approximation (Gerstner and Kistler, 2002; Paninski, 2003).”
MethodOrigin of perseveration in the trade-off between reward and complexity
Gershman, S. J. (2020) · Cognition · doi:10.1016/j.cognition.2020.104394 · Cited by 123
An Automatic Inequality Prover and Instance Optimal Identity Testing
Valiant, G., & Valiant, P. (2017) · SIAM Journal on Computing · doi:10.1137/151002526 · Cited by 93
“[6], [4], [10], [11], [12], [3], [8]), and on establishing lower bounds [14], [18].”
Estimating Entropy Rates with Bayesian Confidence Intervals
Kennel, M. B., Shlens, J., Abarbanel, H. D. I., & Chichilnisky, E. J. (2005) · Neural Computation · doi:10.1162/0899766053723050 · Cited by 115
“One result of this situation is that bias can significantly contaminate the estimate, and variances are underestimated (Miller & Madow, 1954; Nemenman et al., 2002; Paninski, 2003; Roulston, 1999; Treves & Panzeri, 1995).”
BackgroundMethodInfluentialZaidi, A., Aguerri, I. E., & Shamai, S. S. (2020) · Entropy · doi:10.3390/e22020151 · Cited by 123
“We also point out that there exists an extensive literature on building optimal estimators of information quantities (e.g. entropy, mutual information), as well as their Matlab/Python implementations, including in the high-dimensional regime (see, e.g., [44–49] and references therein).”
BackgroundThe Complexity of Approximating the Entropy
Batu, T., Dasgupta, S., Kumar, R., & Rubinfeld, R. (2005) · SIAM Journal on Computing · doi:10.1137/s0097539702403645 · Cited by 102
Quantum Semiparametric Estimation
Tsang, M., Albarelli, F., & Datta, A. (2020) · Physical Review X · doi:10.1103/physrevx.10.031023 · Cited by 126
“In classical statistics, the semiparametric estimation of entropic quantities is a well studied problem with known near-efficient estimators and applications in universal coding, statistical tests, random-number generation, econometrics, spectroscopy, and even neuroscience [48].”
BackgroundTight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques
Montemurro, M. A., Senatore, R., & Panzeri, S. (2007) · Neural Computation · doi:10.1162/neco.2007.19.11.2913 · Cited by 105
“8We also considered the performance of the procedure of Paninski (2003), which also aims at working beyond the asymptotic sampling regime.”
BackgroundYu, H., Khan, F. I., & Garaniya, V. (2015) · Industrial & Engineering Chemistry Research · doi:10.1021/ie503530v · Cited by 104
Demystifying Fixed $k$ -Nearest Neighbor Information Estimators
Gao, W., Oh, S., & Viswanath, P. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2807481 · Cited by 99
“Significant understanding of the minimax rate-optimal estimators has been attained via [13], [14], [18], [21], and [22].”
BackgroundEdgeworth Approximation of Multivariate Differential Entropy
Hulle, M. M. V. (2005) · Neural Computation · doi:10.1162/0899766054323026 · Cited by 99
“As an application of our method, we suggest mutual information (MI) estimation, since it is regarded as not an easy problem (e.g., see Paninski, 2003).”
MethodImproving the Learning of Multi-column Convolutional Neural Network for Crowd Counting
Cheng, Z.-Q., Li, J.-X., Dai, Q., Wu, X. Y., He, J.-Y., & Hauptmann, A. G. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3350898 · Cited by 104
“Since the joint distribution P C θ 1 C θ 2 and the product of marginal distributions P C θ 1 ⊗ P C θ 2 are unknown in our case, the mutual information of two columns is challenging to compute [40].”
BackgroundLearning Speaker Representations with Mutual Information
Ravanelli, M., & Bengio, Y. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2380 · Cited by 96
Nonlinear Information Processing in a Model Sensory System
Chacron, M. J. (2006) · Journal of Neurophysiology · doi:10.1152/jn.01296.2005 · Cited by 112
Benchmarking and Process Diagnostics of Land Models
Nearing, G., Ruddell, B. L., Clark, M., Nijssen, B., & Peters‐Lidard, C. D. (2018) · Journal of Hydrometeorology · doi:10.1175/jhm-d-17-0209.1 · Cited by 101
“The code was validated by comparison with a community code, ProcessNetwork version 1.5 (Ruddell 2016), and found to be in good agreement.”
MethodInfluentialSketching and Streaming Entropy via Approximation Theory
Harvey, N. J. A., Nelson, J., & Onak, K. (2008) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2008.76 · Cited by 96
“One conceivable approach to design a streaming algorithm for approximating entropy is to leverage the existing work on approximating the entropy of a discrete probability distribution from independent random samples [3, 25, 26, 29, 36].”
BackgroundMIST: Maximum Information Spanning Trees for dimension reduction of biological data sets
King, B. M., & Tidor, B. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp109 · Cited by 92
Baptista, M. S., Kakmeni, F. M. M., & Grebogi, C. (2010) · Physical Review E · doi:10.1103/physreve.82.036203 · Cited by 98
Understanding Convolutional Neural Networks With Information Theory: An Initial Exploration
Yu, S., Wickstrøm, K., Jenssen, R., & Príncipe, J. C. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2968509 · Cited by 117
“Instead, we often forget that, rarely, estimators share all the properties of the statistically defined quantities [36].”
BackgroundEstimation of KL Divergence: Optimal Minimax Rate
Bu, Y., Zou, S., Liang, Y., & Veeravalli, V. V. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2805844 · Cited by 106
“The first term in (43) is the bias of the plug-in estimator for entropy estimation, which can be bounded as in [14]:”
BackgroundInfluentialStatistical Measurement of Information Leakage
Chatzikokolakis, K., Chothia, T., & Guha, A. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-12002-2_33 · Cited by 111
“Using continuous versions of mutual information [18] it may be possible to extend our methods to handle an infinite number of possible observations, as long as they are continuous, e.”
BackgroundEntropy estimation of very short symbolic sequences
Lesne, A., Blanc, J.-L., & Pézard, L. (2009) · Physical Review E · doi:10.1103/physreve.79.046208 · Cited by 101
An Automatic Inequality Prover and Instance Optimal Identity Testing
Valiant, G. J., & Valiant, P. (2014) · Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2014.14 · Cited by 89
Extracting biomarkers of autism from MEG resting-state functional connectivity networks
Tsiaras, V., Simos, P. G., Rezaie, R., Sheth, B. R., Garyfallidis, E., Castillo, E. M., & Papanicolaou, A. C. (2011) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2011.04.004 · Cited by 96
“Given a time series {x(t)}, parameter t is the argument of the first local minimum in the mutual information between time series {x(t)} and its delayed version fxðtþtÞg [40].”
BackgroundUnsupervised Feature Learning by Cross-Level Instance-Group Discrimination
Wang, X., Liu, Z., & Yu, S. X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01240 · Cited by 115
Computational mechanisms of mechanosensory processing in the cricket
Jacobs, G. A., Miller, J. P., & Aldworth, Z. (2008) · Journal of Experimental Biology · doi:10.1242/jeb.016402 · Cited by 88
“…kernel approach assuming linearity) grossly underestimates the information content in the neural spike train, as assayed by a model-independent approach called the ‘direct method’ (Strong et al., 1998; Victor, 2002; Paninski, 2003; Nemenman et al., 2004; Kennel et al., 2005; Shlens et al., 2007).”
MethodMeasuring Disentanglement: A Review of Metrics
Carbonneau, M., Zaïdi, J., Boilard, J., & Gagnon, G. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3218982 · Cited by 132
“Even assessing the quality of the estimators remains an open problem [36].”
BackgroundEVALUATION OF MUTUAL INFORMATION ESTIMATORS FOR TIME SERIES
Papana, A., & Kugiumtzis, D. (2009) · International Journal of Bifurcation and Chaos · doi:10.1142/s0218127409025298 · Cited by 99
“Keywords: mutual information, probability density, time sries, nonlinear systems running title: Mutual Information Estimators ∗Email: agpapana@gen.auth.gr”
BackgroundA mutual information approach to calculating nonlinearity
Smith, R. D. (2015) · Stat · doi:10.1002/sta4.96 · Cited by 92
“In addition, if both N and m are large, error bars for confidence intervals may be inaccurate and underestimated (Paninski, 2003).”
BackgroundDistributed Variational Representation Learning
Aguerri, I. E., & Zaidi, A. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2928806 · Cited by 84
“[45], [46], [47], [48] and references therein.”
Estimating Renyi Entropy of Discrete Distributions
Acharya, J., Orlitsky, A., Suresh, A. T., & Tyagi, H. (2016) · IEEE Transactions on Information Theory · doi:10.1109/tit.2016.2620435 · Cited by 110
“The estimation of Shannon entropy has several applications, including measuring genetic diversity [37], quantifying neural activity [29], [32], network anomaly detection [20], and others.”
MethodPopescu, L., Ceșco, T., Gurev, A., Ghendov‐Moșanu, A., Sturza, R., & Tarna, R. (2022) · Foods · doi:10.3390/foods11223565 · Cited by 91
“The more pronounced the influence of concentration of AP powder and storage period on textural parameters and overall acceptability, the higher the value of the bit [41].”
BackgroundTkačik, G., Dubuis, J. O., Petkova, M. D., & Gregor, T. (2015) · Genetics · doi:10.1534/genetics.114.171850 · Cited by 82
Deep Dictionary Learning: A PARametric NETwork Approach
Mahdizadehaghdam, S., Panahi, A., Krim, H., & Dai, L. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2914376 · Cited by 89
“information is estimated (from samples [33], [23]) at different snapshots.”
BackgroundSource-Free Domain Adaptation via Target Prediction Distribution Searching
Tang, S. Y., Chang, A., Zhang, F., Zhu, X., Ye, M., & Zhang, C. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01892-w · Cited by 83
“Inspired by the theoretical correlation of mutual information and pairwise losses (Boudiaf et al., 2020), we use the following objective to reach the goal where I ( · , · ) is the mutual information function (Paninski, 2003) whose computation is the same as the way in (Ji et al., 2019).”
Spike-Timing Precision Underlies the Coding Efficiency of Auditory Receptor Neurons
Rokem, A., Watzl, S., Gollisch, T., Stemmler, M., Herz, A. V. M., & Samengo, I. (2005) · Journal of Neurophysiology · doi:10.1152/jn.00891.2005 · Cited by 93
Estimating the Unseen: Improved Estimators for Entropy and other Properties
Valiant, P., & Valiant, G. J. (2013) · neural information processing systems · Cited by 111
Statistical Mechanics of the US Supreme Court
Lee, E. D., Broedersz, C. P., & Bialek, W. (2015) · Journal of Statistical Physics · doi:10.1007/s10955-015-1253-6 · Cited by 82
“[29] L Paninski, Estimation of entropy and mutual informa-”
BackgroundInferring Connectivity of Genetic Regulatory Networks Using Information-Theoretic Criteria
Zhao, W., Serpedin, E., & Dougherty, E. R. (2008) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2007.1067 · Cited by 92
“We remark that the recent progress in the area of estimating information theoretic quantities has led to a number of alternatives for estimating the entropy, for example, [27], [28], [29], [30], [31].”
BackgroundHacisuleyman, A., & Erman, B. (2017) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005319 · Cited by 86
A Closer Look at Few-shot Image Generation
Zhao, Y., Ding, H., Huang, H., & Cheung, N. M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00893 · Cited by 111
“As directly maximizing MI is challenging [36], we apply contrastive learning [34] to solve Eqn.”
MethodBatch Normalization is a Cause of Adversarial VulnerabilityPreprint
Galloway, A., Golubeva, A., Tanay, T., Moussa, M. A., & Taylor, G. W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02161 · Cited by 88
“We use the “plug-in” maximum-likelihood estimate of the entropy H(T ) using the full MNIST training set [17].”
MethodInfluentialNew Quantum Algorithms for Computing Quantum Entropies and Distances
Wang, Q., Guan, J., Liu, J., Zhang, Z., & Ying, M. (2024) · IEEE Transactions on Information Theory · doi:10.1109/tit.2024.3399014 · Cited by 93
“Apart from property testing between distributions, properties of a single distribution are well studied in the literature, e.g., [78], [79], and [80].”
BackgroundEstimating the Unseen: Improved Estimators for Entropy and Other Properties
Valiant, G. J., & Valiant, P. (2017) · Journal of the ACM · doi:10.1145/3125643 · Cited by 126
“The first three are standard, and are, perhaps, the most commonly used estimators [15].”
BackgroundMethodInfluentialThe Interplay Between Entropy and Variational Distance
Ho, S., & Yeung, R. W. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2080452 · Cited by 78
A Hierarchical Adaptive Approach to Optimal Experimental Design
Kim, W., Pitt, M. A., Lu, Z., Steyvers, M., & Myung, J. I. (2014) · Neural Computation · doi:10.1162/neco_a_00654 · Cited by 85
“LETTER Communicated by Todd Coleman”
Olsen, C., Meyer, P. E., & Bontempi, G. (2009) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1155/2009/308959 · Cited by 78
“It has been shown in [10] that the asymptotic bias of this estimator amounts to bias”
REVIEW ARTICLE: Neuronal coding and spiking randomness
Kostal, L., Lánský, P., & Rospars, J.-P. (2007) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2007.05880.x · Cited by 94
“To answer this question, methods to compare different spike trains are needed first (Rieke et al. , 1997; Victor & Purpura, 1997; Buracas & Albright, 1999; Paninski, 2003; Bhumbra et al. , 2004; Nemenman et al. , 2004).”
MethodHEDGE: Efficient Traffic Classification of Encrypted and Compressed Packets
Casino, F., Choo, K.-K. R., & Patsakis, C. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2911156 · Cited by 77
“However, the entropy estimation approach is not effective when the number of samples is small [50], [51].”
BackgroundSong, D. B., Wang, H., Tu, C. Y., Marmarelis, V. Z., Hampson, R. E., Deadwyler, S. A., & Berger, T. W. (2013) · Journal of Computational Neuroscience · doi:10.1007/s10827-013-0455-7 · Cited by 93
“, mutual information (Li et al. 2012; Paninski 2003), directed information (Quinn et al.”
Efficient Markov Chain Monte Carlo Methods for Decoding Neural Spike Trains
Ahmadian, Y., Pillow, J. W., & Paninski, L. (2010) · Neural Computation · doi:10.1162/neco_a_00059 · Cited by 85
“Estimating this quantity is known to be often computationally quite difficult, particularly for high-dimensional stimuli and responses (Paninski, 2003).”
BackgroundSelf-citationPartial transfer entropy on rank vectors
Kugiumtzis, D. (2013) · The European Physical Journal Special Topics · doi:10.1140/epjst/e2013-01849-4 · Cited by 82
“We considered approximations with Gaussian and Gamma distributions expressing their parameters in terms of the estimated bias and variance, and also using known results on Gamma approximation of conditional mutual information.”
BackgroundInfluentialBayesian Entropy Estimation for Countable Discrete DistributionsPreprint
Archer, E., Park, I. M., & Pillow, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2627435.2697056 · Cited by 102
“Many previous studies have taken a frequentist approach and focused on methods for computing and reducing this bias (Miller, 1955; Panzeri and Treves, 1996; Strong et al., 1998; Paninski, 2003; Grassberger, 2008).”
InfluentialAn Exploration of the Kolmogorov-Smirnov Test as a Competitor to Mutual Information Analysis
Whitnall, C., Oswald, E., & Mather, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-27257-8_15 · Cited by 82
“However, estimation of MI is notoriously problematic ([11]); all known estimators are biased and no `ideal' estimator exists (di erent estimators perform di erently depending on the underlying structure of the data).”
BackgroundAn approximation to the distribution of finite sample size mutual information estimates
Goebel, B., Dawy, Z., Hagenauer, J., & Mueller, J. C. (2005) · International Conference on Communications · doi:10.1109/icc.2005.1494518 · Cited by 72
Legg, P., Rosin, P. L., Marshall, D., & Morgan, J. E. (2013) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2013.08.004 · Cited by 76
“Paninski [8] also looks at the estimation of entropy methods, and extends this to consider the estimation of MI.”
MethodGlobal Warming: Temperature Estimation in Annealers
Raymond, J., Yarkoni, S., & Andriyash, E. (2016) · Frontiers in ICT · doi:10.3389/fict.2016.00023 · Cited by 55
“perature estimation and evaluation of the objectives. The evaluation of the KL-divergence is one exception: our approximation (6) is known to fail when applied to the entropy term P x P A(x)logP A(x) [38, 39]. In the supplementary materials we discuss why evaluation is problematic, and propose a mitigation strategy. W…”
MethodModels of Neuronal Stimulus-Response Functions: Elaboration, Estimation, and Evaluation
Meyer, A. F., Williamson, R. S., Linden, J. F., & Sahani, M. (2017) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2016.00109 · Cited by 85
“While it is possible to correct for this bias to some degree, the optimal number of histogram bins also depends on the amount of data (Paninski, 2003b).”
BackgroundBonev, B., Escolano, F., & Cazorla, M. (2008) · Pattern Analysis and Applications · doi:10.1007/s10044-008-0107-0 · Cited by 70
“Given a set of samples, we want to estimate the Shannon entropy of the probability density function of these samples [16, 17, 18].”
BackgroundReshef, D. N., Reshef, Y., Sabeti, P. C., & Mitzenmacher, M. (2018) · The Annals of Applied Statistics · doi:10.1214/17-aoas1093 · Cited by 70
Improved Information Gain Estimates for Decision Tree InductionPreprint
Nowozin, S. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.4620 · Cited by 92
“Estimating the discrete entropy is challenging (Paninski, 2003; Antos & Kon-toyiannis, 2001), but superior corrections than the one of Miller have been proposed recently.”
BackgroundMeasuring dependence powerfully and equitablyPreprint
Reshef, Y., Reshef, D. N., Finucane, H. K., Sabeti, P. C., & Mitzenmacher, M. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02213 · Cited by 70
“For instance, later in this paper we introduce an additional property of the characteristic matrix, the total information coefficient, that is useful for testing for the presence or absence of a relationship rather than quantifying relationship strength.”
BackgroundA Robust-Equitable Measure for Feature Ranking and Selection
Ding, A. A., Dy, J., Li, Y., & Chang, Y. (2017) · Journal of Machine Learning Research · Cited by 69
Acharya, J., Issa, I., Shende, N. V., & Wagner, A. B. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.3015235 · Cited by 90
Adaptive Compressed Image Sensing Using Dictionaries
Averbuch, A. Z., Dekel, S., & Deutsch, S. (2012) · SIAM Journal on Imaging Sciences · doi:10.1137/110820579 · Cited by 72
“Studies of this problem resort to nonparametric histogram estimation of the PDFs [8], [58], [70].”
BackgroundChao, Z. C., Bakkum, D. J., & Potter, S. M. (2007) · Journal of Neural Engineering · doi:10.1088/1741-2560/4/3/015 · Cited by 74
“MI was estimated by 21 using the histogram-based mutual information methods described by 22 Moddemeijer (Moddemeijer, 1989).”
Qiu, J., Crow, W. T., Nearing, G., Mo, X., & Liu, S. (2014) · Geophysical Research Letters · doi:10.1002/2014gl060017 · Cited by 69
“It should be stressed that the entropy and MI estimates in this study are maximum likelihood estimators (MLEs) for discrete random variables, and the bias of an MLE estimator is determined by both the number of observations and, generally, the magnitude of MI [Paninski, 2003].”
Convergence of Smoothed Empirical Measures With Applications to Entropy Estimation
Goldfeld, Z., Greenewald, K. H., Weed, J., & Polyanskiy, Y. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.2975480 · Cited by 105
“Plug-in techniques are suboptimal for vanilla discrete (Shannon) and differential entropy estimation (see [4] and [5], respectively).”
BackgroundLee, K. S., Tran, N.-T., & Cheung, N. M. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00399 · Cited by 98
“Yet, since directly maximizing mutual information is often intractable in practice [26], these works often maximize on the InfoNCE [21] lower bound instead, which involves a contrastive loss minimized through having a critic correctly finding a positive sample in contrast to a set of negative samples.”
Background
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.