Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 601–700 of 1,004
Fast Approximation of Empirical Entropy via Subsampling
Wang, C., & Ding, B. (2019) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3292500.3330938 · Cited by 13
“The early work dates back to 1950’s [26], and the modern interest is revived by Antos and Kontoyiannis [1] and Paninski [27].”
BackgroundMethodMarx, A., Yang, L., & van Leeuwen, M. (2021) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976700.44 · Cited by 13
Trade-Off between Information Format and Capacity in the Olfactory System
Aldworth, Z., & Stopfer, M. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.3562-14.2015 · Cited by 13
“We debiased our estimates of the various response probabilities in Equation 2 using the “Best Upper Bound” technique (Paninski, 2003).”
MethodInference of combinatorial Boolean rules of synergistic gene sets from cancer microarray datasets
Park, I. H., Lee, K. H., & Lee, D. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq207 · Cited by 15
Prince, L. Y., Tran, M. M., Grey, D., Saad, L., Chasiotis, H., Kwag, J., Kohl, M. M., & Richards, B. A. (2021) · Communications Biology · doi:10.1038/s42003-021-02437-y · Cited by 15
“…is that they require exponentially longer experiments to adequately sample from the distribution of responses to obtain a low variance estimate of mutual information, or one must use approximations that can have quite high bias (see, e.g. Paninski, 2003, Neural Computation, 15(6): 1191-1253).”
BackgroundData Amplification: Instance-Optimal Property EstimationPreprint
Hao, Y., & Orlitsky, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.01432 · Cited by 15
“First, for empirical estimator ĤE, Paninski (2003) [see Proposition 1] provides a short argument showing that its worst-case MAE, for all n and k, satisfies LĤE(n) ≤ log ( 1 + k − 1 n ) + log n√ n .”
Resultinfotheory: A C++/Python package for multivariate information theoretic analysis
Candadai, M., & Izquierdo, E. J. (2020) · The Journal of Open Source Software · doi:10.21105/joss.01609 · Cited by 13
“One of the major challenges in utilizing information theoretic measures in experimental settings is the availability of sufficient data to infer the data distributions correctly (Paninski, 2003).”
BackgroundRobotics: Science and Systems XVII
(2021) · doi:10.15607/rss.2021.xvii · Cited by 13
Training LLMs over Neurally Compressed Text
Lester, B., Lee, J., Alemi, A., Pennington, J., Roberts, A., Sohl‐Dickstein, J., & Constant, N. (2024) · Transactions on machine learning research · doi:10.48550/arxiv.2404.03626 · Cited by 24
“The maximum likelihood, or plug-in, estimator of entropy, ˆ H = − x ∈X ˆ p ( x ) log 2 ˆ p ( x ), is negatively biased— in fact, all entropy estimators are biased (Paninski, 2003).”
BackgroundF2AT: Feature-Focusing Adversarial Training via Disentanglement of Natural and Perturbed Patterns
Qian, Y., Zhao, C., Gu, Z., Wang, B., Ji, S., Wang, W., & Zhang, Y. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3580116 · Cited by 12
“Theorem 1: Let X , X nat , X pert , F be four random variables, such that X = X nat + X pert , we have Unfortunately, computing H ( F (cid:8) | X (cid:8) ) , H ( F (cid:8) | X nat , X pert ) , and I ( X nat ; X pert ; F ) is non-trivial [51].”
Sahani, S. K. (2026) · Journal of Intelligent Decision Making and Information Science · doi:10.59543/jidmis.v3.1549 · Cited by 15
“For continuous features, the support must be discretized (binned) before applying this estimator, with the bin count growing sublinearly in 𝑛 to control the bias-variance trade-off; see Paninski (2003) for the precise rate.”
Analysing the importance of variables for sewer failure prediction
Carvalho, G., Amado, C., Brito, R. S., Coelho, S. T., & Leitão, J. P. (2018) · Urban Water Journal · doi:10.1080/1573062x.2018.1459748 · Cited by 13
“Despite this correction, no unbiased estimator for the entropy can be found, see Paninski (2003).”
BackgroundMethodDirectly estimating the Holevo capacity of discrete Weyl channels
Rehman, J. U., Jeong, Y., & Shin, H. (2019) · Physical Review A · doi:10.1103/physreva.99.042312 · Cited by 12
Evaluating performance of neural codes in model neural communication networks
Antonopoulos, C. G., Bianco-Martínez, E., & Baptista, M. S. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.10.008 · Cited by 12
“There is a huge body of work on the estimation of MI in dynamical systems and in neuroscience, for example [24, 27, 28, 29, 30, 31, 32, 23, 17].”
BackgroundOptimal Estimation of Wasserstein Distance on a Tree With an Application to Microbiome Studies
Wang, S., Cai, T., & Li, H. (2019) · Journal of the American Statistical Association · doi:10.1080/01621459.2019.1699422 · Cited by 14
“Similar phenomena are observed in functional estimation of single distribution (see, e.g., Paninski 2003; Jiao et al. 2015; Wu and Yang 2016).”
BackgroundInferring entropy from structure
Ariel, G., & Diamant, H. (2020) · Physical review. E · doi:10.1103/physreve.102.022110 · Cited by 12
“Despite a large number of suggested algorithms [15, 17], the problem of estimating the entropy, in particular differential entropy, from independent sampling of the distributions remains a challenge in high dimensions [16, 17].”
BackgroundEfficient Approximate Algorithms for Empirical Entropy and Mutual Information
Chen, X., & Wang, S. (2021) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3448016.3457255 · Cited by 12
“Paninski [25] finds the connection between the bias of entropy estimators and a certain polynomial approximation problem.”
BackgroundAn entropy-based measure of correlation for time series
Pernagallo, G. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119272 · Cited by 12
Vejmelka, M., & Hlaváčková‐Schindler, K. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-71618-1_88 · Cited by 12
“Mutual information (MI) is important to many areas and is difficult to be estimated [13].”
BackgroundDelay independence of mutual-information rate of two symbolic sequences
Blanc, J.-L., Pézard, L., & Lesne, A. (2011) · Physical Review E · doi:10.1103/physreve.84.036214 · Cited by 12
Dong, K., Long, L., Zhang, H., & Gao, Y. (2018) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2018.04.059 · Cited by 12
Li, B., & Good, S. P. (2021) · Hydrology and earth system sciences · doi:10.5194/hess-25-5029-2021 · Cited by 12
Anderson, B., Sanderson, M. I., & Sheinberg, D. L. (2006) · Experimental Brain Research · doi:10.1007/s00221-006-0594-4 · Cited by 13
“Also, there are more involved ways of computing the mutual information (Paninski 2003).”
BackgroundPires, C., & Perdigão, R. A. P. (2013) · Entropy · doi:10.3390/e15030721 · Cited by 12
“The leading variance term is N-scaled as generally deduced in [15].”
Mutual Information-Based Hierarchies on Warsaw Stock Exchange
Fiedor, P. (2015) · Acta Physica Polonica A · doi:10.12693/aphyspola.127.a-33 · Cited by 12
Anthropic Correction of Information Estimates and Its Application to Neural Coding
Gastpar, M., Gill, P. R., Huth, A. G., & Theunissen, F. E. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2009.2037053 · Cited by 12
“For a very simple proof of this fact, see [13].”
BackgroundComprehensive review of association estimators for the inference of gene networks
Kurt, Z., Aydın, N., & Altay, G. (2016) · TURKISH JOURNAL OF ELECTRICAL ENGINEERING & COMPUTER SCIENCES · doi:10.3906/elk-1312-90 · Cited by 12
“Paninski proposed using the BUB estimator for analyzing neuroscientific data and denoted that this estimator is a version of bias correction of the MM estimator.”
BackgroundMethodInfluentialOn Lower Bounds for Statistical Learning Theory
Loh, P. (2017) · Entropy · doi:10.3390/e19110617 · Cited by 12
Detection of Multiway Gamma Coordination Reveals How Frequency Mixing Shapes Neural Dynamics
Haufler, D., & Paré, D. (2019) · Neuron · doi:10.1016/j.neuron.2018.12.028 · Cited by 12
An information-theoretic study of fish swimming in the wake of a pitching airfoil
Zhang, P., Krasner, E., Peterson, S. D., & Porfiri, M. (2019) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2019.02.014 · Cited by 12
“values were then computed using plug-in estimators based on the experimental time series [32].”
MethodDeep Learning L2 Norm Fusion for Infrared & Visible Images
Shihabudeen, H., & Rajeesh, J. (2022) · IEEE Access · doi:10.1109/access.2022.3164426 · Cited by 12
“A larger MI metric indicates more details are retained from the individual to the fused image [37].”
BackgroundAlgorithmic approaches to statistical questions
Papadimitriou, C. H., & Valiant, G. J. (2012) · Cited by 12
A Novel Information Theory Method for Filter Feature Selection
Bonev, B., Escolano, F., & Cazorla, M. (2007) · Springer eBooks · doi:10.1007/978-3-540-76631-5_41 · Cited by 12
“The estimation of the Shannon entropy of a probability density given a set of samples has been studied widely in the past [9][10][11][12][13][14].”
BackgroundEstimating Functions of Distributions Defined over Spaces of Unknown Size
Wolpert, D. H., & DeDeo, S. (2013) · Entropy · doi:10.3390/e15114668 · Cited by 15
“over crather than condition on it, so long as we assume IUV. Entropy 2013, 15 4679 Similarly to how the posterior first moment of mutual information can be defined using either Equation (2) or Equation (3), so can the posterior second moment: E(I2 j~n) = E(H(X)2 j~n X) + E(H2 Y j~n Y) + E(H 2 X;Y j~n) 2E(H(X)H(Y) j~n) 2…”
BackgroundMethodInfluentialOnline Learning of Noisy Data with KernelsPreprint
Cesa‐Bianchi, N., Shalev‐Shwartz, S., & Shamir, O. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1005.2296 · Cited by 12
“In fact, when X takes finitely many values and f is not a polynomial function, one can prove that no unbiased estimator can exist (see [13], Proposition 8 and its proof).”
BackgroundUnsupervised Feature Learning by Cross-Level Discrimination between Instances and GroupsPreprint
Wang, X., Liu, Z., & Yu, S. X. (2020) · arXiv (Cornell University) · Cited by 13
Mao, W., Pin, L., & Huang, J. (2024) · Entropy · doi:10.3390/e26060478 · Cited by 14
“Since the optimal mean µ X p , X SF is theoretically intractable [82], to determine µ X p , X SF , we follow the work of [83], using the outputs of the empirically optimal predictor in experiments to approximate µ X p , X SF .”
MethodNot on my watch: ransomware detection through classification of high-entropy file segments
Casino, F., Hurley-Smith, D., Hernandez-Castro, J. C., & Patsakis, C. (2025) · Journal of Cybersecurity · doi:10.1093/cybsec/tyaf009 · Cited by 14
Wei, X. L., Li, L., & Liu, J. (2022) · Journal of Quantitative Linguistics · doi:10.1080/09296174.2022.2037275 · Cited by 11
“…estimator, which is designed and later proved by a series of experiments to be able to counterbalance the underestimation bias of small text sizes, by adding a correction to Shannon entropy (Bentz et al., 2017; Hausser & Strimmer, 2009; Meyer et al., 2008; Miller, 1955; Paninski, 2003).”
BackgroundRevisiting “bursts” in wall-bounded turbulent flows
Chowdhuri, S., & Banerjee, T. (2023) · Physical Review Fluids · doi:10.1103/physrevfluids.8.044606 · Cited by 12
“Therefore, their PDFs (P (Np)) transform to probability mass functions (PMFs) whose computation does not suffer from any arbitrary binning (Paninski, 2003).”
BackgroundInformation Bounds and Convergence Rates for Side-Channel Security Evaluators
Masure, L., Cassiers, G., Hendrickx, J. M., & Standaert, F. (2023) · IACR Transactions on Cryptographic Hardware and Embedded Systems · doi:10.46586/tches.v2023.i3.522-569 · Cited by 11
“Moreover, The proof of this statement is directly inspired from Paninski’s work [Pan03], and is detailed in Appendix A. Note that as a consequence of Equation 5, the upper bound of Equation 4 is asymptotically tight, thereby meaning that the lower bound is asymptotically loose.”
BackgroundMethodInfluentialA statistical description of neural ensemble dynamics
Long, J. D., & Carmena, J. M. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00052 · Cited by 11
“This is in line with observations made by others about the calculation of Shannon Entropy from ensemble data (Paninski, 2003).”
ResultMorgado, P. M., & Silveira, M. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.12.070 · Cited by 12
“In a fi nal note, it should be mentioned that the estimation of mutual information using frequency counts is biased due to the concave shape of the logarithmic function as pointed out by Paninski [33].”
Pinpointing side-channel information leaks in web applications
Mather, L., & Oswald, E. (2012) · Journal of Cryptographic Engineering · doi:10.1007/s13389-012-0036-0 · Cited by 12
Recovering Social Networks From Massive Track Datasets
Connolly, C. I., Burns, J. B., & Bui, H. (2008) · Proceedings · doi:10.1109/wacv.2008.4544042 · Cited by 11
“Entropy computation is prone to sampling error, and several estimators have been proposed to address this problem [6, 7 ,8].”
BackgroundA mutual information estimator with exponentially decaying bias
Zhang, Z., & Zheng, L. (2015) · Statistical Applications in Genetics and Molecular Biology · doi:10.1515/sagmb-2014-0047 · Cited by 11
Information-Theoretic Bounds and Approximations in Neural Population Coding
Huang, W., & Zhang, K. (2018) · Neural Computation · doi:10.1162/neco_a_01056 · Cited by 16
“Various asymptotic expansion methods have been proposed to reduce the bias in an MI estimate (Miller, 1955; Carlton, 1969; Treves & Panzeri, 1995; Victor, 2000; Paninski, 2003).”
An adaptive information-theoretic approach for identifying temporal correlations in big data sets
Ho, N., Vo, H. T., & Vu, M. H. (2016) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2016.7840659 · Cited by 11
Nonlinear Canonical Correlation Analysis:A Compressed Representation Approach
Painsky, A., Feder, M., & Tishby, N. (2020) · Entropy · doi:10.3390/e22020208 · Cited by 11
“Estimating the entropy values of U and V is quite challenging when the number of samples is relatively small, compared to the number of cells [53].”
BackgroundGan, D., Yang, B., & Tang, Y. (2020) · Mathematics · doi:10.3390/math8122137 · Cited by 11
“The existing theoretical tools for dealing with uncertain information include probability theory [3], fuzzy set theory [4], Dempster–Shafer evidence theory [5,6], information entropy theory [7,8], and so on.”
Toward an Information Theory of Quantitative Genetics
Galas, D. J., Kunert-Graf, J. M., Uechi, L., & Sakhanenko, N. A. (2021) · Journal of Computational Biology · doi:10.1089/cmb.2020.0032 · Cited by 11
Felix, L. M., Alves, M. M., Santos, M. T., Jamison, L., & Golino, H. F. (2021) · Archives of Gerontology and Geriatrics · doi:10.1016/j.archger.2021.104337 · Cited by 12
Partial Mutual Information Analysis of Financial Networks
Fiedor, P. (2015) · Acta Physica Polonica A · doi:10.12693/aphyspola.127.863 · Cited by 11
Non-parametric entropy estimators based on simple linear regression
Hino, H., Koshijima, K., & Murata, N. (2015) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2015.03.011 · Cited by 11
How biased are maximum entropy models
Macke, J. H., Murray, I., & Latham, P. E. (2011) · Max Planck Digital Library · Cited by 11
Information processing in the LGN: a comparison of neural codes and cell types
Pręgowska, A., Casti, A., Kaplan, E., Wajnryb, E., & Szczepański, J. (2019) · Biological Cybernetics · doi:10.1007/s00422-019-00801-0 · Cited by 11
“…difference of entropies (Bialek et al. 1991; Borst and Theunissen 1999; Nemenman et al. 2004; Paprocki and Szczepanski 2013a; Pregowska et al. 2015), entropy estimation has received a great deal of attention (Paninski 2003; Amigo et al. 2004; Kennel et al. 2005; Gao et al. 2008; Lesne et al. 2009).”
BackgroundEntropy Estimation of Physically Unclonable Functions via Chow Parameters
Schaub, A., Rioul, O., & Boutros, J. J. (2019) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2019.8919927 · Cited by 11
“However, because we generated much more PUFs than equivalence classes (by a factor of at least 100000), the plugin estimator, based on the empirical frequency estimates, performs quite well: Its bias can be upper bounded as described in [22] and was found to be less than 0.”
BackgroundMethodZhang, Y., & Wang, A. (2020) · Shock and Vibration · doi:10.1155/2020/6216903 · Cited by 11
“Mutual information (MI) [31] can quantitatively represent the correlation between two random variables, and it is more accurate than the correlation coefficient method.”
MethodVictor, J. D., & Purpura, K. P. (2010) · Springer eBooks · doi:10.1007/978-1-4419-5675-0_7 · Cited by 13
Estimation of generalized entropies with sample spacing
Wachowiak, M. P., Smolikova, R., Tourassi, G. D., & Elmaghraby, A. S. (2005) · Pattern Analysis and Applications · doi:10.1007/s10044-005-0012-8 · Cited by 11
“Different methods for computing the Shannon [ 21 ] and generalized entropies are found in the literature [16– 20].”
MethodA Kernel-Based Calculation of Information on a Metric Space
Tobin, R., & Houghton, C. J. (2013) · Entropy · doi:10.3390/e15104540 · Cited by 11
“The actual probability distributions are known: the probability of finding a point r generated by a source, s, depends only on the distance d = |r − s| and is given by the χ-distribution: p(d) = 21−nd/2 Γ(nd/2) ( d σ )nd−1 e−d 2/2σ2 (24) There is a bias in estimating the mutual information, in fact, bias is common to…”
BackgroundMulti-classification by categorical features via clustering
Seldin, Y., & Tishby, N. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390272 · Cited by 11
“See (Paninski, 2003) for a more detailed proof of (8) and (9).”
BackgroundAn efficient entropy estimation approach
Paavola, M. (2011) · Cited by 11
Feature Selection Via Joint Likelihood
Pocock, A. (2013) · Research Explorer (The University of Manchester) · Cited by 11
Differentially private anonymized histogramsPreprint
Suresh, A. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03553 · Cited by 12
“Anonymized histograms are also referred to as histogram of histograms [1], histogram order statistics [2], profiles [3], unattributed histograms [4], fingerprints [5], and frequency lists [6].”
BackgroundMethodHow Efficiency Shapes Human Language, TICS 2019Preprint
Gibson, E. A. F., Futrell, R., Piantadosi, S. T., Dautriche, I., Mahowald, K., Bergen, L., & Lévy, R. (2019) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/w5m38 · Cited by 11
Removing Inter-Experimental Variability from Functional Data in Systems NeurosciencePreprint
Gonschorek, D., Höfling, L., Szatko, K. P., Franke, K., Schubert, T., Dunn, B. A., Berens, P., Klindt, D. A., & Euler, T. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.10.29.466492 · Cited by 16
Sliced Mutual Information: A Scalable Measure of Statistical DependencePreprint
Goldfeld, Z., & Greenewald, K. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.05279 · Cited by 52
“While this topic has received considerable attention [14, 15, 16, 17, 18], MI is fundamentally hard to estimate in high-dimensional settings due to an exponentially large (in dimen-sion) sample complexity [19].”
Faster Hoeffding Racing: Bernstein Races via Jackknife Estimates
Loh, P., & Nowozin, S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40935-6_15 · Cited by 10
Information Diversity in Structure and Dynamics of Simulated Neuronal Networks
Mäki‐Marttunen, T., Aćimović, J., Nykter, M., Kesseli, J., Ruohonen, K., Yli-Harja, O. P., & Linne, M. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00026 · Cited by 10
“Another examples of application of information theoretic methods in the analysis of spike train data can be found in Paninski (2003).”
MethodA novel entropy optimized kernel least-mean mixed-norm algorithm
Luo, X., Deng, J., Liu, J., Li, A., Wang, W., & Zhao, W. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727405 · Cited by 10
“The greater the information entropy, the more invalid information contained in the variable [21], [22].”
Embedding optimization reveals long-lasting history dependence in neural spiking activity
Rudelt, L., Marx, D. G., Wibral, M., & Priesemann, V. (2021) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008927 · Cited by 12
New refinements of Shannon’s entropy upper bounds
Sayyari, Y. (2021) · Journal of Information and Optimization Sciences · doi:10.1080/02522667.2021.1966947 · Cited by 10
“See [9] for other methods to estimate the Shannon entropy and [12] for a review on entropy estimation.”
MethodAn empowerment-based solution to robotic manipulation tasks with sparse rewards
Dai, S., Xu, W., Hofmann, A., & Williams, B. C. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10087-8 · Cited by 15
“MI is also often expressed in terms of Shannon entropy (Paninski, 2003) as well as Kullback-Leibler (KL) divergence:”
BackgroundDisturbed Augmentation Invariance for Unsupervised Visual Representation Learning
Cheng, H., Li, H., Wu, Q., Qiu, H., Zhang, X., Meng, F., & Zhao, T. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3272741 · Cited by 11
“However, mutual information is notoriously difficult to compute in continuous and high-dimensional settings [64].”
BackgroundMorphological entropy encodes cellular migration strategies on multiple length scales
Liu, Y., Jiao, Y., Fan, Q., Li, X., Liu, Z., Qin, D., Hu, J., Liu, L., Shuai, J., & Li, Z. (2024) · npj Systems Biology and Applications · doi:10.1038/s41540-024-00353-5 · Cited by 10
“Actually, entropy is technically quite dif fi cult to compute reliably for continuous variables 57 .”
Liao, D., Liu, C., Christensen, B. W., Tong, A., Huguet, G., Wolf, G., Nickel, M., Adelstein, I., & Krishnaswamy, S. (2024) · Conference on Information Sciences and Systems · doi:10.1109/ciss59072.2024.10480166 · Cited by 10
“Indeed, recent investigations along this line are often restricted to small toy models: e.g., in [6], [7], most experiments were performed on a 7-layer network of width 12-10-7-5-4-3-2.”
MethodON THE ESTIMATION OF MUTUAL INFORMATION
Marek, T., & Tichavský, P. (2009) · Cited by 10
Towards an Information Model of Consistency Maintenance in Distributed Interactive Applications
Zhang, X., Ward, T. E., & McLoone, S. (2008) · International Journal of Computer Games Technology · doi:10.1155/2008/371872 · Cited by 11
“For the purpose of this paper, we use a simple approach to estimate probability and mutual information from experimental data, though other more advanced and complicated algorithms exist [19, 24].”
MethodOnline Entropy Estimation for Non-Binary Sources and Applications on iPhone
Lauradoux, C., Ponge, J., & Roeck, A. (2011) · Cited by 10
Event recognition in photo albums using probabilistic graphical models and feature relevance
Bacha, S., Allili, M. S., & Benblidia, N. (2016) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2016.07.021 · Cited by 11
Structured Generative Models of Continuous Features for Word Sense Induction.
Komninos, A., & Manandhar, S. (2016) · International Conference on Computational Linguistics · Cited by 10
Semi-supervised Attentive Mutual-info Generative Adversarial Network for Brain Tumor Segmentation
Xi, N. (2019) · International Conference on Image and Vision Computing New Zealand (IVCNZ) · doi:10.1109/ivcnz48456.2019.8961008 · Cited by 11
“Despite being a vital measurement indicator, mutual information is notoriously difficult to compute [17].”
BackgroundElectrical & Computer Engineering
Kehoe, J., & Perkovic, O. (2018) · Cited by 10
Uniform Partitioning of Data Grid for Association Detection
Mousavi, A., & Baraniuk, R. G. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3029487 · Cited by 10
Sharma, K., Futrell, R., & Husain, S. (2020) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2020.cmcl-1.1 · Cited by 11
Dong, Z., Li, L., Yan, L., Sun, M. Y., & Li, J. (2020) · Catalysts · doi:10.3390/catal10111364 · Cited by 10
“Therefore, it can express a linear and nonlinear correlation [9].”
BackgroundRicci, L. (2021) · Physical review. E · doi:10.1103/physreve.103.022215 · Cited by 10
Merbis, W., de Mulatier, C., & Corboz, P. (2023) · Physical review. E · doi:10.1103/physreve.108.024303 · Cited by 12
Pietrzak, M., Rempała, G. A., Seweryn, M. T., & Wesołowski, J. (2016) · Test · doi:10.1007/s11749-016-0489-z · Cited by 10
“Their respective hypotheses (iii) may be viewed as complementing the analogous results established for the Shannon entropy and the Kullback–Leibler divergence (Paninski 2003; Zhang and Zhang 2012; Zhang and Grabchak 2014).”
ResultAn Effective Non-rigid Registration Approach for Ultrasound Image Based On “Demons” Algorithm
Liu, Y., Cheng, H.-D., Huang, J., Zhang, Y., Tang, X., & Tian, J. (2012) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-012-9532-0 · Cited by 10
“where S i; j ð Þ is the average of S i; j ð Þ andD i; j ð Þ is the average of D i; j ð Þ: Normalized mutual information (NMI) can be quantitatively considered as a measure of how well one image matching another [22].”
SuperMIC: Analyzing Large Biological Datasets in Bioinformatics with Maximal Information Coefficient
Wang, C., Dai, D., Li, X., Wang, A., & Zhou, X. (2016) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2016.2550430 · Cited by 10
“Paninski [28] presents some results on the nonparametric estimation of entropy and mutual information.”
BackgroundEstimation information bounds using the I-MMSE formula and Gaussian mixture models
Paul, B., & Bliss, D. W. (2016) · Conference on Information Sciences and Systems · doi:10.1109/ciss.2016.7460514 · Cited by 10
“In addition, simulation or histogram methods can suffer from large biases [7].”
BackgroundLiu, C., Lu, Z., Ma, L., Wang, L., Jin, X., & Si, W. (2019) · Medical Physics · doi:10.1002/mp.13770 · Cited by 10
Functional analysis of information rates conveyed by rat whisker-related trigeminal nuclei neurons
Laturnus, S., Hoffmann, A., Chakrabarti, S., & Schwarz, C. (2021) · Journal of Neurophysiology · doi:10.1152/jn.00350.2020 · Cited by 11
Approximating entropy from sublinear samples
Brautbar, M., & Samorodnitsky, A. (2007) · doi:10.5555/1283383.1283422 · Cited by 10
An Information Theoretic Approach to the Study of Auditory Coding
Chechik, G. (2003) · Cited by 10
Calculation of Molecular Entropies Using Temperature Integration
Sharp, K. A. (2012) · Journal of Chemical Theory and Computation · doi:10.1021/ct300901x · Cited by 11
Usable Information and Evolution of Optimal Representations During TrainingPreprint
Kleinman, M., Achille, A., Idnani, D., & Kao, J. C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.02459 · Cited by 11
“In general, estimating mutual information from samples is challenging for high-dimensional random vectors (Paninski, 2003).”
Spectrally and temporally resolved estimation of neural signal diversityPreprint
Mediano, P. A. M., Rosas, F. E., Luppi, A. I., Noreika, V., Seth, A. K., Carhart-Harris, R. L., Barnett, L., & Bor, D. (2023) · doi:10.7554/elife.88683 · Cited by 11
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.