Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 701–800 of 1,004
Detecting translations of the same text and data with common source
Koroutchev, K., & Cebrián, M. (2006) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/2006/10/p10009 · Cited by 9
“The scale parameter B can be chosen in a way to achieve enough statistics for the estimation of the density of each bin k , which in practical terms means having some ten edges per bin [17].”
BackgroundSampling bias corrections for discrete and Gaussian partial information decompositions
Lorenz, G. M., Engel, N. M., Koçillari, L., Celotto, M., Orsenigo, D., Curreli, S., Malerba, S. B., Engel, A. K., Kayser, C., Fellin, T., et al. (2026) · Patterns · doi:10.1016/j.patter.2026.101619 · Cited by 9
“…by simply estimating estimating the probabilities as the empirical occurrences across experimental Although the bias of Shannon information has been studied extensively for both discrete 24,25,27 and Gaussian estimators, 80,81 the bias of individual PID components has remained largely unexplored.”
Dong, L., Chen, Z., & Pan, E. (2025) · Reliability Engineering & System Safety · doi:10.1016/j.ress.2025.112059 · Cited by 9
Chiverton, J. P., Ige, O. A., Barnett, S. J., & Parry, T. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2722234 · Cited by 9
“The “plug-in” formula in (10) for Entropy estimation is also a negatively biased estimator according to Antos and Kontoyiannis [46] and Paninski [47], i.”
BackgroundMethodBowman, H., Jones, W., Pincham, H. L., Fleming, S., Cleeremans, A., & Smith, M. (2022) · Neuroscience of Consciousness · doi:10.1093/nc/niac003 · Cited by 9
“However, from the investigations in Jones (2020), with s the number of bins (21 for us: the number of stimuli and of responses) and a sample size of n, in theworst case, the bias isO(s/n)=O(21/n) (Paninski 2003).”
The Role of Thermodynamic and Informational Entropy in Improving Real Estate Valuation Methods
Özdilek, Ü. (2023) · Entropy · doi:10.3390/e25060907 · Cited by 9
“Other methods that have been developed for this purpose include the quadratic extrapolation method [82], the Panzeri-Treves Bayesian estimation [83], the Best Universal Bound estimation [84], the Nemenman-Shafee-Bialek method [85] and entropy-integrating fuzzy ranking [86].”
MethodDetecting and characterizing phase transitions in active matter using entropy
Sorkin, B., Be’er, A., Diamant, H., & Ariel, G. (2023) · Soft Matter · doi:10.1039/d3sm00482a · Cited by 9
Diffeomorphic Information Neural Estimation
Duong, B., & Nguyen, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25908 · Cited by 17
Weng, G., Clark, K., Akbarian, A., Noudoost, B., & Nategh, N. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1273053 · Cited by 10
Nеtrеbа, N., Sergheeva, E., Gurev, A., Dragancea, V., Codină, G. G., Sturza, R., & Ghendov‐Moșanu, A. (2024) · Applied Sciences · doi:10.3390/app14156671 · Cited by 9
Sequential Fixed-Point ICA Based on Mutual Information Minimization
Hulle, M. M. V. (2007) · Neural Computation · doi:10.1162/neco.2008.09-06-335 · Cited by 9
“…which we define as the difference between the sum of the marginal entropies and the joint entropy: (2.3) Since MI estimation is considered to be a hard problem (for review, see Paninski, 2003), usually the link with negentropy is made by constraining the variables to be uncorrelated (prewhitening).”
BackgroundInfluentialEstimation Bias in Maximum Entropy Models
Macke, J. H., Murray, I., & Latham, P. E. (2013) · Entropy · doi:10.3390/e15083109 · Cited by 9
“Information theory [3,4] has played a major role in our effort to address this problem, but its usefulness is limited by the fact that the information-theoretic quantity of interest, mutual information (usually between stimuli and neuronal responses), is hard to compute from data [5].”
BackgroundMethodInfluentialPhysical Layer Secret Key Capacity Using Correlated Wireless Channel Samples
Wang, X., Jin, L., Huang, K., Li, M., & Ming, Y. (2016) · Globecom · doi:10.1109/glocom.2016.7842141 · Cited by 9
“Specifically, the secret key capacity is estimated by the Copula entropy estimation approach [23, 24] through 10 5 independent Monte-Carlo simulations.”
MethodMutual Information and Information Gating in Synfire Chains
Xiao, Z.-C., Wang, B., Sornborger, A. T., & Tao, L. (2018) · Entropy · doi:10.3390/e20020102 · Cited by 9
Information-Based Hierarchical Planning for a Mobile Sensing Network in Environmental Mapping
Li, T., Tong, K., Xia, M., Li, B., & de Silva, C. W. (2019) · IEEE Systems Journal · doi:10.1109/jsyst.2019.2939250 · Cited by 9
“In addition, conditional entropy (CE) and mutual information (MI) have been adopted as information-theoretic metrics to evaluate the high-information sites [19], [20], which quantify the information gain of a sampling frame in terms of prediction uncertainty.”
An, Z., Li, S., Jiang, X., Xin, Y., & Wang, J. (2019) · IEEE Access · doi:10.1109/access.2019.2961711 · Cited by 9
“If we want estimate the indeterminacy of event, the information entropy is provided, which is the mathematic expectation of the amount of information and is defined as follows [28]:”
BackgroundChernoff-Type Concentration of Empirical Probabilities in Relative Entropy
Guo, F. R., & Richardson, T. S. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.3034539 · Cited by 9
“[10], Paninski [13].”
Panday, S. K., & Ghosh, I. (2020) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.0c00764 · Cited by 9
STDNet: Rethinking Disentanglement Learning With Information Theory
Liu, Z., Li, M., Han, C., Tang, S., & Guo, T. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3241791 · Cited by 9
“With the daunting scale and nonlinearity involved in machine learning network structure, the analysis is challenging if not intractable [47], [48].”
BackgroundThe MAPK/ERK channel capacity exceeds 6 bit/hour
Nałęcz‐Jawecki, P., Gagliardi, P. A., Kochańczyk, M., Dessauges, C., Pertz, O. C., & Tomasz Lipniacki, B. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011155 · Cited by 9
“where (Miller–Madow) bias correction is 1/(2k ln 2) if both n0 and n1 are non-zero, and 0 otherwise [45,46].”
BackgroundNondestructive Evaluation of Early Contact Fatigue Using Eddy Current Pulsed Thermography
Liu, J., Ren, W., Tian, G. Y., Gao, B., Wang, Y., Zhang, J., Shaw, B., Yin, A., & King-Alale, N. O. (2015) · IEEE Sensors Journal · doi:10.1109/jsen.2015.2416394 · Cited by 9
Does dirichlet prior smoothing solve the Shannon entropy estimation problem?
Han, Y., Jiao, J., & Weissman, T. (2015) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2015.7282679 · Cited by 9
“On the other hand, it was shown in [16] [3] that the MLE requiresn ≍ S samples, implying that MLE is strictly sub-optimal in terms of sample complexity.”
BackgroundA Primer on Information Theory with Applications to Neuroscience
Effenberger, F. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8785-2_5 · Cited by 12
Measures of Qualitative Variation in the Case of Maximum Entropy
Evren, A., & Ustaoğlu, E. (2017) · Entropy · doi:10.3390/e19050204 · Cited by 9
“Although this estimator is biased, the amount of bias can be reduced by increasing the sample size [17].”
BackgroundCircularLogo: A lightweight web application to visualize intra-motif dependencies
Ye, Z., Ma, T., Kalmbach, M. T., Dasari, S., Kocher, J., & Wang, L. (2017) · BMC Bioinformatics · doi:10.1186/s12859-017-1680-2 · Cited by 9
Mahalunkar, A., & Kelleher, J. D. (2019) · doi:10.18653/v1/w19-3904 · Cited by 9
“This dependence can be computed using information theoretic measure i.e. Mutual Information (Cover and Thomas, 1991; Paninski, 2003; Bouma, 2009; Lin and Tegmark, 2017).”
MethodFingerprinting Cryptographic Protocols with Key Exchange Using an Entropy Measure
Luo, S., Seideman, J. D., & Dietrich, S. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00032 · Cited by 9
Li, P., Ren, Y., Yan, Y., & Wang, G. (2019) · Artificial intelligence for engineering design analysis and manufacturing · doi:10.1017/s0890060419000258 · Cited by 9
“It can be regarded as a random variable containing information on another random variable (Paninski, 2014).”
BackgroundMore Flexible Integration of Functional Systems After Musical Training in Young Adults
Li, Q., Wang, X., Wang, S., Xie, Y., Xie, Y., & Li, S. (2020) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2020.2977250 · Cited by 10
“dependent coefficients computed using the histogram [40].”
Application of Positional Entropy to Fast Shannon Entropy Estimation for Samples of Digital Signals
Cholewa, M., & Płaczek, B. (2020) · Entropy · doi:10.3390/e22101173 · Cited by 9
“Other estimators for finite samples are the James–Stein (JS) [13] and the best upper bound (BUB) estimator [14].”
Dynamic Bayesian Network Modeling Based on Structure Prediction for Gene Regulatory Network
Qu, L., Wang, Z., Li, C., Guo, S., Xin, J., Zhou, Y., & Wang, W. (2021) · IEEE Access · doi:10.1109/access.2021.3109133 · Cited by 9
“Mutual Information (MI) [29], [30] is the expected value of mutual information between two nodes and is used to measure the degree of dependence between two variables X and Y .”
BackgroundCNN-Aided Factor Graphs with Estimated Mutual Information Features for Seizure Detection
Salafian, B., Ben-Knaan, E. F., Shlezinger, N., de Ribaupierre, S., & Farsad, N. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746730 · Cited by 10
“Using (5) to compute MI as a measure of statistical dependence for EEG samples, with unknwon probability distributions is a challenging task [26].”
MethodChen, T., Hsieh, F., & Chou, E. P. (2022) · Entropy · doi:10.3390/e24101382 · Cited by 9
“This concept makes possible for [TC-1] by doing without the nonparametric estimations of Shannon entropy for a continuous distribution function as well as mutual information for two sets of continuous variables, which have been the long standing problems in physics and neural computing, see theoretical details in [19]…”
BackgroundDynamics of information and emergent computation in generic neural microcircuit models
Natschläger, T., & Maass, W. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.05.004 · Cited by 12
“This undersampling effect has been addressed in a number of studies (see e.g. Hertz, 1999; Paninski, 2003; Pola et al., 2003; and the references therein), and has turned out to be a serious obstacle for a wide-spread application of information theoretic methods to the analysis of neural computation.”
BackgroundMethodInfluentialBeyond Maximum Likelihood: from Theory to PracticePreprint
Jiao, J., Venkat, K., Han, Y., & Weissman, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.7458 · Cited by 9
“Then (6) implies that the empirical entropy estimator for H ( X 1 , X 2 ) is also consistent with o ( |X| 2 ) samples, which violates the results in [24], [25].”
BackgroundResultInfluentialMutual Information as a Figure of Merit for Optical Fiber SystemsPreprint
Fehenberger, T., & Hanik, N. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.2029 · Cited by 9
“, genetics [23] and neural computation [24].”
Conditional Dependence via Shannon Capacity: Axioms, Estimators and ApplicationsPreprint
Gao, W., Kannan, S., Oh, S., & Viswanath, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03476 · Cited by 12
The distribution of information content in English sentencesPreprint
Yu, S., Cong, J., Liang, J., & Liu, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.07681 · Cited by 10
“The reason for this is that the accurate calculation of information content is far from easy (35) due to two features.”
BackgroundMahalunkar, A., & Kelleher, J. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02966 · Cited by 9
“verbs to agree, compare: “Thedoginthathouseisaggressive" with “Thedogsinthathouseare aggressive". This dependence can be computed using information theoretic measure i.e. Mutual Information [4, 10, 28, 35]. One of the early attempts at addressing this issue was by El Hihi and Bengio [12] who proposed a Hiera…”
MethodEstimators for Multivariate Information Measures in General Probability SpacesPreprint
Rahimzamani, A., Asnani, H., Viswanath, P., & Kannan, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.11551 · Cited by 9
“Discrete entropy estimation is a well-studied topic and minimax rate of this problem is well-understood as a function of the alphabet size [33–35].”
BackgroundFast Rates for Maximum Entropy ExplorationPreprint
Tiapkin, D., Belomestny, D. V., Calandriello, D., Moulines, É., Munos, R., Naumov, A., Perrault, P., Tang, Y., Valko, M., & Ménard, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.08059 · Cited by 33
“In the following our analysis differs from (Paninski, 2003) since we obtain a direct estimate on the KL-divergence using Theorem G.1, E[nKL(p̂n, p)] = ∫ ∞ 0 P[nKL(p̂n, p) > t]dt ≤ (m− 1) log(e(1 + n/(m− 1))) + ∫ ∞ 0 e−tdt.”
MethodInfluentialGenerative vocal plasticity in chimpanzees
Lameira, A. R., Caneco, B., Kershenbaum, A., Santamaría-Bonfil, G., & Call, J. (2025) · iScience · doi:10.1016/j.isci.2025.112381 · Cited by 8
Towards a robust approach to infer causality from molecular dynamics simulations
Tatto, V. D., Banerjee, D., Hassanali, A. A., & Laio, A. (2025) · The Journal of Chemical Physics · doi:10.1063/5.0267926 · Cited by 8
Li, R., Li, H., & Tang, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2975989 · Cited by 8
“Theoretical tools for processing uncertain information include probability theory [1], fuzzy set theory [2], rough set theory [3], Dempster-Shafer evidence theory (D-S evidence theory) [4]–[6], information entropy theory [7], [8] etc.”
MethodEstimating and Maximizing Mutual Information for Knowledge Distillation
Shrivastava, A., Qi, Y., & Ordóñez, V. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00010 · Cited by 8
“Mutual information is a fundamental quantity that measures the relationship between random variables but it is notoriously difficult to measure [17].”
BackgroundOptimal binning for a variance based alternative of mutual information in pattern recognition
Fazekas, A., & Kovács, G. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.11.037 · Cited by 8
Li, P., Qi, G., Zhao, S., & Guan, W. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106922 · Cited by 10
DiMA: sequence diversity dynamics analyser for viruses
Tharanga, S., Ünlü, E. S., Hu, Y., Sjaugi, M. F., Celik, M. A., Hekimoğlu, H., Miotto, O., Öncel, M. M., & Khan, A. M. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbae607 · Cited by 8
“Raw entropy values for each k-mer position are corrected for sample size bias [37] by applying to the input alignment a statistical adjustment that estimates entropy values for infinitely sized resampled alignments with analogous k-mer distribution [23].”
Tegnitsap, J. V. N., Njitacke, Z. T., Barà, C., Fozin, T. F., Fotsin, H. B., Valdes-Sosa, P. A., Yoshimura, N., & Minati, L. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.115886 · Cited by 8
Understanding encoder–decoder structures in machine learning using information measures
Silva, J. F., Faraggi, V., Ramírez, C., Egaña, Á. F., & Pavéz, E. (2025) · Signal Processing · doi:10.1016/j.sigpro.2025.109983 · Cited by 8
How to infer the informational energy from small datasets
Cafaron, A., & Andonie, R. (2012) · International Conference on Optimization of Electrical and Electronic Equipment · doi:10.1109/optim.2012.6231921 · Cited by 8
“Several methods were proposed for density and MI estimation [19], [20], [21].”
MethodInformation-Selectivity of Beta-Amyloid Pathology in an Associative Memory Model
Rowan, M. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00002 · Cited by 9
“It is noted that biases appear when applying information theoretic measures to data, particularly with limited sample sets, and therefore the “true” information contribution measure of neurons in this work are subject to such biases (Paninski, 2003).”
BackgroundKuzmanovski, V., Todorovski, L., & Džeroski, S. (2018) · GigaScience · doi:10.1093/gigascience/giy118 · Cited by 8
Asymptotic Normality for Plug-In Estimators of Generalized Shannon’s Entropy
Zhang, J., & Shi, J. (2022) · Entropy · doi:10.3390/e24050683 · Cited by 8
“In particular, [9] proved that an unbiased estimator of Shannon’s entropy does not exist.”
Learning photonic neural network initialization for noise-aware end-to-end fiber transmission
Kirtas, M., Passalis, N., Mourgias-Alexandris, G., Dabos, G., Pleros, N., & Tefas, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909781 · Cited by 8
“At the same time, using an information-theoretic measure for estimating information flow, such as mutual information, is especially challenging and inefficient in high-dimension spaces [30].”
MethodDetecting Information Relays in Deep Neural Networks
Hintze, A., & Adami, C. (2023) · Entropy · doi:10.3390/e25030401 · Cited by 8
“First, estimates of entropies from finite ensembles can be inaccurate: these estimates are both noisy and biased (see, for example, [44]), leading to the removal of the wrong node from the set YR.”
BackgroundTruly Bayesian Entropy Estimation
Papageorgiou, I., & Kontoyiannis, I. (2023) · IEEE Information Theory Workshop (ITW) · doi:10.1109/itw55543.2023.10161645 · Cited by 10
“Since then it has received a lot of attention, perhaps most notably in connection with neuroscience [1, 30, 15, 20, 21], where the entropy rate plays a crucial role in efforts to describe and quantify the amount of information communicated between neurons.”
BackgroundOptimal entropy estimation on large alphabets via best polynomial approximation
Wu, Y., & Yang, P. (2015) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2015.7282570 · Cited by 8
A simple method for estimating the entropy of neural activity
Berry, M. J., II, Tkačik, G., Dubuis, J., Marre, O., & da Silveira, R. A. (2013) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/2013/03/p03015 · Cited by 8
“In his beautiful piece of work [11], Paninski lays out a number of general results and puts forth the so-called best upper bound estimator.”
BackgroundMethodInfluentialSynthesized feature space for multiclass emotion classification
Thomas, B., Dhanya, K. A., & Vinod, P. (2014) · International Conference on Networks and Soft Computing (ICNSoC) · doi:10.1109/cnsc.2014.6906656 · Cited by 8
“The feature selection techniques used here to select relevant features from the large dataset containing redundant features are (a) Weighted Log Likelihood (WLLS) [6] (b) Normalized Google Distance (NGD) [7] and (c) Mutual Information (MI) [13].”
MethodThe effect of automated taxa identification errors on biological indices
Ärje, J., Kärkkäinen, S., Meissner, K. K., Iosifidis, A., İnce, T., Gabbouj, M., & Serkan Kiranyaz, B. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.12.015 · Cited by 10
“Paninski (2003) studies nonparametric estimation of H 0 and gives an overview on its bias and variance.”
Preserving physically important variables in optimal event selections: a case study in Higgs physics
Windischhofer, P., Zgubič, M., & Bortoletto, D. (2020) · Journal of High Energy Physics · doi:10.1007/jhep07(2020)001 · Cited by 9
Controlling Costs in Feature Selection: Information Theoretic Approach
Teisseyre, P., & Klonecki, T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77964-1_37 · Cited by 8
Bird’s Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach
Hou, Y., & Sachan, M. (2021) · doi:10.18653/v1/2021.acl-long.145 · Cited by 13
“Traditional methods (Moon et al., 1995; Steuer et al., 2002; Paninski, 2003) for MI estimation do not scale well with large sample size or dimension (Gao et al., 2015).”
InfluentialAn improved estimator of Shannon entropy with applications to systems with memory
Gregorio, J. D., Sánchez, D., & Toral, R. (2022) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2022.112797 · Cited by 13
“There is not known unbiased estimator for the entropy [27].”
BackgroundHsieh, F., Chou, E. P., & Chen, T. (2023) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2023.129227 · Cited by 8
“The rigorous discussion of related issues of how to make reliable CE-drop evaluations is given in [19], where results are sharply contrasting with results reported in [20].”
ResultImprecise probabilistic inference from sequential data.
Prat-Carrabin, A., & Woodford, M. (2024) · Psychological Review · doi:10.1037/rev0000469 · Cited by 12
Reading Between the Spikes of the Hypothalamic Neural Code
Bhumbra, G. S., & Dyball, R. E. J. (2010) · Journal of Neuroendocrinology · doi:10.1111/j.1365-2826.2010.02068.x · Cited by 8
Coding processes involved in the cortical representation of complex tactile stimuli
Blanc, J.-L., & Coq, J. (2007) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2007.10.004 · Cited by 9
“Such a high-dimensionality leads to large undersampled probability distributions and to large biases (Panzeri and Treves, 1996; Paninski , 2003).”
BackgroundFaes, L., & Nollo, G. (2013) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2013.6610683 · Cited by 8
Bifurcation Structure of a Class of SN-invariant Constrained Optimization Problems
Parker, A. E., & Gedeon, T. (2004) · Journal of Dynamics and Differential Equations · doi:10.1007/s10884-004-6112-0 · Cited by 8
“The limited data problem makes a nonparametric determination of p(X,Y ) impossible [30], and makes parametric estimations tenuous at best.”
BackgroundNormal Laws for Two Entropy Estimators on Infinite Alphabets
Chen, C., Grabchak, M., Stewart, A., Zhang, J., & Zhang, Z. (2018) · Entropy · doi:10.3390/e20050371 · Cited by 8
“The problem of statistical estimation of entropy has a long history (see the survey paper [3] or the recent book [4]).”
BackgroundA Regression Approach to Certain Information Transmission Problems
Zhang, W., Wang, Y., Shen, C., & Liang, N. (2019) · IEEE Journal on Selected Areas in Communications · doi:10.1109/jsac.2019.2933964 · Cited by 9
“First, nonparametric estimation of mutual information with training data only (see, e.g., [2]) and maximization of mutual information with respect W. Zhang, Y. Wang, and N. Liang are with CAS Key Laboratory of Wireless-Optical Communications, University of Science and Technology of China, China.”
MethodOptimality of the Plug-in Estimator for Differential Entropy Estimation under Gaussian Convolutions
Goldfeld, Z., Greenewald, K. H., Weed, J., & Polyanskiy, Y. (2019) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2019.8849414 · Cited by 8
“Despite the suboptimality of plug-in techniques for vanilla discrete (Shannon) and differential entropy estimation (see [4] and [5], respectively), we show that h ( P̂Xn ∗ Nσ ) attains the”
He, Y., Sheng, Z., Zhu, Y., & He, M. (2020) · Journal of Atmospheric and Oceanic Technology · doi:10.1175/jtech-d-20-0004.1 · Cited by 8
Generalization from Observed to Unobserved Features by Clustering
Krupka, E., & Tishby, N. (2008) · Journal of Machine Learning Research · doi:10.5555/1390681.1390692 · Cited by 10
Critical Values of a Kernel Density-based Mutual Information Estimator
May, R., Dandy, G. C., Maier, H. R., & Fernando, T. M. K. G. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2006.247170 · Cited by 8
“In the case of stochastic measures of dependency, such as in the case of the MI estimator described, these cases must be considered within a statistical context [1].”
BackgroundSpeeding Up Feature Subset Selection Through Mutual Information Relevance Filtering
Dijck, G. V., & Hulle, M. M. V. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74976-9_27 · Cited by 8
“Estimators of the mutual information based on histograms are biased [8], due to the discretization in intervals and thus need to be avoided.”
BackgroundInstance-by-instance optimal identity testing.Preprint
Valiant, G. J., & Valiant, P. (2013) · Electronic colloquium on computational complexity · Cited by 8
Trace Estimation of Quantum State Powers: Sample Complexity and Computational HardnessPreprint
Chen, K., Liu, Y., & Wang, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.09563 · Cited by 9
“In addition, Shannon entropy estimation was studied in [Pan03, Pan04, VV11a, VV11b, VV17, WY16].”
End-to-end optimal detector design with mutual information surrogates
Wozniak, K. A., Mulligan, S. B., Kieseler, J., Klute, M., Fleuret, F., & Golling, T. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ae1acb · Cited by 8
“The study therefore demonstrates optimization under idealized conditions, with robustness to real-world effects left for future work.”
CATOM: Causal Topology Map for Spatiotemporal Traffic Analysis With Granger Causality in Urban Areas
Jung, C., Yim, S., Park, G., Oh, S., & Jang, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3489676 · Cited by 7
“SinceMutualInformation(MI)doesnotassume aparticulardatadistributionorrestrictrelationshipstolinearor nonlinearforms[68],itremainsinsensitivetospecificpatterns liketrendeffectsandseasonaleffects.”
Njitacke, Z. T., Tegnitsap, J. V. N., Zhao, M., Barà, C., Fozin, T. F., Awrejcewicz, J., Yoshimura, N., Valdes-Sosa, P. A., Letellier, C., & Minati, L. (2025) · Chinese Journal of Physics · doi:10.1016/j.cjph.2025.04.006 · Cited by 7
On the Efficiency of Mathematics in Intrusion Detection: The NetEntropy Case
Goubault-Larrecq, J., & Olivain, J. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-05302-8_1 · Cited by 7
“The sample entropy Ĥ N , introduced above, is an estimator, sometimes called the plug-in estimate, or maximum likelihood estimator [17].”
BackgroundApplying Information Theory Analysis for the Solution of Biomedical Data Processing Problems
Blokh, D., & Stambler, I. (2014) · Current Research in Bioinformatics · doi:10.3844/ajbsp.2014.17.29 · Cited by 7
“…represents a pinnacle of statistical research: The ideas are at once beautiful and applicable to a remarkably wide variety of questions” (Paninski, 2003) and according to one of the pioneers of the use of information theory in biomedicine, Quastler (1958), “The basic concepts of…”
BackgroundBeyond maximum likelihood: Boosting the Chow-Liu algorithm for large alphabets
Jiao, J., Han, Y., & Weissman, T. (2016) · Asilomar Conference on Signals, Systems and Computers (ACSSC) · doi:10.1109/acssc.2016.7869051 · Cited by 7
“In contrast to all these schemes which require n = Θ(S/ lnS) samples, the MLE requires n = Θ(S) samples [11], [19].”
Modeling Collaboration Using Point Processes
Halpin, P. F., & von Davier, A. A. (2017) · Methodology of educational measurement and assessment · doi:10.1007/978-3-319-33261-1_15 · Cited by 7
Complex Dietary Topologies in Non-alcoholic Fatty Liver Disease: A Network Science Analysis
Xia, Y., Zhao, Z., Zhang, S., Liu, Y., Meng, G., Zhang, Q., Liu, L., Wu, H., Gu, Y., Wang, Y., et al. (2020) · Frontiers in Nutrition · doi:10.3389/fnut.2020.579086 · Cited by 7
“First, we computed the MI matrix for cases and controls using the Miller–Madow estimator (34) using the build.”
MethodHuman-in-the-loop handling of knowledge drift
Bontempelli, A., Giunchiglia, F., Passerini, A., & Teso, S. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00845-0 · Cited by 10
“4 Unlike other theoretically well-behaved discrepancies (like total variation distance and A-distance [14]), the MMD can be estimated efficiently (in linear or quadratic time, depending on the estimator) even for higher-dimensional data [19, 21, 10].”
BackgroundCausality in Time Series: Its Detection and Quantification by Means of Information Theory
Hlaváčková‐Schindler, K. (2008) · Information Theory and Statistical Learning · doi:10.1007/978-0-387-84816-7_8 · Cited by 7
“Paninski (64) used an exact local expansion of the entropy function and proved almost sure consistency (strong consistency) for three of the most commonly used discretized information estimators, namely the maximum likelihood (MLE) estimator the MLE with the so-called Miller-Madow bias correction…”
MethodUnderstanding Deep Neural Networks Using Sliced Mutual Information
Wongso, S., Ghosh, R., & Motani, M. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834357 · Cited by 7
InfoShape: Task-Based Neural Data Shaping via Mutual Information
Esfahanizadeh, H., Wu, W., Ghobadi, M., Barzilay, R., & Médard, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095632 · Cited by 12
“Consequently, MI is difficult to compute for a finite dataset of high-dimensional inputs [22].”
BackgroundBayesian estimation of information-theoretic metrics for sparsely sampled distributions
Piga, A., Font-Pomarol, L., Sales‐Pardo, M., & Guimerà, R. (2024) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2024.114564 · Cited by 7
Camaglia, F., Nemenman, I. M., Mora, T., & Walczak, A. M. (2024) · Physical review. E · doi:10.1103/physreve.109.024305 · Cited by 7
Brinkman, B. A. W. (2025) · PRX Life · doi:10.1103/kjx5-1hwy · Cited by 7
Zheng, T., Li, B., Wu, S., Wan, B., Mu, G., Liu, S., Ding, S., & Wang, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.131388 · Cited by 7
“Mutual information measures the relationship between different random variables [32].”
BackgroundDetecting Genetic Interactions for Quantitative Traits Using m -Spacing Entropy Measure
Yee, J., Kwon, M., Jin, S., Park, T., & Park, M. (2015) · BioMed Research International · doi:10.1155/2015/523641 · Cited by 7
Informative Priors for Markov Blanket Discovery
Pocock, A., Luján, M., & Gavin Brown, B. (2012) · Research Explorer (The University of Manchester) · Cited by 7
An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewards
Dai, S., Xu, W., Hofmann, A., & Williams, B. C. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.001 · Cited by 7
Active Feature Selection for the Mutual Information Criterion
Schnapp, S., & Sivan Sabato, A. (2021) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v35i11.17144 · Cited by 7
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.