Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 201–300 of 1,004
Rose Hips, a Valuable Source of Antioxidants to Improve Gingerbread Characteristics
Ghendov‐Moșanu, A., Cristea, E., Patraș, A., Sturza, R., & Niculaua, M. (2020) · Molecules · doi:10.3390/molecules25235659 · Cited by 68
“Entropy and information were the main concepts employed, while the bit was the unit of measure [56,57].”
BackgroundUniversal Entropy Estimation Via Block Sorting
Cai, H., Kulkarni, S. R., & Verdú, S. (2004) · IEEE Transactions on Information Theory · doi:10.1109/tit.2004.830771 · Cited by 73
“In [21], it is pointed out that in the undersampled regime, i.e., the alphabet size is comparable to n, the plug-in estimator can be contaminated by the bias, although the variance converges to zero fast.”
BackgroundApplications of Entropy in Data Analysis and Machine Learning: A Review
Sepúlveda-Fontaine, S. A., & Amigó, J. M. (2024) · Entropy · doi:10.3390/e26121126 · Cited by 85
“See [273] for a general review on the estimation of entropy.”
Empirical Estimation of Information Measures: A Literature Guide
Verdú, S. (2019) · Entropy · doi:10.3390/e21080720 · Cited by 81
“In addition to standard statistical bias-reduction techniques (e.g., [33–37]) and shrinkage techniques [38], one way to ameliorate this issue is to distort the function f ( · ) in Equation (6) by replacing the initial portion with a polynomial (see, e.g., [39–42]).”
BackgroundMethodInformation theory in auditory research
Nelken, I., & Chechik, G. (2007) · Hearing Research · doi:10.1016/j.heares.2007.01.012 · Cited by 66
“This means that it tends to produce overestimates of the MI relative to the MI of the true distribution, Ið~ pðS; RÞÞ > IðpðS; RÞÞ (but not always: the theoretical bias may be negative, see the discussion in Paninski, 2003).”
BackgroundMethodInfluentialA fair evaluation framework for comparing side-channel distinguishers
Whitnall, C., & Oswald, E. (2011) · Journal of Cryptographic Engineering · doi:10.1007/s13389-011-0011-1 · Cited by 64
“Unfortunately, estimators for MI do not behave so ‘nicely’ as the sample correlation coefficient; in fact, there are no universal rates of convergence [23], so that whatever estimator we pick, we can always find a distribution for which the error vanishes arbitrarily slowly.”
BackgroundA CLT and tight lower bounds for estimating entropy.Preprint
Valiant, G. J., & Valiant, P. (2010) · Electronic colloquium on computational complexity · Cited by 64
Information-theoretic methods for studying population codes
Ince, R. A. A., Senatore, R., Arabzadeh, E., Montani, F., Diamond, M. E., & Panzeri, S. (2010) · Neural Networks · doi:10.1016/j.neunet.2010.05.008 · Cited by 70
“This is the case for any stimulus–response probability distribution (Paninski, 2003).”
A Large-Deviation Analysis of the Maximum-Likelihood Learning of Markov Tree Structures
Tan, V. Y. F., Anandkumar, A., Tong, L., & Willsky, A. S. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2104513 · Cited by 73
“Importantly, if P is not a tree, there may be several trees that are optimal projections [9] and this requires careful consideration of the error events.”
BackgroundOpen-World Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding
Liu, Q., Wen, Y., Han, J., Xu, C., Xu, H., & Liang, X. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-20044-1_16 · Cited by 89
Ruddell, B. L., Drewry, D. T., & Nearing, G. (2019) · Water Resources Research · doi:10.1029/2018wr023692 · Cited by 62
“The bin ‐ edge and small ‐ sample errors are a known systematic error tend to cause underestimation of Shannon entropy and overestimation of the transfer entropy (Paninski, 2003; Ruddell & Kumar, 2009a).”
BackgroundDorval, A. D. (2008) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2008.05.013 · Cited by 62
“However, despite major advances in entropy estimation techniques (e.g., Treves and Panzeri, 1995; Strong et al., 1998; Paninski, 2003) and substantial insights information theory has helped illuminate (e.g., Reinagel et al., 1999; Koch et al., 2006), its application to neuroscience remains a niche…”
BackgroundMethodInfluentialBarraza, N. R., Moro, S., Ferreyra, M., & de la Peña, A. (2018) · Journal of Information Science · doi:10.1177/0165551518770967 · Cited by 64
“Thus, it is linked to the entropy of a random feature, given by the amount of information held in the feature [Paninski, 2003].”
Kwak, N.-S., & Lee, S. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2924237 · Cited by 65
“They also formulated that the relation between ear-EEG and scalp-EEG was linear based on the mutual information theory [53]; note that the predictions based on the mutual information theory eventually led to the multiple linear regression (MLR) method used in our experiment.”
MethodGoldberg, D. H., Victor, J. D., Gardner, E. P., & Gardner, D. (2009) · Neuroinformatics · doi:10.1007/s12021-009-9049-y · Cited by 78
“& Best upper bound (Paninski 2003) & Coverage-adjusted (Chao and Shen 2003).”
BackgroundMethodInfluentialThe Efficiency of Data Assimilation
Nearing, G., Yatheendradas, S., Crow, W. T., Zhan, X., Liu, J., & Chen, F. (2018) · Water Resources Research · doi:10.1029/2017wr020991 · Cited by 60
“Paninski (2003) gives a review of some simple empirical estimators for entropy and mutual information metrics, and we use those estimators here.”
MethodBootstrap Methods for the Empirical Study of Decision-Making and Information Flows in Social Systems
DeDeo, S., Hawkins, R., Klingenstein, S., & Hitchcock, T. (2013) · Entropy · doi:10.3390/e15062246 · Cited by 80
“As proven in [2], any estimator of entropy is necessarily biased—meaning that estimates of H made on a finite sample will, on average, disagree with the asymptotic value.”
BackgroundMethodMinimax Estimation of Discrete Distributions Under $\ell _{1}$ Loss
Han, Y., Jiao, J., & Weissman, T. (2015) · IEEE Transactions on Information Theory · doi:10.1109/tit.2015.2478816 · Cited by 59
Chase, S. M., & Young, E. D. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.4986-05.2006 · Cited by 59
“Estimates of MI based on finite datasets are subject to bias (Treves and Panzeri, 1995; Panzeri and Treves 1996; Paninski, 2003).”
BackgroundEstimation of Information Theoretic Measures for Continuous Random Variables
Pérez‐Cruz, F. (2008) · neural information processing systems · Cited by 73
Mäki-Marttunen, V., Diez, I. P., Cortés, J. M., Chialvo, D. R., & Villarreal, M. F. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00024 · Cited by 61
Palomar, D. P., & Verdú, S. (2008) · IEEE Transactions on Information Theory · doi:10.1109/tit.2007.915715 · Cited by 58
“Mutual information has also proven a popular measure of statistical dependence in many experimental applications such as neurobiology [2], genetics [3], machine learning [4], [5], medical imaging [6], linguistics [7], artificial intelligence [8], authentication [9], and signal processing [10].”
BackgroundOptimal Sensor Configuration and Feature Selection for AHU Fault Detection and Diagnosis
Li, D., Zhou, Y., Hu, G., & Spanos, C. J. (2016) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2016.2644669 · Cited by 58
“In this work, we use [24] for mutual information estimation.”
MethodQuantifying Process Connectivity With Transfer Entropy in Hydrologic Models
Bennett, A. R., Nijssen, B., Ou, G., Clark, M., & Nearing, G. (2019) · Water Resources Research · doi:10.1029/2018wr024555 · Cited by 64
“There are many ways to estimate these distributions with varying degrees of complexity, bias, and stability (Gong et al., 2014; Hlaváčková‐Schindler et al., 2007; Paninski, 2003).”
BackgroundLeakage Certification Revisited: Bounding Model Errors in Side-Channel Security Evaluations
Bronchain, O., Hendrickx, J. M., Massart, C., Olshevsky, A., & Standaert, F. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-26948-7_25 · Cited by 69
A Distribution-Based Approach to Anomaly Detection and Application to 3G Mobile Traffic
D’Alconzo, A., Coluccia, A., Ricciato, F., & Romirer‐Maierhofer, P. (2009) · Globecom · doi:10.1109/glocom.2009.5425651 · Cited by 57
Quantum Query Complexity of Entropy Estimation
Li, T., & Wu, X. (2018) · IEEE Transactions on Information Theory · doi:10.1109/tit.2018.2883306 · Cited by 79
An Entropy-Based Approach to Portfolio Optimization
Mercurio, P. J., Wu, Y., & Xie, H. (2020) · Entropy · doi:10.3390/e22030332 · Cited by 67
“In fact, it has been shown by Paninski (2003) [64] that there does not exist an unbiased estimator of entropy.”
BackgroundConsistent distribution-free $K$-sample and independence tests for univariate random variablesPreprint
Heller, R., Heller, Y., Kaufman, S., Brill, B., & Gorfine, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.6758 · Cited by 57
“…1 (logN)2 (B.9) where the first inequality follows from the fact that {| ∑ I(Ĥ I UV−E(ĤIUV ))| ≥ |Π| /3} ⊆ {∪I∈Π(ĤIUV −E(ĤIUV )) ≥ /3} and Bonferroni’s inequality, and the second inequality follows from the upper bound (3.4) in Paninski (2003) for the plug in estimator for a given partition I.”
BackgroundMethodInfluentialEvaluation of Mutual Information Estimators for Time SeriesPreprint
Papana, A., & Kugiumtzis, D. (2009) · Cited by 57
How (Non-)Optimal is the Lexicon?
Pimentel, T., Nikkarinen, I., Mahowald, K., Cotterell, R., & Blasí, D. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.350 · Cited by 61
Mahmud, M., & Vassanelli, S. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00248 · Cited by 61
“BUB (Paninski, 2003); 8.”
Arleo, A., Nieus, T. R., Bezzi, M., D’Errico, A., D'Angelo, E. U., & Coenen, O. J. M. D. (2010) · Neural Computation · doi:10.1162/neco_a_00006-arleo · Cited by 62
“Both MI and the surprise measure have a systematic bias due to the limited data samples available (Nemenman, Bialek, & de Ruyter van Steveninck, 2004; Paninski, 2003; Strong,Koberle, deRuyter van Steveninck, & Bialek, 1998; Treves & Panzeri, 1995).”
BackgroundMaximum Entropy Approaches to Living Neural Networks
Yeh, F.-C., Tang, A., Hobbs, J., Hottowy, P., Dąbrowski, W., Sher, A., Litke, A. M., & Beggs, J. M. (2010) · Entropy · doi:10.3390/e12010089 · Cited by 63
Feature-Level Deeper Self-Attention Network With Contrastive Learning for Sequential Recommendation
Hao, Y., Zhang, T., Zhao, P., Liu, Y., Sheng, V. S., Xu, J., Liu, G., & Zhou, X. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3250463 · Cited by 61
“It is based on mutual information [56], which can”
BackgroundPAC-Bayesian Analysis of Co-clustering and Beyond
Seldin, Y., & Tishby, N. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1953046 · Cited by 81
“Theorem 10 (Paninski, 2003)Let X1, ..,XN be i.i.d. distributed byp(x) and let p̂(x) be their empirical distribution.”
BackgroundMethodInfluentialCoincidences and Estimation of Entropies of Random Variables with Large Cardinalities
Nemenman, I. M. (2011) · Entropy · doi:10.3390/e13122013 · Cited by 53
“Developments driven in part by computational biology applications have solved this problem in the moderately undersampled regime, N ∼ K and N ∼ e [1–9].”
BackgroundA Theory of Associating Food Types with Their Postingestive Consequences
Yearsley, J. M., Villalba, J. J., Gordon, I. J., Kyriazakis, I., Speakman, J. R., Tolkamp, B. J., Illius, A. W., & Duncan, A. J. (2006) · The American Naturalist · doi:10.2307/3844777 · Cited by 60
Forecastable Component Analysis
Goerg, G. M. (2013) · Cited by 53
A Mutual Information-Based Two-Phase Memetic Algorithm for Large-Scale Fuzzy Cognitive Map Learning
Zou, X., & Liu, J. (2017) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2017.2764445 · Cited by 56
Privacy With Estimation Guarantees
Wang, H., Vo, L., Calmon, F. P., Médard, M., Duffy, K. R., & Varia, M. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2019.2934414 · Cited by 64
“Also of note, the study of robustness of estimated distributions with finite sample size has appeared in [48]–[51].”
BackgroundSharpee, T. O., Miller, K. D., & Stryker, M. P. (2008) · Journal of Neurophysiology · doi:10.1152/jn.01397.2007 · Cited by 53
“This is as expected theoretically for white Gaussian or other uncorrelated (spherically symmetric) stimuli (Chichilnisky 2001; de Boer and Kuyper 1968; Paninski 2003a; Rieke et al. 1997; Sharpee et al. 2004).”
BackgroundMethodInfluentialMeasuring Disentanglement: A Review of Metrics
Carbonneau, M.-A., Zaidi, J., Boilard, J., & Gagnon, G. (2020) · arXiv · doi:10.48550/arxiv.2012.09276 · Cited by 52
“Even assessing the quality of the estimators remains an open problem [36].”
BackgroundGurev, A., Ceșko, T., Dragancea, V., Ghendov‐Moșanu, A., Pintea, A., & Sturza, R. (2023) · Foods · doi:10.3390/foods12142773 · Cited by 55
“The more pronounced the influence of the pH of the medium in extracts on the investigated results, the higher the bit value [69].”
BackgroundVariational Gibbs state preparation on noisy intermediate-scale quantum devices
Consiglio, M., Settino, J., Giordano, A., Mastroianni, C., Plastina, F., Lorenzo, S., Maniscalco, S., Goold, J., & Apollaro, T. J. G. (2024) · Physical Review A · doi:10.1103/physreva.110.012445 · Cited by 57
Performance Assessment of MIMO-BICM Demodulators Based on Mutual Information
Fertl, P., Jaldén, J., & Matz, G. (2011) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2011.2177823 · Cited by 52
“Based on the results in [48], we numerically optimized the bin-”
BackgroundMethodInfluentialUniversal Estimation of Information Measures for Analog Sources
Wang, Q., Kulkarni, S. R., & Verdú, S. (2009) · Foundations and Trends® in Communications and Information Theory · doi:10.1561/0100000021 · Cited by 59
“Information-theoretic methods have also been used in neuroscience to study the dependence between stimuli and neural response [36, 100, 176, 184, 232] and to classify neurons according to their functions [126, 213].”
BackgroundMethodMillisecond-Scale Motor Encoding in a Cortical Vocal Area
Tang, C., Chehayeb, D., Srivastava, K. H., Nemenman, I. M., & Sober, S. J. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1002018 · Cited by 52
“On the other hand, we know that no method can be universally unbiased for every underlying probability distribution in the severely undersampled, square-root, regime [58].”
MethodEntropy Estimation in Turing's Perspective
Zhang, Z. (2012) · Neural Computation · doi:10.1162/neco_a_00266 · Cited by 52
“The two most commonly cited review articles are by Beirlant, Dudewicz, Gy¨orfi, and Meulen (2001) and Paninski (2003).”
BackgroundA Statistical Test for Information Leaks Using Continuous Mutual Information
Chothia, T., & Guha, A. (2011) · IEEE Computer Security Foundations Workshop (CSFW) · doi:10.1109/csf.2011.19 · Cited by 51
“It is known that there is no universal rate at which the error in estimation of mutual information goes to zero, no matter what estimator we pick, see [2] and [29].”
BackgroundCoverage‐adjusted entropy estimation
Vu, V. Q., Yu, B., & Kass, R. E. (2007) · Statistics in Medicine · doi:10.1002/sim.2942 · Cited by 63
“A more principled correction based on the jack-knife technique was proposed earlier by Zahl [17] Unfortunately, it is an asymptotic correction that depends on the unknown parameter m. Paninski [10] observed that both the MLE and Miller–Maddow estimates fall into a class of estimators that are linear in the frequencies…”
BackgroundMethodInfluentialMehralian, F., Salehnamadi, N., & Malek, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468604 · Cited by 63
“To measure the dependency between two random variables, we calculate their Mutual Information (MI) [41].”
MethodDistributional Property Testing in a Quantum World
Gilyén, A., & Li, T. (2019) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itcs.2020.25 · Cited by 54
Followers Are Not Enough: A Multifaceted Approach to Community Detection in Online Social Networks
Darmon, D. M., Omodei, E., & Garland, J. (2015) · PLoS ONE · doi:10.1371/journal.pone.0134860 · Cited by 54
“|X t−1 −k −Hˆ X |Xt−1,Yt−1 (16) where we use the plug-in estimatorsHˆ X t|X t−1 t−k and Hˆ X t|X t−1 t−k,Y t−1 t−k for the entropies. It is well known that the plug-in estimator for entropy is biased [37]. To account for this bias, we use the Miller-Madow adjustment to the plug-in estimator [31]. For a random variable…”
MethodBayesian and Quasi-Bayesian Estimators for Mutual Information from Discrete Data
Archer, E., Park, I. M., & Pillow, J. W. (2013) · Entropy · doi:10.3390/e15051738 · Cited by 50
“Efficient Computation Computation of the posterior mean and variance under the NSB prior can be carried out more efficiently using a representation in terms of multiplicities, also known as the empirical histogram distribution function [8], which is the number of bins in the empirical distribution with each count.”
BackgroundMethodEstimation of mutual information for real-valued data with error bars and controlled bias
Holmes, C. M., & Nemenman, I. M. (2019) · Physical review. E · doi:10.1103/physreve.100.022404 · Cited by 50
Demystifying fixed k-nearest neighbor information estimators
Gao, W., Oh, S., & Viswanath, P. (2017) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2017.8006732 · Cited by 50
“A canonical question in all these applications is to estimate the information theoretic quantities from samples, typically supposed to be drawn i.i.d. from an unknown distribution.”
BackgroundAldworth, Z., Miller, J. P., Gedeon, T., Cummins, G. I., & Dimitrov, A. G. (2005) · Journal of Neuroscience · doi:10.1523/jneurosci.0359-05.2005 · Cited by 55
“There are alternate approaches for estimating encoding rates that make no explicit assumptions about details of stimulus representation, such as the direct method (Strong et al., 1998; Paninski, 2003).”
MethodInfluentialInferring Nonlinear Gene Regulatory Networks from Gene Expression Data Based on Distance Correlation
Guo, X., Zhang, Y., Hu, W., Tan, H., & Wang, X. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087446 · Cited by 59
“, empirical estimator [15], Miller-Madow [15], shrinkage [16] and the Schurmann-Grassberger mutual information estimator [17].”
MethodLPSE: Lightweight password-strength estimation for password meters
Guo, Y., & Zhang, Z. (2017) · Computers & Security · doi:10.1016/j.cose.2017.07.012 · Cited by 61
“An accurate measure of the password probability distribution requires a large number of password samples (Bonneau, 2012a; Paninski, 2003).”
BackgroundGhendov‐Moșanu, A., Cojocari, D., Bălan, G., Patraș, A., Lung, I., Soran, M.-L., Opriș, O., Cristea, E., & Sturza, R. (2022) · Molecules · doi:10.3390/molecules27051610 · Cited by 50
“Thus, the higher the values of mutual information, the lower the uncertainties [101].”
BackgroundCesa‐Bianchi, N., Shalev‐Shwartz, S., & Shamir, O. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2164053 · Cited by 49
“Of course, this example pertains to a very simple RKHS with a finite dimensional representation.”
BackgroundDecoding Stimulus Variance from a Distributional Neural Code of Interspike Intervals
Lundstrom, B. N., & Fairhall, A. L. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.0225-06.2006 · Cited by 56
“3481), the bias-corrected estimators are effective (Paninski,2003).”
BackgroundAnalysis of neural coding through quantization with an information-based distortion measure
Dimitrov, A. G., Miller, J. P., Gedeon, T., Aldworth, Z., & Parker, A. E. (2003) · Network Computation in Neural Systems · doi:10.1080/net.14.1.151.176 · Cited by 51
“In fact, as noted in [28], both can be seen as extensions of Grenander’s method of sieves [16] for estimating expectations of probabilistic quantities.”
BackgroundInformation-Theoretic Inference of Gene Networks Using Backward Elimination
Meyer, P. E., Marbach, D., Roy, S., & Kellis, M. (2010) · ORBi (University of Liège) · Cited by 59
Jackknife approach to the estimation of mutual information
Zeng, X., Xia, Y., & Tong, H. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1715593115 · Cited by 57
“This definition can be easily extended to other types of random variables that may not have density functions (13).”
BackgroundInce, R. A. A., Mazzoni, A., Bartels, A., Logothetis, N. K., & Panzeri, S. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.11.013 · Cited by 50
Assessing functional connectivity of neural ensembles using directed information
So, K., Koralek, A. C., Ganguly, K., Gastpar, M., & Carmena, J. M. (2012) · Journal of Neural Engineering · doi:10.1088/1741-2560/9/2/026004 · Cited by 52
Generalization of the Ignorance Score: Continuous Ranked Version and Its Decomposition
Tödter, J., & Ahrens, B. (2012) · Monthly Weather Review · doi:10.1175/mwr-d-11-00266.1 · Cited by 54
“The Miller series (Miller 1955; Paninski 2003) describes the bias when estimating the entropy of a probability distribution by using the relative frequencies in the dataset:”
BackgroundGerken, T., Ruddell, B. L., Fuentes, J. D., de Araújo, A. C., Brunsell, N. A., Maia, J. M. F., Manzi, A. O., Mercer, J. R., dos Santos, R. N., von Randow, C., et al. (2017) · Agricultural and Forest Meteorology · doi:10.1016/j.agrformet.2017.03.023 · Cited by 49
“The number of data used for each calculation ranges from 269 to 825 observations, which is enough for qualitatively robust estimation of I (Paninski, 2003; Ruddell and Kumar, 2009a).”
MethodSymbolic transfer entropy test for causality in longitudinal data
Camacho, M., Romeu, A., & Marín, M. R. (2020) · Economic Modelling · doi:10.1016/j.econmod.2020.02.007 · Cited by 47
Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
Jin, W., Liu, X., Ma, Y., Aggarwal, C. C., & Tang, J. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539445 · Cited by 66
“However, maximizing mutual information directly is generally intractable when A or B is obtained through neural networks [29], thus we resort to maximizing a lower bound on MI(A, B).”
BackgroundA novel approach for distributed application scheduling based on prediction of communication events
Dodonov, E., & de Mello, R. F. (2009) · Future Generation Computer Systems · doi:10.1016/j.future.2009.05.004 · Cited by 47
Jiang, Z., Bi, Y., Zhou, M., Hu, Y., Burke, M., & Navab, N. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649231223547 · Cited by 45
“Nevertheless, the fast computation of MI is restricted to discrete variables (Paninski 2003).”
BackgroundSublinear Algorithms for Approximating String Compressibility
Raskhodnikova, S., Ron, D., Rubinfeld, R., & Smith, A. (2012) · Algorithmica · doi:10.1007/s00453-012-9618-6 · Cited by 61
“[9], Paninski [33,34], Brautbar and Samorodnitsky [6].”
Mardia, J., Jiao, J., Tánczos, E., Nowak, R. D., & Weissman, T. (2019) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaz025 · Cited by 65
Information-theoretic wavelet packet subband selection for texture classification
Huang, K., & Aviyente, S. (2005) · Signal Processing · doi:10.1016/j.sigpro.2005.07.032 · Cited by 45
“This problem of estimation has been studied in past research [28,29].”
ResultTsubo, Y., Isomura, Y., & Fukai, T. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002461 · Cited by 48
“Shannon’s information theory and mutual information have often been used to extract the information represented by neural activity [4–15].”
MethodThe Linux Pseudorandom Number Generator RevisitedPreprint
Lacharme, P., Röck, A., Strubel, V., & Videau, M. (2012) · Cited by 45
Unbiased and efficient log-likelihood estimation with inverse binomial sampling
van Opheusden, B., Acerbi, L., & Ma, W. J. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008483 · Cited by 57
“Incidentally, the lack of bias in entropy estimates by IBS may be surprising in light of a theorem stating that uniformly unbiased estimators of the entropy given a finite set of samples cannot exist [53].”
BackgroundA Novel Estimator of Mutual Information for Learning to Disentangle Textual Representations
Colombo, P., Piantanida, P., & Clavel, C. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.511 · Cited by 56
“However, estimating MI has been a long-standing challenge, in particular when dealing with high-dimensional data (Paninski, 2003; Pichler et al., 2020).”
BackgroundErdmenger, J., Grosvenor, K. T., & Jefferson, R. (2022) · SciPost Physics · doi:10.21468/scipostphys.12.1.041 · Cited by 45
Gallistel, C. R., Craig, A. R., & Shahan, T. A. (2013) · Behavioural Processes · doi:10.1016/j.beproc.2013.08.012 · Cited by 43
Tradeoffs and Constraints on Neural Representation in Networks of Cortical Neurons
Kermany, E., Gal, A., Lyakhov, V., Meir, R., Marom, S., & Eytan, D. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.0661-10.2010 · Cited by 44
“…the information content of a specific neural response feature with regard to a specific input feature (each regraded as a random variable) (Paninski, 2003; Nelken and Chechik, 2007), we chose the decoding approach, which put a lower limit on the information content (in the sense that an…”
On the Use of Financial Data as a Random Beacon.Preprint
Clark, J., & Hengartner, U. (2010) · IACR Cryptology ePrint Archive · Cited by 53
Optimized dissimilarity space embedding for labeled graphs
Livi, L., Rizzi, A., & Sadeghian, A. (2014) · Information Sciences · doi:10.1016/j.ins.2014.01.005 · Cited by 41
“A key issue in this context is the estimation of information-theoretic descriptors from a data sample, such as entropy and mutual information [29,40,44,45].”
BackgroundMinimax Estimation of Divergences Between Discrete Distributions
Han, Y., Jiao, J., & Weissman, T. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.3041036 · Cited by 49
“Specifically, the minimax-optimal estimation procedures have been found for many non-smooth 1D functionals, including the entropy [21], [22], [24], [27], [28], L1 norm of the mean vector [23], support size [26], [35], support coverage [29], [30], distance to uniformity [25], [34], and general 1-Lipschitz functions [36…”
BackgroundMethodNabavi, S. O., Haimberger, L., Abbasi, R., & Samimi, C. (2018) · Aeolian Research · doi:10.1016/j.aeolia.2018.10.002 · Cited by 44
Zhou, K., Wang, W., Wu, C., & Hu, T. (2020) · ETRI Journal · doi:10.4218/etrij.2019-0190 · Cited by 48
“Paninski [20] concluded that at least N > m samples were needed for the uniformity test.”
MethodA Resilient and Hierarchical IoT-Based Solution for Stress Monitoring in Everyday Settings
Jiang, S., Firouzi, F., Chakrabarty, K., & Elbogen, E. B. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3122015 · Cited by 44
Predicting the price of WTI crude oil futures using artificial intelligence model with chaos
Yin, T., & Wang, Y. (2022) · Fuel · doi:10.1016/j.fuel.2021.122523 · Cited by 40
CENCALC: A computational tool for conformational entropy calculations from molecular simulations
Suárez, E., Díaz, N., Méndez‐Hurtado, J., & Suárez, D. (2013) · Journal of Computational Chemistry · doi:10.1002/jcc.23350 · Cited by 40
Dependency Reduction with Divisive Normalization: Justification and Effectiveness
Lyu, S. (2011) · Neural Computation · doi:10.1162/neco_a_00197 · Cited by 43
“However, nonparametric estimation of entropy and MI from data may be difficult, as straightforward estimation using histograms are prone to strong biases (Paninski, 2003), and this may be further exacerbated by the curse of dimensionality for high-dimensional data.”
BackgroundMethodInfluentialInformation Flow through a Model of the C. elegans Klinotaxis Circuit
Izquierdo, E. J., Williams, P. L., & Beer, R. D. (2015) · PLoS ONE · doi:10.1371/journal.pone.0140397 · Cited by 47
Entropy Estimation for Real-Time Encrypted Traffic Identification (Short Paper)
Dorfinger, P., Panholzer, G., & John, W. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-20305-3_14 · Cited by 39
“In accordance to Paninski [8] we do not use the entropy estimation for payload with less than 16 bytes.”
MethodInformation Theory and Neural Information Processing
Johnson, D. H. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2009.2037047 · Cited by 41
On the Keyhole Hypothesis: High Mutual Information between Ear and Scalp EEG
Mikkelsen, K. B., Kidmose, P., & Hansen, L. K. (2017) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2017.00341 · Cited by 39
“With Equation (4) and the data processing equation (Paninski, 2003), the mutual information between an ear electrode and the scalp electrodes may be lower bounded by the mutual information between y and its linear predictor y ( x ) = w T x , I ≥ I y , y = p ( y , y ) log p ( y , y ) p ( y ) p ( y )…”
BackgroundNon-Gaussianity and Asymmetry of the Winter Monthly Precipitation Estimation from the NAO
Pires, C., & Perdigão, R. A. P. (2007) · Monthly Weather Review · doi:10.1175/mwr3407.1 · Cited by 38
“The numerical estimation of MI is rather difficult and has no unbiased estimators ( Paninski 2003, 2004 ).”
BackgroundEC3: Combining Clustering and Classification for Ensemble Learning
Chakraborty, T. (2017) · Proceedings · doi:10.1109/icdm.2017.92 · Cited by 43
“The accuracy is reported in terms of Normalized Mutual Information (NMI) [58].”
MethodIdentifying key factors of the seawater intrusion model of Dagu river basin, Jiaozhou Bay
Zeng, X., Dong, J., Wang, D., Wu, J., Zhu, X., Xu, S., Zheng, X., & Xin, J. (2017) · Environmental Research · doi:10.1016/j.envres.2017.10.039 · Cited by 40
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.