Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 801–900 of 1,004
Painsky, A., & Feder, M. (2021) · Entropy · doi:10.3390/e23060773 · Cited by 9
“In fact, Paninski [44] showed that in the large alphabet setup, the minimal expected worst-case divergence is unbounded, and grows like log ( m / n ) .”
BackgroundStructure Learning for Directed Trees
Jakobsen, M. E., Shah, R. D., Bühlmann, P., & Peters, J. (2021) · Research at the University of Copenhagen (University of Copenhagen) · doi:10.17863/cam.125386 · Cited by 14
Dynamic feature selection combining standard deviation and interaction information
Wu, P., Zhang, Q., Wang, G., Yang, F., & Xue, F. (2022) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-022-01706-4 · Cited by 7
“Given a discrete random variable Z = { z 1 , z 2 , … , z k } , the conditional mutual information between X and Y is expressed as [23]: The difference between the two probability distributions X and Y can be measured by the information gain I ( X ; Y ), which is asymmetric.”
BackgroundFa, Z., Yan, K., Qiu, Z., Zhang, Y., Liu, Q., & Li, X. (2024) · Resources Policy · doi:10.1016/j.resourpol.2024.104891 · Cited by 7
Optimal Bounds for Estimating Entropy with PMF Queries
Caferov, C., Kaya, B., O’Donnell, R., & Say, A. C. C. (2015) · Lecture notes in computer science · doi:10.1007/978-3-662-48054-0_16 · Cited by 7
“For an extended survey and results on the fundamental task of estimating entropy, see Paninski [14]; this survey includes justification of discretization, as well as discussion of applications to neuroscience (e.”
BackgroundTransmission efficiency in ring, brain inspired neuronal networks. Information and energetic aspects
Paprocki, B., & Szczepański, J. (2013) · Brain Research · doi:10.1016/j.brainres.2013.07.024 · Cited by 7
“Entropy rate estimation is broadly discussed in the literature (Kontoyiannis et al., 1998; Paninski, 2003; Amigo et al., 2004; Kennel et al., 2005; Lesne et al., 2009).”
Improved Estimation of Entropy for Evaluation of Word Sense Induction
Li, L., Titov, I., & Sporleder, C. (2014) · Computational Linguistics · doi:10.1162/coli_a_00196 · Cited by 7
“In this work, we use the universal worst-case bounds advocated in Paninski (2003).”
BackgroundMethodInfluentialBridging Mixture Model Estimation and Information Bounds Using I-MMSE
Paul, B., Chapman, C. D., Chiriyath, A. R., & Bliss, D. W. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2716903 · Cited by 7
“The results in Reference [1] were extended to include arbitrary noise distributions, also modeled using Gaussian mixtures, as well as a new isolated upper bound.”
BackgroundMethodInfluentialValle, M. A., Ruz, G. A., & Rica, S. (2019) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2019.03.001 · Cited by 9
Statistical Estimation of Mutual Information for Mixed Model
Bulinski, A. V., & Kozhevin, A. (2020) · Methodology And Computing In Applied Probability · doi:10.1007/s11009-020-09802-0 · Cited by 8
Computing the trajectory mutual information between a point process and an analog stochastic process
Pasha, S. A., & Solo, V. (2012) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2012.6346992 · Cited by 7
“This involves brute force calculation based on the definition [1], [2], [5].”
Generalization in Clustering with Unobserved Features
Krupka, E., & Tishby, N. (2005) · Cited by 7
Using Shannon's entropy to sample heterogeneous and high‐dimensional atmospheric datasets
Paul, M., & Aires, F. (2014) · Quarterly Journal of the Royal Meteorological Society · doi:10.1002/qj.2367 · Cited by 7
“The binned-based entropy is an estimation of the entropy and, as for all entropy estimators, it is negatively biased (Paninski, 2003), so the binning choice is an important factor.”
BackgroundUniversal estimation of information measures
Verdú, S. (2005) · IEEE Information Theory Workshop (ITW) · doi:10.1109/itw.2005.1531895 · Cited by 7
Non-asymptotic Theory for the Plug-in Rule in Functional EstimationPreprint
Jiao, J., Venkat, K., Han, Y., & Weissman, T. (2014) · arXiv (Cornell University) · Cited by 9
Theoretical Neuroanatomy:Analyzing the Structure, Dynamics,and Function of Neuronal Networks
Seth, A. K., & Edelman, G. M. (2004) · Lecture notes in physics · doi:10.1007/978-3-540-44485-5_22 · Cited by 10
“A contrasting approach, which focuses on these correlations, is provided by a growing literature concerned with measuring the information content of single spike trains [21–24].”
BackgroundDiscussion of “Data-driven Confounder Selection via Markov and Bayesian Networks” by Jenny Häggström
Kennedy, E. H., & Balakrishnan, S. (2017) · Biometrics · doi:10.1111/biom.12787 · Cited by 7
“Independence testing is a difficult nonparametric problem in its own right (Paninski, 2003; Jiao et al., 2015), and the method of testing independence to facilitate a subsequently more efficient lower-dimensional regression is unfortunately not a panacea that avoids the curse of dimensionality.”
BackgroundMethodSpatEntropy: Spatial Entropy Measures in RPreprint
Altieri, L., Cocchi, D., & Roli, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.05521 · Cited by 7
“It can also be seen as an estimator Ĥ(X), where the probability distribution is estimated by the so-called plug-in estimator (Paninski, 2003), which is the nonparametric as well as the maximum likelihood estimator: p̂(xi) = ni/n substitutes the probabilities with the observed proportions over n…”
BackgroundInfluentialPRI-VAE: Principle-of-Relevant-Information Variational AutoencodersPreprint
Li, Y., Yu, S., Príncipe, J. C., Li, X., & Wu, D. O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.06503 · Cited by 7
“Albeit easy-to-understand, estimating information-theoretic quantities (such as mutual information) exactly in high-dimensional space is problematic [20].”
BackgroundPartial Label Learning via Mutual Information Representation Learning
Feng, J., Huang, L., Li, J., Ren, K., Fan, J., Bu, Q., Gan, M., & Chen, C. L. P. (2025) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2025.3621307 · Cited by 9
“However, its accurate estimation is a challenging task, mainly attributed to the fact that we usually only have access to sample data and cannot directly touch the underlying distribution behind the data [28], [29], [30].”
Assessing Web Fingerprinting Risk
Bacis, E., Bilogrevic, I., Busa‐Fekete, R., Herath, A., Sartori, A., & Syed, U. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3648322 · Cited by 6
“Based on (4.1) of Paninski (2003), we can rewrite the ML estimate of entropy as and n .”
MethodInfluentialTask-specific dynamical entropy variations in EEG as a biomarker for Parkinson’s disease progression
Onay, F., & Karaçalı, B. (2025) · GeroScience · doi:10.1007/s11357-025-01821-4 · Cited by 7
Data Valuation for Vertical Federated Learning: A Model-Free and Privacy-Preserving Method
Xiao, H., Wang, L., Wu, J., & Fang, X. (2025) · MIS Quarterly · doi:10.25300/misq/2025/19161 · Cited by 10
Universal no reference image quality assessment metrics based on local dependency
Gao, F., Gao, X., Tao, D., Li, X., He, L., & Lu, W. B. (2011) · Asian Conference on Pattern Recognition · doi:10.1109/acpr.2011.6166657 · Cited by 6
“In order to select the most significant neighbours, as [19] did, the greedy algorithm is utilized to sort the 17 neighbours according to their contribution to the mutual information [21] between C and P.”
MethodHam-Spam Filtering Using Different PCA Scenarios
Dagher, I. J., & Antoun, R. (2016) · IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) · doi:10.1109/cse-euc-dcabes.2016.238 · Cited by 6
“They are words with frequency greater than a threshold value [11].”
BackgroundLocalizing category-related information in speech with multi-scale analyses
Tilsen, S., Kim, S.-E., & Wang, C. (2021) · PLoS ONE · doi:10.1371/journal.pone.0258178 · Cited by 6
“Second, although there are many ways to obtain estimates of the relevant probability distributions [7, 8], these depend on smoothing or binning parameters.”
BackgroundDatseris, G., & Haaga, K. A. (2025) · PLoS ONE · doi:10.1371/journal.pone.0324431 · Cited by 6
Braiman, V., Malyarenko, A., Mishura, Y. S., & Rudyk, Y. A. (2024) · Nonlinear Analysis Modelling and Control · doi:10.15388/namc.2024.29.35560 · Cited by 6
Mutual Information Analysis in Multimodal Learning Systems
Hadizadeh, H., Yeganli, S. F., Rashidi, B., & Bajić, I. V. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00068 · Cited by 8
“Nonetheless, due to the importance of estimating MI empirically from data, many MI estimators have been proposed in the literature [8], [9], [10], [11], [12], [13], relying on concepts such as kernel density estimation, variational approaches, or estimating bounds on MI.”
BackgroundYang, X., Li, G., Wu, J., Zhou, K., Li, J., & Yang, W. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2025.3533648 · Cited by 6
“InBackPriv, Graph Mutual Information (GMI) will be employed to compute the correlation [31], [32], [33], [34], [35].”
Maximum Likelihood Estimation of information measures
Jiao, J., Venkat, K., Han, Y., & Weissman, T. (2015) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2015.7282573 · Cited by 6
“In the same paper [11], Paninski demonstrated that if n = cS, where c > 0 is a constant, then the maximum squared bias of H(Pn), and of the Miller–Madow bias-corrected estimator HMM(Pn) would be bounded from zero.”
BackgroundInfluentialAn Information Theoretic Approach Toward Assessing Perceptual Audio Quality Using EEG
Mehta, K., & Kliewer, J. (2015) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2015.2501744 · Cited by 6
“The superposition of these activity patterns can be noninvasively recorded via electroencephalography (EEG) by placing sensors on the scalp.”
MethodAn Oral Exam for Measuring a Dialog System’s Capabilities
Cohen, D., & Lane, I. R. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10060 · Cited by 6
“The MLE estimate of entropy is negatively biased as described in (Paninski 2003), but estimates have low vari-”
BackgroundMethodInfluentialAsymptotic normality for plug-in estimators of diversity indices on countable alphabets
Grabchak, M., & Zhang, Z. (2018) · Journal of nonparametric statistics · doi:10.1080/10485252.2018.1482294 · Cited by 6
“For which, the result was proved, in two steps, in Paninski (2003) and Zhang and Zhang (2012). There the approach was to approximate a distribution on a countably infinite alphabet by a sequence of distributions on finite alphabets. We do not require such approximations here, and, instead, verify asymptotic normality…”
BackgroundMethodResultInfluentialA Note on the Shannon Entropy of Short Sequences
de Oliveira, H. M., & Ospina, R. (2018) · Anais de XXXV Simpósio Brasileiro de Telecomunicações e Processamento de Sinais · doi:10.14209/sbrt.2018.8 · Cited by 6
Pareto-Optimal Clustering with the Primal Deterministic Information Bottleneck
Tan, A. K., Tegmark, M., & Chuang, I. L. (2022) · Entropy · doi:10.3390/e24060771 · Cited by 7
“Estimation of mutual information and entropy for finite datasets is known to be a difficult problem with many common estimators either being biased, having high variance, or both [21, 22, 23, 24, 25].”
BackgroundPre-training local and non-local geographical influences with contrastive learning
Oh, B., Suh, I., Cha, K., Kim, J., Park, G., & Jeong, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110016 · Cited by 8
Jensen-Mercer inequality for uniformly convex functions with some applications
Sayyari, Y., & Barsam, H. (2023) · Afrika Matematika · doi:10.1007/s13370-023-01084-2 · Cited by 6
Sublinear Algorithms for Approximating String Compressibility
Raskhodnikova, S., Ron, D., Rubinfeld, R., & Smith, A. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74208-1_44 · Cited by 6
An Introduction to Information Theoretic Learning, Part I: Foundations
Silva, D. G., Fantinato, D. G., Canuto, J. C., Duarte, L. T., Neves, A. O., Suyama, R., Montalvão, J., & Attux, R. (2016) · Journal of Communication and Information Systems · doi:10.14209/jcis.2016.6 · Cited by 6
“(46) More recently, Paninski [37] applied Bernstein approximating polynomials to obtain corrections of order greater than or equal to two.”
BackgroundMethodDima, B., & Dima, Ş. M. (2017) · International Review of Economics & Finance · doi:10.1016/j.iref.2017.05.008 · Cited by 6
“Estimates of mutual information based on this approach are commonly called ’naive’ estimates (Paninski, 2003).”
BackgroundTopological features determining the error in the inference of networks using transfer entropy
Goodman, R. H., & Porfiri, M. (2019) · Mathematics in Engineering · doi:10.3934/mine.2020003 · Cited by 6
Limitations to Estimating Mutual Information in Large Neural Populations
Mölter, J., & Goodhill, G. J. (2020) · Entropy · doi:10.3390/e22040490 · Cited by 6
“A crucial result in this context is the inconsistency theorem, which implies that under some circumstances estimators that are frequently employed will yield arbitrarily poor estimates [18].”
BackgroundStudying Maximum Information Leakage Using Karush-Kuhn-Tucker Conditions
Chen, H., & Malacaria, P. (2009) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.7.1 · Cited by 7
“Futureworksaimtoinvestigateothercontinuousdefinitionsofinformationleakage[ 23 ]usingKarush-”
A Maximum Entropy Test for Evaluating Higher-Order Correlations in Spike Counts
Onken, A., Dragoi, V., & Obermayer, K. (2012) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002539 · Cited by 7
“Bias correction techniques have been developed for alleviating this problem [16–18].”
Osmotic modulation of stimulus‐evoked responses in the rat supraoptic nucleus
Bhumbra, G. S., Orlans, H. O., & Dyball, R. E. J. (2008) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2008.06163.x · Cited by 7
“…S½h; x The relative entropy D(h, x||hx) is an naive estimate of the mutual information between the phase angle and log interval I(h;x) However, its value is likely to be an overestimate because random coincidences will occur as a consequence of discretization of a finite data set (Paninski, 2003).”
BackgroundPerformance analysis for canonical Huffman coding with fixed window size
Konstantinides, J. M., & Andreadis, I. T. (2016) · Electronics Letters · doi:10.1049/el.2015.3299 · Cited by 6
Quantifying Side-Channel Information Leakage from Web Applications.Preprint
Mather, L., & Oswald, E. (2012) · IACR Cryptology ePrint Archive · Cited by 6
Amigó, J. M. (2010) · Springer series in synergetics · doi:10.1007/978-3-642-04084-9_1 · Cited by 6
Forecastable Component Analysis (ForeCA)Preprint
Goerg, G. M. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1205.4591 · Cited by 6
“(15) can be improved by a better spectral density (Fryzlewicz, Nason, and von Sachs, 2008; Lees and Park, 1995; Trobs and Heinzel, 2006) and entropy estimation (Paninski, 2003).”
BackgroundA PAC-Bayesian Approach to Structure Learning
Seldin, Y. (2009) · MPG.PuRe (Max Planck Society) · Cited by 7
Baudot, P., Tapia, M., & Goaillard, J. (2018) · Preprints.org · doi:10.20944/preprints201804.0157.v1 · Cited by 7
“Paninski developed a complete framework for information estimation [194], which unfortunately does not apply here, since an important part of his results assumes binary variables and independent identically distributed (iid) processes such that central limit theorems can apply and asymptotic convergence is obtained.”
BackgroundUnified Sample-Optimal Property Estimation in Near-Linear TimePreprint
Hao, Y., & Orlitsky, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03105 · Cited by 6
“Approximation-empirical The approximation-empirical method [21, 22, 28, 34, 35] identifies a non-smooth part of the underlying function f i , replaces it by a polynomial ˜ f i , and estimates the value of p i by its empirical frequency ˆ p i .”
MethodActive Structure Learning of Bayesian Networks in an Observational SettingPreprint
Ben-David, N., & Sabato, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13796 · Cited by 6
Learning is Forgetting: LLM Training As Lossy Compression
Conklin, H., Hosking, T., Yi-Chern, T., Gold, J., Cohen, J. D., Griffiths, T. L., Bartolo, M., & Goldfarb-Tarrant, S. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2604.07569 · Cited by 9
“Given this estimator resembles a differentiable relaxation of a binning-based estimate, it is relevant to note that in binning based approaches increasing the number of bins reduces mutual information by assigning similar representations to an increasing number of different bins (Paninski, 2003).”
Optimal von Neumann Entropy EstimationPreprint
Chen, Z., Gao, M., & Wang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.11151 · Cited by 6
“The study of entropy estimation from the perspective of algorithms and complexity dates back to [Pan03].”
Chen, X., Xu, G., He, B., Zhang, S., Su, Z., Jia, Y., Zhang, X., & Zhao, Z. (2023) · Royal Society Open Science · doi:10.1098/rsos.221067 · Cited by 7
Toward Efficient Deep Learning in RF Fingerprint Identification With OverlapConv
Shen, Y., Zeng, S., Tan, Z., Shen, Y., Zhao, D., & Song, H. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3670435 · Cited by 5
“Although the theory of information bottleneck has been widely studied, it still faces computational di ffi culties [41], [42], [43].”
Raimondi, G., Bonato, J., & Maucieri, C. (2026) · European Journal of Agronomy · doi:10.1016/j.eja.2026.128132 · Cited by 6
Buehler, M. J. (2026) · Interdisciplinary materials · doi:10.1002/idm2.70074 · Cited by 7
Profile Maximum Likelihood is Optimal for Estimating KL Divergence
Acharya, J. (2018) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2018.8437461 · Cited by 5
“In the recent years we have gained a significant understanding of the sample complexity of estimating information measures such as entropy, mutual information, divergence, and other properties of discrete distributions [1]–[6].”
BackgroundCloud-Based Data Offloading for Multi-focus and Multi-views Image Fusion in Mobile Applications
Shi, Y., Sun, J., Liu, D., Kou, L., Li, B., Yang, Q. B., & Zhang, L. (2019) · Mobile Networks and Applications · doi:10.1007/s11036-019-01326-3 · Cited by 5
“In experiments we use mutual information (MI) [33], margin maintenance (Q AB/F ) [34], standard deviation (SD) [35], and runtime to evaluate performance of proposed method and four comparing algorithms.”
MethodZhang, J. (2022) · Machine Learning and Knowledge Extraction · doi:10.3390/make4040044 · Cited by 5
“[24] proved that for finite K , an unbiased estimator for entropy does not exist.”
BackgroundNear-Optimal Learning of Tree-Structured Distributions by Chow and Liu
Bhattacharyya, A., Gayen, S., Price, E., Tan, V. Y. F., & Vinodchandran, N. V. (2023) · SIAM Journal on Computing · doi:10.1137/22m1489678 · Cited by 14
Multi-Armed Bandits With Self-Information Rewards
Weinberger, N., & Yemini, M. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3299460 · Cited by 9
“The general problem of entropy estimation is well studied [33], [34], [35], [37], [38], [39], [40], [41], [42], [43], [44].”
BackgroundMethodInfluentialTowards federated feature selection: Logarithmic division for resource-conscious methods
Suárez‐Marcote, S., Morán‐Fernández, L., & Bolón‐Canedo, V. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128099 · Cited by 5
Ahmadzai, M., & Nguyen, G. (2024) · Future Internet · doi:10.3390/fi16070220 · Cited by 7
“Mutual information, entropy, and ( ε , δ ) -DP are examples of these concepts [35–37].”
BackgroundShakeel, M., Abbas, H., Waseem, A., & Ali, Z. (2025) · Theoretical and Applied Climatology · doi:10.1007/s00704-025-05516-w · Cited by 5
Information‐Theoretic Gene Selection In Expression Data
Meyer, P. E., & Bontempi, G. (2013) · Wiley eBooks · doi:10.1002/9781118617151.ch17 · Cited by 7
“We refer the reader to [42, 13, 4, 40, 12] for alternative approaches of entropy and mutual information estimation.”
Statistical dependencies beyond linear correlations in light scattered by disordered media
Starshynov, I., Turpin, A., Binner, P., & Faccio, D. (2022) · Physical Review Research · doi:10.1103/physrevresearch.4.l022033 · Cited by 5
“…distributions given a particular output pattern outcome and H is the standard entropy H [ P ( x )] = − i P ( x i ) log 2 ( P ( x i )). the P In order to remove the bias of conditional entropy estimator, we use its jack-knife version [31] (see Supplemental Material [30] for more details).”
Complexity changes in functional state dynamics suggest focal connectivity reductions
Blair, D. S., Soriano‐Mas, C., Cabral, J., Moreira, P. S., Morgado, P. F., & Deco, G. (2022) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2022.958706 · Cited by 5
“While microarray studies of functional complexity are not new (Paninski, 2003; Pereda et al., 2005; Quian Quiroga and Panzeri, 2009), the whole-brain level presents two serious problems.”
BackgroundResolving entropy contributions in nonequilibrium transitions
Sorkin, B., Ricouvier, J., Diamant, H., & Ariel, G. (2023) · Physical review. E · doi:10.1103/physreve.107.014138 · Cited by 7
On spatially correlated observations in importance sampling methods for subsidence estimation
Kim, S. S. R., & Vossepoel, F. C. (2023) · Computational Geosciences · doi:10.1007/s10596-023-10264-9 · Cited by 5
“A more practical computation of the mutual information [35], written as a function of the entropy and the joint entropy H ( z , x ) is Also Eq.”
Advocating for the Silent: Enhancing Federated Generalization for Nonparticipating Clients
Wu, Z., Xu, Z., Zeng, D., Wang, Q., & Liu, J. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3516692 · Cited by 6
“The proposed maximum entropy aggregation can be applied to other distribution shift scenarios if we can estimate the empirical entropy of data source [39].”
Information Theoretic Derivations for Causality Detection: Application to Human Gait
Dijck, G. V., Vaerenbergh, J. V., & Hulle, M. M. V. (2007) · Lecture notes in computer science · doi:10.1007/978-3-540-74695-9_17 · Cited by 5
“On the other hand, estimators of the entropy and mutual information see e.g. [11], have gained a lot of attention in the literature.”
BackgroundRandomness of spontaneous activity and information transfer in neurons
Kostal, L., & Lánský, P. (2008) · Physiological Research · doi:10.33549/physiolres.931607 · Cited by 5
Testing the Beta-Lognormal Model in Amazonian Rainfall Fields Using the Generalized Space q-Entropy
Salas, H. D., Poveda, G., & Mesa, Ó. J. (2017) · Entropy · doi:10.3390/e19120685 · Cited by 5
“On the other hand, we revisit the classical problem [92] of how and how well diverse information-theoretic quantities, can be estimated given a finite set of i.”
BackgroundComparing Entropy and Beta as Measures of Risk in Asset Pricing
Deeva, G. (2017) · Acta Universitatis Agriculturae et Silviculturae Mendelianae Brunensis · doi:10.11118/actaun201765061889 · Cited by 5
Nonparametric Independence Tests and k-sample Tests for Large Sample Sizes Using Package HHG
Brill, B., Heller, Y., & Heller, R. (2018) · The R Journal · doi:10.32614/rj-2018-008 · Cited by 5
“Another classic approach is based on the measure of mutual information following partitioning of the data into a 2-dimensional grid (Paninski, 2003).”
BackgroundShannon versus semantic information processing in the brain
Majumdar, K. (2018) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.1284 · Cited by 7
Huang, W., & Zhang, K. (2019) · Entropy · doi:10.3390/e21030243 · Cited by 8
“…difficult to effectively calculate Shannon mutual information in many practical applications [8], various approximation methods have been proposed to estimate the mutual information, such as those based on asymptotic expansion [9–13], k -nearest neighbor [14], and minimal spanning trees [15].”
MethodInformation content analysis reveals desirable aspects of in vivo experiments of a synthetic circuit
Gomez-Cabeza, D., Bandiera, L., Balsa‐Canto, E., & Menolascina, F. (2019) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb.2019.8791449 · Cited by 5
“joint posterior, multiple approximations have been proposed in scientific literature [16], [17].”
BackgroundOn the Competitive Analysis and High Accuracy Optimality of Profile Maximum Likelihood
Han, Y., & Shiragur, K. (2021) · Society for Industrial and Applied Mathematics eBooks · doi:10.1137/1.9781611976465.80 · Cited by 5
Adaptive Label Smoothing for Classifier-based Mutual Information Neural Estimation
Wang, X., Al-Bashabsheh, A., Zhao, C., & Chan, C. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9518097 · Cited by 8
Entropy Estimation Using a Linguistic Zipf–Mandelbrot–Li Model for Natural Sequences
Back, A. D., & Wiles, J. (2021) · Entropy · doi:10.3390/e23091100 · Cited by 6
Data Structures and Algorithms for k-th Nearest Neighbours Conformational Entropy Estimation
Borelli, R., Dovier, A., & Fogolari, F. (2022) · Biophysica · doi:10.3390/biophysica2040031 · Cited by 5
Synchrony-Division Neural Multiplexing: An Encoding Model
Rezaei, M. R., Fard, R. S., Popović, M. R., Prescott, S. A., & Lankarany, M. (2023) · Entropy · doi:10.3390/e25040589 · Cited by 5
“One can find the most informative subspaces that maximize the mutual information between stimulus and response [23,24].”
BackgroundSpeckle statistics in adaptive optics images at visible wavelengths
Stangalini, M., Pedichini, F., Ambrosino, F., Centrone, M., & del Moro, D. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2232377 · Cited by 5
Agglomerative Info-Clustering: Maximizing Normalized Total Correlation
Chan, C., Al-Bashabsheh, A., & Zhou, Q. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.3040492 · Cited by 5
An analytical comparison of the information in sorted and non-sorted cosine-tuned spike activity
Won, D. S., Tiesinga, P. H. E., Henriquez, C. S., & Wolf, P. D. (2007) · Journal of Neural Engineering · doi:10.1088/1741-2560/4/3/017 · Cited by 5
Universal Estimation of Erasure Entropy
Yu, J., & Verdú, S. (2009) · IEEE Transactions on Information Theory · doi:10.1109/tit.2008.2008117 · Cited by 5
“As in [20], this results in underestimation of the th-order erasure entropy with a bias.”
ResultConvergence of Monte Carlo distribution estimates from rival samplers
Heard, N. A., & Turcotte, M. (2015) · Statistics and Computing · doi:10.1007/s11222-015-9595-0 · Cited by 5
“In information theory, it is common practice to discretise distributions of any continuous random variables (see Paninski 2003).”
BackgroundEstimating mutual information in high dimensions via classification errorPreprint
Zheng, C. Y., & Benjamini, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05229 · Cited by 5
“, experiments studying the properties of a single or a small number of neurons often make use of the concept of mutual information in summarizing or interpreting their results [2]. See discussions in [3]. However, estimating mutual information for multiple channels requires large and over-parameterized generative mode…”
The Effectiveness of Privacy Enhancing Technologies against FingerprintingPreprint
Datta, A. J., Lu, J., & Tschantz, M. C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.03920 · Cited by 5
“kwithasample. The qualityofthemetricscomputedfromthesampledepends upon both the nature of the metric and the sample. Forexample, arandomsamplewillprovide a reasonable estimation of the entropy (e.g., [50]). However,estimatingtheproportionofusersinsmall anonymity sets from even a random sample proves 13 difficult since t…”
BackgroundEstimating Entropy of Distributions in Constant SpacePreprint
Acharya, J., Bhadane, S., Indyk, P., & Sun, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.07976 · Cited by 5
“Its sample complexity [47, 20] is S(k, ") = ⇥ k/"+ (log(2) k)/"(2) .”
MethodOn Predictive Information in RNNsPreprint
Dong, Z., Oktay, D., Poole, B., & Alemi, A. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.09578 · Cited by 7
“While mutual information estimation is difficult in general (Paninski, 2003; McAllester & Stratos, 2019), recent progress has been made on a wide range of variational bounds on mutual information (Alemi et al.”
BackgroundPrivacy-Constrained Policies via Mutual Information Regularized Policy GradientsPreprint
Cundy, C., Desai, R., & Ermon, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15019 · Cited by 5
“Since even evaluating the mutual information in high dimensions is challenging Paninski (2003), this is not trivial.”
BackgroundWang, Q. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2509.03496 · Cited by 12
“…probability distribution p with the sample space of size N , the Shannon entropy [Sha48a, Sha48b] of p is defined by The estimation of Shannon entropy has a wide range of applications, e.g., measuring the variability of genetic sequences [SEM91], analyzing neural data [Pan03 ) in [JVHW15, WY16].”
Diffusion model-based parameter estimation in dynamic power systems
Zhu, F., Torbunov, D., Jiang, Z., Zhao, T., Yogarathnam, A., Ren, Y., & Yue, M. (2026) · Communications Engineering · doi:10.1038/s44172-026-00670-z · Cited by 5
Lee, S., Lee, H., Lee, H., Yum, K. S., Baek, J., Khil, J., Lee, J. Y., Shin, S., Cho, M., Ahn, N. Y., et al. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf099 · Cited by 4
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.