Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 901–1,000 of 1,004
Duan, T., Liang, H., Yue, L., & Dai, Z. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3587234 · Cited by 4
“It is defined as follows : Furthermore, it is possible to calculate IFMI by combining entropy [43] where )) is the joint entropy.”
He, S., Sun, Y., Zeng, N., Wang, L., Cao, Z., & He, Z. (2025) · Frontiers in Environmental Science · doi:10.3389/fenvs.2025.1658538 · Cited by 5
Yamada, L., Oskotsky, T., & Nuyujukian, P. (2025) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2025.3563789 · Cited by 4
“Moreover, large bias and variance have been reported in common estimators; the true value may lie outside the computed confidence intervals [78].”
Information-Theoretic Measures of Metacognition: Bounds and Relation to Group Performance
Meyen, S., Göppert, F., Schrenk, C., von Luxburg, U., & Franz, V. H. (2025) · Open Mind · doi:10.1162/opmi.a.40 · Cited by 5
“…information-theoretic measures but not for SDT-based measures because, without distributional assumptions, the transmitted information is notoriously difficult to estimate: The estimation bias can only be mitigated (Archer et al., 2014; Jiao et al., 2017) but not removed entirely (Paninski, 2003).”
Information-Theoretical Analysis of the Neural Code in the Rodent Temporal Lobe
Capitán, M. M., Kropff, E., & Samengo, I. (2018) · Entropy · doi:10.3390/e20080571 · Cited by 4
Inferring a Property of a Large System from a Small Number of Samples
Hernández, D. G., & Samengo, I. (2022) · Entropy · doi:10.3390/e24010125 · Cited by 4
Understanding Concept Identification as Consistent Data Clustering Across Multiple Feature Spaces
Lanfermann, F., Schmitt, S., & Wollstadt, P. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00032 · Cited by 4
“Note that it is a well-known problem that MI estimates from finite data suffer from estimation bias (e.g., [26]).”
BackgroundLow-probability states, data statistics, and entropy estimation
Hernández, D. G., Roman, A., & Nemenman, I. M. (2023) · Physical review. E · doi:10.1103/physreve.108.014101 · Cited by 6
“Although these Bayesian and non-parametric estimators perform well on some data sets, it is known that no estimator can be universally unbiased in this regime [4, 14].”
BackgroundMethodNormalization and cross-entropy connectivity in brain disease classification
Wu, H., Li, S., & Zeng, Y. (2025) · iScience · doi:10.1016/j.isci.2025.112226 · Cited by 4
Mutual Information Neural-Estimation-Driven Constellation Shaping Design and Performance Analysis
Ji, X., Wang, Q., Qian, L., & Kam, P. (2025) · Entropy · doi:10.3390/e27040451 · Cited by 4
“However, it is very challenging to directly and accurately evaluate MI, since the actual data are highly discrete and the specific distribution is unknown in practice [14].”
Asymptotically Unbiased Estimator of the Informational Energy with kNN
Caţaron, A., Andonie, R., & Chueh, Y. (2013) · International Journal of Computers Communications & Control · doi:10.15837/ijccc.2013.5.643 · Cited by 4
“Several methods were proposed for MI estimation [18], [22], [13]: histogram based estimators, kernel density estimators, B-spline estimators, k-th nearest neighbor (kNN) estimators, and wavelet density estimators.”
MethodInformation-theoretic approach to quantifying currency risk
Fiedor, P., & Hołda, A. (2016) · The Journal of Risk Finance · doi:10.1108/jrf-03-2015-0029 · Cited by 4
Mutual Information Flows in a Bivariate Point Process
Pasha, S. A., & Solo, V. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414484 · Cited by 4
“An early reference is [4], but it only deals with instantaneous mutual information or marginal mutual information.”
BackgroundZhang, L., Pan, J., Ma, X., & Yang, C. (2023) · International Journal of Modern Physics B · doi:10.1142/s0217979223502314 · Cited by 4
Efficiently Estimating Mutual Information Between Attributes Across Tables
Santos, A. S. R., Korn, F. R., & Freire, J. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00022 · Cited by 4
“All MI estimators used in this paper have been proven to be consistent estimators [44], [44], [49] under some assumptions, such as i.i.d samples.”
BackgroundMethodExploring language relations through syntactic distances and geographic proximity
Gregorio, J. D., Toral, R., & Sánchez, D. (2024) · EPJ Data Science · doi:10.1140/epjds/s13688-024-00498-7 · Cited by 7
“Entropy estimation is a largely analyzed problem and, even though there is not known unbiased estimator for the entropy [35], numerous useful estimators can be found in the literature [36].”
BackgroundS, D. D. B., Sharma, P., Chadaga, K., Sampathila, N., Bairy, G. M., Belurkar, S., Prabhu, S., & Swathi, K. S. (2024) · Systems Science & Control Engineering · doi:10.1080/21642583.2024.2420927 · Cited by 4
“Information gain and entropy are concepts that are used in mutual information (Paninski, 2003).”
Temporal resolution of spike coding in feedforward networks with signal convergence and divergence
Mobille, Z., Sikandar, U. B., Sponberg, S., & Choi, H. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012971 · Cited by 4
“For large populations of neurons and small ∆ t ’s, the response matrix R becomes very high dimensional, rendering the quantity I m ( S, R ) difficult to estimate directly [70].”
Effect of Anesthesia and Diurnal Variation on Chronic Vagus Nerve Activity in Rats
Rodrigues, A. J., Marmerstein, J. T., Kotamraju, B. P., McCallum, G. A., & Durand, D. M. (2025) · Journal of Neuroscience Research · doi:10.1002/jnr.70045 · Cited by 5
“…entropy for entropy approximation of vagus nerve activity complements other approximation methods, such as Shannon entropy (Kotamraju et al. 2023; Nagaraj and Balasubramanian 2017; Paninski 2003) or Lempel-Ziv complexity (Aboy et al. 2006; Amigó et al. 2004; Nagaraj and Balasubramanian 2017).”
PCMINN: A GPU-Accelerated Conditional Mutual Information-Based Feature Selection Method
Papaioannou, N., Myllis, G., Tsimpiris, A., Aggelopoulos, S., & Vrana, V. G. (2025) · Information · doi:10.3390/info16060445 · Cited by 4
Towards an Information Theoretic Framework for Evolutionary Learning
Card, S. W. (2011) · Syracuse University Libraries (Syracuse University) · Cited by 6
How do the amplitude fluctuations affect the neuronal transmission efficiency
Paprocki, B., & Szczepański, J. (2012) · Neurocomputing · doi:10.1016/j.neucom.2012.11.001 · Cited by 4
“Using the high quality entropy estimators [14–16] we determined maximal values of the mutual information between input and neurons output signals for simple neuronal architecture.”
BackgroundMethodThe confidence interval of entropy estimation through a noisy channel
Ho, S., Chan, T., & Grant, A. J. (2010) · IEEE Information Theory Workshop (ITW) · doi:10.1109/cig.2010.5592695 · Cited by 4
“The algorithms in [1], [6], [7] can provide improved estimates compared to (1), but they do not yield confidence intervals.”
BackgroundSánchez, L., Couso, I., Palacios, A. M., & Palacios, J. L. (2012) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2012.02.009 · Cited by 4
Detection of Cooperative Interactions in Logistic Regression Models
Xu, L., Qian, X., Liu, T., & Cui, S. R. (2016) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2016.2646664 · Cited by 4
“Next, we compare the performances of Algorithm 1 against two existing feature-selection methods: the mRMR forwardi ng selection [11] and feature ranking based on mutual informat ion estimates [14].”
MethodSearching for structure in collective systems
Twomey, C. R., Hartnett, A. T., Sosna, M. M. G., & Romańczuk, P. (2020) · Theory in Biosciences · doi:10.1007/s12064-020-00311-9 · Cited by 4
Entropy, mutual information, and systematic measures of structured spiking neural networks
Li, W., & Li, Y. (2020) · Journal of Theoretical Biology · doi:10.1016/j.jtbi.2020.110310 · Cited by 5
“sampled from a certain given probability distribution [1] (also see [23]).”
BackgroundSupport Estimation with Sampling Artifacts and Errors
Chien, E., Milenković, O., & Nedić, A. (2021) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit45174.2021.9517892 · Cited by 5
Precise spatial memory in local random networks
Natale, J. L., Hentschel, H. G. E., & Nemenman, I. M. (2020) · Physical review. E · doi:10.1103/physreve.102.022405 · Cited by 8
Wiebel-Herboth, C. B., Krüger, M., & Wollstadt, P. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248166 · Cited by 5
Asymptotic Properties of the Plug-in Estimator of the Discrete Entropy Under Dependence
Seri, R., & Martinoli, M. (2021) · IEEE Transactions on Information Theory · doi:10.1109/tit.2021.3109307 · Cited by 4
Implicit Feature Decoupling with Depthwise Quantization
Fostiropoulos, I., & BOEHM, B. W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00049 · Cited by 4
“Quantization post-hoc is sensitive to sample size but performs at par with other state-of-the-art approaches [12, 23, 24, 35].”
BackgroundLess is more: Low-precision feature selection for wearables
Suárez‐Marcote, S., Morán‐Fernández, L., & Bolón‐Canedo, V. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892143 · Cited by 4
“These estimated distributions can be calculated using sample estimates of the probability by maximum-likelihood estimator [22]: Nowadays, Mutual Information has many different applications, among which are Machine Learning and Statistics.”
A nonprofiled side-channel analysis based on variational lower bound related to mutual information
Zhang, C., Lu, X., Cao, P., Gu, D., Guo, Z., & Xu, S. (2022) · Science China Information Sciences · doi:10.1007/s11432-021-3451-1 · Cited by 6
Interpreting Training Aspects of Deep-Learned Error-Correcting Codes
Devroye, N., Mulgund, A., Shekhar, R., Turán, G., Žefran, M., & Zhou, Y. (2023) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit54713.2023.10206783 · Cited by 4
“We opt for a naive approach to get around this, but more sophisticated approaches can be found in [25], [26].”
MethodVoQs: A Web Application for Visualization of Questionnaire Surveys
Zhang, X., Klawonn, F., Grigull, L., & Lechner, W. M. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24465-5_29 · Cited by 4
“The entropy (see for instance [5]) for the answers to question Qj is then”
BackgroundAttribute Vector Guided Groupwise Registration
Wang, Q., Yap, P., Wu, G., & Shen, D. G. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04268-3_81 · Cited by 4
“Nevertheless, to accurately estimate the entropy is challenging when sample size is not large enough (Paninski, 2003).”
BackgroundAn optimal entropy estimator for discrete random variables
Shiga, M., & Yokota, Y. (2006) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2005.1556038 · Cited by 4
“The minimax entropy estimator It has been pointed out that no entropy estimator can simultaneously minimize both the bias error and variance of its estimate[4].”
BackgroundMethodInfluentialFactorization and structure of natural 4×4 patch densities
Koroutchev, K., & Dorronsoro, J. R. (2006) · Optical Engineering · doi:10.1117/1.2402520 · Cited by 5
Marrelec, G., & Benali, H. (2011) · Entropy · doi:10.3390/e13040805 · Cited by 4
“The present work provides a quantification in the case of multivariate normal samples for the well-known negative bias for this estimator [7].”
BackgroundRuan, Y., Jayaweera, S. K., & Abdallah, C. T. (2013) · Journal of Control Theory and Applications · doi:10.1007/s11768-013-2005-2 · Cited by 4
“However, it still could be closely estimated by Monte Carlo techniques given large enough amount [13, 14].”
BackgroundMethodSynchrony: A spiking-based mechanism for processing sensory stimuli
Glackin, C., Maguire, L., Mcdaid, L. J., & Wade, J. J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.020 · Cited by 4
“For this reason, neuroscientists when trying to determine the information transmission between stimulus and neural response typically use mutual information estimators (for a detailed review of the different estimators see [12]) to cope with the variability of neural response.”
BackgroundQuantifying Information Conveyed by Large Neuronal Populations
Berkowitz, J. A., & Sharpee, T. O. (2019) · Neural Computation · doi:10.1162/neco_a_01193 · Cited by 4
“If we were always constrained to drawing samples from P( r | sμ), then we would once again be limited to making biased estimates of Ĥ( R) (Paninski, 2003).”
InfluentialStatistical Approaches for the Analysis of Dependency Among Neurons Under Noise
Gençağa, D., Ayan, S. Ş., Farnoudkıa, H., & Okuyucu, S. (2020) · Entropy · doi:10.3390/e22040387 · Cited by 4
“For the latter case, information theory quantities, such as the MI, are much more effective [39,40].”
BackgroundReliability as Lindley Information
Markon, K. E. (2022) · Multivariate Behavioral Research · doi:10.1080/00273171.2022.2136613 · Cited by 4
“…is consistent with related research showing that estimates of mutual information between observed variables are susceptible to positive bias (e.g., Paninski, 2003), as are set correlations between observed variables (Cohen & Nee, 1984), and maximal reliability estimates (AguirreUrreta et al.,…”
ResultSYMMETRY BREAKING BIFURCATIONS OF THE INFORMATION DISTORTION
Parker, A. E. (2003) · Montana State University ScholarWorks (Montana State University) · Cited by 4
Contributions to theory and algorithms of independent component analysis and signal separation
Eriksson, J. (2004) · Aaltodoc (Aalto University) · Cited by 4
Hypothesis Testing and Feature Selection in Semi-Supervised Data
Sechidis, K. (2015) · Research Explorer (The University of Manchester) · Cited by 4
Universal Clustering via CrowdsourcingPreprint
Raman, R. K., & Varshney, L. R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02276 · Cited by 4
“2) ML estimate, Ĥ is negatively biased [27]:”
BackgroundAnalytic Formulas for Renyi Entropy of Hidden Markov ModelsPreprint
Breitner, J., & Skórski, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.09699 · Cited by 4
Entropy Analysis of Financial Time SeriesPreprint
Schwill, S. (2018) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.1807.09423 · Cited by 6
“tropy and mutual informa26 tion have both been estimated in the literature based on the kernel density method (see Schreiber (2000); and Blumentritt and Schmid (2011)). The maximum likelihood method (Paninski (2003)) has also been applied successfully. In the following, we will focus on estimation methods which assume…”
MethodParametric Information Bottleneck to \\Optimize Stochastic Neural Networks
Nguyên, T. T., & Choi, J. (2018) · Scholarworks@UNIST (Ulsan National Institute of Science and Technology) · Cited by 4
Motor control by precisely timed spike patternsPreprint
Srivastava, K. H., Holmes, C. M., Vellema, M., Pack, A. R., Elemans, C. P. H., Nemenman, I. M., & Sober, S. J. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/056010 · Cited by 4
On the Estimation of Information Measures of Continuous DistributionsPreprint
Pichler, G., Piantanida, P., & Koliander, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02851 · Cited by 6
“More negative results on the estimation of entropy and mutual information can be found in [28].”
BackgroundNIT: an open-source tool for information theoretic analysis of neural population dataPreprint
Maffulli, R., Casal, M. A., Celotto, M., Zucca, S., Safaai, H., Fellin, T., & Panzeri, S. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.12.11.519966 · Cited by 4
“NIT is equipped with a sets of well-used for 292 limited sampling bis correction in MI measure: Panzeri-Treves [23], linear and quadratic extrapolation 293 [94], the shuffling procedure [14], the Best Upper Bounds (BUB) estimator[95], and the bootstrap 294 correction [96].”
MethodSparse networks of conformational fluctuations communicate signals within proteinsPreprint
Trenfield, K., & Lin, M. M. (2025) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2025.05.28.656549 · Cited by 4
Nearly Sample-Optimal Estimators for Quantum Rényi and Tsallis EntropiesPreprint
Chen, K., & Wang, Q. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.18070 · Cited by 4
“The estimation of Shannon entropy [Sha48a, Sha48b] has attracted a lot of attention in the literature [Pan03, Pan04, BDKR05, Val11, VV11a, VV11b, VV17, JVHW15, JVHW17, WY16].”
Approximating the Entropy of Large AlphabetsPreprint
Brautbar, M., & Samorodnitsky, A. (2005) · Electronic colloquium on computational complexity · Cited by 4
Learning dependence from samples
Seth, S., & Príncipe, J. C. (2014) · International Journal of Bioinformatics Research and Applications · doi:10.1504/ijbra.2014.058777 · Cited by 3
“Although mathematically sound, these concepts, however, suffer from extensive computational issues when the input domain is non-categorical (Paninski, 2003).”
BackgroundPre-neuronal processing of haptic sensory cues via dispersive high-frequency vibrational modes
Ding, Y., & Vlasov, Y. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-40675-8 · Cited by 4
Karell-Albo, J. A., Legón-Pérez, C. M., Llanes, R. S., Rojas, O., & Sosa-Gómez, G. (2023) · Entropy · doi:10.3390/e25111545 · Cited by 3
“Some of the estimators proposed in the literature use discretization [26,27], kernels [28,29] or correlation integrals [30], k-nearest neighbors [31,32], B-splines [33], or Gram–Charlier polynomial expansion [34].”
MethodBidirectional dynamic threshold SNN for enhanced object detection with rich spike information
Wu, S., Wang, G., Song, Y., Zhao, Y., Zhou, Y., Meng, Q., & Liao, Y. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2025.1661916 · Cited by 3
“To address this issue, we employed information entropy theory (Paninski, 2003) into the object detection task and showed that the binary spike activation maps of SNNs carry substantially less information than ANN activations, leading to reduced accuracy.”
Transfer entropy for finite data
Kirkley, A. (2025) · Physical review. E · doi:10.1103/tcss-5hn3 · Cited by 3
Estimating Tsallis entropy of several exponentially distributed populations
Kumar, N., Dixit, A., & Vijay, V. (2026) · Communication in Statistics- Theory and Methods · doi:10.1080/03610926.2026.2632358 · Cited by 3
“Several widely used non parametric estimators of Shannon entropy include the precise local expansion based estimator Paninski (2003), the weighted affine combination of estimators Sricharan et al.(2013), and the Bayes estimator Holste et al. (1998).”
A modular and generalized machine learning approach for non-Newtonian EHL film thickness prediction
Zhang, C., Tošić, M., Kelley, J., Habchi, W., Marian, M., & Lohner, T. (2026) · Tribology International · doi:10.1016/j.triboint.2026.112211 · Cited by 3
Prognostic models in coronary artery disease: Cox and network approaches
Mora, A., Sicari, R., Cortigiani, L., Carpeggiani, C., Picano, E., & Capobianco, E. (2015) · Royal Society Open Science · doi:10.1098/rsos.140270 · Cited by 3
“In particular, two estimation methods were employed: the MM estimator [12] and the ARACNE algorithm [13], implemented in the ‘minet’ R package [14].”
MethodQuantitative Trait Specific Differential Expression (qtDE)
Sarkar, M., & Majumder, A. (2015) · Procedia Computer Science · doi:10.1016/j.procs.2015.02.131 · Cited by 3
Luo, X., Liu, J., Zhang, D., Wang, W., & Zhu, Y. (2016) · Entropy · doi:10.3390/e18070274 · Cited by 3
“In addition, those random errors could be eliminated by calculating entropy weights [22].”
BackgroundAn information-theoretic approach to study spatial dependencies in small datasets
Porfiri, M., & Marín, M. R. (2020) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2020.0113 · Cited by 3
“The estimation of the mutual information statistic can be carried out by binning the variables under consideration, thereby facilitating the study of small datasets [26,27].”
MethodLarge-scale multiple inference of collective dependence with applications to protein function
Jernigan, R., Jia, K., Ren, Z., & Zhou, W. (2021) · The Annals of Applied Statistics · doi:10.1214/20-aoas1431 · Cited by 3
“…dependence accommodates discrete and categorical data more naturally as its building block, the information entropy originally was for an arbitrary discrete distribution and can be effectively estimated based on independent and identically distributed (i.i.d.) observations (Paninski, 2003).”
BackgroundMethodGhandeharioun, H. (2021) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2021.12605 · Cited by 4
Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers
Lin, B. (2021) · Entropy · doi:10.3390/e24010059 · Cited by 4
“Despite various recent attempts to estimate the mutual information [40,41], the information bottleneck approach can arrive at different conclusions under different assumptions for these estimation and misleading causal connections between compression and generalization [42].”
BackgroundMINE-based Geometric Constellation Shaping in AWGN Channel
Wang, Q., Ji, X., Qian, L., Liu, Z., Du, X., & Kam, P. (2023) · IEEE/CIC International Conference on Communications in China (ICCC) · doi:10.1109/icccworkshops57813.2023.10233820 · Cited by 3
“However, direct and accurate calculation of MI is difficult, since estimating MI depends on the underlying joint probability density function which should match the channel [16].”
BackgroundData Efficiency, Dimensionality Reduction, and the Generalized Symmetric Information Bottleneck
Martini, K. M., & Nemenman, I. M. (2024) · Neural Computation · doi:10.1162/neco_a_01667 · Cited by 3
“Estimation of information-theoretic quantities is a hard task, potentially as hard as estimating the underlying distributions themselves, largely due to the estimation bias (Antos & Kontoyiannis, 2001; Paninski, 2003).”
BackgroundMethodHwang, S., Lee, E., & Choi, S.-K. (2024) · International Journal of Precision Engineering and Manufacturing-Smart Technology · doi:10.57062/ijpem-st.2023.00136 · Cited by 3
Limit Distribution Theory for Quantum Divergences
Sreekumar, S., & Berta, M. (2024) · IEEE Transactions on Information Theory · doi:10.1109/tit.2024.3482168 · Cited by 6
“Consistency and/or convergence rates for various estimators of f -divergences, which subsumes entropy and mutual information as special cases, have been studied in [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], and [40].”
Wildfire Risk Assessment to Overhead Transmission‐Line Based on Improved Analytic Hierarchy Process
Xu, J., Fang, C., & Cheng, Y. (2024) · Fire and Materials · doi:10.1002/fam.3267 · Cited by 3
“The MI [28, 29] algorithm is applied to calculate the importance of each factor, reducing the subjective bias inherent in expert ratings.”
Text Information Mining in Cyberspace: An Information Extraction Method Based on T5 and KeyBERT
Liu, K., Li, Y., Qi, Y., Qi, N., & Zhai, M. (2024) · IEEE International Conference on Data Science in Cyberspace · doi:10.1109/dsc63484.2024.00092 · Cited by 3
Correcting convexity bias in function and functional estimate
Ma, C., & Ying, L. (2025) · Research in the Mathematical Sciences · doi:10.1007/s40687-024-00483-6 · Cited by 4
“…for a long time, with many methods proposed, for instance, the James–Stein estimator of the expectation of Gaussian distributions [15,21], the estimate of entropy [11,14,19,20,22,24,28], mutual information [10,17,18,20], other functionals of probability distributions [1,13,16,26], etc.”
Cheung, A. H., Sane, A., & Fox‐Kemper, B. (2026) · Climate Dynamics · doi:10.1007/s00382-025-07996-y · Cited by 3
“Estimating the probability distributions of these two information theory metrics is difficult because the sample size of each random variable is limited, which leads to a bias in estimating these metrics (Papana and Kugiumtzis 2008 ; Paninski 2003 ; Treves and Panzeri 1995 ).”
Two-Sided Exponential Concentration Bounds for Bayes Error Rate and Shannon Entropy
Honorio, J., & Jaakkola, T. (2013) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 3
Leveraging a Semantically Annotated Corpus to Disambiguate Prepositional Phrase Attachment
Emerson, G., & Copestake, A. (2015) · Apollo (University of Cambridge) · Cited by 3
ANALYSIS OF THE TIME EVOLUTION OF NON-LINEAR FINANCIAL NETWORKS
Fiedor, P. (2015) · Acta Universitatis Lodziensis Folia oeconomica · doi:10.18778/0208-6018.314.09 · Cited by 3
“Using a relatively short running window makes it hard to use mutual information due to it being slow to converge (Cover & Thomas 1991, Paninski 2003).”
BackgroundSysoev, I. V. (2016) · Izvestiya VUZ Applied Nonlinear Dynamics · doi:10.18500/0869-6632-2016-24-4-86-95 · Cited by 3
“…данных, широко используемая в задачах различных областей наук, в частности, в нелиней-ной динамике [1] для оптимальной реконструкции вектора состояния по скалярной временной реализации, в медицине для обработки объёмных изображений [2, 3], в нейрофизиологии [4] и даже в лингвистике [5].”
Optimality of Feature Selection After Dimensionality Reduction (Latent Semantic Analysis)
Das, A., Upadhya, P., & Sanyal, S. (2018) · Lecture notes in networks and systems · doi:10.1007/978-981-10-8201-6_31 · Cited by 3
Farzadian, O., Oikonomou, T., Good, M. R. R., & Niry, M. D. (2019) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2019.123350 · Cited by 3
“[33] L.”
BackgroundIntelligence level analysis for crowd networks based on business entropy
Li, Z., Pan, Z., Wang, X., Ji, W., & Feng, Y. (2019) · International Journal of Crowd Science · doi:10.1108/ijcs-05-2019-0014 · Cited by 3
“Imitating the concept of information entropy (Paninski, 2003), this paper will introduce business entropy to analyze crowd network’s intelligence level so that the correlation between professional category of the crowd subjects and crowd network’s intelligence level can be described.”
BackgroundCASMI—An Entropic Feature Selection Method in Turing’s Perspective
Shi, J., Zhang, J., & Ge, Y. (2019) · Entropy · doi:10.3390/e21121179 · Cited by 3
“The plug-in entropy estimator performs very poorly when the sample size is not sufficiently large [33][34], and we have discussed that the sample size is usually relatively limited in healthcare datasets.”
BackgroundDimension-Free Empirical Entropy Estimation
Cohen, D., Kontorovich, A., Koolyk, A., & Wolfer, G. (2022) · IEEE Transactions on Information Theory · doi:10.1109/tit.2022.3232739 · Cited by 3
“For fixed alphabet size d < ∞, the plug-in estimate is asymptotically minimax optimal [29].”
BackgroundThe impact of Clean Spark Spread expectations on storage hydropower generation
Condemi, C., Mastroeni, L., & Vellucci, P. (2021) · Decisions in Economics and Finance · doi:10.1007/s10203-021-00355-6 · Cited by 3
“| (Meyer 2008; Paninski 2003).”
Sharp concentration inequalities for the centred relative entropy
Bhatt, A., & Pensia, A. (2022) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iaac009 · Cited by 3
Upper Confidence Interval Strategies for Multi-Armed Bandits with Entropy Rewards
Weinberger, N., & Yemini, M. (2022) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit50566.2022.9834746 · Cited by 3
“Then, the plug-in estimator H (ˆ p ( n )) is biased, and, as was proved in [21], where for B ( n ) The concentration result of the plug-in estimator from [22, p. 168] implies the following confidence interval bound: Then, assuming n ≥ 2 , it holds for any δ ∈ (0 , 1) that with probability larger…”
BackgroundMethodInfluentialRoy, D., Sengupta, D., & Kulkarni, K. A. (2022) · Biochemical and Biophysical Research Communications · doi:10.1016/j.bbrc.2022.09.059 · Cited by 3
Identification of Systems Based on Minimax Entropy-Information Criteria
Chernyshov, K. R. (2022) · International Russian Automation Conference (RusAutoCon) · doi:10.1109/rusautocon54946.2022.9896396 · Cited by 3
Generalized Maxwell Distribution in the Tsallis Entropy Formalism
Bakiev, T. N., Викторович, Н. Д., Savchenko, А. М., & Semenov, K. M. (2022) · Moscow University Physics Bulletin · doi:10.3103/s0027134922050046 · Cited by 3
Sun, K., & Wang, R. (2022) · Journal of Cognitive Psychology · doi:10.1080/20445911.2022.2154781 · Cited by 4
On the Use of Variability Measures to Analyze Source Coding Data Based on the Shannon Entropy
de Oliveira, H. M., Ospina, R., Martín-Barreiro, C., Leiva, V., & Chesneau, C. (2023) · Mathematics · doi:10.3390/math11020293 · Cited by 3
Not so optimal: The evolution of mutual information in potassium voltage-gated channels
Duran-Urriago, A., & Marzen, S. (2023) · PLoS ONE · doi:10.1371/journal.pone.0264424 · Cited by 4
“Different relationships 133 between X and Y result in different expressions for mutual information, which have 134 been reviewed in other works [24–26].”
BackgroundSeveral Basic Elements of Entropic Statistics
Zhang, Z. (2023) · Entropy · doi:10.3390/e25071060 · Cited by 3
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.