Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 501–600 of 1,004
HLA Preferences for Conserved Epitopes: A Potential Mechanism for Hepatitis C Clearance
Rao, X., Hoof, I., van Baarle, D., Keşmir, C., & Textor, J. C. (2015) · Frontiers in Immunology · doi:10.3389/fimmu.2015.00552 · Cited by 22
“To prevent a bias due to different numbers of sequences in the alignments, we used the Miller-Madow entropy estimator which corrects for such bias (27).”
Anderson, R. P., Jiménez, G., Bae, J. Y., Silver, D., Macinko, J. A., & Porfiri, M. (2016) · SIAM Journal on Applied Dynamical Systems · doi:10.1137/15m1041584 · Cited by 19
“7) requires even more data, as entropy estimators can be notoriously biased [61].”
A new hybrid classifier selection model based on mRMR method and diversity measures
Cheriguene, S., Azizi, N., Dey, N., Ashour, A. S., & Ziani, A. (2018) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-018-0797-6 · Cited by 18
“Such measurements include information measures, which determine the information gain that can be provided with a random variable [20, 36].”
BackgroundRobotic Exploration for Learning Human Motion Patterns
Molina, S., Cielniak, G., & Duckett, T. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3101358 · Cited by 18
“Although this estimate still retains some bias when N << k or N ∼ k [41], this is not the case for this application as the number of person detections tends to be greater than the number of bins.”
BackgroundWu, Z., Qiu, J., Crow, W. T., Wang, D., Wang, Z., & Zhang, X. (2021) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2021.3136565 · Cited by 18
Sensor Adaptation and Development in Robots by Entropy Maximization of Sensory Data
Olsson, L. V., Nehaniv, C. L., & Polani, D. (2005) · IEEE International Symposium on Computational Intelligence in Robotics and Automation (CIRA) · doi:10.1109/cira.2005.1554340 · Cited by 23
“For example, what is the best method to estimate entropy given distributions with empty bins? Here other methods like Miller-Madow bias correction [ 14 ] could be investigated.”
MethodWei, H., & Billings, S. A. (2008) · International Journal of Bifurcation and Chaos · doi:10.1142/s0218127408022585 · Cited by 18
Entropy-based traffic filtering to support real-time Skype detection
Dorfinger, P., Panholzer, G., Trammell, B., & Pepe, T. (2010) · International Wireless Communications and Mobile Computing Conference (IWCMC) · doi:10.1145/1815396.1815568 · Cited by 18
“They also show that the Paninski estimator [6] can be used to calculate HN (U) as”
MethodEmmert‐Streib, F. (2013) · PeerJ · doi:10.7717/peerj.10 · Cited by 18
“That means for each time step t, we generate D = {D1(tj),...,DE(tj)} different data sets and each of these data sets contains S = 300 samples (as described in section ‘Simulation design of our study’).”
InfluentialDiscrete camera calibration from pixel streams
Grossmann, E., Gaspar, J. A., & Orabona, F. (2009) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2009.03.009 · Cited by 18
“In addition to the plug-in and Miller-Madow estimators, we also consider here the estimator of Paninski [22].”
MethodInfluentialEstimating the Mutual Information between Two Discrete, Asymmetric Variables with Limited Samples
Hernández, D. G., & Samengo, I. (2019) · Entropy · doi:10.3390/e21060623 · Cited by 18
“In fact, when nx ∼ O(1), the plug-in estimator typically underestimates H(Y|x) severely [5], and often leads to an overestimation of I(X, Y).”
Analysis of KNN Information Estimators for Smooth Distributions
Zhao, P., & Lai, L. (2019) · IEEE Transactions on Information Theory · doi:10.1109/tit.2019.2945041 · Cited by 23
Estimating Information Theoretic Measures via Multidimensional Gaussianization
Laparra, V., Johnson, J. E., Camps‐Valls, G., Santos‐Rodríguez, R., & Malo, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3495827 · Cited by 22
Using mutual information as a cocitation similarity measure
Zheng, L. (2019) · Scientometrics · doi:10.1007/s11192-019-03098-9 · Cited by 18
“Admittedly, the use of mutual information as a similarity measure is met with some difficulties as pointed out in Paninski (2003).”
MethodRecent trends in morbidity and mortality rates for abdominal aortic aneurysms.
Semenciw, R. M., MORRISON, H. I., Wigle, D., Cole, W., & Hill, G. (1992) · PubMed · Cited by 19
ForeCA: Forecastable Component Analysis
Goerg, G. M. (2012) · doi:10.32614/cran.package.foreca · Cited by 18
Estimating the Entropy of Linguistic Distributions
Arora, A., Meister, C., & Cotterell, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.20 · Cited by 29
Entropy Estimators for Markovian Sequences: A Comparative Analysis
Gregorio, J. D., Sánchez, D., & Toral, R. (2024) · Entropy · doi:10.3390/e26010079 · Cited by 17
“It is well known that Equation (11) is an unbiased estimator of p ( x i ) , but the MLE estimator, given by is negatively biased [15], i.e., ⟨ ˆ H MLE ⟩ − H < 0.”
BackgroundLimits to extreme event forecasting in chaotic systems
Yuan, Y., & Lozano-Durán, A. (2024) · Physica D Nonlinear Phenomena · doi:10.1016/j.physd.2024.134246 · Cited by 17
“The two curves also allow us to discern between errors from uncertainty in initial conditions ( P c e,I ) and errors from the unobserved variables θ 1 and θ 2 ( P ce,O ).”
BackgroundEstimating causal interaction between prefrontal cortex and striatum by transfer entropy
Chaofei, M., Pan, X., Wang, R., & Sakagami, M. (2013) · Cognitive Neurodynamics · doi:10.1007/s11571-012-9239-4 · Cited by 17
“To characterize nonlinear interactions between two systems, many researchers have used the concept of mutual information (Tsukada et al. 1975, 1976; Dan et al. 1998; Paninski 2003), which is quantified in Shannon’s information theory (Shannon and Weaver 1949).”
MethodFlow complexity in open systems: interlacing complexity index based on mutual information
Pozo, J. M., Geers, A. J., Villa‐Uriol, M., & Frangi, A. F. (2017) · Journal of Fluid Mechanics · doi:10.1017/jfm.2017.392 · Cited by 17
“This bias for small samples is a known effect for experimental MI estimators (Paninski 2003).”
BackgroundAnalysis of Precipitation Extremes in the Source Region of the Yangtze River during 1960–2016
Zhou, B., Liang, C., Zhao, P., & Qiong, D. (2018) · Water · doi:10.3390/w10111691 · Cited by 18
“Potential correlation analysis methods for two climatic variables includes mutual information [59,60], contingency tables [61], Spearman’s rank, and Pearson correlation [62].”
MethodDetecting switching leadership in collective motion
Butail, S., & Porfiri, M. (2019) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5079869 · Cited by 17
Polynomial Methods in Statistical Inference: Theory and Practice
Wu, Y., & Yang, P. (2020) · Foundations and Trends® in Communications and Information Theory · doi:10.1561/0100000095 · Cited by 37
On Generalized Schürmann Entropy Estimators
Grassberger, P. (2022) · Entropy · doi:10.3390/e24050680 · Cited by 17
“Notice that these covariances were not taken into account in [13,17], whence the variance estimations in these papers are, at best, approximate.”
BackgroundAsymptotic Normality of an Entropy Estimator With Exponentially Decaying Bias
Zhang, Z. (2012) · IEEE Transactions on Information Theory · doi:10.1109/tit.2012.2217393 · Cited by 17
“Among many difficult issues of nonparametric entropy estimation, much research effort in the literature seems to be placed on reducing the bias of the estimators.”
BackgroundMutual information and redundancy in spontaneous communication between cortical neurons
Szczepański, J., ARNOLD, M. M., Wajnryb, E., Amigó, J. M., & Sanchez-Vives, M. V. (2011) · Biological Cybernetics · doi:10.1007/s00422-011-0425-y · Cited by 18
“The answer is given by the mutual information I (X,Y ) between X and Y (Cover and Thomas 1991; Paninski 2003; London et al. 2008),”
BackgroundCausality and Bayesian Network PDEs for multiscale representations of porous media
Um, K., Hall, E. J., Katsoulakis, M. A., & Tartakovsky, D. M. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.06.007 · Cited by 17
“Indeed, a benefit to using the mutual information is the availability of methods relying on plug-in estimators for (29) with corresponding numerical analysis (including [21, 32, 3]).”
MethodVariance estimators for the Lempel-Ziv entropy rate estimator
Amigó, J. M., & Kennel, M. B. (2006) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.2347102 · Cited by 18
Information Dynamics: Its Theory and Application to Embodied Cognitive Systems
Williams, P. L. (2011) · doi:10.13140/2.1.1401.2162 · Cited by 28
Information-theoretic generalization bounds for black-box learning algorithmsPreprint
Harutyunyan, H., Raginsky, M., Steeg, G. V., & Galstyan, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.01584 · Cited by 74
Voita, E., Sennrich, R., & Titov, I. (2019) · arXiv · doi:10.48550/arxiv.1909.01380 · Cited by 17
“Even though the information-theoretic view provides insights into processes shaping the representations, direct MI estimation from finite samples for densities on multi-dimensional spaces is challenging (Paninski, 2003).”
BackgroundMonitoring Crack Propagation using Data-Driven Causal Discovery and Supervised Learning
Liu, R., & Misra, S. (2021) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2021.108791 · Cited by 18
Posterior Representations for Bayesian Context Trees: Sampling, Estimation and Convergence
Papageorgiou, I., & Kontoyiannis, I. (2023) · Bayesian Analysis · doi:10.1214/23-ba1362 · Cited by 21
“Entropy estimation has also received a lot of attention in the neuroscience literature (Strong et al., 1998; London et al., 2002; Nemenman et al., 2004; Paninski, 2003), in an effort to describe and quantify the amount of information transmitted by neurons.”
BackgroundFeature Selection for Network Intrusion Detection
Westphal, C., Hailes, S. M. V., & Musolesi, M. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709339 · Cited by 24
“One of the most commonly estimated information theoretic quantities is MI [5, 40, 46, 49, 63].”
Speckle statistics in adaptive optics images at visible wavelengths
Stangalini, M., Pedichini, F., Pinna, E., Christou, J. C., Hill, J. M., Puglisi, A. T., Bailey, V. P., Centrone, M., del Moro, D., Esposito, S., et al. (2017) · Journal of Astronomical Telescopes Instruments and Systems · doi:10.1117/1.jatis.3.2.025001 · Cited by 16
Direct Estimation of Density Functionals Using a Polynomial Basis
Wisler, A. A., Berisha, V., Spanias, A., & Hero, A. O. (2017) · IEEE Transactions on Signal Processing · doi:10.1109/tsp.2017.2775587 · Cited by 16
“When using a similar approach for entropy estimation, Paninski observed that the good approximation properties were a result of “large oscillating coefficients” which magnify the variance of the corresponding estimator [11].”
MethodResultA hierarchical Bayesian model to incorporate uncertainty into methods for diversity partitioning
Marion, Z. H., Fordyce, J. A., & Fitzpatrick, B. M. (2018) · Ecology · doi:10.1002/ecy.2174 · Cited by 16
Gribkova, E., Ibrahim, B. A., & Llano, D. A. (2018) · Journal of Neurophysiology · doi:10.1152/jn.00012.2018 · Cited by 18
Preserving domain private information via mutual information maximization
Chen, J., Wang, J., Lin, W., Zhang, K., & de Silva, C. W. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106112 · Cited by 17
Biological Network Inference Using Redundancy Analysis
Meyer, P. E., Kontos, K., & Bontempi, G. (2007) · Springer eBooks · doi:10.1007/978-3-540-71233-6_2 · Cited by 16
“Note however that other entropy estimators exist [22].”
BackgroundMethodWark, A. R., Wark, B., Lageson, T. J., & Peichel, C. L. (2010) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1130-x · Cited by 20
“Correct estimation of entropy is not trivial, and several methods exist in the literature (Paninski 2003).”
MethodInformation estimators for weighted observations
Hino, H., & Murata, N. (2013) · Neural Networks · doi:10.1016/j.neunet.2013.06.005 · Cited by 18
“…approaches, there are two representative non-parametric density estimation methods, the kernel density estimator (KDE) and the k -nearest neighbor ( k -NN) based estimator, (see Wand & Jones (1994); Beirlant et al. (1997); Gy¨orfi & van der Meulen (1987); Paninski (2003) for examples).”
MethodFeature selection with limited bit depth mutual information for portable embedded systems
Morán‐Fernández, L., Sechidis, K., Bolón‐Canedo, V., Alonso‐Betanzos, A., & Brown, G. (2020) · Knowledge-Based Systems · doi:10.1016/j.knosys.2020.105885 · Cited by 19
Pires, C., & Hannachi, A. (2017) · Complexity · doi:10.1155/2017/3076810 · Cited by 17
“It also provides a lower-bound of the positive JEd bias for nonGaussian pdfs, as it occurs for any negentropy estimator [46].”
BackgroundEstimation of generalized entropies with sample spacing
Wachowiak, M. P., Smolikova, R., Tourassi, G. D., & Elmaghraby, A. S. (2005) · Pattern Analysis and Applications · doi:10.1007/s10044-005-0247-4 · Cited by 16
“An empiric entropy estimator of order m, proposed for Shannon entropy [11], can also be used for estimating Renyi entropy.”
MethodInfluentialTemporal structure and gain-loss asymmetry for real and artificial stock indices
Siven, J. V., & Lins, J. T. (2009) · Physical Review E · doi:10.1103/physreve.80.057102 · Cited by 16
“Mutual information can estimated from a finite set { ( X t , Y t ) } t =1 ,...,n of joint samples of ( X, Y ) in a number of different ways, see Paninski [5].”
BackgroundPAC-Bayesian Generalization Bound for Density Estimation with Application to Co-clustering
Seldin, Y., & Tishby, N. (2009) · MPG.PuRe (Max Planck Society) · Cited by 19
Pufferfish Privacy: An Information-Theoretic Study
Nuradha, T., & Goldfeld, Z. (2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.2023.3296288 · Cited by 24
“For continuous, high-dimensional random variables (which is often the regime of interest in modern privacy applications), the sample complexity of estimating mutual information grows exponentially with dimension [47], [48], making tests based on ϵ -MI DP infeasible.”
BackgroundToosi, A., Samadzadegan, F., & Javan, F. D. (2025) · Advances in Space Research · doi:10.1016/j.asr.2025.02.019 · Cited by 15
Statistical Hypothesis Testing in Positive Unlabelled Data
Sechidis, K., Calvo, B., & Gavin Brown, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44845-8_5 · Cited by 15
“where Î(X ;Y ) is the maximum likelihood estimator of the mutual information between X and Y [17].”
BackgroundAmigó, J. M., Monetti, R., Tort‐Colet, N., & Sanchez-Vives, M. V. (2015) · Journal of Computational Neuroscience · doi:10.1007/s10827-015-0563-7 · Cited by 16
“This is usually done via relative frequencies if the time series is sufficiently long, or, otherwise, via estimators that take finite-size effects into account (Paninski 2003).”
A scalable saliency-based feature selection method with instance-level information
Cancela, B., Bolón‐Canedo, V., Alonso‐Betanzos, A., & da Gama, J. M. P. (2019) · Knowledge-Based Systems · doi:10.1016/j.knosys.2019.105326 · Cited by 21
Open-Source Tools for Processing and Analysis of In Vitro Extracellular Neuronal Signals
Mahmud, M., & Vassanelli, S. (2019) · Advances in neurobiology · doi:10.1007/978-3-030-11135-9_10 · Cited by 15
A Model-based Approach to Realize Privacy and Data Protection by Design
Pedroza, G., Muntés-Mulero, V., Martín, Y. S., & Mockly, G. (2021) · IEEE European Symposium on Security and Privacy Workshops (EuroS&PW) · doi:10.1109/eurospw54576.2021.00042 · Cited by 15
“These strategies can be applied to process-oriented models to validate the allocated privacy requirements.”
BackgroundBias-free estimation of information content in temporally sparse neuronal activity
Sheintuch, L., Rubin, A., & Ziv, Y. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1009832 · Cited by 16
Detecting continuous-variable entanglement in phase space with the Q distribution
Gärttner, M., Haas, T., & Noll, J. (2023) · Physical Review A · doi:10.1103/physreva.108.042410 · Cited by 17
“Moreover, for specific choices of f direct ways of extracting f ( Q ± ) , or its phase space integral [111, 112], from samples exist, circumventing full density estimation, which may lead to even more data-efficient estimates.”
BackgroundMethodRole of Relaxation Time Scale in Noisy Signal Transduction
Maity, A. K., Chaudhury, P., & Banik, S. K. (2015) · PLoS ONE · doi:10.1371/journal.pone.0123242 · Cited by 17
“In the calculation of mutual information using theoretical and experimental data, specially the last one, bias estimation has a significant role [52, 66].”
BackgroundPienkowski, M., Shaw, G., & Eggermont, J. J. (2009) · Journal of Neurophysiology · doi:10.1152/jn.91242.2008 · Cited by 17
Gordon, N., Shackleton, T. M., Palmer, A. R., & Nelken, I. (2007) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2007.11.005 · Cited by 15
Ku, W. L., Duggal, G., Li, Y., Girvan, M., & Ott, E. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031969 · Cited by 18
“F for yeast using both empirical [9] and Miller-Madow [23] estimators with both equal-width and equal-frequency binning (10 bins for both).”
BackgroundMethodInfluentialEstimating mutual information on data streams
Keller, F., Müller, E., & Böhm, K. (2015) · International Conference on Scientific and Statistical Database Management · doi:10.1145/2791347.2791348 · Cited by 15
“This leads to a heavily biased estimate of mutual information, which has been studied extensively [27, 29, 11, 13].”
BackgroundMarkov chain order estimation with parametric significance tests of conditional mutual information
Papapetrou, M., & Kugiumtzis, D. (2015) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2015.11.002 · Cited by 15
“Sequence analysis, initially developed mostly for biological applications [1], has expanded with regard to both applications and methodologies, and sequence mining techniques are constantly being developed [2].”
BackgroundHo, N., Vo, H., Vu, M. H., & Pedersen, T. B. (2019) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2907987 · Cited by 21
“Among several estimationmethods [39] (e.”
Transitive Sequential Pattern Mining for Discrete Clinical Data
Estiri, H., Vasey, S., & Murphy, S. N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59137-3_37 · Cited by 16
Emotion Recognition Based on DEAP Database Physiological Signals
Stajić, T., Jovanović, J., Jovanović, N., & Janković, M. M. (2021) · Telecommunications Forum (TELFOR) · doi:10.1109/telfor52709.2021.9653286 · Cited by 17
Information Dynamics and Emergent Computation in Recurrent Circuits of Spiking Neurons
Natschläger, T., & Maass, W. (2003) · neural information processing systems · Cited by 19
Modeling the Adversary to Evaluate Password Strength With Limited Samples
Komanduri, S. (2016) · KiltHub Repository · doi:10.1184/r1/6720701 · Cited by 28
The Broad Optimality of Profile Maximum LikelihoodPreprint
Hao, Y., & Orlitsky, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03794 · Cited by 15
“ll as the empirical estimator, and the empirical estimator with a larger nlognsample size. Additional entropy estimators such as the Miller-Mallow estimator [17], the best upper bound (BUB) estimator [67], and the Valiant-Valiant estimator [79] were compared in [84, 48] and found to perform similarly to or worse than…”
ResultSpectrally and temporally resolved estimation of neural signal diversityPreprint
Mediano, P. A. M., Rosas, F. E., Luppi, A. I., Noreika, V., Seth, A. K., Carhart-Harris, R. L., Barnett, L., & Bor, D. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.03.30.534922 · Cited by 18
“This is due to several factors, most prominently the “curse of dimensionality” ( Paninski, 2003 ).”
BackgroundSpectrally and temporally resolved estimation of neural signal diversityPreprint
Mediano, P. A. M., Rosas, F. E., Luppi, A. I., Noreika, V., Seth, A. K., Carhart-Harris, R. L., Barnett, L., & Bor, D. (2023) · doi:10.7554/elife.88683.1 · Cited by 19
Stabilization of Sunflower Oil with Biologically Active Compounds from Berries
Ghendov‐Moșanu, A., Popovici, V., Constantinescu, C. G., Deseatnicova, O., Siminiuc, R., Subotin, I., Druţă, R., Pintea, A., Socaciu, C., & Sturza, R. (2023) · Molecules · doi:10.3390/molecules28083596 · Cited by 17
“The more pronounced the influence of storage time on the physicochemical quality parameters, the higher the bit value [74].”
BackgroundBulgaru, V., Gurev, A., Baerle, A., Dragancea, V., Bălan, G., Cojocari, D., Sturza, R., Soran, M.-L., & Ghendov‐Moșanu, A. (2024) · Antioxidants · doi:10.3390/antiox13080890 · Cited by 14
“The higher the bit value, the more pronounced the influence of the extraction methods, pH, and optimal extraction conditions [52].”
Step-Wise Distribution-Aligned Style Prompt Tuning for Source-Free Cross-Domain Few-Shot Learning
Xu, H., Liu, L., Liu, T., Zhi, S., Sun, S., & Cheng, M. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3610039 · Cited by 20
“Here, I ( · , · ) denotes the mutual information function [62], computed in a manner similar to [63].”
Bayesian estimation of discrete entropy with mixtures of stick-breaking priors
Archer, E., Park, I. M., & Pillow, J. W. (2012) · neural information processing systems · Cited by 14
Nonparametric triple collocation
Nearing, G., Yatheendradas, S., Crow, W. T., Bosch, D. D., Cosh, M. H., Goodrich, D. C., Seyfried, M. S., & Starks, P. J. (2017) · Water Resources Research · doi:10.1002/2017wr020359 · Cited by 14
“These ratios are the statistics that we would like to estimate, and if our variables are discrete then each ratio is bounded between 0 and 1 with the usual implications of independence or identity at the extremes [Paninski, 2003].”
BackgroundMethodPerinelli, A., Castelluzzo, M., Tabarelli, D., Mazza, V., & Ricci, L. (2021) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0053857 · Cited by 15
Learn from Your Faults: Leakage Assessment in Fault Attacks Using Deep Learning
Saha, S., Alam, M., Bag, A., Mukhopadhyay, D., & Dasgupta, P. (2023) · Journal of Cryptology · doi:10.1007/s00145-023-09462-6 · Cited by 14
“Although such data-based estimation of mutual information is possible, it is challenging and still an active area of research [50,51].”
BackgroundAnarchists, Unite: Practical Entropy Approximation for Distributed Streams
Gabel, M., Keren, D., & Schuster, A. (2017) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3097983.3098092 · Cited by 14
“Entropy fluctuates widely, both due to large hourly differences, but also because local windows only have nj = 24 observations in the window so the MLE estimator is at the lower range of accuracy (where nj ∼ k [39, 41]).”
BackgroundMethodInfluentialGötz, T., Lahmer, G., Strnad, V., Bert, C., Hensel, B., Tomé, A. M., & Lang, E. W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183608 · Cited by 17
Directional and Causal Information Flow in EEG for Assessing Perceived Audio Quality
Mehta, K., & Kliewer, J. (2017) · IEEE Transactions on Molecular Biological and Multi-Scale Communications · doi:10.1109/tmbmc.2018.2806454 · Cited by 16
“For example, [7], [8] presented mutual information (MI) [9] estimates to assess the correlation between the spike timing of an ensemble of neurons.”
BackgroundStatistical estimation of conditional Shannon entropy
Bulinski, A. V., & Kozhevin, A. (2018) · ESAIM Probability and Statistics · doi:10.1051/ps/2018026 · Cited by 17
Dynamics of respiratory symptoms during infancy and associations with wheezing at school age
Usemann, J., Xu, B., Delgado‐Eckert, E., Korten, I., Anagnostopoulou, P., Gorlanova, O., Kuehni, C. E., Röösli, M., Latzin, P., & Frey, U. (2018) · ERJ Open Research · doi:10.1183/23120541.00037-2018 · Cited by 15
“[7] For each infant, we calculated the average estimated entropy[6] of all the conditional probability distributions defined by the rows of the participant’s Markov matrix.”
MethodDetecting and testing altered brain connectivity networks with k-partite network topology
Chen, S., Bowman, F. D., & Xing, Y. (2019) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2019.06.007 · Cited by 14
“T c0 is the contrast between empirical estimates of information entropy (Paninski, 2003) We leverage the edge permutation and graph combinatorics to examine the hypotheses H 0 and H 1 , that is whether differentially expressed edges are concentrated in the detected topological pattern of G c .”
BackgroundApplying the Multivariate Time-Rescaling Theorem to Neural Population Models
Gerhard, F., Haslinger, R., & Pipa, G. (2011) · Neural Computation · doi:10.1162/neco_a_00126 · Cited by 19
“…the neuron’s own spike history on the output spikes (Brillinger, 1988; Chornoboy, Schramm, & Karr, 1988; Kass & Ventura, 2001; Brown et al., 2003; Paninski, 2003; Truccolo et al., 2005; Okatan, Wilson, & Brown, 2005; Kass et al., 2005; Rigat, de Gunst, & van Pelt, 2006; Pillow, 2007;…”
BackgroundInfluentialJames, R. G., Ayala, B. D. M., Zakirov, B., & Crutchfield, J. P. (2026) · Entropy · doi:10.3390/e28101162 · Cited by 14
Entropy Rate Estimation for Markov Chains with Large State SpacePreprint
Han, Y., Jiao, J., Lee, C.-Z., Weissman, T., Wu, Y., & Yu, T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.07889 · Cited by 14
“The part pertaining to the empirical entropy follows from [2],[31, Proposition 1],[20, Eqn.”
BackgroundTight Mutual Information Estimation With Contrastive Fenchel-Legendre OptimizationPreprint
Guo, Q., Chen, J., Wang, D., Yang, Y., Deng, X., Carin, L., Li, F., Huang, J., & Tao, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.01131 · Cited by 14
“MI estimation has been extensively studied [6, 51, 50, 59, 61, 77, 10].”
BackgroundQuantifying gender bias towards politicians in cross-lingual language models
Stańczak, K., Ray Choudhury, S., Pimentel, T., Cotterell, R., & Augenstein, I. (2023) · PLOS ONE · doi:10.1371/journal.pone.0277640 · Cited by 13
Divergence-guided disentanglement of view-common and view-unique representations for multi-view data
Lu, M., Zhang, Q., & Chen, B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102661 · Cited by 18
Entropy of Neuronal Spike Patterns
Luczak, A. (2024) · Entropy · doi:10.3390/e26110967 · Cited by 16
“This sparsity makes it challenging to obtain accurate probability distributions necessary for entropy calculations since traditional estimation methods require an impracti-cally large amount of data to sample the space adequately [65,66].”
MINT: A toolbox for the analysis of multivariate neural information coding and transmission
Lorenz, G. M., Engel, N. M., Celotto, M., Koçillari, L., Curreli, S., Fellin, T., & Panzeri, S. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012934 · Cited by 14
Renyi Entropy Estimation Revisited
Obremski, M., & Skórski, M. (2017) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.approx-random.2017.20 · Cited by 24
Split-door criterion: Identification of causal effects through auxiliary outcomes
Sharma, A. K., Hofman, J. M., & Watts, D. J. (2018) · The Annals of Applied Statistics · doi:10.1214/18-aoas1179 · Cited by 15
“For example, it is well known that in small sample sizes, testing for independence via mutual information estimation can be heavily biased [Paninski (2003)].”
BackgroundNeural Joint Entropy Estimation
Shalev, Y., Painsky, A., & Ben‐Gal, I. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3204919 · Cited by 20
“We refer the reader to [9], [11], [18], [20], [29], [30] for a comprehensive review of these measures.”
BackgroundQuantifying the Predictability of Visual Scanpaths Using Active Information Storage
Wollstadt, P., Hasenjäger, M., & Wiebel-Herboth, C. B. (2021) · Entropy · doi:10.3390/e23020167 · Cited by 13
“Estimation bias depends on the number of samples and the size of the variables used [43, 44].”
BackgroundFay, L., Cobos, E., Yang, B., Gatidis, S., & Küstner, T. (2023) · IEEE Access · doi:10.1109/access.2023.3289397 · Cited by 19
“However, these approaches faced the challenge of minimizing the mutual information since the calculation of MI is computationally complex, and the MI is non-differentiable [58].”
Binary Graph Convolutional Network With Capacity Exploration
Wang, J., Guo, Y., Yang, L., & Wang, Y. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3342224 · Cited by 15
“Then, given N input samples, the empirical distribution of any floating-point valued neuron, e.g., h i , can be formulated as where δ m denotes the probability measurement concentrated on m [51].”
BackgroundShade: Information-Based Regularization for Deep Learning
Blot, M., Robert, T., Thome, N., & Cord, M. (2018) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2018.8451092 · Cited by 15
Ruiz, R., de la Cruz, F., & Fernández-López, R. (2018) · Science Advances · doi:10.1126/sciadv.aat5771 · Cited by 15
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.