Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 301–400 of 1,004
Estimating Information Rates with Confidence Intervals in Neural Spike Trains
Shlens, J., Kennel, M. B., Abarbanel, H. D. I., & Chichilnisky, E. J. (2007) · Neural Computation · doi:10.1162/neco.2007.19.7.1683 · Cited by 41
“Judging the convergence and bias of conventional estimators of entropy and mutual information is difficult (Treves & Panzeri, 1995; Paninski, 2003a), making the application of these tools particularly problematic in limited data sets.”
BackgroundInfluentialUSING DIRECTED INFORMATION TO BUILD BIOLOGICALLY RELEVANT INFLUENCE NETWORKS
Rao, A. U. K., Hero, A. O., States, D. J., & Engel, J. D. (2008) · Journal of Bioinformatics and Computational Biology · doi:10.1142/s0219720008003515 · Cited by 43
Lossy Compression for Lossless PredictionPreprint
Dubois, Y., Bloem-Reddy, B., Ullrich, K., & Maddison, C. J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.10800 · Cited by 82
“The mutual information is well known to be hard to estimate from samples (Paninski, 2003; McAllester & Stratos, 2020), but fortunaltely many (11)Also called topological information content.”
BackgroundEarly Exiting with Ensemble Internal ClassifiersPreprint
Sun, T., Zhou, Y., Liu, X., Zhang, X., Jiang, H., Cao, Z., Huang, X., & Qiu, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13792 · Cited by 44
“Multi-information is defined as However, directly maximizing mutual information is intractable when the function is parameterized by modern neural networks [27].”
BackgroundToward Practical Usage of the Attention Mechanism as a Tool for Interpretability
Tutek, M., & Šnajder, J. (2022) · IEEE Access · doi:10.1109/access.2022.3169772 · Cited by 46
Metacognitive Information Theory
Dayan, P. (2023) · Open Mind · doi:10.1162/opmi_a_00091 · Cited by 41
“confidence in some experiments, is known to be hard to estimate, because of biases, and so care is necessary (Kozachenko & Leonenko, 1987; Paninski, 2003; Panzeri & Treves, 1996; Witter & Houghton, 2021).”
Kluver, D., Nguyen, T. T., Ekstrand, M. D., Sen, S., & Riedl, J. (2012) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2365952.2365974 · Cited by 37
“Unfortunately, it has been shown [16] that there is no unbiased estimator for mutual information, therefore it is important that large sample sizes (N >> m(2)R) be used to avoid estimator bias.”
BackgroundMethodMutual information regularized identity-aware facial expression recognition in compressed video
Liu, X., Jin, L., Han, X., & You, J. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108105 · Cited by 38
Semi-Supervised Hierarchical Graph Classification
Li, J., Huang, Y., Chang, H., & Rong, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3203703 · Cited by 44
“One naive method to compute HGMI is according to the definition, which inevitably involves with the computation of joint distribution P ðG;E;GÞ and is not tractable in general [41].”
MethodGuo, J., Li, J., Fu, H., Gong, M., Zhang, K., & Tao, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01771 · Cited by 45
“Since computing MI is difficult [42], researchers have taken much effort to improve the estimation of MI.”
MethodThe Tsallis entropy of natural information
Sneddon, R. (2007) · Physica A Statistical Mechanics and its Applications · doi:10.1016/j.physa.2007.05.065 · Cited by 37
Approximate Profile Maximum LikelihoodPreprint
Pavlichin, D. S., Jiao, J., & Weissman, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.07177 · Cited by 37
“4The fingerprint is also called the profile [6], histogram order statistics [19], and histogram of histograms [20].”
Predictive Models for Min-entropy Estimation
Kelsey, J. M., McKay, K. A., & Turan, M. S. (2015) · Lecture notes in computer science · doi:10.1007/978-3-662-48324-4_19 · Cited by 39
“, see [2, 15]).”
Advances in spatial entropy measures
Altieri, L., Cocchi, D., & Roli, G. (2019) · Stochastic Environmental Research and Risk Assessment · doi:10.1007/s00477-019-01686-y · Cited by 36
“In the current literature and in the present work, spatial entropy measures are estimated by substituting the unknown probabilities with the observed relative frequencies, obtaining the well known non parametric and also maximum likelihood entropy estimator (Paninski 2003).”
MethodInformation-theoretic locality properties of natural language
Futrell, R. (2019) · doi:10.18653/v1/w19-7902 · Cited by 51
Tensor networks and efficient descriptions of classical data
Lu, S., Kanász-Nagy, M., Kukuljan, I., & Cirac, J. I. (2025) · Physical Review A · doi:10.1103/physreva.111.032409 · Cited by 39
The New Urban Success: How Culture Pays
Hristova, D., Aiello, L. M., & Quercia, D. (2018) · Frontiers in Physics · doi:10.3389/fphy.2018.00027 · Cited by 36
“To overcome this problem we apply the Miller–Madow’s correction to the entropy computation [22].”
MethodInformation Processing in Biochemical Networks
Tkačik, G., & ten Wolde, P. R. (2025) · Annual Review of Biophysics · doi:10.1146/annurev-biophys-060524-102720 · Cited by 46
Simoes, R. D. M., & Emmert‐Streib, F. (2011) · PLoS ONE · doi:10.1371/journal.pone.0029279 · Cited by 39
“The Miller-Madow estimator [3] accounts for the under-sampling bias by adjusting the estimate by a constant factor that is proportional to the bin size and sample size: for b number of bins and N number of samples.”
BackgroundMethodInfluentialLeast-Squares Independent Component Analysis
Suzuki, T., & Sugiyama, M. (2010) · Neural Computation · doi:10.1162/neco_a_00062 · Cited by 35
“Note that SMI is the Pearson divergence (Pearson, 1900; Paninsky, 2003; Liese & Vajda, 2006; Cichocki, Zdunek, Phan, & Amari, 2009) between q (z) and r (z), while ordinary MI is the Kullback-Leibler divergence (Kullback & Leibler, 1951).”
BackgroundInfluentialNetwork Analysis of the Shanghai Stock Exchange Based on Partial Mutual Information
You, T., Fiedor, P., & Hołda, A. (2015) · Journal of risk and financial management · doi:10.3390/jrfm8020266 · Cited by 35
Ensemble Estimation of Information Divergence †
Moon, K. R., Sricharan, K., Greenewald, K. H., & Hero, A. O. (2018) · Entropy · doi:10.3390/e20080560 · Cited by 49
λPSI: exact inference for higher-order probabilistic programs
Gehr, T., Steffen, S., & Vechev, M. (2020) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/3385412.3386006 · Cited by 47
“Note that there exists no (purely sampling-based) unbiased estimator for entropy or mutual information [16].”
BackgroundEntropy as a measure of variability and stemness in single-cell transcriptomics
Gandrillon, O., Gaillard, M., Espinasse, T., Garnier, N., Dussiau, C., Kosmider, O., & Sujobert, P. (2021) · Current Opinion in Systems Biology · doi:10.1016/j.coisb.2021.05.009 · Cited by 37
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning
Lee, S., Chung, J., Yu, Y., Kim, G., Breuel, T. M., Chechik, G., & Song, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01011 · Cited by 84
“MI estimation has a long history of research [53, 35], including the recent self-supervised approaches [50, 28, 12] that use noise contrastive estimation [23] as the learning objective.”
MethodTemporal Encoding in a Nervous System
Aldworth, Z., Dimitrov, A. G., Cummins, G. I., Gedeon, T., & Miller, J. P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002041 · Cited by 40
“Assuming that biases are appropriately accounted for [6,7], the former method gives an accurate estimate of the true information rate contained in neural activity and allows for encoding of stimulus parameters by temporal patterns of responses (as well as all other types of responses), however it gives no model for ho…”
MethodRecurrent, Robust and Scalable Patterns Underlie Human Approach and Avoidance
Kim, B. W., Kennedy, D. N., Lehár, J., Lee, M. J., Blood, A. J., Lee, S., Perlis, R. H., Smoller, J. W., Morris, R. R., Fava, M., et al. (2010) · PLoS ONE · doi:10.1371/journal.pone.0010613 · Cited by 37
“This information theoretic approach [64,66,67,137–143] is grounded in the classical theory of entropy, but does not necessitate the temporal evolution of an ensemble.”
Mutual Information Rate and Bounds for It
Baptista, M. S., Rubinger, R. M., Viana, E. R., Sartorelli, J. C., Parlitz, U., & Grebogi, C. (2012) · PLoS ONE · doi:10.1371/journal.pone.0046745 · Cited by 35
“As a consequence, mutual information can often be calculated with a bias, only [4,11–13].”
BackgroundInformation estimation using nonparametric copulas
Safaai, H., Onken, A., Harvey, C. D., & Panzeri, S. (2018) · Physical review. E · doi:10.1103/physreve.98.053302 · Cited by 44
An information-theoretic analysis of deep latent-variable modelsPreprint
Alemi, A. A., Poole, B., Fischer, I., Dillon, J. V., Saurous, R. A., & Murphy, K. P. (2018) · arXiv (Cornell University) · Cited by 48
What can spike train distances tell us about the neural code?
Chicharro, D., Kreuz, T., & Andrzejak, R. G. (2011) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2011.05.002 · Cited by 41
“…estimation of this quantity from experimental data is hindered by the curse of dimensionality caused by sampling limitations (see for example Borst and Theunissen, 1999; Paninski, 2003; Victor, 2006; Panzeri et al., 2007, for a discussion of different estimation techniques and bias corrections).”
A Unified Definition of Mutual Information with Applications in Machine Learning
Zeng, G. (2015) · Mathematical Problems in Engineering · doi:10.1155/2015/201874 · Cited by 34
“On the one hand, the joint density function involved is unknown inmost cases and hence must be estimated [13, 14].”
BackgroundAttribute vector guided groupwise registration
Wang, Q., Wu, G., Yap, P., & Shen, D. G. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.01.040 · Cited by 37
Learning Gaussian Mixture Models With Entropy-Based Criteria
Benavent, A. P., Ruiz, F. J. E., & Saez, J. M. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2009.2030190 · Cited by 34
Cateni, S., Colla, V., & Vannucci, M. (2020) · International Journal of Simulation Systems Science & Technology · doi:10.5013/ijssst.a.12.02.04 · Cited by 34
“2 ISSN: 1473-804x online, 1473-8031 print 32 The Mutual Information (MI) [34] between each variable and the target is calculated to obtain a measure of dependency between the density of each variable and the density of the target.”
BackgroundSyntactic dependencies correspond to word pairs with high mutual information
Futrell, R., Qian, P., Gibson, E. A. F., Fedorenko, E. G., & Blank, I. (2019) · International Conference on Dependency Linguistics (Depling) · doi:10.18653/v1/w19-7703 · Cited by 34
“It is notoriously challenging to estimate information-theoretic quantities such as entropy and mutual information from count data (Miller, 1955; Paninski, 2003; Archer et al., 2013), especially for distributions with long tails, such as wordforms of natural language.”
BackgroundA Bayesian Nonparametric Approach to Testing for Dependence Between Random Variables
Filippi, S., & Holmes, C. C. (2016) · Bayesian Analysis · doi:10.1214/16-ba1027 · Cited by 33
“It is often based on approximations of the probability distributions (Paninski 2003; Cellucci et al. 2005; Khan et al. 2007) either using bin-based procedures, kernel densities (Moon et al. 1995), or k-nearest neighbours (Kraskov et al. 2004).”
MethodMeasuring heterogeneity in urban expansion via spatial entropy
Altieri, L., Cocchi, D., & Roli, G. (2018) · Environmetrics · doi:10.1002/env.2548 · Cited by 33
“In the literature and in this work, spatial entropy measures are proposed as descriptive indices, substituting the unknown probabilities with the observed relative frequencies, obtaining the nonparametric and the maximum likelihood entropy estimators (Paninski, 2003).”
MethodMicrotubule instability driven by longitudinal and lateral strain propagation
Igaev, M., & Grubmüller, H. (2020) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1008132 · Cited by 33
“We therefore used the Jack Knifed estimate that is known to be a low bias estimate of the MI and robust to discretization bin size [70].”
MethodMitigating Embedding and Class Assignment Mismatch in Unsupervised Image Classification
Han, S., Park, S., Park, S., Kim, S., & Cha, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_45 · Cited by 41
“Mutual information quan-tifies mutual dependencies of two random variables [28], and measures how much two variables share the same kind of information.”
BackgroundThe Fundamental Limits of Structure-Agnostic Functional Estimation
Balakrishnan, S., Kennedy, E. H., & Wasserman, L. (2026) · Statistical Science · doi:10.1214/25-sts997 · Cited by 33
“Beyond causal inference, functional estimation problems arise routinely in machine learning [25, 44], information theory [24, 29, 35, 53], theoretical computer science [48] and other fields.”
Wang, S., Chen, X., Du, D., Zheng, W. L., Hu, L., Yang, H., Cheng, J., & Gong, M. (2018) · Analytical Chemistry · doi:10.1021/acs.analchem.8b03065 · Cited by 33
Structural Determinants of Functional Brain Dynamics
Sporns, O., & Tononi, G. (2007) · Understanding complex systems · doi:10.1007/978-3-540-71512-2_4 · Cited by 37
“As mentioned above, these estimates can be difficult to derive from small or sparse data sets such as those often encountered in neurobiological applications and, in many cases, additional statistical assumptions have to be made (e.g. Paninski, 2003; Pola et al., 2003).”
BackgroundJahagirdar, S., & Saccenti, E. (2020) · Metabolites · doi:10.3390/metabo10040171 · Cited by 35
Hematopoietic differentiation is characterized by a transient peak of entropy at a single-cell level
Dussiau, C., Boussaroque, A., Gaillard, M., Bravetti, C., Zaroili, L., Knosp, C. A., Friedrich, C., Asquier, P., Willems, L., Quint, L., et al. (2022) · BMC Biology · doi:10.1186/s12915-022-01264-9 · Cited by 37
Efficient Temporal Coding in the Early Visual System: Existing Evidence and Future Directions
Price, B. H., & Gavornik, J. P. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.929348 · Cited by 37
“(3) Efficient coding makes very precise, testable predictions in certain contexts, but, in general, information-theoretic measures are very difficult, if not impossible, to estimate (Paninski, 2003) making the overall utility of applying efficient coding to the cortex unclear.”
BackgroundInformation-theoretic approach to lead-lag effect on financial markets
Fiedor, P. (2014) · The European Physical Journal B · doi:10.1140/epjb/e2014-50108-3 · Cited by 40
THE MUTUAL INFORMATION DIAGRAM FOR UNCERTAINTY VISUALIZATION
Correa, C. D., & Lindström, P. (2012) · International Journal for Uncertainty Quantification · doi:10.1615/int.j.uncertaintyquantification.2012003959 · Cited by 37
Estimating information in time-varying signals
Cepeda-Humerez, S. A., Ruess, J., & Tkačik, G. (2019) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1007290 · Cited by 38
“The simplest such approach is perhaps the direct estimator of information or entropy [5], and a good review is provided in Ref [4].”
BackgroundInfluentialSparsityBoost: A New Scoring Function for Learning Bayesian Network StructurePreprint
Brenner, E., & Sontag, D. A. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1309.6820 · Cited by 35
“We expect that many of the techniques that we developed will be broadly useful beyond Bayesian network structure learning.”
MethodMeasuring dependence powerfully and equitablyPreprint
Reshef, Y., Reshef, D. N., Finucane, H. K., Sabeti, P. C., & Mitzenmacher, M. (2016) · arXiv (Cornell University) · doi:10.5555/2946645.3053493 · Cited by 32
Tang, X., Zhang, Q., & Hu, L. (2020) · IEEE Access · doi:10.1109/access.2020.2984744 · Cited by 31
“…method [22] to give where G (cid:54) ( · ) is the Gaussian function defined as follows: Therefore, almost all of the concepts based on probability density functions, such as entropy, mutual information, etc, can be estimated by date-based approximation using the approach we introduced above [23].”
Spatial entropy for biodiversity and environmental data: The R-package SpatEntropy
Altieri, L., Cocchi, D., & Roli, G. (2021) · Environmental Modelling & Software · doi:10.1016/j.envsoft.2021.105149 · Cited by 31
A mutual information based face clustering algorithm for movie content analysis
Vretos, N., Solachidis, V., & Pitas, I. (2011) · Image and Vision Computing · doi:10.1016/j.imavis.2011.07.006 · Cited by 35
Numata, J., & Knapp, E. (2012) · Journal of Chemical Theory and Computation · doi:10.1021/ct200910z · Cited by 34
Information theoretic approaches for inference of biological networks from continuous-valued data
Budden, D., & Crampin, E. J. (2016) · BMC Systems Biology · doi:10.1186/s12918-016-0331-y · Cited by 33
“Despite the introduction of several bias-correction techniques [22–24] and the observation that I D ( X ; Y ) = lim (cid:5) → 0 I ( X (cid:5) ; Y (cid:5) ) for discretisations X and Y with bin width (cid:5) [25], it is well-established that discretisation is a suboptimal method for handling…”
MethodDąbrowski, M., Mazelanik, M., Parniak, M. P., Leszczyński, A., Lipka, M., & Wasilewski, W. (2018) · Physical Review A · doi:10.1103/physreva.98.042126 · Cited by 36
Qiu, J., Crow, W. T., Dong, J., & Nearing, G. (2020) · Hydrology and earth system sciences · doi:10.5194/hess-24-581-2020 · Cited by 31
“(7) are then approximated as summations over the empirical probability distribution function bins (Paninski, 2003).”
A review of symbolic dynamics and symbolic reconstruction of dynamical systems
Hirata, Y., & Amigó, J. M. (2023) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/5.0146022 · Cited by 35
Analytic Pattern Matching: From DNA to Twitter
Jacquet, P., & Szpankowski, W. (2015) · Cambridge University Press eBooks · doi:10.1017/cbo9780511843204 · Cited by 42
Adaptive estimation of Shannon entropy
Han, Y., Jiao, J., & Weissman, T. (2015) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2015.7282680 · Cited by 34
“Classical theory is mainly concerned with the case where the number of samples n→∞ while the alphabet size S is fixed.”
BackgroundUndersmoothed Kernel Entropy Estimators
Paninski, L., & Yajima, M. (2008) · IEEE Transactions on Information Theory · doi:10.1109/tit.2008.928251 · Cited by 31
Self-citationSteganalysis Feature Selection With Multidimensional Evaluation and Dynamic Threshold Allocation
Ma, Y., Xu, L., Zhang, Y., Zhang, T., & Luo, X. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3295364 · Cited by 30
“So far, scholars have conducted lots of research on feature selection and dimension reduction for steganalysis and proposed a series of methods [33], [34], [35], [36], [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51].”
BackgroundStopping rules for mutual information-based feature selection
Mielniczuk, J., & Teisseyre, P. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.05.048 · Cited by 30
“For more information on entropy estimation procedures, we refer the reader to [25] .”
MethodFu, L., Yang, Y., Yao, X., Jiao, X., & Zhu, T. (2019) · Energies · doi:10.3390/en12203817 · Cited by 30
“In the theory of information entropy, mutual information is utilized to represent the correlation between two variables [34,35].”
MethodSynaptic Scaling—An Artificial Neural Network Regularization Inspired by Nature
Hofmann, M. R., & Mäder, P. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3050422 · Cited by 37
“Mutual information is a measure for the information of the input distribution that is passed to the reconstructor and has, therefore, been used to study the superior generalization of deep neural network topologies over shallow ones before [26], [27].”
MethodOn Accuracy of PDF Divergence Estimators and Their Applicability to Representative Data Sampling
Budka, M., Gabryś, B., & Musiał, K. (2011) · Entropy · doi:10.3390/e13071229 · Cited by 34
“, [47,48]) or closed-form solutions are used, restricting the PDFs to be Gaussian (e.”
MethodEfficient High-Order Interaction-Aware Feature Selection Based on Conditional Mutual Information
Shishkin, А., Bezzubtseva, A., Drutsa, A. V., Shishkov, I., Gladkikh, E., Gusev, G., & Serdyukov, P. (2016) · neural information processing systems · Cited by 29
On the Analysis and Improvement of Min-Entropy Estimation on Time-Varying Data
Zhu, S., Ma, Y., Li, X., Yang, J., Lin, J., & Jing, J. (2019) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2019.2947871 · Cited by 29
Estiri, H., Strasser, Z. H., Klann, J. G., McCoy, T. H., Wagholikar, K. B., Vasey, S., Castro, V. M., Murphy, M., & Murphy, S. N. (2020) · Patterns · doi:10.1016/j.patter.2020.100051 · Cited by 31
“Mutual information provides a measurement of the mutual dependence between two random variables, which unlike most correlation measures can capture non-linear relationships.(50,51) We ranked the data representations based on their mutual information with the labeled outcome (in ties, we used prevalence to determine th…”
Nonlinearity and Multivariate Dependencies in the Terrestrial Leg of Land‐Atmosphere Coupling
Hsu, H., & Dirmeyer, P. A. (2021) · Water Resources Research · doi:10.1029/2020wr028179 · Cited by 29
Amer: A New Attribute-Missing Network Embedding Approach
Jin, D., Wang, R., Wang, T., He, D., Ding, W., Huang, Y., Wang, L., & Pedrycz, W. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3166539 · Cited by 34
“A well-known problem of mutual information is that it is difficult to compute [53].”
BackgroundRobust Architecture-Agnostic and Noise Resilient Training of Photonic Deep Learning Models
Kirtas, M., Passalis, N., Mourgias-Alexandris, G., Dabos, G., Pleros, N., & Tefas, A. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3182765 · Cited by 34
“Moreover, estimating the mutual information in high-dimensional spaces is especially challenging and inefficient [40].”
BackgroundLi, E., Zhang, N., Xi, B., Zhou, J., & Gao, X. (2023) · Frontiers of Structural and Civil Engineering · doi:10.1007/s11709-023-0997-3 · Cited by 28
Data compression and inference in cosmology with self-supervised machine learning
Akhmetzhanova, A., Mishra-Sharma, S., & Dvorkin, C. (2023) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stad3646 · Cited by 36
Schiavo, M. (2024) · Journal of Hydrology · doi:10.1016/j.jhydrol.2024.131072 · Cited by 29
Variational Graph Embedding for Globally and Locally Consistent Feature Extraction
Yang, S.-H., Hongyuan Zha, B., Zhou, S., & Hu, B.-G. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04174-7_35 · Cited by 29
“Compared with other criteria that only account for up to second-order statistics, an appealing property of MI is that it makes use of higher-order statistics and is able to capture the complete nonlinear dependence between the features and the class label [14,16].”
BackgroundInformation-theoretic selection of high-dimensional spectral features for structural recognition
Bonev, B., Escolano, F., Giorgi, D., & Biasotti, S. (2012) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2012.11.007 · Cited by 29
Peng, Z., Genewein, T., & Braun, D. A. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00168 · Cited by 29
“There are a number of approaches to correct for this bias (Efron and Stein, 1981; Strong et al., 1998; Antos and Ioannis Kontoyiannis, 2001), but in certain data regimes also “bias-corrected” estimators are likely to be contaminated by bias (Paninski, 2003).”
BackgroundAnonymous routing in structured peer-to-peer overlays
Borissov, N., & Brewer, E. (2005) · Cited by 29
Selecting an Effective Entropy Estimator for Short Sequences of Bits and Bytes with Maximum Entropy
Rodríguez, L. C., Madarro-Capó, E. J., Legón-Pérez, C. M., Rojas, O., & Sosa-Gómez, G. (2021) · Entropy · doi:10.3390/e23050561 · Cited by 32
“It is well known that there is not an unbiased estimator [4], and the convergence rate of a consistent estimator can be arbitrarily slow [5].”
BackgroundMethodInfluentialSalafian, B., Ben-Knaan, E. F., Shlezinger, N., de Ribaupierre, S., & Farsad, N. (2023) · IEEE Access · doi:10.1109/access.2023.3252897 · Cited by 28
“Although MI is a reliable measure to capture statistical dependence, the exact calculation based on (1) and (3) for finite continuous and non-parametric EEG samples is challenging [53].”
MethodInformation-Based Skill Scores for Probabilistic Forecasts
Ahrens, B., & Walser, A. (2008) · Monthly Weather Review · doi:10.1175/2007mwr1931.1 · Cited by 31
“To be well defined, forecasters should replace any zero forecast probability value in IGNt determination with a very small positive probability , as motivated in Roulston and Smith (2002). The IGNt is a statistic defined for single dates and thus is a scoring rule (Murphy 1997).”
BackgroundMethodEstimating the unseen: A sublinear-sample canonical estimator of distributions.Preprint
Valiant, G. J., & Valiant, P. (2010) · Electronic colloquium on computational complexity · Cited by 28
Copula Correlation: An Equitable Dependence Measure and Extension of Pearson's CorrelationPreprint
Ding, A. A., & Li, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.7214 · Cited by 28
“Over all distributions, even discrete ones, no uniform rate of convergence is possible for MI (Antos and Kontoyiannis, 2001; Paninski, 2003).”
BackgroundWang, K., Zhang, J., Bai, G., Ko, R., & Dong, J. S. (2021) · Proceedings of the Web Conference 2021 · doi:10.1145/3442381.3450008 · Cited by 27
“We estimate the former using a histogram-based approach commonly used for entropy estimation [34].”
MethodAn evolutionary many-objective approach to multiview clustering using feature and relational data
José-García, A., Handl, J., Gómez‐Flores, W., & Garza-Fabre, M. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107425 · Cited by 27
“The clustering performance of the multiview clustering algorithms is also evaluated by using the Normalized Mutual Information (NMI) metric [35].”
MethodBeyond linear regression: mapping models in cognitive neuroscience should align with research goals
Ivanova, A. A., Schrimpf, M., Anzellotti, S., Zaslavsky, N., Fedorenko, E. G., & Işık, L. (2022) · Neurons Behavior Data analysis and Theory · doi:10.51628/001c.37507 · Cited by 31
Robust cone-mediated signaling persists late into rod photoreceptor degeneration
Scalabrino, M. L., Thapa, M., Chew, L. A., Zhang, E., Xu, J., Sampath, A. P., Chen, J., & Field, G. D. (2022) · eLife · doi:10.7554/elife.80271 · Cited by 28
“It has the potential disadvantages that it is a datahungry analysis and can be prone to certain biases (Paninski, 2003).”
BackgroundA robust estimator of mutual information for deep learning interpretability
Piras, D., Peiris, H. V., Pontzen, A., Lucie-Smith, L., Guo, N., & Nord, B. (2023) · Machine Learning Science and Technology · doi:10.1088/2632-2153/acc444 · Cited by 35
Entropic representation and estimation of diversity indices
Zhang, Z., & Grabchak, M. (2016) · Journal of nonparametric statistics · doi:10.1080/10485252.2016.1190357 · Cited by 27
“We now extend our results to the estimator defined in (9).”
BackgroundInfluentialJustifying Additive Noise Model-Based Causal Discovery via Algorithmic Information Theory
Janzing, D., & Steudel, B. (2010) · Open Systems & Information Dynamics · doi:10.1142/s1230161210000126 · Cited by 35
“To give an impression on the amount of information between p Y | X and p Y that can be inferred after n -fold sampling, we recall that the mutual information between E and X can be estimated up to an accuracy of O (1 /n ) [12].”
BackgroundExtracting information in spike time patterns with wavelets and information theory
Lopes‐dos‐Santos, V., Panzeri, S., Kayser, C., Diamond, M. E., & Quiroga, R. Q. (2014) · Journal of Neurophysiology · doi:10.1152/jn.00380.2014 · Cited by 34
“Although several statistical methods have been developed to correct for sampling biases arising in these cases (Montemurro et al. 2007; Nemenman et al. 2004; Paninski 2003; Panzeri et al. 2007; Panzeri and Treves 1996; Strong et al. 1998), these methods are still of limited value when dealing with long response spaces…”
MethodTemporal structure of mouse courtship vocalizations facilitates syllable labeling
Hertz, S., Weiner, B., Perets, N., & London, M. (2020) · Communications Biology · doi:10.1038/s42003-020-1053-7 · Cited by 28
Multibody local approximation: Application to conformational entropy calculations on biomolecules
Suárez, E., & Suárez, D. (2012) · The Journal of Chemical Physics · doi:10.1063/1.4748104 · Cited by 27
Information divergence estimation based on data-dependent partitions
Silva, J. F., & Narayanan, S. S. (2010) · Journal of Statistical Planning and Inference · doi:10.1016/j.jspi.2010.04.011 · Cited by 27
“An excellent review can be found in Beirlant et al. (1997) and some recent contributions in Darbellay and Vajda (1999) and Paninski (2003, 2008).”
BackgroundRelating Granger causality to directed information theory for networks of stochastic processesPreprint
Amblard, P., & Michel, O. (2009) · arXiv (Cornell University) · doi:10.48550/arxiv.0911.2873 · Cited by 27
Gene regulatory network inference from single-cell data using multivariate information measuresPreprint
Chan, T. E., Stumpf, M. P. H., & Babtie, A. C. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/082099 · Cited by 28
“…a number of methods have been developed either for estimating the probability distribution from a set of frequencies, such as the Dirichlet estimator and the shrinkage estimator, or for estimating the entropy directly, as in the Miller-Madow estimator (Hausser & Strimmer 2009, Paninski 2003).”
MethodDetecting Subverted Cryptographic Protocols by Entropy Checking
Olivain, J., & Goubault-Larrecq, J. (2006) · HAL · Cited by 27
Trends of news diffusion in social media based on crowd phenomena
Kim, M., Newth, D., & Christen, P. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2567948.2579325 · Cited by 27
“Extensive research on information-theoretic measures has been conducted in diverse areas including computer science [4, 27, 28, 29], neuroscience [21, 22, 24], and economics [14, 18, 19].”
Background
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.