Estimation of Entropy and Mutual Information
The DOI matches its Crossref record; 62 of the 63 references it cites were found in a registry or an index.
Abstract
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods. Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The information-theoretic neuroscience literature has long recognized the importance of correcting for the limited-sampling bias and has provided effective tools for correcting the Shannon information sampling bias.24,25,26,27,28”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“Earlier applications of information theory to neuroscience have recognized that Shannon information measures from neural activity suffer from a prominent limited-sampling bias.24,25,26”
From Sampling bias corrections for discrete and Gaussian partial information decompositions (2026) · cites this paper as [25]
In the citing paper
“It is also well known that such histogram approach leads to a biased estimate of the entropy [58].”
From Maximizing the information content of experiments in systems biology (2013) · cites this paper as [58]
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 48,949
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
Numerical recipes in Pascal: the art of scientific computing
(1990) · Choice Reviews Online · doi:10.5860/choice.27-5158 · Cited by 11,811
The Mathematical Theory of Communication
Shannon, C. E., Weaver, W., & Wiener, N. (1950) · Physics Today · doi:10.1063/1.3067010 · Cited by 9,801
Loève, M. (1963) · Andalas University Repository (Andalas University) · Cited by 5,938
Approximation Theorems of Mathematical Statistics
Serfling, R. J. (1980) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316481 · Cited by 5,851
Weak Convergence and Empirical Processes
van der Vaart, A. W., & Wellner, J. A. (1996) · Springer Series in Statistics · doi:10.1007/978-1-4757-2545-2 · Cited by 5,014 · cited 1×
“This asymptotically deterministic behavior of the sorted histograms is perhaps surprising, given that there is no such corresponding deterministic behavior for the unsorted histograms (although, by the Glivenko-Cantelli theorem, van der Vaart & Wellner, 1996, there is well-known deterministic behavior for the integral…”
BackgroundInfluentialA Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550 · cited 2×
“One such slice through parameter space is given by the “central lines” of the m-simplex: these are the subsets formed by linearly interpolating between the trivial (minimal entropy) and flat (maximal entropy) distributions (there are m of these lines, by symmetry).”
Background
Research works printed but not linked: 1
- Beirlant J., 1997, International Journal of the Mathematical Statistics Sciences, 6, 17
Cited by
Items 401–500 of 1,004
Sundaram, P., Luessi, M., Bianciardi, M., Stufflebeam, S. M., Hämäläinen, M., & Solo, V. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2962517 · Cited by 27
“MI originated in communication engineering [33] and has been widely used in imaging applications for, e.g., in image registration [34], and also in computational neuroscience [35].”
MethodHigh-throughput phenotyping with temporal sequences
Estiri, H., Strasser, Z. H., & Murphy, S. N. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa288 · Cited by 26
Reinhardt, M., Tkačik, G., & ten Wolde, P. R. (2023) · Physical Review X · doi:10.1103/physrevx.13.041017 · Cited by 37
“Moreover, any nonzero bin size leads to a systematic bias in the entropy estimates, even in one dimension [17].”
BackgroundHigh Stimulus-Related Information in Barrel Cortex Inhibitory Interneurons
Reyes‐Puerta, V., Kim, S., Sun, J., Imbrosci, B., Kilb, W., & Luhmann, H. J. (2015) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004121 · Cited by 26
“Other methods, such as the best upper bound [80], the Chao-Shen [81] and the Nemenman-Shafee-Bialek [82] methods yielded inaccurate (i.”
MethodExponential Concentration for Mutual Information Estimation with Application to Forests
Liu, H., Wasserman, L., & Lafferty, J. (2012) · Cited by 26
Temporal pairwise spike correlations fully capture single-neuron information
Dettner, A., Münzberg, S., & Tchumatchenko, T. (2016) · Nature Communications · doi:10.1038/ncomms13805 · Cited by 34
“Second, providing exact solutions for all basis projections was often computationally demanding or required strong approximations, for example, Gaussian assumptions, and offered only a lower bound estimate 3,10,12 .”
InfluentialFinite-Sample Concentration of the Multinomial in Relative Entropy
Agrawal, R. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.2996134 · Cited by 39
“Most notably, Paninski [7] showed by comparison to the χ2-statistic that E[Vn,k,P ] ≤ log 1 + k−1n ≤ k−1n .”
BackgroundInfluentialVariational Representations and Neural Network Estimation of Rényi Divergences
Birrell, J., Dupuis, P. G., Katsoulakis, M. A., Rey-Bellet, L., & Wang, J. (2021) · SIAM Journal on Mathematics of Data Science · doi:10.1137/20m1368926 · Cited by 26
“eck method [5], independence testing [6], as well as in the analysis and design of generativeadversarial networks (GANs) [7, 8, 9, 10, 11]. Estimation of divergences is known to be a difficult problem [12, 13]. Likelihood ratio methods such as those in [14] are known to work best in low dimensions. However, recent work…”
MethodInformation transmission with spiking Bayesian neurons
Lochmann, T., & Denève, S. (2008) · New Journal of Physics · doi:10.1088/1367-2630/10/5/055019 · Cited by 26
“Unfortunately, estimating the mutual information using formula (7) gives a systematically biased information estimate if the sample is too small to cover the space well [29, 30].”
BackgroundKlekota, J., Brauner, E., Roth, F. P., & Schreiber, S. L. (2006) · Journal of Chemical Information and Modeling · doi:10.1021/ci050495h · Cited by 27
Probability Estimation in the Rare-Events Regime
Wagner, A. B., Viswanath, P., & Kulkarni, S. R. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2137210 · Cited by 27
“In particular, we assume that the source that generates the length-n string has an alphabet of size Θ( n ) with each symbol having probability Θ(1 /n ) .”
BackgroundZheng, H., Jiang, Q., & Yan, X. (2019) · Journal of Chemometrics · doi:10.1002/cem.3110 · Cited by 25
“4,25 First, the proposed method divides all measured process variables into quality ‐ relevant and quality ‐ independent sub ‐ blocks in accordance with their MI values with quality variables.”
MethodCross-modal image retrieval with deep mutual information maximization
Gu, C., Bu, J., Zhou, X., Yao, C., Ma, D., Yu, Z., & Yan, X. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.078 · Cited by 30
Rényi entropy, abundance distribution, and the equivalence of ensembles
Mora, T., & Walczak, A. M. (2016) · Physical review. E · doi:10.1103/physreve.93.052418 · Cited by 26
Exploring Multiobjective Optimization for Multiview Clustering
Saha, S., Mitra, S., & Krämer, S. (2018) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3182181 · Cited by 28
“In order to measure the quality of the obtained results, some cluster quality measures like the Minkowski Score (Ben-Hur and Guyon 2003), the Adjusted Rand Index (Hubert and Arabie 1985) and Mutual Information (Paninski 2003) are used.”
MethodBaravalle, R., Rosso, O. A., & Montani, F. (2018) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5025187 · Cited by 27
Chong, L. C., & Khan, A. M. (2019) · BMC Genomics · doi:10.1186/s12864-019-6311-z · Cited by 28
“Sequence count in the alignment affects the entropy calculation due to the inverse relationship between sample size and alignment bias [48].”
BackgroundEstimation of the entropy based on its polynomial representation
Vinck, M. A., Battaglia, F. P., Balakirsky, V. B., Vinck, A. J. H., & Pennartz, C. M. A. (2012) · Physical Review E · doi:10.1103/physreve.85.051139 · Cited by 25
Information theoretic approaches to functional neuroimaging
Ostwald, D., & Bagshaw, A. P. (2011) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2011.07.013 · Cited by 30
“Information theoretic approaches to neural data analysis have been pioneered in the application to invasive electrophysiological recordings [5] and can be considered an established signal processing methodology in electrophysiology [6].”
BackgroundAn information-theoretic approach to network monitoring and measurement
Liu, Y., Towsley, D., Ye, T., & Bolot, J. C. (2005) · doi:10.5555/1251086.1251100 · Cited by 25
Testing Closeness With Unequal Sized SamplesPreprint
Bhattacharya, B. B., & Valiant, G. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.04599 · Cited by 36
“[4, 6, 7, 9, 12, 19, 20, 26, 27, 28]), and results on developing techniques and tools for establishing lower bounds (e.”
BackgroundApparent remote synchronization of amplitudes: A demodulation and interference effect
Minati, L., Faes, L., Frasca, M., Oświȩcimka, P., & Drożdż, S. (2018) · Chaos An Interdisciplinary Journal of Nonlinear Science · doi:10.1063/1.5026980 · Cited by 25
Quantifying information of intracellular signaling: progress with machine learning
Tang, Y., & Hoffmann, A. (2022) · Reports on Progress in Physics · doi:10.1088/1361-6633/ac7a4a · Cited by 27
Learning Disentangled Textual Representations via Statistical Measures of Similarity
Colombo, P., Staerman, G., Noiry, N., & Piantanida, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.187 · Cited by 26
A systematic literature review on sentiment analysis techniques, challenges, and future trends
Ali, H. M. U., Farooq, Q., Imran, A., & Hindi, K. M. E. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02365-x · Cited by 29
When do microcircuits produce beyond-pairwise correlations?
Barreiro, A. K., Gjorgjieva, J., Rieke, F. M., & Shea‐Brown, E. (2014) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2014.00010 · Cited by 29
“Cauchy (blue solid), Gaussian (red dashed), and bimodal (green dash-dotted) stimuli are shown.”
BackgroundNonparametric Estimation of Küllback-Leibler Divergence
Zhang, Z., & Grabchak, M. (2014) · Neural Computation · doi:10.1162/neco_a_00646 · Cited by 24
“We now discuss the bias of D̂∗n. Miller (1951) (see also Paninski, 2003, for a more formal treatment) shows that E[Ân] − A ≥ 0 and E[Ân] − A = O(1/n).”
BackgroundTemporal Adaptation Enhances Efficient Contrast Gain Control on Natural Images
Sinz, F. H., & Bethge, M. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1002889 · Cited by 26
“For the marginal entropies H ½ Y i (cid:2) , we use a jackknifed estimator for the discrete entropy on the binned values [39].”
MethodSeok, J., & Kang, Y. S. (2015) · Scientific Reports · doi:10.1038/srep10981 · Cited by 26
A multivariate extension of mutual information for growing neural networks
Ball, K., Grant, C., Mundy, W. R., & Shafer, T. J. (2017) · Neural Networks · doi:10.1016/j.neunet.2017.07.009 · Cited by 26
“Entropy of a discrete random variable X that takes M possible values depends on the probability mass function p(X) as: h(X) = M∑ i=1 −p(xi)log2p(xi), (6) where the log base 2 is taken to return information theoretic units of entropy as bits (Paninski, 2003; Rényi et al., 1961; Shannon, 1949).”
BackgroundHe, L., O’Sullivan, J. A., Politte, D. V., Powell, K. E., & Arvidson, R. E. (2019) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2019.2900644 · Cited by 24
“According to Paninski [36], the bias error for d̂KL is”
BackgroundMethodInfluentialEfficient feature selection using shrinkage estimators
Sechidis, K., Azzimonti, L., Pocock, A., Corani, G., Weatherall, J., & Brown, G. (2019) · Machine Learning · doi:10.1007/s10994-019-05795-1 · Cited by 27
Finger movements recognition using minimally redundant features of wavelet denoised EMG
Phukan, N., Kakoty, N. M., Shivam, P., & Gan, J. Q. (2019) · Health and Technology · doi:10.1007/s12553-019-00338-z · Cited by 25
A mutual information‐based metric for evaluation of fMRI data‐processing approaches
Afshin‐Pour, B., Hamid Soltanian-zadeh, A., Hossein‐Zadeh, G., Grady, C. L., & Strother, S. C. (2010) · Human Brain Mapping · doi:10.1002/hbm.21057 · Cited by 24
“There are many approaches in the literature for the estimation of MI that are applicable to our problem [Blinnikov and Moessner, 1998; Cellucci et al., 2005; Darbellay and Vajda, 1999; Daub et al., 2004; Khan et al., 2007; Moon et al., 1995; Paninski, 2003].”
MethodBayesian entropy estimation for binary spike train data using parametric prior knowledge
Archer, E., Park, I. M., & Pillow, J. W. (2013) · Cited by 24
A comprehensive comparison of association estimators for gene network inference algorithms
Kurt, Z., Aydın, N., & Altay, G. (2014) · Bioinformatics · doi:10.1093/bioinformatics/btu182 · Cited by 25
“The purpose of this study is not to investigate the GNI algorithms, but it is to examine the association estimators that are the most important process of the GNI algorithms.”
MethodMutual information scaling for tensor network machine learning
Convy, I., Huggins, W. J., Liao, H., & Whaley, K. B. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac44a9 · Cited by 31
Mutual Information Regularized Feature-level Frankenstein for Discriminative Recognition
Liu, X., Yang, C., You, J., Kuo, C. ‐. J., & Vijayakumar, B. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3077397 · Cited by 26
Depth-first random forests with improved Grassberger entropy for small object detection
Ma, J., Pan, Q., & Guo, Y. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105138 · Cited by 24
Estimation of the Rate–Distortion Function
Harrison, M. T., & Kontoyiannis, I. (2008) · IEEE Transactions on Information Theory · doi:10.1109/tit.2008.926387 · Cited by 26
“In many theoretical and practical problems arising in a wide variety of scientific contexts, it is desirable – and often important – to obtain accurate estimates of the entropy H(P ) of the source, based on the observed data x1 ; see, for example, [35] [26] [30] [29] [32] [31] [8] and the references therein.”
BackgroundLower bounds on the redundancy of natural images
Hosseini, R., Sinz, F. H., & Bethge, M. (2010) · Vision Research · doi:10.1016/j.visres.2010.07.025 · Cited by 25
“To this end we use a histogram based jackknifed maximum likelihood estimator (see e.g. Paninski, 2003).”
BackgroundMethodVariational Minimax Estimation of Discrete Distributions under KL Loss
Paninski, L. (2004) · neural information processing systems · Cited by 23
Self-citationMapping HLA-A2, -A3 and -B7 supertype-restricted T-cell epitopes in the ebolavirus proteome
Lim, W. C., & Khan, A. M. (2018) · BMC Genomics · doi:10.1186/s12864-017-4328-8 · Cited by 23
“The entropy bias is inversely proportional to the sample size N [44], where as N increases to ∞, bias reduces, approaching zero.”
BackgroundDetecting pathway biomarkers of diabetic progression with differential entropy
Liu, Z., & Gao, R. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.05.006 · Cited by 24
Unsupervised Algorithm for Brain Anomalies Localization in Electromagnetic Imaging
Brankovic, A., Zamani, A., Trakic, A., Bialkowski, K. S., Mohammed, B. J., Cook, D., Walsham, J., & Abbosh, A. M. (2020) · IEEE Transactions on Computational Imaging · doi:10.1109/tci.2020.3041922 · Cited by 23
“knowledge of the joint probability distribution of two considered signals on the corresponding ranges of space, which can be very difficult given the limited amount of data [28].”
BackgroundA comparison study of image spatial entropy
Razlighi, Q. R., & Kehtarnavaz, N. (2008) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.814439 · Cited by 25
“There exist many papers on how to estimate the amount of this underestimation or bias [15, 16, 17, 18, 19, 20, 21, 22].”
BackgroundSimilarity of interspike interval distributions and information gain in a stationary neuronal firing
Kostal, L., & Lánský, P. (2005) · Biological Cybernetics · doi:10.1007/s00422-005-0036-6 · Cited by 23
“Therefore there is a need for the quantification of the differences among these firing patterns and the information they transfer (Bhumbra et al., 2004; Buracas and Albright, 1999; Nemenman, 2004; Paninski, 2003; Rieke et al., 1997; Stein, 1967; Strong et al., 1998; Zador, 1998).”
BackgroundFrom the Entropy to the Statistical Structure of Spike Trains
Gao, Y., Kontoyiannis, I., & Bienenstock, E. (2006) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2006.261864 · Cited by 23
Hierarchical Modeling of Local Image Features through L_p-Nested Symmetric Distributions
Bethge, M., Simoncelli, E. P., & Sinz, F. H. (2009) · neural information processing systems · Cited by 23
Analysis of viral diversity for vaccine target discovery
Khan, A. M., Hu, Y., Miotto, O., Thevasagayam, N. M., Sukumaran, R., Raman, H. S. A., Brusić, V., Tan, T. W., & August, J. T. (2017) · BMC Medical Genomics · doi:10.1186/s12920-017-0301-2 · Cited by 23
“Briefly, given the relationship that alignment size bias is proportional to 1/N where N is the alignment size [43], correction for size bias for a nonamer position x can be made through repeated random sampling of the sequences at the position, to create smaller alignments of varying size whose entropies are measured.”
BackgroundBias Correction With Jackknife, Bootstrap, and Taylor Series
Jiao, J., & Han, Y. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.2969439 · Cited by 30
“In fact, it was shown first in [20] that for f(p) = p ln(1/p), there exist universal constants C1 > 0, C2 > 0 such that for any n ≥ 1, sup p∈[0,1] |e1,n(p)| ≤ C1 n (8)”
BackgroundPredicting Declension Class from Form and Meaning
Williams, A., Pimentel, T., Blix, H., McCarthy, A. D., Chodroff, E., & Cotterell, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.597 · Cited by 29
Finding the Last Bits of Positional Information
McGough, L., Casademunt, H., Nikolić, M., Aridor, Z., Petkova, M. D., Gregor, T., & Bialek, W. (2024) · PRX Life · doi:10.1103/prxlife.2.013016 · Cited by 23
“This gives us confidence in the extrapolation N em → ∞ [26,30–33].”
BackgroundA frequency-resolved mutual information rate and its application to neural systems
Bernardi, D., & Lindner, B. (2014) · Journal of Neurophysiology · doi:10.1152/jn.00354.2014 · Cited by 26
Near-optimal learning of tree-structured distributions by Chow-Liu
Bhattacharyya, A., Gayen, S., Price, E., & Vinodchandran, N. V. (2021) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/3406325.3451066 · Cited by 37
An approach to quantifying the efficiency of a Bayesian filter
Nearing, G., Gupta, H. V., Crow, W. T., & Gong, W. (2013) · Water Resources Research · doi:10.1002/wrcr.20177 · Cited by 22
“The entropy and mutual information estimates in (6) and (7) are maximum likelihood estimators (MLEs) for discrete random variables, and the asymptotic bias of an MLE estimator of entropy (Bmle ) is bounded by –ln 1þ m 1 k Bmle 0, where m is the number of histogram bins; the upper bound at zero is tight when k=m >1 [Pa…”
BackgroundMotif Discovery in Tissue-Specific Regulatory Sequences Using Directed Information
Rao, A. U. K., Hero, A. O., States, D. J., & Engel, J. D. (2007) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1155/2007/13853 · Cited by 31
“Several such algorithms for adaptive density estimation have been proposed ([42],[27],[29]) and can find potential application in this procedure.”
Coding of Stimuli by Ampullary Afferents inGnathonemus petersii
Engelmann, J., Gertz, S., Goulet, J., Schuh, A., & von der Emde, G. (2010) · Journal of Neurophysiology · doi:10.1152/jn.00503.2009 · Cited by 26
Neural Entropic Estimation: A faster path to mutual information estimationPreprint
Chan, C., Al-Bashabsheh, A., Huang, H. P., Lim, M., Tam, D. S. H., & Zhao, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.12957 · Cited by 25
“Despite the limitation of the well-known histogram approach [20, 28], there are various other estimation methods, including different density estimations using a kernel [18] and the nearest-neighbor distance [15].”
Enterprise Social Link Recommendation
Zhang, J., Lv, Y., & Yu, P. S. (2015) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2806416.2806549 · Cited by 21
“To reward/penalize the importance of different common followees (and common followers), we propose to weight them by both an idf-like measure and the pointwise mutual information [24]: , where U is the set of all users in the network.”
BackgroundEstimating the variance of Shannon entropy
Ricci, L., Perinelli, A., & Castelluzzo, M. (2021) · Physical review. E · doi:10.1103/physreve.104.024220 · Cited by 21
Zhao, K., Yang, B., Lin, C., Rong, W., Villavicencio, A., & Cui, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.33 · Cited by 33
Evaluating Anti-Fingerprinting Privacy Enhancing Technologies
Datta, A. J., Lu, J., & Tschantz, M. C. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308558.3313703 · Cited by 21
Jo, Y., Genel, S., Wandelt, B. D., Somerville, R. S., Villaescusa-Navarro, F., Bryan, G. L., Anglés‐Alcázar, D., Foreman-Mackey, D. T., Nelson, D., & Kim, J.-H. (2023) · The Astrophysical Journal · doi:10.3847/1538-4357/aca8fe · Cited by 22
“However, the estimation of MI is challenging and only tractable for discrete variables or when probability distributions are known (Paninski 2003).”
BackgroundA Normal Law for the Plug-in Estimator of Entropy
Zhang, Z., & Zhang, X. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2011.2179702 · Cited by 21
“When is assumed to dynamically vary as the sample size increases, i.e., , Paninski [7] established a normal law for , stated as Lemma 1.”
BackgroundSignal enumeration in Gaussian and non-Gaussian noise using entropy estimation of eigenvalues
Asadi, H., & Seyfe, B. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2018.03.005 · Cited by 21
“In [30], Paninski showed that the estimation of probability spaces, which are needed for an entropy estimator, is an important challenge and could be a much difficult problem, and the solution to solve this problem is: nonparametric estimators .”
BackgroundWollstadt, P., Schmitt, S., & Wibral, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04187 · Cited by 29
“While in theory the criterion is zero for (conditionally) independent variables, in practice estimators may return non-zero results also for independent variables due to finite sample size (Paninski, 2003; Kraskov et al., 2004; Hlav´aˇckov´a-Schindler et al., 2007).”
BackgroundTowards an informational account of interpersonal coordination
Chidichimo, E., Luppi, A. I., Mediano, P. A. M., Leong, V., Dumas, G., Canales‐Johnson, A., & Bethlehem, R. A. I. (2025) · Nature reviews. Neuroscience · doi:10.1038/s41583-025-00989-0 · Cited by 24
“For example, multiple debiasing methods are available and continually emerging 141,177–180 .”
Correlating paleoclimate time series: Sources of uncertainty and potential pitfalls
Franke, J., & Donner, R. V. (2019) · Quaternary Science Reviews · doi:10.1016/j.quascirev.2019.03.017 · Cited by 22
“Some methods, like mutual information (Paninski, 2003), operate on the joint and marginal probability distributions directly.”
BackgroundMethodConfidence Decision Trees via Online and Active Learning for Streaming Data
Rosa, R. D., & Cesa‐Bianchi, N. (2017) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5440 · Cited by 24
A Correlation-Based Feature Selection Algorithm for Operating Data of Nuclear Power Plants
He, Y., Yu, H., Yu, R., Song, J. G., Lian, H., He, J., & Yuan, J. (2021) · Science and Technology of Nuclear Installations · doi:10.1155/2021/9994340 · Cited by 20
“Mutual information can be expressed by equations (1) and (2), given by [27]”
BackgroundInformation Flow in Computational Systems
Venkatesh, P., Dutta, S., & Grover, P. (2020) · IEEE Transactions on Information Theory · doi:10.1109/tit.2020.2987806 · Cited by 23
“might be essential for an understanding of quantification of M -information flow [82]–[85].”
BackgroundZamani, R., Moghaddam, M. P., & Haghifam, M. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3144463 · Cited by 22
“The entropy of X is defined ( H ( X ) ) using the three most popular estimators [28].”
MethodTorkkola, K. (2008) · Studies in fuzziness and soft computing · doi:10.1007/978-3-540-35488-8_7 · Cited by 20
Li, P., & Zhang, C. (2011) · Cited by 20
Sample complexity of the distinct elements problem
Wu, Y., & Yang, P. (2018) · Mathematical Statistics and Learning · doi:10.4171/msl/1-1-2 · Cited by 20
“Furthermore, the fingerprints fˆj gj 1, which are the histogram of the histograms, are also sufficient for estimating any permutation-invariant distributional property [35,43], in particular, the number of colors.”
BackgroundDivisive normalization: Justification and effectiveness as efficient coding transform
Lyu, S. (2010) · Cited by 28
Splitting with confidence in decision trees with application to stream mining
Rosa, R. D., & Cesa‐Bianchi, N. (2015) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2015.7280392 · Cited by 25
Linaro, D., Levy, M. J., & Hunt, D. L. (2022) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010071 · Cited by 21
Estimation of entropy rate and Rényi entropy rate for Markov chains
Kamath, S., & Verdú, S. (2016) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2016.7541386 · Cited by 27
“Efforts to reduce this bias should generally improve performance [21].”
BackgroundGafar, M., Elhoseny, M., & Gunasekaran, M. (2018) · The Journal of Supercomputing · doi:10.1007/s11227-018-2512-5 · Cited by 19
INSPECTRE: Privately Estimating the Unseen
Acharya, J., Kamath, G., Sun, Z., & Zhang, H. (2020) · Journal of Privacy and Confidentiality · doi:10.29012/jpc.724 · Cited by 20
Conditional Mutual Information Neural Estimator
Molavipour, S., Bassi, G., & Skoglund, M. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053422 · Cited by 28
“, [17, 18]), the capacity in many communication setups (such as the relay channel, channels with random state, wiretap channels) is described by the conditional mutual information (CMI), which requires specialized estimators.”
BackgroundHierarchical Interaction Structure of Neural Activities in Cortical Slice Cultures
Santos, G. S., Gireesh, E. D., Plenz, D., & Nakahara, H. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.6141-09.2010 · Cited by 24
“This was because directly evaluating Fpair with a larger number of elements is contaminated by a strong bias that is imposed by limited sampling in the estimation of the divergence between empirical and model distributions (Nemenman et al., 2002; Paninski, 2003).”
BackgroundMethodInfluentialMutual information against correlations in binary communication channels
Pręgowska, A., Szczepański, J., & Wajnryb, E. (2015) · BMC Neuroscience · doi:10.1186/s12868-015-0168-0 · Cited by 19
“The mutual information ( MI ) is defined as where H ( X ) is the entropy of X , H ( Y ) is the entropy of Y , H ( X , Y ) is the joint entropy of X and Y , and H ( X | Y ) is the conditional entropy [4,17,21,27-29].”
BackgroundBias Adjustment for a Nonparametric Entropy Estimator
Zhang, Z., & Grabchak, M. (2013) · Entropy · doi:10.3390/e15061999 · Cited by 19
“Many important references can be found in [4] and [5].”
BackgroundEstimating Neuronal Information: Logarithmic Binning of Neuronal Inter-Spike Intervals
Dorval, A. D. (2011) · Entropy · doi:10.3390/e13020485 · Cited by 19
“And yet, despite theoretical advances [14,15,22], and the recent dissemination of practical recipes [23–25] and computational tools [26–29], information theory remains a niche subfield within neuroscience.”
BackgroundSamee, N. M. A., Solouma, N. H., & Kadah, Y. M. (2012) · Theoretical Biology and Medical Modelling · doi:10.1186/1742-4682-9-34 · Cited by 20
“The formula of calculating the Mutual information [11,12] is given as,”
MethodLearning and Generalization with the Information Bottleneck
Shamir, O., Sabato, S., & Tishby, N. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-87987-9_12 · Cited by 24
Spatial Sensor Selection via Gaussian Markov Random Fields
Nguyen, L. V., Kodagoda, S., & Ranasinghe, R. (2015) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2015.2503382 · Cited by 19
Macro-level information transfer in social media: Reflections of crowd phenomena
Kim, M., Newth, D., & Christen, P. (2015) · Neurocomputing · doi:10.1016/j.neucom.2014.12.107 · Cited by 22
Analyzing bin-width effect on the computed entropy
Purwani, S., Nahar, J., & Twining, C. (2017) · AIP conference proceedings · doi:10.1063/1.4995123 · Cited by 19
“According to Paninski and Carlton [11, 12], the entropy estimated from the sample using this probability estimate is always an underestimate on average.”
BackgroundEstimating Differential Entropy using Recursive Copula Splitting
Ariel, G., & Louzoun, Y. (2020) · Entropy · doi:10.3390/e22020236 · Cited by 19
“Estimating entropy in higher dimensions is significantly more challenging [4, 26].”
BackgroundData-Robust Tight Lower Bounds to the Information Carried by Spike Times of a Neuronal Population
Pola, G., Petersen, R. S., Thiele, A., Young, M. P., & Panzeri, S. (2005) · Neural Computation · doi:10.1162/0899766054322955 · Cited by 19
“The extent to which these corrected estimates might be improved further by applying alternative methods to control the bias (Victor, 2002; Nemenman, Shafee, & Bialek, 2002; Paninski, 2003) will not be investigated here.”
BackgroundMethodResultInfluentialUSING DIRECTED INFORMATION TO BUILD BIOLOGICALLY RELEVANT INFLUENCE NETWORKS
Rao, A. U. K., Hero, A. O., States, D. J., & Engel, J. D. (2007) · IEEE Computational Systems Bioinformatics Conference (CSB) · doi:10.1142/9781860948732_0018 · Cited by 25
Information theory and adaptationPreprint
Nemenman, I. M. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1011.5466 · Cited by 19
“Some of the reasons for the relative lack of progress may be practical considerations that estimation of informations among nonlinearly related multidimensional variables [16, 53, 54] or extracting the predictive aspects of the information [55] from empirical data is hard, while for simple Gaussian signals and respons…”
BackgroundUnsupervised Discovery of Temporal Structure in Noisy Data with Dynamical Components AnalysisPreprint
Clark, D. G., Livezey, J. A., & Bouchard, K. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09944 · Cited by 29
“Directly estimating mutual information from multidimensional data with continuous support is possible, and popular nonparametric methods include those based on binning [17, 18], kernel density estimation [19] and k-nearest neighbor (kNN) statistics [20].”
MethodHe, J., Sun, C., & Wang, P. (2019) · Sensors · doi:10.3390/s19204382 · Cited by 21
“High sample variance based on AMA indicates the locations of transition [31,32] where the signal should be processed with smaller window size to avoid distortion.”
Effect of Microencapsulated Basil Extract on Cream Cheese Quality and Stability
Popescu, L., Cojocari, D., Lung, I., Kacsó, I., Ciorîţă, A., Ghendov‐Moșanu, A., Bălan, G., Pintea, A., & Sturza, R. (2023) · Molecules · doi:10.3390/molecules28083305 · Cited by 21
“The more pronounced the influence of storage time and concentrations of MBE addition on textural parameters, pH, and overall acceptability, the higher the bit value [74].”
BackgroundDiscriminative Learning of Receptive Fields from Responses to Non-Gaussian Stimulus Ensembles
Meyer, A. F., Diepenbrock, J.-P., Happel, M. F. K., Ohl, F. W., & Anemüller, J. (2014) · PLoS ONE · doi:10.1371/journal.pone.0093062 · Cited by 20
“In contrast, information-theoretic approaches use histogram-based estimation of probability distributions, which is prone to be biased in some data regimes even if the correct estimator is used [62].”
Metadata
- Authors
- Paninski, L.
- Affiliations
- New York University (US)
- Corresponding author
- Paninski, Liam
- ORCID
- Paninski, Liam
- Venue
- Neural Computation
- Publisher
- MIT Press
- Published
- 2003-06-01
- Volume
- 15
- Issue
- 6
- Pages
- 1191-1253
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2114771311
- Cited by
- 1,442
- References
- 63
- Citations by year
- 2026: 88 · 2025: 84 · 2024: 73 · 2023: 93 · 2022: 84 · 2021: 100 · 2020: 90 · 2019: 100 · 2018: 83 · 2017: 52 · 2016: 62 · 2015: 60 · 2014: 52 · 2013: 56 · 2012: 51
- FWCI
- 28.49
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Neural Computation (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Statistical Mechanics and Entropy
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- entropy estimation, mutual information estimation, consistency, central limit theorem, confidence intervals, polynomial approximation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.