The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Kullback-Leibler divergence estimation of continuous distributions
Abstract
We present a method for estimating the KL divergence between continuous densities and we prove it converges almost surely. Divergence estimation is typically solved estimating the densities first. Our main result shows this intermediate step is unnecessary and that the divergence can be either estimated using the empirical cdf or k-nearest-neighbour density estimation, which does not converge to the true measure for finite k. The convergence proof is based on describing the statistics of our estimator using waiting-times distributions, as the exponential or Erlang. We illustrate the proposed estimators and show how they compare to existing methods based on density estimation, and we also outline how our divergence estimators can be used for solving the two-sample problem.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Wu, Y. (1999) · Technometrics · doi:10.1080/00401706.1999.10485951 · Cited by 27,221
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Information theory, inference, and learning algorithms
(2004) · Choice Reviews Online · doi:10.5860/choice.41-5949 · Cited by 6,362
Information Theory, Inference, and Learning Algorithms
(2004) · Kybernetes · doi:10.1108/03684920410534506 · Cited by 4,937
Kraskov, A., Stögbauer, H., & Grassberger, P. (2004) · Physical Review E · doi:10.1103/physreve.69.066138 · Cited by 3,512
Probability and Random Processes
(2004) · Elsevier eBooks · doi:10.1016/b978-0-12-172651-5.x5000-3 · Cited by 2,535
A Kernel Method for the Two-Sample-Problem
Gretton, A., Borgwardt, K. M., Rasch, M., Schölkopf, B., & Smola, A. J. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0069 · Cited by 2,167
Probability and Random Processes.
Grimmett, G. R., & Stirzaker, D. R. (2001) · doi:10.1093/oso/9780198572237.001.0001 · Cited by 1,850
Estimation of entropy and mutual information
Paninski, L. (2003) · Neural Computation · doi:10.1162/089976603321780272 · Cited by 1,442
A family of algorithms for approximate bayesian inference
Minka, T. P., & Picard, R. W. (2001) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 882
(1997) · SZTAKI Publication Repository (Hungarian Academy of Sciences) · Cited by 598
A divisive information theoretic feature clustering algorithm for text classification
Dhillon, I. S., Mallela, S., & Kumar, R. (2003) · doi:10.5555/944919.944973 · Cited by 500
Estimation of the information by an adaptive partitioning of the observation space
Darbellay, G. A., & Vajda, I. (1999) · IEEE Transactions on Information Theory · doi:10.1109/18.761290 · Cited by 493
A Kullback-Leibler Divergence Based Kernel for SVM Classification in Multimedia Applications
Moreno, P. J., Ho, P., & Vasconcelos, N. M. (2003) · Cited by 380
The Exponential Distribution—Theory, Methods and Applications
Singh, A. K. (1997) · Technometrics · doi:10.1080/00401706.1997.10485138 · Cited by 312
A class of Rényi information estimators for multidimensional densities
Leonenko, N., Pronzato, L., & Savani, V. (2008) · The Annals of Statistics · doi:10.1214/07-aos539 · Cited by 240
Anderson, N. H., Hall, P., & Titterington, D. M. (1994) · Journal of Multivariate Analysis · doi:10.1006/jmva.1994.1033 · Cited by 205
Divergence Estimation of Continuous Distributions Based on Data-Dependent Partitions
Wang, Q., Kulkarni, S. R., & Verdu, S. (2005) · IEEE Transactions on Information Theory · doi:10.1109/tit.2005.853314 · Cited by 180
Consistency of data-driven histogram methods for density estimation and classification
Lugosi, G., & Nobel, A. (1996) · The Annals of Statistics · doi:10.1214/aos/1032894460 · Cited by 169
A Nearest-Neighbor Approach to Estimating Divergence between Continuous Random Vectors
Wang, Q., Kulkarni, S. R., & Verdú, S. (2006) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2006.261842 · Cited by 88
Nonparametric estimation of the likelihood ratio and divergence functionals
Nguyen, X., Wainwright, M. J., & Jordan, M. I. (2007) · IEEE International Symposium on Information Theory (ISIT) · doi:10.1109/isit.2007.4557517 · Cited by 27
Cited by
Considerate approaches to constructing summary statistics for ABC model selection
Barnes, C. P., Filippi, S., Stumpf, M. P. H., & Thorne, T. (2012) · Statistics and Computing · doi:10.1007/s11222-012-9335-7 · Cited by 61
Effective Molecular Dynamics from Neural Network-Based Structure Prediction Models
Jussupow, A., & Kaila, V. R. I. (2023) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.2c01027 · Cited by 42
Metadata
- Authors
- Perez-Cruz, F.
- Venue
- 2008 IEEE International Symposium on Information Theory
- Publisher
- IEEE
- Published
- 2008-07
- Pages
- 1666-1670
- OpenAlex
- W2001150023
- Cited by
- 400
- References
- 21
- FWCI
- 8.08
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.