The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Information Theory and Statistics: A Tutorial
Abstract
This tutorial is concerned with applications of information theory concepts in statistics, in the finite alphabet setting. The information measure known as information divergence or Kullback-Leibler distance or relative entropy plays a key role, often with a geometric flavor as an analogue of squared Euclidean distance, as in the concepts of I-projection, I-radius and I-centroid. The topics covered include large deviations, hypothesis testing, maximum likelihood estimation in exponential families, analysis of contingency tables, and iterative algorithms with an “information geometry” background. Also, an introduction is provided to the theory of universal coding, and to statistical inference via the minimum description length principle motivated by that theory.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Estimating the Dimension of a Model
Schwarz, G. (1978) · The Annals of Statistics · doi:10.1214/aos/1176344136 · Cited by 47,033
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Modeling by shortest data description
Rissanen, J. (1978) · Automatica · doi:10.1016/0005-1098(78)90005-5 · Cited by 6,011
A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations
Chernoff, H. (1952) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729330 · Cited by 3,550
Cox, D. R., & Hinkley, D. V. (1974) · doi:10.1007/978-1-4899-2887-0 · Cited by 3,089
Differential-Geometrical Methods in Statistics
Amari, S.-I. (1985) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-5056-2 · Cited by 1,673
$I$-Divergence Geometry of Probability Distributions and Minimization Problems
Csiszar, I. (1975) · The Annals of Probability · doi:10.1214/aop/1176996454 · Cited by 1,645
Generalized Iterative Scaling for Log-Linear Models
Darroch, J. N., & Ratcliff, D. (1972) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177692379 · Cited by 1,206
A General Class of Coefficients of Divergence of One Distribution from Another
Ali, S. M., & Silvey, S. D. (1966) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1966.tb00626.x · Cited by 1,205
The context-tree weighting method: basic properties
Willems, F. M. J., Shtarkov, Y. M., & Tjalkens, T. J. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382012 · Cited by 919
Universal coding, information, prediction, and estimation
Rissanen, J. (1984) · IEEE Transactions on Information Theory · doi:10.1109/tit.1984.1056936 · Cited by 898
Csiszar, I. (1991) · The Annals of Statistics · doi:10.1214/aos/1176348385 · Cited by 790
Rissanen, J., & Langdon, G. G. (1979) · IBM Journal of Research and Development · doi:10.1147/rd.232.0149 · Cited by 751
The performance of universal encoding
Krichevsky, R., & Trofimov, V. (1981) · IEEE Transactions on Information Theory · doi:10.1109/tit.1981.1056331 · Cited by 535
Generalized Kraft Inequality and Arithmetic Coding
Rissanen, J. J. (1976) · IBM Journal of Research and Development · doi:10.1147/rd.203.0198 · Cited by 510
Davisson, L. (1973) · IEEE Transactions on Information Theory · doi:10.1109/tit.1973.1055092 · Cited by 365
Information projections revisited
Csiszar, I., & Matus, F. (2003) · IEEE Transactions on Information Theory · doi:10.1109/tit.2003.810633 · Cited by 169
An algorithm for maximizing expected log investment return
Cover, T. M. (1984) · IEEE Transactions on Information Theory · doi:10.1109/tit.1984.1056869 · Cited by 148
A source matching approach to finding minimax codes
Davisson, L., & Leon‐Garcia, A. (1980) · IEEE Transactions on Information Theory · doi:10.1109/tit.1980.1056167 · Cited by 97
A Geometric Interpretation of Darroch and Ratcliff's Generalized Iterative Scaling
Csiszár, I. (1989) · The Annals of Statistics · doi:10.1214/aos/1176347279 · Cited by 81
Self-citation
Cited by
A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
Gui, J., Sun, Z., Wen, Y., Tao, D., & Ye, J. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3130191 · Cited by 1,231
Decision trees for uplift modeling with single and multiple treatments
Rzepakowski, P., & Jaroszewicz, S. (2012) · Knowledge and Information Systems · doi:10.1007/s10115-011-0434-0 · Cited by 181
Efficient compression in color naming and its evolution
Zaslavsky, N., Kemp, C., Regier, T., & Tishby, N. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1800521115 · Cited by 178
From classification to quantification in tweet sentiment analysis
Gao, W., & Sebastiani, F. (2016) · Social Network Analysis and Mining · doi:10.1007/s13278-016-0327-z · Cited by 75
Metadata
- Authors
- Csiszár, I., & Shields, P. C.
- Venue
- Foundations and Trends™ in Communications and Information Theory
- Publisher
- Emerald
- Published
- 2004
- Volume
- 1
- Issue
- 4
- Pages
- 417-528
- ISSN
- 1567-2190 (ISSN-L)
- 1567-2328
- OpenAlex
- W4293404332
- Cited by
- 420
- References
- 22
- FWCI
- 2.62
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
