The DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index.
Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
Open accessAbstract
We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to the other. Although Topsøe described this connection for the Shannon entropy over 20 years ago, it does not appear to be widely known even in that important special case. We here generalize this theory to apply to arbitrary decision problems and loss functions. We indicate how an appropriate generalized definition of entropy can be associated with such a problem, and we show that, subject to certain regularity conditions, the above-mentioned duality continues to apply in this extended context. This simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust Bayes acts. We also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions. This leads to an extension, to arbitrary discrepancies, of a well-known minimax theorem for the case of Kullback–Leibler divergence (the “redundancy-capacity theorem” of information theory). For the important case of families of distributions having certain mean values specified, we develop simple sufficient conditions and methods for identifying the desired solutions. We use this theory to introduce a new concept of “generalized exponential family” linked to the specific decision problem under consideration, and we demonstrate that this shares many of the properties of standard exponential families. Finally, we show that the existence of an equilibrium in our game can be rephrased in terms of a “Pythagorean property” of the related divergence, thus generalizing previously announced results for Kullback–Leibler and Bregman divergences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 36,808
Convergence of Probability Measures
Billingsley, P. (1999) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316962 · Cited by 13,910
Information theory and statistical mechanics
Jaynes, E. T. (1957) · Physical Review · doi:10.1103/physrev.106.620 · Cited by 12,931
Rockafellar, R. T. (1970) · Princeton University Press eBooks · doi:10.1515/9781400873173 · Cited by 10,801
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
Information theory and statistics
(1959) · Journal of the Franklin Institute · doi:10.1016/0016-0032(59)90368-0 · Cited by 7,094
On Measures of Entropy and Information
Rényi, A. (1961) · Project Euclid (Cornell University) · Cited by 4,031
Optimal Statistical Decisions.
Siegmund, D., & DeGroot, M. H. (1972) · Journal of the American Statistical Association · doi:10.2307/2284675 · Cited by 3,386
Stastical Decision Theory and Bayesian Analysis.
Ghosh, M., & Berger, J. (1988) · Journal of the American Statistical Association · doi:10.2307/2288950 · Cited by 2,905
Statistical Reasoning with Imprecise Probabilities
Walley, P. (1991) · Springer eBooks · doi:10.1007/978-1-4899-3472-7 · Cited by 2,655
Statistical Reasoning With Imprecise Probabilities
Wasserman, L., & Walley, P. (1993) · Journal of the American Statistical Association · doi:10.2307/2290362 · Cited by 2,649
Bregman, L. M. (1967) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(67)90040-7 · Cited by 2,618
Zur Theorie der Gesellschaftsspiele
v. Neumann, J. (1928) · Mathematische Annalen · doi:10.1007/bf01448847 · Cited by 2,572
Diversity and dissimilarity coefficients: A unified approach
Rao, C. R. (1982) · Theoretical Population Biology · doi:10.1016/0040-5809(82)90004-1 · Cited by 1,760
$I$-Divergence Geometry of Probability Distributions and Minimization Problems
Csiszar, I. (1975) · The Annals of Probability · doi:10.1214/aop/1176996454 · Cited by 1,645
On a measure of the information provided by an experiment
Lindley, D. V. (1956) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177728069 · Cited by 1,509
Mathematical Statistics A Decision Theoretic Approach
Rogers, G. S. (1968) · Technometrics · doi:10.1080/00401706.1968.10490555 · Cited by 1,488
Information and Exponential Families in Statistical Theory.
Kingman, J. F. C., & Barndorff-Nielsen, O. (1979) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2344658 · Cited by 1,173
Reference Posterior Distributions for Bayesian Inference
Bernardo, J. M. (1979) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1979.tb01066.x · Cited by 1,095
Good, I. J. (1952) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1952.tb00104.x · Cited by 976
Convexity and Optimization in Finite Dimensions I
Stoer, J., & Witzgall, C. (1970) · Springer eBooks · doi:10.1007/978-3-642-46216-0 · Cited by 885
Entropy optimization principles with applications
Kapur, J. N., & Kesavan, H. K. (1992) · Medical Entomology and Zoology · Cited by 858
Parallel optimization: Theory, algorithms, and applications
(1998) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(98)90713-1 · Cited by 813
Csiszar, I. (1991) · The Annals of Statistics · doi:10.1214/aos/1176348385 · Cited by 790
Information and information stability of random variables and processes
(1964) · Journal of the Franklin Institute · doi:10.1016/0016-0032(64)90319-9 · Cited by 758
Entropy Optimization Principles and Their Applications
Kapur, J. N., & Kesavan, H. K. (1992) · Water Science and Technology Library · doi:10.1007/978-94-011-2430-0_1 · Cited by 509
Information and Exponential Families in Statistical Theory
Mathai, A. M. (1980) · Technometrics · doi:10.1080/00401706.1980.10486149 · Cited by 498
Probability Theory, An Analytic View
Stroock, D. W. (2024) · Cambridge University Press eBooks · doi:10.1017/9781009549035 · Cited by 466
Information-theoretic asymptotics of Bayes methods
Clarke, B. S., & Barron, A. R. (1990) · IEEE Transactions on Information Theory · doi:10.1109/18.54897 · Cited by 431
Relative Loss Bounds for On-Line Density Estimation with the Exponential Family of Distributions
Azoury, K. S., & Warmuth, M. K. (2001) · Machine Learning · doi:10.1023/a:1010896012157 · Cited by 298
Jeffreys' prior is asymptotically least favorable under entropy risk
Clarke, B. S., & Barron, A. R. (1994) · Journal of Statistical Planning and Inference · doi:10.1016/0378-3758(94)90153-8 · Cited by 292
Uncertainty, Information, and Sequential Experiments
DeGroot, M. H. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704567 · Cited by 266
The Search for a Naturalistic World View
Shimony, A. (1993) · Cambridge University Press eBooks · doi:10.1017/cbo9780511621147 · Cited by 216
Can the maximum entropy principle be explained as a consistency requirement?
Uffink, J. (1995) · Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics · doi:10.1016/1355-2198(95)00015-1 · Cited by 195
Coherent dispersion criteria for optimal experimental design
Dawid, A. P., & Sebastiani, P. (1999) · The Annals of Statistics · doi:10.1214/aos/1018031101 · Cited by 162
Self-citationThe Search for a Naturalistic World View
Shimony, A. (1993) · Cambridge University Press eBooks · doi:10.1017/cbo9781139172196 · Cited by 153
A PROBLEM FOR RELATIVE INFORMATION MINIMIZERS IN PROBABILITY KINEMATICS
van Fraassen, B. C. (1981) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/32.4.375 · Cited by 153
Harremoës, P., & Topsøe, F. (2001) · Entropy · doi:10.3390/e3030191 · Cited by 143
Information-theoretical optimization techniques
Topsøe, F. (1979) · Czech digital mathematics library · Cited by 134
Seidenfeld, T. (1986) · Philosophy of Science · doi:10.1086/289336 · Cited by 130
Boosting as entropy projection
Kivinen, J., & Warmuth, M. K. (1999) · Conference on Learning Theory · doi:10.1145/307400.307424 · Cited by 113
A strong version of the redundancy-capacity theorem of universal coding
Merhav, N., & Feder, M. (1995) · IEEE Transactions on Information Theory · doi:10.1109/18.382017 · Cited by 101
A source matching approach to finding minimax codes
Davisson, L., & Leon‐Garcia, A. (1980) · IEEE Transactions on Information Theory · doi:10.1109/tit.1980.1056167 · Cited by 97
Jones, L. K., & Byrne, C. L. (1990) · IEEE Transactions on Information Theory · doi:10.1109/18.50370 · Cited by 96
Seidenfeld, T. (1987) · Springer eBooks · doi:10.1007/978-94-009-4788-7_23 · Cited by 92
The constraint rule of the maximum entropy principle
Uffink, J. (1996) · Studies in History and Philosophy of Science Part B Studies in History and Philosophy of Modern Physics · doi:10.1016/1355-2198(95)00022-4 · Cited by 88
The Minimum Description Length Principle and Reasoning under Uncertainty
Grünwald, P. (1998) · Cited by 81
Self-citationAdditive models, boosting, and inference for generalized divergences
Lafferty, J. (1999) · Conference on Learning Theory · doi:10.1145/307400.307422 · Cited by 78
Γ-Minimax: A Paradigm for Conservative Robust Bayesians
Vidaković, B. (2000) · Lecture notes in statistics · doi:10.1007/978-1-4612-1306-2_13 · Cited by 58
Maximum entropy inference as a special case of conditionalization
Skyrms, B. (1985) · Synthese · doi:10.1007/bf00485955 · Cited by 57
Basic Concepts, Identities and Inequalities - the Toolkit of Information Theory
Topsøe, F. (2001) · Entropy · doi:10.3390/e3030162 · Cited by 50
Jaynes, E. T. (1985) · Synthese · doi:10.1007/bf00485957 · Cited by 28
Maximum entropy versus minimum risk and applications to some classical discrete distributions
Topsøe, F. (2002) · IEEE Transactions on Information Theory · doi:10.1109/tit.2002.800479 · Cited by 24
Game theory, maximum generalized entropy, minimum discrepancy, robust Bayes and Pythagoras
Grünwald, P., & Dawid, A. P. (2003) · IEEE Information Theory Workshop (ITW) · doi:10.1109/itw.2002.1115425 · Cited by 17
Self-citationVerification of forecasts expressed in terms of probability
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078%3c0001:vofeit%3e2.0.co;2
VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY
BRIER, G. W. (1950) · Monthly Weather Review · doi:10.1175/1520-0493(1950)078<0001:vofeit>2.0.co;2
Cited by
Modeling of species distributions with Maxent: new extensions and a comprehensive evaluation
Phillips, S. J., & Dudík, M. (2008) · Ecography · doi:10.1111/j.0906-7590.2008.5203.x · Cited by 5,890
Strictly Proper Scoring Rules, Prediction, and Estimation
Gneiting, T., & Raftery, A. E. (2007) · Journal of the American Statistical Association · doi:10.1198/016214506000001437 · Cited by 5,793
Clustering with Bregman Divergences
Banerjee, A., Merugu, S., Dhillon, I., & Ghosh, J. (2004) · Proceedings of the 2004 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972740.22 · Cited by 1,472
A survey of inverse reinforcement learning: Challenges, methods and progress
Arora, S., & Doshi, P. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2021.103500 · Cited by 610
Frameworks and Results in Distributionally Robust Optimization
Rahimian, H., & Mehrotra, S. (2022) · Open Journal of Mathematical Optimization · doi:10.5802/ojmo.15 · Cited by 506
Metadata
- Authors
- Grünwald, P. D., & Dawid, A. P.
- Venue
- The Annals of Statistics
- Publisher
- Institute of Mathematical Statistics
- Published
- 2004-08-01
- Volume
- 32
- Issue
- 4
- Pages
- 1367-1433
- ISSN
- 0090-5364 (ISSN-L)
- OpenAlex
- W1590693676
- Cited by
- 454
- References
- 56
- FWCI
- 22.53
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.