The DOI matches its DataCite record; 39 of the 39 references it cites were found in a registry or an index; none of the 11 sources its claims cite is checked for support yet.
A Theory of Usable Information Under Computational Constraints
Open accessAbstract
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive $\mathcal{V}$-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, $\mathcal{V}$-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, $\mathcal{V}$-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive $\mathcal{V}$-information is more effective than mutual information for structure learning and fair representation learning.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes predictive V-information, a variational extension of Shannon information that incorporates the observer’s computational constraints and can be created through computation, violating the data processing inequality. It provides PAC-style estimation guarantees and shows empirical advantages over mutual information for structure learning and fair representation learning.
Key findings
- Predictive V-information generalizes Shannon mutual information when the predictive family is unrestricted, and recovers other notions such as mean absolute deviation, trace of covariance, and the coefficient of determination \(R^2\) under specific restricted families.
- Unlike Shannon mutual information, V-information can increase under preprocessing/computation, so the data processing inequality does not hold; this is framed as consistent with representation learning and decryption examples.
- V-information can be estimated with PAC-style guarantees: Theorem 1 bounds estimation error by \(4R_{|D|}(G_V) + 2B\sqrt{\frac{2\log(1/\delta)}{|D|}}\), and Corollary 1.1 gives a concrete linear-Gaussian bound.
- In experiments, V-information-based Chow-Liu tree learning recovers ground-truth trees with about 100 samples in simulation-A, outperforms CPC/NWJ/MINE alternatives across data regimes, and outperforms non-parametric Shannon estimators on DREAM5 gene regulatory network inference using 20 genes and 660 datapoints.
- On Moving-MNIST, V-information decreases with frame distance and Algorithm 1 perfectly recovers relative frame order; on Deterministic-Moving-MNIST it recovers order when frame distances are less than 9.
- In fairness experiments on Adult, German, and Heritage datasets, features trained to minimize V-information against one attacker family often leak to another; in one special case, a \(V_B\) attacker achieves AUC 0.857 despite \(V_A\)-information minimization.
Methods
- The paper defines a predictive family \(V\) of computationally bounded predictive models, then defines predictive conditional V-entropy and predictive V-information as \(I_V(X \to Y)=H_V(Y|\emptyset)-H_V(Y|X)\).
- It proves elementary properties (monotonicity, non-negativity, independence), PAC bounds for empirical V-information using Rademacher complexity, and a finite-sample guarantee for directed Chow-Liu tree structure learning solved via the Chu-Liu algorithm.
- Experiments cover synthetic high-dimensional tree structures with 7–20 variables each 10-dimensional, DREAM5 gene regulatory network inference, Moving-MNIST and Deterministic-Moving-MNIST frame ordering, and fair representation learning on Adult, German, and Heritage datasets.
Limitations
- The paper states in Appendix F that general V-information lacks additive algebra such as \(H(X,Y)=H(X)+H(Y|X)\), though this may hold for simple \(V\) such as polynomial-time computable functions.
- The paper states future work is needed to better integrate V-information with representation learning, active learning, classification, and reinforcement learning.
- Evident from the text: the synthetic experiments use V-families misspecified with respect to the true conditional distributions; PixelCNN++ tends to overfit in the deterministic Moving-MNIST case; and fair representations trained with one V-information objective do not reliably generalize to other attacker families.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 No basis given · 11 Own interpretation · 10 Own result · 5 Check pending · 4 No full text of the sources · 1 Not checked yet
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Without constraints, predictive V-information specializes to classic mutual information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In practice, when the observer employs deep neural networks as a prediction scheme, V-information outperforms methods that approximate Shannon information in various applications, including Chow-Liu tree contruction in high dimension and gene regulatory network inference.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination.”
- Also stated in: § 1 INTRODUCTION · plain
“Without constraints, predictive V-information specializes to classic mutual information.Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees.”
- Also stated in: § 1 INTRODUCTION · plain
“As an additional benefit, we show that predictive V-information can be estimated with statistical guarantees using the Probably Approximately Correct framework (Valiant, 1984).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
What the sources say
- Not checked — no full text of the sourcecites Valiant, 1984 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“I V (X → Y ) -ÎV (X → Y ; D) ≤ M 4|D| 1 + 4 2 log 1 δ”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
What the sources say
- Check pendingcites Kakade et al. (2008)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“2. Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, if the function class does not satisfy optional ignorance, then the V-information could be negative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 5
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The leaned features by V A -information minimization appear more evenly spread as expected, however, the attacker functions in V B can still achieve a high AUC of 0.857.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can see that although the two V-families used are misspecified with respect to the true underlying (conditional) distributions, the estimated Chow-Liu trees are much more accurate across all data regimes, with CPC (blue) being the best alternative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if the V is the largest possible predictive family that includes all possible models, i.e.V = Ω, then Definition 2 reduces to Shannon entropy: H Ω (Y |X) = H(Y |X) and H V (Y |∅) = H Ω (Y ) = H(Y ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“1. Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let V = {f : {∅} → Q t,θ , θ ∈ Θ}, where Q t,θ is a distribution in a minimal exponential family with sufficient statistics t : Y → R d and set of natural parameters Θ.For a random variable Y with expected sufficient statistics µ Y = E[t(Y )], the V-entropy of Y is the maximum Shannon entropy over all random variables Ŷ with identical expected sufficient statistics, i.e.E[t( Ŷ )] = µ Y .”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On three datasets commonly used in the fairness literature (Adult, German, Heritage), previous methods work well at preventing information”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let Y = R d and V = {f : {∅} → N (µ, Σ) | µ ∈ R d , Σ = 1/2I d×d }, then the V-entropy of a random variable Y equals the trace of its covariance tr (Cov(Y )), up to an additive constant.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, V-information(Gaussian) works consistently well in all cases and only requires about 100 samples to recover the ground-truth Chow-Liu tree in simulation-A.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The diagonal elements in the matrix are usually the smallest in rows, indicating that the attacker function family V i extracts more information on featured trained with V j(j =i) -information minimization.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation.”
- Also stated in: § 1 INTRODUCTION · plain
“A consequence of this new formulation is that computation can”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Shannon information can be manipulated with certain additive algebra (e.g.H(X, Y ) = H(X) + H(Y | X)), while the same does not hold true for general V-Information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“V-information is empirically useful, has several intuitive theoretical properties, but exhibits certain limitations.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, V-information can be provably estimated, and can thus be more effective for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 8 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Chow & Wagner (1973) show that the Chow-Liu algorithm is consistent, i.e, it recovers the true solution as the dataset size goes to infinity.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
What the sources say
- Not checked — no full text of the sourcecites Chow & Wagner, 1973 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Certain functions h are one-way: there exists a polynomial algorithm to compute h(x) but no polynomial algorithm to compute h^{-1}(y).
“modern cryptography is founded on the assumption that certain functions h : X → Y are one-way, meaning that there exists an polynomial algorithm to compute h(x) but no polynomial algorithm to compute h -1 (y).”
The authors state this as an assumption.
The random variable Y ∈ R^d and the predictive family V is the Gaussian family V = {f : x → N(φ(x), Σ), x ∈ X; ∅ → N(µ, Σ) | µ ∈ R^d; Σ = 1/2 I_{d×d}; φ ∈ Φ}.
“Assume random variable Y ∈ R d , V = {f : x → N (φ(x), Σ), x ∈ X ; ∅ → N (µ, Σ)|µ ∈ R d ; Σ = 1 2 I d×d ; φ ∈ Φ}.”
The authors state this as an assumption.
For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
“Assume ∀f ∈ V, x ∈ X , y ∈ Y, log f [x](y) ∈ [-B, B].”
The authors state this as an assumption.
For all f ∈ V, y ∈ Y, log f[∅](y) ∈ [-B, B].
“Assume ∀f ∈ V, y ∈ Y, log f [∅](y) ∈ [-B, B].”
The authors state this as an assumption.
X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
“Assume X = {x ∈ R dx , x 2 ≤ k x } and Y = {y ∈ R dy , y 2 ≤ k y }. If V = {f : f [x] = N (W x + b, I), f [∅] = N (c, I), W ∈ R dy×dx , b, c ∈ R dy , (W, b) 2 ≤ 1}”
The authors state this as an assumption.
The function f_θ achieves the optimum value for I_NWJ.
“Assume that f θ achieves the optimum value for I NWJ .”
The authors state this as an assumption.
U is discrete and V belongs to predictive family 1.
“Assume U is discrete and V belongs to preditive family 1.”
The authors state this as an assumption.
The discussion is restricted to distributions with probability density functions or probability mass functions with respect to the underlying measure, and ∅ ∈ X.
“Regularity Conditions: To minimize technical overhead we restrict out discussion only to distributions with probability density functions (PDF) or probability mass functions (PMF) with respect to the underlying measure.Also ∅ ∈ X .”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
“min φ {I V (u; z)} = min φ sup f ∈V E z,u∼q φ (z,u) [log P f (z|u)]”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
“features trained with I Vi (Z → U ) minimization”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A.5 PROOF OF THEOREM 2
“Let C D (g * ) be the estimated sum of edge weights on dataset D of the tree g * , i.e., C(g * ) = m i=2 ÎV t(g * )(i),i (X t(g)(i) → X i ; D).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
“Denote C(g) = m i=2 I V t(g)(i),i (X t(g)(i) → X i ),”
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
A Mathematical Theory of Communication
Shannon, C. E. (1948) · The MIT Press eBooks · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 50,398
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 37,139
(2008) · Journal of Machine Learning Research · Cited by 33,051
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
CAUSALITY: MODELS, REASONING AND INFERENCE
Hitchcock, C., & Pearl, J. (2001) · The Philosophical Review · doi:10.2307/3182612 · Cited by 4,827
Kraskov, A., Stögbauer, H., & Grassberger, P. (2004) · Physical Review E · doi:10.1103/physreve.69.066138 · Cited by 3,512
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981851 · Cited by 3,070
Ledoux, M., & Talagrand, M. (1991) · Springer eBooks · doi:10.1007/978-3-642-20212-4 · Cited by 2,532
Using mutual information for selecting features in supervised neural net learning
Battiti, R. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.298224 · Cited by 2,148
Rademacher and Gaussian Complexities: Risk Bounds and Structural Results
Bartlett, P. L., & Mendelson, S. (2001) · Lecture notes in computer science · doi:10.1007/3-540-44581-1_15 · Cited by 1,989
Approximating discrete probability distributions with dependence trees
Chow, C., & Liu, C. (1968) · IEEE Transactions on Information Theory · doi:10.1109/tit.1968.1054142 · Cited by 1,720
Wisdom of crowds for robust gene network inference
The DREAM5 Consortium, Marbach, D., Costello, J. C., Küffner, R., Vega, N. M., Prill, R. J., Camacho, D. M., Allison, K. R., Kellis, M., Collins, J. J., et al. (2012) · Nature Methods · doi:10.1038/nmeth.2016 · Cited by 1,667
On the rationale of maximum-entropy methods
Jaynes, E. T. (1982) · Proceedings of the IEEE · doi:10.1109/proc.1982.12425 · Cited by 1,297
Mutual Information Neural Estimation.
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A. C., & Hjelm, D. (2018) · International Conference on Machine Learning · Cited by 920
Elements of Causal Inference: Foundations and Learning Algorithms
Peters, J., Janzing, D., & Schölkopf, B. (2017) · OAPEN (The OAPEN Foundation) · Cited by 789
Black Box Variational InferencePreprint
Ranganath, R., Gerrish, S., & Blei, D. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0118 · Cited by 735
Salimans, T., Karpathy, A., Chen, X., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1701.05517 · Cited by 584
Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
Grunwald, P., & Dawid, A. P. (2004) · The Annals of Statistics · doi:10.1214/009053604000000553 · Cited by 575
Estimation of the information by an adaptive partitioning of the observation space
Darbellay, G. A., & Vajda, I. (1999) · IEEE Transactions on Information Theory · doi:10.1109/18.761290 · Cited by 499
Censoring Representations with an AdversaryPreprint
Edwards, H. S., & Storkey, A. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05897 · Cited by 488
On Variational Bounds of Mutual Information
Poole, B., Ozair, S., Oord, A. V. D., Alemi, A. A., & Tucker, G. (2019) · arXiv · doi:10.48550/arxiv.1905.06922 · Cited by 480
Learning Adversarially Fair and Transferable RepresentationsPreprint
Madras, D., Creager, E., Pitassi, T., & Zemel, R. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06309 · Cited by 479
Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk Minimization
Nguyen, X., Wainwright, M. J., & Jordan, M. I. (2010) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2068870 · Cited by 379
The Variational Fair Autoencoder
Louizos, C., Swersky, K., Li, Y., Welling, M., & Zemel, R. S. (2016) · Data Archiving and Networked Services (DANS) · Cited by 311
Information-Theoretic Inference of Large Transcriptional Regulatory Networks
Meyer, P. E., Kontos, K., Lafitte, F., & Bontempi, G. (2007) · EURASIP Journal on Bioinformatics and Systems Biology · doi:10.1155/2007/79879 · Cited by 273
Uncertainty, Information, and Sequential Experiments
DeGroot, M. H. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704567 · Cited by 268
Understanding the Limitations of Variational Mutual Information EstimatorsPreprint
Song, J., & Ermon, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.06222 · Cited by 147
Self-citationStatistical mechanics based on Renyi entropy
Lenzi, E. K., Mendes, R. S., & da Silva, L. R. (2000) · Physica A Statistical Mechanics and its Applications · doi:10.1016/s0378-4371(00)00007-8 · Cited by 128
Estimating Mutual Information for Discrete-Continuous MixturesPreprint
Gao, W., Kannan, S., Oh, S., & Viswanath, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.06212 · Cited by 113
Learning Controllable Fair RepresentationsPreprint
Song, J., Kalluri, P., Grover, A., Zhao, S., & Ermon, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.04218 · Cited by 106
Self-citationInformation Theoretical Estimators ToolboxPreprint
Szabó, Z. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.2106 · Cited by 99
Dropout Rademacher complexity of deep neural networks
Gao, W., & Zhou, Z. (2016) · Science China Information Sciences · doi:10.1007/s11432-015-5470-z · Cited by 66
Consistency of an estimate of tree-dependent probability distributions (Corresp.)
Chow, C., & Wagner, T. (1973) · IEEE Transactions on Information Theory · doi:10.1109/tit.1973.1055013 · Cited by 33
Undersmoothed Kernel Entropy Estimators
Paninski, L., & Yajima, M. (2008) · IEEE Transactions on Information Theory · doi:10.1109/tit.2008.928251 · Cited by 30
Multiclass classification, information, divergence and surrogate risk
Duchi, J. C., Khosravi, K., & Ruan, F. (2018) · The Annals of Statistics · doi:10.1214/17-aos1657 · Cited by 26
Valiant, L. G. (2026) · Foundations of Computation and Machine Learning · doi:10.1145/3749740.3749746
(2008) · ScholarlyCommons (University of Pennsylvania)
Cited by
Sorted by citations
On Mutual Information Maximization for Representation LearningPreprint
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lučić, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13625 · Cited by 437
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 283
Acquisition of chess knowledge in AlphaZero
McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U., & Kramnik, V. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2206625119 · Cited by 166
To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review
Shwartz-Ziv, R., & LeCun, Y. (2024) · Entropy · doi:10.3390/e26030252 · Cited by 140
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120
Metadata
- Authors
- Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S.
- Affiliations
- Peking University (CN)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.10689
- OpenAlex
- W2996336810
- Cited by
- 142
- References
- 39
- Citations by year
- 2026: 28 · 2025: 22 · 2024: 21 · 2023: 30 · 2022: 8 · 2021: 20 · 2020: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1651565, 1522054, 1733686)
- Office of Naval Research
- Air Force Office of Scientific Research (FA9550-19-1-0024)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Diverse Interdisciplinary Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- Shannon mutual information, computational resource constraints, data processing inequality, representation learning, feature hierarchy, structure learning, fair representation learning, high-dimensional estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
