The DOI matches its DataCite record; 39 of the 39 references it cites were found in a registry or an index; none of the 11 sources its claims cite is checked for support yet.
A Theory of Usable Information Under Computational Constraints
Open accessAbstract
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive $\mathcal{V}$-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, $\mathcal{V}$-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, $\mathcal{V}$-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive $\mathcal{V}$-information is more effective than mutual information for structure learning and fair representation learning.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes predictive V-information, a variational extension of Shannon information that incorporates the observer’s computational constraints and can be created through computation, violating the data processing inequality. It provides PAC-style estimation guarantees and shows empirical advantages over mutual information for structure learning and fair representation learning.
Key findings
- Predictive V-information generalizes Shannon mutual information when the predictive family is unrestricted, and recovers other notions such as mean absolute deviation, trace of covariance, and the coefficient of determination \(R^2\) under specific restricted families.
- Unlike Shannon mutual information, V-information can increase under preprocessing/computation, so the data processing inequality does not hold; this is framed as consistent with representation learning and decryption examples.
- V-information can be estimated with PAC-style guarantees: Theorem 1 bounds estimation error by \(4R_{|D|}(G_V) + 2B\sqrt{\frac{2\log(1/\delta)}{|D|}}\), and Corollary 1.1 gives a concrete linear-Gaussian bound.
- In experiments, V-information-based Chow-Liu tree learning recovers ground-truth trees with about 100 samples in simulation-A, outperforms CPC/NWJ/MINE alternatives across data regimes, and outperforms non-parametric Shannon estimators on DREAM5 gene regulatory network inference using 20 genes and 660 datapoints.
- On Moving-MNIST, V-information decreases with frame distance and Algorithm 1 perfectly recovers relative frame order; on Deterministic-Moving-MNIST it recovers order when frame distances are less than 9.
- In fairness experiments on Adult, German, and Heritage datasets, features trained to minimize V-information against one attacker family often leak to another; in one special case, a \(V_B\) attacker achieves AUC 0.857 despite \(V_A\)-information minimization.
Methods
- The paper defines a predictive family \(V\) of computationally bounded predictive models, then defines predictive conditional V-entropy and predictive V-information as \(I_V(X \to Y)=H_V(Y|\emptyset)-H_V(Y|X)\).
- It proves elementary properties (monotonicity, non-negativity, independence), PAC bounds for empirical V-information using Rademacher complexity, and a finite-sample guarantee for directed Chow-Liu tree structure learning solved via the Chu-Liu algorithm.
- Experiments cover synthetic high-dimensional tree structures with 7–20 variables each 10-dimensional, DREAM5 gene regulatory network inference, Moving-MNIST and Deterministic-Moving-MNIST frame ordering, and fair representation learning on Adult, German, and Heritage datasets.
Limitations
- The paper states in Appendix F that general V-information lacks additive algebra such as \(H(X,Y)=H(X)+H(Y|X)\), though this may hold for simple \(V\) such as polynomial-time computable functions.
- The paper states future work is needed to better integrate V-information with representation learning, active learning, classification, and reinforcement learning.
- Evident from the text: the synthetic experiments use V-families misspecified with respect to the true conditional distributions; PixelCNN++ tends to overfit in the deterministic Moving-MNIST case; and fair representations trained with one V-information objective do not reliably generalize to other attacker families.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 No basis given · 11 Own interpretation · 10 Own result · 5 Check pending · 4 No full text of the sources · 1 Not checked yet
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Without constraints, predictive V-information specializes to classic mutual information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In practice, when the observer employs deep neural networks as a prediction scheme, V-information outperforms methods that approximate Shannon information in various applications, including Chow-Liu tree contruction in high dimension and gene regulatory network inference.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination.”
- Also stated in: § 1 INTRODUCTION · plain
“Without constraints, predictive V-information specializes to classic mutual information.Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees.”
- Also stated in: § 1 INTRODUCTION · plain
“As an additional benefit, we show that predictive V-information can be estimated with statistical guarantees using the Probably Approximately Correct framework (Valiant, 1984).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
What the sources say
- Not checked — no full text of the sourcecites Valiant, 1984 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“I V (X → Y ) -ÎV (X → Y ; D) ≤ M 4|D| 1 + 4 2 log 1 δ”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
What the sources say
- Check pendingcites Kakade et al. (2008)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“2. Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, if the function class does not satisfy optional ignorance, then the V-information could be negative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 5
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The leaned features by V A -information minimization appear more evenly spread as expected, however, the attacker functions in V B can still achieve a high AUC of 0.857.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can see that although the two V-families used are misspecified with respect to the true underlying (conditional) distributions, the estimated Chow-Liu trees are much more accurate across all data regimes, with CPC (blue) being the best alternative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if the V is the largest possible predictive family that includes all possible models, i.e.V = Ω, then Definition 2 reduces to Shannon entropy: H Ω (Y |X) = H(Y |X) and H V (Y |∅) = H Ω (Y ) = H(Y ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“1. Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let V = {f : {∅} → Q t,θ , θ ∈ Θ}, where Q t,θ is a distribution in a minimal exponential family with sufficient statistics t : Y → R d and set of natural parameters Θ.For a random variable Y with expected sufficient statistics µ Y = E[t(Y )], the V-entropy of Y is the maximum Shannon entropy over all random variables Ŷ with identical expected sufficient statistics, i.e.E[t( Ŷ )] = µ Y .”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On three datasets commonly used in the fairness literature (Adult, German, Heritage), previous methods work well at preventing information”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let Y = R d and V = {f : {∅} → N (µ, Σ) | µ ∈ R d , Σ = 1/2I d×d }, then the V-entropy of a random variable Y equals the trace of its covariance tr (Cov(Y )), up to an additive constant.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, V-information(Gaussian) works consistently well in all cases and only requires about 100 samples to recover the ground-truth Chow-Liu tree in simulation-A.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The diagonal elements in the matrix are usually the smallest in rows, indicating that the attacker function family V i extracts more information on featured trained with V j(j =i) -information minimization.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation.”
- Also stated in: § 1 INTRODUCTION · plain
“A consequence of this new formulation is that computation can”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Shannon information can be manipulated with certain additive algebra (e.g.H(X, Y ) = H(X) + H(Y | X)), while the same does not hold true for general V-Information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“V-information is empirically useful, has several intuitive theoretical properties, but exhibits certain limitations.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, V-information can be provably estimated, and can thus be more effective for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 8 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Chow & Wagner (1973) show that the Chow-Liu algorithm is consistent, i.e, it recovers the true solution as the dataset size goes to infinity.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
What the sources say
- Not checked — no full text of the sourcecites Chow & Wagner, 1973 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Certain functions h are one-way: there exists a polynomial algorithm to compute h(x) but no polynomial algorithm to compute h^{-1}(y).
“modern cryptography is founded on the assumption that certain functions h : X → Y are one-way, meaning that there exists an polynomial algorithm to compute h(x) but no polynomial algorithm to compute h -1 (y).”
The authors state this as an assumption.
The random variable Y ∈ R^d and the predictive family V is the Gaussian family V = {f : x → N(φ(x), Σ), x ∈ X; ∅ → N(µ, Σ) | µ ∈ R^d; Σ = 1/2 I_{d×d}; φ ∈ Φ}.
“Assume random variable Y ∈ R d , V = {f : x → N (φ(x), Σ), x ∈ X ; ∅ → N (µ, Σ)|µ ∈ R d ; Σ = 1 2 I d×d ; φ ∈ Φ}.”
The authors state this as an assumption.
For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
“Assume ∀f ∈ V, x ∈ X , y ∈ Y, log f [x](y) ∈ [-B, B].”
The authors state this as an assumption.
For all f ∈ V, y ∈ Y, log f[∅](y) ∈ [-B, B].
“Assume ∀f ∈ V, y ∈ Y, log f [∅](y) ∈ [-B, B].”
The authors state this as an assumption.
X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
“Assume X = {x ∈ R dx , x 2 ≤ k x } and Y = {y ∈ R dy , y 2 ≤ k y }. If V = {f : f [x] = N (W x + b, I), f [∅] = N (c, I), W ∈ R dy×dx , b, c ∈ R dy , (W, b) 2 ≤ 1}”
The authors state this as an assumption.
The function f_θ achieves the optimum value for I_NWJ.
“Assume that f θ achieves the optimum value for I NWJ .”
The authors state this as an assumption.
U is discrete and V belongs to predictive family 1.
“Assume U is discrete and V belongs to preditive family 1.”
The authors state this as an assumption.
The discussion is restricted to distributions with probability density functions or probability mass functions with respect to the underlying measure, and ∅ ∈ X.
“Regularity Conditions: To minimize technical overhead we restrict out discussion only to distributions with probability density functions (PDF) or probability mass functions (PMF) with respect to the underlying measure.Also ∅ ∈ X .”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
“min φ {I V (u; z)} = min φ sup f ∈V E z,u∼q φ (z,u) [log P f (z|u)]”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
“features trained with I Vi (Z → U ) minimization”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A.5 PROOF OF THEOREM 2
“Let C D (g * ) be the estimated sum of edge weights on dataset D of the tree g * , i.e., C(g * ) = m i=2 ÎV t(g * )(i),i (X t(g)(i) → X i ; D).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
“Denote C(g) = m i=2 I V t(g)(i),i (X t(g)(i) → X i ),”
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
A Mathematical Theory of Communication
Shannon, C. E. (1948) · The MIT Press eBooks · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 50,398
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 37,139
(2008) · Journal of Machine Learning Research · Cited by 33,051
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
CAUSALITY: MODELS, REASONING AND INFERENCE
Hitchcock, C., & Pearl, J. (2001) · The Philosophical Review · doi:10.2307/3182612 · Cited by 4,827
Kraskov, A., Stögbauer, H., & Grassberger, P. (2004) · Physical Review E · doi:10.1103/physreve.69.066138 · Cited by 3,512
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981851 · Cited by 3,070
Ledoux, M., & Talagrand, M. (1991) · Springer eBooks · doi:10.1007/978-3-642-20212-4 · Cited by 2,532
Cited by
Sorted by citations
Items 101–143 of 143
Cook, R. A., Lalor, J. P., & Abbasi, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.129 · Cited by 1
Usable Information and Evolution of Optimal Representations During TrainingPreprint
Kleinman, M., Achille, A., Idnani, D., & Kao, J. C. (2021) · arXiv (Cornell University) · Cited by 1
Quantifying and Localizing Private Information Leakage from Neural Network GradientsPreprint
Mo, F., Borovykh, A., Malekzadeh, M., Haddadi, H., & Demetriou, S. (2021) · arXiv (Cornell University) · Cited by 1
Geometric Median (GM) Matching for Robust Data PruningPreprint
Acharya, A., Dhillon, I. S., & Sanghavi, S. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.17188 · Cited by 1
OneDSE: Metric-Conditioned Inverse Modeling and Active Search for Sample-Efficient DSEPreprint
Raj, R., Ramachandran, A., Samuel, D., Nye, J., Nemawarkar, S. S., & Krishna, T. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.03771 · Cited by 1
A Representation-Learning Game for Classes of Prediction TasksPreprint
Uzan, N., & Weinberger, N. (2024) · arXiv (Cornell University) · doi:10.3929/ethz-b-000664555 · Cited by 1
Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMsPreprint
Pramanik, V., Maliha, M., Bastian, N. D., & Jha, S. K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.13258 · Cited by 1
On the Monitorability of Natural Language Autoencoders: Identification vs DiscriminationPreprint
Denamganaï, K. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.18727 · Cited by 1
To Fix the Image in Memory: Adaptive Analog Coding in Emerging Memory Systems
Zarcone, R. V. (2020) · eScholarship (California Digital Library)
Backpropagation-free learning with an information surrogate
Ma, W.-D. (2022) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.21331005
Improving Variational Methods for Representation Learning and Generative Modelling of Speech
Braithwaite, D. T. (2023) · Open Access Victoria University of Wellington | Te Herenga Waka (Figshare) · doi:10.26686/wgtn.23462480
Syntax and Geometry of Information
Bailly, R., Leblond, L., & Gábor, K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.590
Kleinman, M., Wang, T., Xiao, D., Feghhi, E., Lee, E., Carr, N., Li, Y., Hadidi, N., Chandrasekaran, C., & Kao, J. C. (2023) · eLife · doi:10.7554/elife.89369
A State-Vector Framework for Dataset Effects
Sahak, E., Zhu, Z., & Rudzicz, F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.942
Data Checklist: On Unit-Testing Datasets with Usable InformationPreprint
Zhang, H. C., Behzad, S., Ethayarajh, K., & Jurafsky, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2408.02919
Investigating usable information for assessing the impact of medical image processing
Lu, C., Sengupta, S., Li, H., & Anastasio, M. A. (2025) · SPIE Medical Imaging: Image Perception, Observer Performance, and Technology Assessment · doi:10.1117/12.3047516
Observer-Usable Information as a Task-Specific Image Quality Metric
Lu, C., Sengupta, S., Li, H., & Anastasio, M. A. (2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2026.3680092
Laplace Sample Information: Data Informativeness Through a Bayesian LensPreprint
Kaiser, J., Schwethelm, K., Rueckert, D., & Kaissis, G. A. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2505.15303
Simple and Effective Specialized Representations for Fair Classifiers
Sinigaglia, A., Sartor, D., Ceccon, M., & Susto, G. A. (2025) · neural information processing systems · doi:10.52202/085713-0610
Βαθεία εκμάθηση για κωδικοποίηση σηματοδοτικών δικτύων
Alevizos, G. (2020) · DSpace - NTUA (National Technical University of Athens) · doi:10.26240/heal.ntua.20201
Analyzing Chain-of-thought Prompting in Black-Box Large Language Models via Estimated V-information
Wang, Z., Li, C., Yang, Z., Liu, Q., Hao, Y., Chen, X., Chu, D., & Sui, D. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/56y48yrxazk3
GRIP: Geometric Refinement and Adaptive Information Potential for Data EfficiencyPreprint
Wang, C., Yang, J., Yao, X., Yu, Y., Peng, J., Yu, L., Fang, J., Cantoro, R., Cui, Q., & Zhou, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.00031
Ambiguity and Invariance in Machine Listening
Perera, D. (2025) · theses.fr (ABES) · doi:10.70675/164092fdz865cz4fe7za2a9z2f88f6f38360
Parametric Prior Mapping Framework for Non-stationary Probabilistic Time Series ForecastingPreprint
Li, J., Tan, J., Fang, Q., & Gui, N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2605.23402
Principles and Practice of Deep Representation Learning: or a Mathematical Theory of MemoryPreprint
Buchanan, S., Pai, D., Wang, P., & Ma, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2606.06624
Pediatric Sepsis Cohort Detection Using In-Context Pointwise V-Usable Information
Li, Y., Geva, A., Bethard, S. J., Miller, T. A., Madden, K., Eisenberg, M. A., Kelly, D. P., & Savova, G. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/38bj4pwcnt3q
If Grid Cells are the Answer, What is the Question? A Review of Normative Grid Cell TheoryPreprint
Dorrell, W., & Whittington, J. C. R. (2026) · eLife · doi:10.7554/elife.111058
If Grid Cells are the Answer, What is the Question? A Review of Normative Grid Cell TheoryPreprint
Dorrell, W., & Whittington, J. C. R. (2026) · eLife · doi:10.7554/elife.111058.1
Label-free segmentation of mitochondria for simultaneous morphological and metabolic studies
Bae, K., Özbey, M., Ho, A., Aksamitiene, E., Tan, K. K. D., Sorrells, J. E., Iyer, R. R., Anastasio, M. A., & Boppart, S. A. (2026) · Communications Biology · doi:10.1038/s42003-026-10687-x
Trustworthy AI: Exploring Reliability, Transparency and Adaptability of Machine Learning Systems.
Sartor, D. (n.d.) · Padua Research Archive (University of Padova)
MARAGE: Multi-Model Adversarial Attack for Retrieval-Augmented Generation Database Extraction
Xiao, H. U. (2026) · TSpace (University of Toronto)
Comparing Corrupted Constrained Learning ProblemsPreprint
Iacovissi, L., Derr, R., & Williamson, R. C. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2608.25745
Andrews, E., & Mengaldo, G. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.11282
Understanding Generalization Requires Universal InductionPreprint
Ebtekar, A., Hütter, M., & Sutherland, D. J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.34458
The Layer Mystery of VLA: An Information-Theoretical Analysis of VLA Latent InterfacePreprint
Liu, Y., Yang, L., Xie, F., Ames, A. D., & Yue, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.36118
What Visual Generators Need from Teachers: Rethinking Representation AlignmentPreprint
Wang, Y., Chen, H., Fan, M., Jiang, S., Zhang, T., Sun, Y., Wang, Z., Lu, Y., & Shen, C. (2026) · arXiv (Cornell University)
Liang, W., Yue, L., Zhang, W. E., Guo, M., Maennel, O., & Chen, W. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.38956
It Is Not Seeing the Hazard: A Frozen Vision-Language Safety Score Measures Its Caption BankPreprint
Tetteh, S., & Fleming, C. H. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.09517
Santoso, K. P., Fadil, N. Z., Harsanti, F. P., Ginardi, R. V. H., & Iyer, G. N. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.07906
The Value of Memory in Language AgentsPreprint
Lin, Z., Zhu, E., Hu, H., Hao, X., Li, C., Chen, K., & Xiong, F. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2610.31303
Scaling Science: Toward a Chain Reaction of DiscoveryPreprint
Ding, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.00580
Mostajabdaveh, M., & Zhou, Z. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2611.05565
Metadata
- Authors
- Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S.
- Affiliations
- Peking University (CN)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.10689
- OpenAlex
- W2996336810
- Cited by
- 142
- References
- 39
- Citations by year
- 2026: 28 · 2025: 22 · 2024: 21 · 2023: 30 · 2022: 8 · 2021: 20 · 2020: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1651565, 1522054, 1733686)
- Office of Naval Research
- Air Force Office of Scientific Research (FA9550-19-1-0024)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Diverse Interdisciplinary Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- Shannon mutual information, computational resource constraints, data processing inequality, representation learning, feature hierarchy, structure learning, fair representation learning, high-dimensional estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
