The DOI matches its DataCite record; 39 of the 39 references it cites were found in a registry or an index; none of the 11 sources its claims cite is checked for support yet.
A Theory of Usable Information Under Computational Constraints
Open accessAbstract
We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive $\mathcal{V}$-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, $\mathcal{V}$-information can be created through computation. This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning. Additionally, we show that by incorporating computational constraints, $\mathcal{V}$-information can be reliably estimated from data even in high dimensions with PAC-style guarantees. Empirically, we demonstrate predictive $\mathcal{V}$-information is more effective than mutual information for structure learning and fair representation learning.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes predictive V-information, a variational extension of Shannon information that incorporates the observer’s computational constraints and can be created through computation, violating the data processing inequality. It provides PAC-style estimation guarantees and shows empirical advantages over mutual information for structure learning and fair representation learning.
Key findings
- Predictive V-information generalizes Shannon mutual information when the predictive family is unrestricted, and recovers other notions such as mean absolute deviation, trace of covariance, and the coefficient of determination \(R^2\) under specific restricted families.
- Unlike Shannon mutual information, V-information can increase under preprocessing/computation, so the data processing inequality does not hold; this is framed as consistent with representation learning and decryption examples.
- V-information can be estimated with PAC-style guarantees: Theorem 1 bounds estimation error by \(4R_{|D|}(G_V) + 2B\sqrt{\frac{2\log(1/\delta)}{|D|}}\), and Corollary 1.1 gives a concrete linear-Gaussian bound.
- In experiments, V-information-based Chow-Liu tree learning recovers ground-truth trees with about 100 samples in simulation-A, outperforms CPC/NWJ/MINE alternatives across data regimes, and outperforms non-parametric Shannon estimators on DREAM5 gene regulatory network inference using 20 genes and 660 datapoints.
- On Moving-MNIST, V-information decreases with frame distance and Algorithm 1 perfectly recovers relative frame order; on Deterministic-Moving-MNIST it recovers order when frame distances are less than 9.
- In fairness experiments on Adult, German, and Heritage datasets, features trained to minimize V-information against one attacker family often leak to another; in one special case, a \(V_B\) attacker achieves AUC 0.857 despite \(V_A\)-information minimization.
Methods
- The paper defines a predictive family \(V\) of computationally bounded predictive models, then defines predictive conditional V-entropy and predictive V-information as \(I_V(X \to Y)=H_V(Y|\emptyset)-H_V(Y|X)\).
- It proves elementary properties (monotonicity, non-negativity, independence), PAC bounds for empirical V-information using Rademacher complexity, and a finite-sample guarantee for directed Chow-Liu tree structure learning solved via the Chu-Liu algorithm.
- Experiments cover synthetic high-dimensional tree structures with 7–20 variables each 10-dimensional, DREAM5 gene regulatory network inference, Moving-MNIST and Deterministic-Moving-MNIST frame ordering, and fair representation learning on Adult, German, and Heritage datasets.
Limitations
- The paper states in Appendix F that general V-information lacks additive algebra such as \(H(X,Y)=H(X)+H(Y|X)\), though this may hold for simple \(V\) such as polynomial-time computable functions.
- The paper states future work is needed to better integrate V-information with representation learning, active learning, classification, and reinforcement learning.
- Evident from the text: the synthetic experiments use V-families misspecified with respect to the true conditional distributions; PixelCNN++ tends to overfit in the deterministic Moving-MNIST case; and fair representations trained with one V-information objective do not reliably generalize to other attacker families.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
26 No basis given · 11 Own interpretation · 10 Own result · 5 Check pending · 4 No full text of the sources · 1 Not checked yet
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Without constraints, predictive V-information specializes to classic mutual information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Empirically, we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In practice, when the observer employs deep neural networks as a prediction scheme, V-information outperforms methods that approximate Shannon information in various applications, including Chow-Liu tree contruction in high dimension and gene regulatory network inference.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination.”
- Also stated in: § 1 INTRODUCTION · plain
“Without constraints, predictive V-information specializes to classic mutual information.Under natural restrictions, V-information specializes to other well-known notions of predictiveness, such as the coefficient of determination (R 2 ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“Additionally, we show that by incorporating computational constraints, V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees.”
- Also stated in: § 1 INTRODUCTION · plain
“As an additional benefit, we show that predictive V-information can be estimated with statistical guarantees using the Probably Approximately Correct framework (Valiant, 1984).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
What the sources say
- Not checked — no full text of the sourcecites Valiant, 1984 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
In the paper
“I V (X → Y ) -ÎV (X → Y ; D) ≤ M 4|D| 1 + 4 2 log 1 δ”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
What the sources say
- Check pendingcites Kakade et al. (2008)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“2. Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Non-Negativity: I V (X → Y ) ≥ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, if the function class does not satisfy optional ignorance, then the V-information could be negative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A check of a cited source is scheduled, or we don’t hold its full text yet; the claim is checked once it runs.
In the paper
“Using a conditional PixelCNN++ (Salimans et al., 2017) as predictive family V, we shown in Figure 1c that predictive V-information does indeed decrease with frame distance, despite some fluctuations when the frame distances are large.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.3 RECOVERING THE ORDER OF VIDEO FRAMES
What the sources say
- Not checked yetcites Salimans et al., 2017 source
A check is scheduled.
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not checked yet
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Independence: If X is independent of Y , I V (X → Y ) = I V (Y → X) = 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“3. Independence: If X is independent of Y , I V (X → Y ) = I V (Y → X) = 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“C(ĝ) ≥ C(g * ) -2(m -1) max i,j 2R Di,j (G Vi,j ) + 2R Dj (G Vj ) + B 2 log 1 δ (|D j | -1 2 + |D i,j | -1 2 )(8)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A.5 PROOF OF THEOREM 2
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“C(ĝ) ≥ C(g * ) -2(m -1) max i,j 2R Di,j (G Vi,j ) + 2R Dj (G Vj ) + B 2 log 1 δ (|D j | -1 2 + |D i,j | -1 2 ) (8 )”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“I V (X → Y ) -ÎV (X → Y ; D) ≤ 4R |D| (G V ) + 2B 2 log 1 δ |D| (5”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“H V (Y |X) - 1 |D| xi,yi∈D -log f [x i ](y i ) ≤ 2R |D| (G V ) + 2B 2 log 1 δ |D|(16)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using Algorithm 1 to construct a Chow-Liu tree, we find that the tree perfectly recovers the relative order of the frames.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.3 RECOVERING THE ORDER OF VIDEO FRAMES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Proposition 4. Assume that f θ achieves the optimum value for I NWJ .Then the variance of the empirical NWJ estimator satisfies Var ÎNWJ ≥ e I(X;Y ) -1 N”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
What the sources say
- Check pendingcites Nguyen et al., 2010 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The function f_θ achieves the optimum value for I_NWJ.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let Y = R d , X be any vector space, and V = {f : x → N (φ(x), Σ), x ∈ X ; ∅ → N (µ, Σ)|µ ∈ R d ; Σ = 1/2I d×d , φ ∈ Φ}, where Φ is the set of linear functions {φ : X → R d }, then V-information I V (X → Y ) equals the (unnormalized) maximum coefficient of determination R 2 • tr (Cov(Y )) for linear regression.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A check of a cited source is scheduled, or we don’t hold its full text yet; the claim is checked once it runs.
In the paper
“Proposition 3. ∀f θ : X × Y → R + , I CPC ≤ log N.(43)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > B ANALYSIS OF APPROXIMATE ESTIMATORS FOR SHANNON INFORMATION
What the sources say
- Not checked yetcites van den Oord et al., 2018 source
A check is scheduled.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although larger C(g) does not necessarily lead to better Chow-Liu trees, empirically we find that the optimal tree in the sense of equation ( 7) is consistent with the optimal tree in equation ( 6) under commonly used V.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“H V (Y ) = inf f ∈V E y∼Y [-log f [∅](y)] = inf θ∈Θ E y∼Y [-log exp (θ • E y∼Y [t(y)] -A(θ))] = -sup θ∈Θ (θ • E y∼Y [t(y)] -A(θ))] = -A * (E y∼Y [t(y)])(12)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“H V (Y ) - 1 |D| yi∈D -log f [∅](y i ) ≤ 2R |D| (G V ∅ ) + B 2 log 1 δ |D| (29 ) ≤ 2R |D| (G V ) + B 2 log 1 δ |D|(30)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: For all f ∈ V, y ∈ Y, log f[∅](y) ∈ [-B, B].
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, once we constrain the observer to PixelCNN++ models, algorithm 1 with predictive V-information can still recover the order of different frames when the frame distances are relatively small (less than 9).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.3 RECOVERING THE ORDER OF VIDEO FRAMES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this case, minimizing I V (Z → U ) equals to minimize -H V (Y |X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: U is discrete and V belongs to predictive family 1.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Thus, if the function class does not satisfy optional ignorance, then the V-information does not take value 0 when the two distributions are independent.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Figure 1b, our method outperforms all other methods in network inference under different fractions of data used for estimation.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“I V (X → Y ) -ÎV (X → Y ; D) ≤ M 4|D| 1 + 4 2 log 1 δ”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Typically, the Rademacher complexity term satisfies R
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“= tr (Cov(Y )) R 2(13)”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: The random variable Y ∈ R^d and the predictive family V is the Gaussian family V = {f : x → N(φ(x), Σ), x ∈ X; ∅ → N(µ, Σ) | µ ∈ R^d; Σ = 1/2 I_{d×d}; φ ∈ Φ}.
In the paper
“This is in sharp contrast with Shannon information, which is well known to be difficult to estimate for high dimensional or continuous random variables (Battiti, 1994).”
- Also stated in: § 4 PAC GUARANTEES FOR V -INFORMATION ESTIMATION · boosted
“Shannon information is notoriously difficult to estimate for high dimensional random variables.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
What the sources say
- Not checked — no full text of the sourcecites Battiti, 1994 source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 PAC GUARANTEES FOR V -INFORMATION ESTIMATION · boosted
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Theoretically we show that the statistical guarantees of estimating V information translate to statistical guarantees for a variant of the Chow-Liu algorithm for structure learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 1 INTRODUCTION
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore H entropy still suffers from difficulty of estimation in high dimension because the definitions do not restrict to functions with small complexity (e.g.Rademacher complexity).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Alternative definitions of Information
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More formally, denoting t as the decryption algorithm and V as a class of natural language processing functions, we have that: I V (t(X) → Y ) > I V (X → Y ) ≈ 0.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.2 ON THE PRODUCTION OF INFORMATION THROUGH PREPROCESSING
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“Additionally, methods based on mutual information cannot take advantage of intrinsically asymmetric relationships, which are common for example in gene regulatory networks (Meyer et al., 2007).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
What the sources say
- Check pendingcites Meyer et al., 2007 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the finite sample behavior of the Chow-Liu algorithm for high dimensional problems is much less studied, due to the difficulty of estimating mutual information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Hence, standard Chow-Liu tree learning algorithm would fail to discover the natural ordering of the frames (causal structure).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.3 RECOVERING THE ORDER OF VIDEO FRAMES
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Under mild conditions (Wainwright et al., 2008) -A * (µ) = H(P µ ) where P µ is the maximum entropy distribution out of all distributions satisfying E y∼Pµ [t(y)] = µ (Jaynes, 1982), and H(•) is the Shannon entropy.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
What the sources say
- Not matchedcites Wainwright et al., 2008
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Jaynes, 1982 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The optional ignorance requirement in Eq.( 1) is a technical condition needed for these properties to hold.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“In particular, the Chow-Liu algorithm (Chow & Liu, 1968) can efficiently learn tree graphs (i.e.G is the set of trees).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
What the sources say
- Not checked — no full text of the sourcecites Chow & Liu, 1968 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This challenges the generalizability of fair representations in previous works.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means that if V contains all polynomial-time computable functions, then I V (X → h(X)) I V (h(X) → X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.3 ON THE ASYMMETRY OF PREDICTIVE V -INFORMATION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- The authors state this as an assumption: Certain functions h are one-way: there exists a polynomial algorithm to compute h(x) but no polynomial algorithm to compute h^{-1}(y).
- It may state two separate claims as one. 12
- Probably not a substantial claim. 5
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The leaned features by V A -information minimization appear more evenly spread as expected, however, the attacker functions in V B can still achieve a high AUC of 0.857.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can see that although the two V-families used are misspecified with respect to the true underlying (conditional) distributions, the estimated Chow-Liu trees are much more accurate across all data regimes, with CPC (blue) being the best alternative.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“if the V is the largest possible predictive family that includes all possible models, i.e.V = Ω, then Definition 2 reduces to Shannon entropy: H Ω (Y |X) = H(Y |X) and H V (Y |∅) = H Ω (Y ) = H(Y ).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“1. Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A PROOFS A.1 PROOF OF PROPOSITION 1
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Monotonicity: If V ⊆ U, then H V (Y ) ≥ H U (Y ), H V (Y | X) ≥ H U (Y | X).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 3.1 ELEMENTARY PROPERTIES
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let V = {f : {∅} → Q t,θ , θ ∈ Θ}, where Q t,θ is a distribution in a minimal exponential family with sufficient statistics t : Y → R d and set of natural parameters Θ.For a random variable Y with expected sufficient statistics µ Y = E[t(Y )], the V-entropy of Y is the maximum Shannon entropy over all random variables Ŷ with identical expected sufficient statistics, i.e.E[t( Ŷ )] = µ Y .”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On three datasets commonly used in the fairness literature (Adult, German, Heritage), previous methods work well at preventing information”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Let Y = R d and V = {f : {∅} → N (µ, Σ) | µ ∈ R d , Σ = 1/2I d×d }, then the V-entropy of a random variable Y equals the trace of its covariance tr (Cov(Y )), up to an additive constant.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 2 DEFINITIONS AND NOTATIONS
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Surprisingly, V-information(Gaussian) works consistently well in all cases and only requires about 100 samples to recover the ground-truth Chow-Liu tree in simulation-A.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.1 STRUCTURE LEARNING WITH CONTINUOUS HIGH-DIMENSIONAL DATA
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The diagonal elements in the matrix are usually the smallest in rows, indicating that the attacker function family V i extracts more information on featured trained with V j(j =i) -information minimization.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Additional information
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unlike Shannon's mutual information and in violation of the data processing inequality, V-information can be created through computation.”
- Also stated in: § 1 INTRODUCTION · plain
“A consequence of this new formulation is that computation can”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 1 INTRODUCTION · plain
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Shannon information can be manipulated with certain additive algebra (e.g.H(X, Y ) = H(X) + H(Y | X)), while the same does not hold true for general V-Information.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“V-information is empirically useful, has several intuitive theoretical properties, but exhibits certain limitations.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > F LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, V-information can be provably estimated, and can thus be more effective for structure learning and fair representation learning.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 8 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
It may state two separate claims as one.
In the paper
“Chow & Wagner (1973) show that the Chow-Liu algorithm is consistent, i.e, it recovers the true solution as the dataset size goes to infinity.”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
What the sources say
- Not checked — no full text of the sourcecites Chow & Wagner, 1973 source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Certain functions h are one-way: there exists a polynomial algorithm to compute h(x) but no polynomial algorithm to compute h^{-1}(y).
“modern cryptography is founded on the assumption that certain functions h : X → Y are one-way, meaning that there exists an polynomial algorithm to compute h(x) but no polynomial algorithm to compute h -1 (y).”
The authors state this as an assumption.
The random variable Y ∈ R^d and the predictive family V is the Gaussian family V = {f : x → N(φ(x), Σ), x ∈ X; ∅ → N(µ, Σ) | µ ∈ R^d; Σ = 1/2 I_{d×d}; φ ∈ Φ}.
“Assume random variable Y ∈ R d , V = {f : x → N (φ(x), Σ), x ∈ X ; ∅ → N (µ, Σ)|µ ∈ R d ; Σ = 1 2 I d×d ; φ ∈ Φ}.”
The authors state this as an assumption.
For all f ∈ V, x ∈ X, y ∈ Y, log f[x](y) ∈ [-B, B].
“Assume ∀f ∈ V, x ∈ X , y ∈ Y, log f [x](y) ∈ [-B, B].”
The authors state this as an assumption.
For all f ∈ V, y ∈ Y, log f[∅](y) ∈ [-B, B].
“Assume ∀f ∈ V, y ∈ Y, log f [∅](y) ∈ [-B, B].”
The authors state this as an assumption.
X = {x ∈ R^{dx}, ||x||_2 ≤ k_x}, Y = {y ∈ R^{dy}, ||y||_2 ≤ k_y}, and V = {f : f[x] = N(Wx + b, I), f[∅] = N(c, I), W ∈ R^{dy×dx}, b, c ∈ R^{dy}, ||(W, b)||_2 ≤ 1}.
“Assume X = {x ∈ R dx , x 2 ≤ k x } and Y = {y ∈ R dy , y 2 ≤ k y }. If V = {f : f [x] = N (W x + b, I), f [∅] = N (c, I), W ∈ R dy×dx , b, c ∈ R dy , (W, b) 2 ≤ 1}”
The authors state this as an assumption.
The function f_θ achieves the optimum value for I_NWJ.
“Assume that f θ achieves the optimum value for I NWJ .”
The authors state this as an assumption.
U is discrete and V belongs to predictive family 1.
“Assume U is discrete and V belongs to preditive family 1.”
The authors state this as an assumption.
The discussion is restricted to distributions with probability density functions or probability mass functions with respect to the underlying measure, and ∅ ∈ X.
“Regularity Conditions: To minimize technical overhead we restrict out discussion only to distributions with probability density functions (PDF) or probability mass functions (PMF) with respect to the underlying measure.Also ∅ ∈ X .”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > D.2 FAIRNESS
“min φ {I V (u; z)} = min φ sup f ∈V E z,u∼q φ (z,u) [log P f (z|u)]”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 6.4 INFORMATION THEORETIC APPROACHES TO FAIRNESS
“features trained with I Vi (Z → U ) minimization”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > A.5 PROOF OF THEOREM 2
“Let C D (g * ) be the estimated sum of edge weights on dataset D of the tree g * , i.e., C(g * ) = m i=2 ÎV t(g * )(i),i (X t(g)(i) → X i ; D).”
§ A THEORY OF USABLE INFORMATION UNDER COMPUTATIONAL CONSTRAINTS > 5 STRUCTURE LEARNING WITH V -INFORMATION
“Denote C(g) = m i=2 I V t(g)(i),i (X t(g)(i) → X i ),”
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
A Mathematical Theory of Communication
Shannon, C. E. (1948) · The MIT Press eBooks · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 50,398
Elements of Information Theory
Cover, T. M., & Thomas, J. A. (2001) · Wiley eBooks · doi:10.1002/0471200611 · Cited by 37,139
(2008) · Journal of Machine Learning Research · Cited by 33,051
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Representation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242
CAUSALITY: MODELS, REASONING AND INFERENCE
Hitchcock, C., & Pearl, J. (2001) · The Philosophical Review · doi:10.2307/3182612 · Cited by 4,827
Kraskov, A., Stögbauer, H., & Grassberger, P. (2004) · Physical Review E · doi:10.1103/physreve.69.066138 · Cited by 3,512
Graphical Models, Exponential Families, and Variational Inference
Wainwright, M. J., & Jordan, M. I. (2007) · now publishers, Inc. eBooks · doi:10.1561/9781601981851 · Cited by 3,070
Ledoux, M., & Talagrand, M. (1991) · Springer eBooks · doi:10.1007/978-3-642-20212-4 · Cited by 2,532
Cited by
Sorted by citations
On Mutual Information Maximization for Representation LearningPreprint
Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., & Lučić, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.13625 · Cited by 437
From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
Lauscher, A., Ravishankar, V., Vulić, I., & Glavaš, G. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.363 · Cited by 283
Acquisition of chess knowledge in AlphaZero
McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U., & Kramnik, V. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2206625119 · Cited by 166
To Compress or Not to Compress—Self-Supervised Learning and Information Theory: A Review
Shwartz-Ziv, R., & LeCun, Y. (2024) · Entropy · doi:10.3390/e26030252 · Cited by 140
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120
Metadata
- Authors
- Xu, Y., Zhao, S., Song, J., Stewart, R., & Ermon, S.
- Affiliations
- Peking University (CN)
- Stanford University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2002.10689
- OpenAlex
- W2996336810
- Cited by
- 142
- References
- 39
- Citations by year
- 2026: 28 · 2025: 22 · 2024: 21 · 2023: 30 · 2022: 8 · 2021: 20 · 2020: 12
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1651565, 1522054, 1733686)
- Office of Naval Research
- Air Force Office of Scientific Research (FA9550-19-1-0024)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computability, Logic, AI Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Diverse Interdisciplinary Research Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
- Keywords
- Shannon mutual information, computational resource constraints, data processing inequality, representation learning, feature hierarchy, structure learning, fair representation learning, high-dimensional estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
