On the Paradox of Learning to Reason from Data

As more statistical features are jointly removed, the conditional probability Pr(label=1|X) becomes more unbalanced.

“As shown in Table 4, as we try to jointly remove more statistical features X, Pr(label = 1|X) becomes more unbalanced”

A reasoning problem's most basic statistic can reveal substantial information about its correct label.

“even the simplest statistic of a reasoning problem can give away significant information about the true label (Sec.4.1)”

• Also stated in: § 4 BERT Learns Statistical Features · plain

  “even the simplest statistics of reasoning problems can provide significant information about their labels”

BERT trained on RP balance performs even better on RP than BERT trained on RP.

“As shown in Table 3, BERT trained on RP shows large performance drop when tested on RP balance, while BERT trained on RP balance shows even better performance on RP than RP-trained BERT.”

A BERT model trained on combined RP and LP data performs poorly on LP*, a slightly altered version of LP.

“we observe that the model performance again drops significantly on LP * (Table 2)”

The rule count acts as a statistical feature in the RP dataset.

“Hence, we call #rule a statistical feature for the dataset RP.”

T5 shows behavior similar to BERT in this setting.

“This intuition is supported by experiments with T5 [Raffel et al., 2020], which exhibits similar behaviors.”

• cites Raffel et al., 2020Not checked yet

  Not checked in the free run.

Using only the count of rules in a problem allows better-than-chance prediction of its label.

“by only looking at the number of rules in a reasoning problem, we can predict the correct label better than a random guess.”

• Also stated in: § 4.1 Statistical Features Inherently Exists · hedged

  “which makes it possible for the model to guess the label of an example with relatively high accuracy by only using its #rule.”

Training BERT yields near-perfect test accuracy on distributions that span the entire problem space.

“(E3) the BERT model can be trained to achieve near-perfect test accuracy on data distributions covering the whole problem space.”

A BERT model trained on combined RP and LP data reaches almost perfect test accuracy.

“We train BERT on the mixture of RP and LP, and BERT again achieves nearly perfect test accuracy.”

As #fact, branching factor, and #rule are progressively removed, the minimum sampling multiplier grows roughly exponentially: 5.5, 20.0, 55.6.

“in particular, as we progressively remove #fact, branching factor and #rule, the minimum times of examples we need to sample grows roughly exponentially: 5.5 → 20.0 → 55.6.”

T5 generalizes better after #rule is removed from RP.

“the T5 model generalizes better after #rule is removed from RP, suggesting that it is using #rule to make predictions.”

BERT trained on RP balance achieves higher accuracy on LP than BERT trained on RP.

“As shown in Table 3, compared to RP-trained BERT, BERT trained on RP balance achieves higher accuracy when tested on LP; in particular, for examples with reasoning depth 6, the model trained on RP balance attains an accuracy of 68.1%, approximately 10% higher than the model trained on RP.”

BERT possesses sufficient capacity to represent the correct reasoning function for SimpleLogic.

“(E2) we prove by construction that the BERT model has enough capacity to represent the correct reasoning function (Sec 2.2)”

• Also stated in: § 2.2 BERT Has Enough Capacity to Solve SimpleLogic · boosted

  “we explicitly construct a parameterization for BERT such that the fixed-parameter model solves all problem instances in SimlpleLogic.”

LP and the uniform distribution show similar statistical patterns for the rule count.

“In addition to RP, we also verify that both LP and the uniform distribution exhibit similar statistical patterns”

The branching factor serves as a statistical feature in LP and the uniform distribution as well.

“we observe that branching factor is also a statistical feature for both LP and the uniform distribution”

For examples with reasoning depth 6, BERT trained on RP balance attains 68.1% accuracy on LP, about 10% higher than BERT trained on RP.

“As shown in Table 3, compared to RP-trained BERT, BERT trained on RP balance achieves higher accuracy when tested on LP; in particular, for examples with reasoning depth 6, the model trained on RP balance attains an accuracy of 68.1%, approximately 10% higher than the model trained on RP.”

BERT trained on RP shows a large performance drop when tested on RP balance.

“As shown in Table 3, BERT trained on RP shows large performance drop when tested on RP balance, while BERT trained on RP balance shows even better performance on RP than RP-trained BERT.”

Statistical features can impair a model's ability to generalize.

“We show that statistical features can hinder model generalization performance (Sec.4.2)”

Uniform sampling is undesirable because more than 99.99% of generated examples would contain 30 atoms and 120 rules.

“over 99.99% of the examples generated by uniform sampling would have 30 atoms and 120 rules.”

Models fail catastrophically on other distributions.

“while models can attain near-perfect test accuracy on training distributions, they fail catastrophically on other distributions”

Models can attain near-perfect test accuracy on training distributions.

“while models can attain near-perfect test accuracy on training distributions, they fail catastrophically on other distributions”

90% of the examples in RP have #rule ≤ 80.

“Not balancing the label for x > 80 is acceptable as 90% of the examples in RP have #rule ≤ 80.”

If we naively drop the minimum number of examples from RP to satisfy Equation 1, we will be left with only 339 examples with #rule = 80, while 6822 examples with #rule = 38 remain unchanged.

“if we naively drop the minimum number of examples from RP such that Equation 1 is satisfied, we will be left with only 339 examples with #rule = 80, where the number ( 6822) of examples with #rule = 38 remains unchanged.”

Powerful neural models can learn to use statistical features to achieve high in-distribution accuracy.

“thus powerful neural models can learn to use them to achieve high in-distribution accuracy”

It is computationally infeasible to jointly remove multiple statistical features.

“We show that though removing one statistical feature (e.g., #rule) from training data can benefit model generalization, it is computationally infeasible to jointly remove multiple statistical features.”

SimpleLogic can be solved without substantial model capacity.

“Solving SimpleLogic does not require significant model capacity.”

Jointly removing statistical features from data is not feasible.

“We also show that it is infeasible to jointly remove statistical features from data, illustrating the difficulty of learning to reason in general.”

• Also stated in: § 1 Introduction · hedged

  “we argue that there are potentially countless statistical features and it is computationally expensive to jointly remove them from data (Sec.4.3).”

Identifying every statistical feature is not feasible.

“Thus, it is infeasible to identify all of them.”

Statistical features can be compositional, formed by combining several features, which increases their complexity.

“statistical features can also be compositional: one can define a joint statistical feature by combining multiple ones (e.g., branching factor and #rule), which further adds to the complexity.”

The SimpleLogic problem space is limited and simple.

“our problem space is confined and simple (see E1, E2)”

BERT learned statistical features inherent in logical reasoning problems rather than emulating the correct reasoning function.

“instead of learning to emulate the correct reasoning function, BERT has, in fact, learned statistical features that inherently exist in logical reasoning problems.”

• Also stated in: § 1 Introduction · plain

  “the model has learned to use statistical features in logical reasoning problems to make predictions rather than to emulate the correct reasoning function.”

A model that relies on statistical features to make predictions does not generalize to out-of-distribution examples.

“the model that relies on statistical features to make predictions does not generalize to out-of-distribution examples”

• AssumptionRests on an assumption: Correlations between statistical features and labels can change when the data distribution changes.

In contrast to shallow shortcuts in typical NLP tasks, logical reasoning's statistical features can be numerous and highly complex, making them hard to remove from training data.

“unlike the shallow shortcuts found in other typical NLP tasks, such statistical features can be countless and extremely complicated, and thus very difficult to be removed from training data.”

Removing #rule as a statistical feature improves model generalization.

“This is a clear signal that when #rule is removed as a statistical feature, the model generalizes better, suggesting that statistical features can hinder the generalization of the model.”

The paper's conclusions should also hold for other neural models.

“More importantly, as our argument assumes little about model architectures/pre-training procedures, most of our conclusions should also hold for other neural models.”

• AssumptionThe authors state this as an assumption: The paper's argument does not depend much on specific model architectures or pre-training procedures.

A model that performs well only on in-distribution examples has not acquired reasoning.

“the model attaining high accuracy only on indistribution test examples has not learned to reason.”

• Also stated in: § 1 Introduction · plain

  “Since the correct reasoning function does not change across data distributions, it follows that the model has not learned to reason.”

• AssumptionRests on an assumption: The correct reasoning function remains the same across data distributions.

The observed failure is not a typical out-of-distribution generalization failure.

“What we observed is not a common case of out-of-distribution (OOD) generalization failure”

T5 uses #rule to make predictions.

“the T5 model generalizes better after #rule is removed from RP, suggesting that it is using #rule to make predictions.”

These statistical patterns should appear naturally in any non-adversarially constructed dataset.

“Since this intuition assumes nothing about data distributions, it follows that such statistical patterns should naturally exist in any dataset that is not adversarially constructed.”

Statistical features belong to logical reasoning problems generally, rather than to particular data distributions.

“statistical features still inherently exist and are not specific to certain data distributions.”

• Also stated in: § Statistical Features are Inherent to Logical Reasoning · boosted

  “we show that #rule inherently exists as a statistical feature for logical reasoning problems in general; that is, it is not specific to the RP dataset.”

The findings extend to other neural models such as T5.

“Our result naturally extends to other neural models (e.g.T5)”

• Also stated in: § 1 Introduction · plain

  “This intuition is supported by experiments with T5 [Raffel et al., 2020], which exhibits similar behaviors.”

• cites Raffel et al., 2020Not checked yet

  Not checked in the free run.

The rule count is inherently a statistical feature of logical reasoning problems generally, not only of RP.

“we show that #rule inherently exists as a statistical feature for logical reasoning problems in general; that is, it is not specific to the RP dataset.”

The number of facts is also positively associated with labels.

“as facts can be seen as special form of rules, it follows from previous argument that #fact is also positively correlated with labels.”

Rules with three body atoms are less likely to fire than rules with one body atom.

“rules of the form A, B, C → D are less likely to be”

With fixed atoms and facts, adding more rules makes any atom more likely to be proved, so Pr(label=1 | #rule=x) should rise roughly monotonically with x.

“given a fixed set of atoms and facts, any atom is more likely to be proved when more rules are given, that is, Pr(label(e) = 1 | #rule(e) = x) should increase roughly monotonically as x increases.”

A problem's average rule-body size can reveal information about its label.

“the average number of atoms in rules of a reasoning problem can also leak information about its label.”

Statistical features can hinder the generalization of the model.

“This is a clear signal that when #rule is removed as a statistical feature, the model generalizes better, suggesting that statistical features can hinder the generalization of the model.”

Learning to reason differs fundamentally from achieving high NLP benchmark scores through statistical features.

“Our findings unveil the fundamental difference between”

The model appears to imitate a flawed reasoning function tied to its training distribution.

“the model seems to be emulating an incorrect”

No evidence indicates that continually enriching the training distribution will transform the model into one that learns reasoning.

“we find no evidence that consistently enriching the training distribution will bring a transformative change such that the model can learn to reason.”

The accuracy drop from RP to RP balance indicates that BERT trained on RP uses #rule to make predictions.

“Since RP balance is down-sampled from RP, the accuracy drop from RP to RP balance is explained by that BERT trained on RP is using #rule to make predictions.”

Models have learned to exploit statistical features rather than to emulate the correct reasoning function.

“we demonstrate that they have learned to exploit statistical features rather than to emulate the correct reasoning function.”

Training on datasets might not be sufficient for models to learn certain complex behaviors such as reasoning and planning.

“Our study suggests that training on datasets might not be sufficient for model to learn certain complex behaviors such as reasoning and planning.”