Exact-Solution Volume and Length Generalization in Transformers

Classified as probably not a substantial claim.

Hahn & Rofin (2024) report that fitted PARITY solutions become much sharper as input length grows than fitted solutions for FIRST and MAJORITY.

“Hahn & Rofin (2024) report that fitted PARITY solutions become much sharper as input length grows than fitted solutions for FIRST and MAJORITY”

• cites Hahn & Rofin (2024) sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The log n attention scaling was proposed by Chiang & Cholak (2022) and Nakanishi (2025), used in practice in Qwen (Bai et al., 2023), and given additional theoretical justification by Chen et al. (2026).

“The log n factor was proposed by Chiang & Cholak (2022) and Nakanishi (2025) and has been used in practice in Qwen (Bai et al., 2023).Chen et al. (2026) provide additional theoretical justification for this choice.”

• cites Chiang & Cholak (2022) sourceSupported
• cites Nakanishi (2025) sourceSupported
• cites Chen et al. (2026)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Chiang & Cholak (2022) construct exact transformers for FIRST and PARITY, but report that standard training fails on PARITY, and that FIRST does not necessarily extrapolate to longer inputs without log n-scaling.

“Chiang & Cholak (2022) construct exact transformers for FIRST (whether the first symbol is a 1) and PARITY (whether the number of 1's is odd), but report that standard training fails on PARITY, and that FIRST does not necessarily extrapolate to longer inputs without log n-scaling.”

• cites Chiang & Cholak (2022) sourceSupported

May combine two claims.

V_MAJORITY,Agen(h)^B(n) ≤ 4hB^2 E Δρ_{n,log n}(X) ≤ 4hB^2 ||f_X||∞ ∫_R Δρ_{n,log n}(x) dx ≤ 16h artanh(1/n)/log n.

“V MAJORITY,Agen(h) B (n) ≤ 4hB 2 E ∆ρ n,log n (X) ≤ 4hB 2 ∥f X ∥ ∞ R ∆ρ n,log n (x) dx ≤ 16h artanh(1/n) log n .”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims.

In experiments, FIRST remains accurate at the longest test length under A_pos, MAJORITY retains substantial accuracy under A_gen, and INDEX falls much closer to chance under A_pos; these observations agree with the ranking of the three volume bounds.

“Table 1 shows that FIRST remains accurate at the longest test length (under A pos ), MAJORITY retains substantial accuracy (under A gen ), and INDEX falls much closer to chance (under A pos ).These observations agree with the ranking of the three volume bounds.”

May combine two claims.

In A_pos(h), the two effective constraints have widths n^-2 and n^-1; in A_norm(h), the saturation of the quadratic query channel leaves only the n^-1 constraint.

“In A pos (h), the two effective constraints have widths n -2 and n -1 ; in A norm (h), the saturation of the quadratic query channel leaves only the n -1 constraint.”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims.

The original model degrades rapidly beyond the training range, whereas the selectively normalized model retains substantially higher accuracy.

“As shown in Fig. 2, the original model degrades rapidly beyond the training range, whereas the selectively normalized model retains substantially higher accuracy.”

May combine two claims. The quote only partly states this claim.

In experiments, the modified normalized model generalizes substantially better on INDEX, achieving an average accuracy of 85% at ten times the training length, compared with 60% for the original model.

“In our experiments, the modified model also generalizes substantially better on INDEX, achieving an average accuracy of 85% percent at ten times the training length.”

May combine two claims.

The map F(v) = v/(1 + ||v||_2^2) is approximately the identity near the origin, but its output has norm strictly smaller than one.

“The map F is approximately the identity near the origin, but its output has norm strictly smaller than one.”

May combine two claims.

0 < Δρ_{n,β}(x) ≤ 1/n, and ∫_R Δρ_{n,β}(x) dx = 4 artanh(1/n)/β.

“0 < ∆ρ n,β (x) ≤ 1 n , R ∆ρ n,β (x) dx = 4 artanh(1/n) β .”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims.

Despite INDEX being expressible and learnable on lengths up to 51, it showed poor length generalization, much worse than that of MAJORITY.

“Despite the task being expressible and learnable on lengths up to 51, it showed poor length generalization, much worse than that of MAJORITY.”

May combine two claims.

The lower bound constructs fixed neighborhoods of parameters independent of input length that give exact solutions to the FIRST task; the upper bound follows from symmetry between the two labels.

“The lower bound constructs fixed neighborhoods of parameters independent of input length that gives exact solutions to the task.The upper bound follows from symmetry between the two labels.”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims.

For INDEX, volume analysis shows that the volume decreases with longer input lengths and isolates the sources of such shrinkage.

“For INDEX, such analysis not only shows that the volume decreases with longer input lengths, but also isolates the sources of such shrinkage.”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims.

The two exact-solution sets for FIRST and its complementary task are disjoint, so each occupies at most one half of the cube; in particular, the NESV is O(1).

“The two exact-solution sets are disjoint, so each occupies at most one half of the cube.In particular, the NESV is O(1).”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

May combine two claims. The quote only partly states this claim.

For every large-margin solution, the indexed position receives at least an inverse polynomial amount of attention, and comparing with neighboring scores gives |χ| = O(n^-2) and |l| = O(n^-1).

“Repeating the same argument gives |χ| = O(n -2 ), |ℓ| = O(n -1 ).(15)”

May combine two claims.

For FIRST, one can choose parameters such that the attention score decreases quadratically away from position one, and a fixed neighborhood of these parameters preserves that decrease, yielding a constant NESV bound.

“For FIRST, one can choose parameters such that the attention score decreases quadratically away from position one.A fixed neighborhood of these parameters preserves that decrease, and the resulting concentration of attention lets the classifier return the first symbol for all sufficiently large lengths.The neighborhood does not shrink with n, thus ensuring a constant NESV bound.”

May combine two claims.

The only counterexample to the NESV-length-generalization relation was that under the general sinusoidal model, MAJORITY trains as well as under the position-free model A_gen, although the former's volume shrinks faster with n.

“The only counterexample was that, under the general sinusoidal model, MAJORITY trains as well as under the position-free model A gen , although the former's volume shrinks faster with n.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Additional experiments on N = 11, 21, 31, 41 give similar results.

“Additional experiments on N = 11, 21, 31, 41, which give similar results, are in Appendix D.”

May combine two claims.

INDEX demonstrates particularly bad length generalization, with 60% average accuracy at ten times the training length, despite being expressible.

“Among the tasks we examine, INDEX demonstrates particularly bad length generalization (60% average accuracy at ten times training length) despite being expressible (Kozachinskiy et al., 2026a).”

• cites Kozachinskiy et al., 2026aNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Across 100 positional models trained at five maximum lengths, smaller values of N^2|χ| and N|l| are associated with higher mean accuracy on unseen lengths; ranking models separately within each maximum training length gives correlations of -0.655 for N^2|χ| and -0.516 for N|l|, both with p < 10^-3 under permutation tests preserving the five training-length groups.

“Across the 100 positional models trained at five maximum lengths, smaller values of both quantities are associated with higher mean accuracy on unseen lengths; see Fig. 1.Ranking the models separately within each maximum training length gives correlations of -0.655 for N 2 |χ| and -0.516 for N |ℓ|.Both have p < 10 -3 under permutation tests that preserve the five training-length groups.”

May combine two claims.

In the 100 selected INDEX checkpoints on A_pos, the within-N rank correlation between N^2|χ| and seed-level mean OOD accuracy is -0.655, and between N|l| is -0.516; both permutation tests give p < 10^-3.

“The resulting correlations are -0.655 for N 2 |χ| and -0.516 for N |ℓ|.The two-sided permutation test uses 10,000 permutations and shuffles ranks only within the same value of N ; both tests give p < 10 -3 .”

May combine two claims.

F(ak) is collinear with a and equals λ_a(k) a for the stated λ_a(k).

“Since F (ak) is collinear with a, F (ak) = λ a (k) a, λ a (k) = k∥a∥ 2 1 + k 2 ∥a∥ 2 2 .”

May combine two claims.

For k ≥ 1, 0 ≤ 1 - λ_a(k) ≤ 1/(2k^2 ||a||_2^2) and |2k εχ(k)| ≤ ||s||_2 k ||a||_2^2.

“0 ≤ 1 -λ a (k) ≤ 1 2k 2 ∥a∥ 2 2 , |2kε χ (k)| ≤ ∥s∥ 2 k∥a∥ 2 2 .”

May combine two claims.

Outside a measure-zero set of raw parameters, every exact solution at a sufficiently large odd length has an activation pattern σ ⊆ [h] with δσ ≠ 0 and satisfies the positional compensation bounds (138) and (139).

“Lemma 10 (Positional compensation).Outside a set of raw-cube measure zero, every exact solution at a sufficiently large odd length has an activation pattern σ ⊆ [h] such that δ σ ̸ = 0 and”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For the paper's MAJORITY setup, A_pos and A_norm are equivalent and produce the same results for every seed.

“For our setup of MAJORITY, A pos and A norm are equivalent, hence producing same results for every seed.”

May combine two claims.

For PARITY, every increment in the count must flip the output, and a fixed number of hidden ReLU units cannot provide arbitrarily many flips in output.

“For PARITY, by contrast, every increment in the count must flip the output.A fixed number of hidden ReLU units cannot provide arbitrarily many flips in output.”

Classified as probably not a substantial claim. May combine two claims.

The coefficient C_quad(k) controls the curvature of the score around the indexed position, while C_lin(k) measures its first-order drift and therefore whether the score remains centered at j = k; we want C_lin to be zero.

“The coefficient C quad (k) controls the curvature of the score around the indexed position, while C lin (k) measures its first-order drift and therefore whether the score remains centered at j = k; we want it to be zero.”

Probably not a substantial claim: may not pass S2 (checkable).

This suggests a broader use of volume analysis as a diagnostic for transformer model refinement, especially task-specific ones.

“This suggests a broader use of volume analysis as a diagnostic for transformer model refinement, especially task-specific ones.”

May combine two claims.

Parameter-space volume is useful not only for measuring how exact solutions change with input length, but also for revealing why they change.

“Parameter-space volume is useful not only for measuring how exact solutions change with input length, but also for revealing why they change.”

Classified as probably not a substantial claim. May combine two claims.

Checkpoints are selected using validation data at length N, without inspecting longer inputs, and tested on random examples at lengths up to 511.

“Checkpoints are selected using validation data at length N , without inspecting longer inputs, and tested on random examples at lengths up to 511.”

May combine two claims.

The k^2 drift in the score gap is replaced by a linear term and a bounded error 2kεχ(k) = O(k^-1), leaving only the coefficient l to be confined to an interval of width O(n^-1).

“Thus the k 2 drift in (10) is replaced by a linear term and a bounded error 2kε χ (k) = O(k -1 ).Only the coefficient ℓ remains to be confined to an interval of width O(n -1 ).”

May combine two claims.

Expressing the difference in attention scores between neighboring positions as a function of k reveals two key error components, one proportional to k^2 and another proportional to k, which are the sources of failing length generalization.

“Expressing the difference in attention scores between neighboring positions as a function of k reveals two key error components, one proportional to k 2 and another proportional to k, which are the sources of failing length generalization.”

Classified as probably not a substantial claim.

The theoretical results concern fixed-width, one-layer, one-head transformers.

“Our theoretical results concern fixed-width, one-layer, one-head transformers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Chiang et al. (2023) argue that the geometry and volume of low-loss regions can explain part of neural-network generalization without relying entirely on the implicit bias of gradient descent.

“Chiang et al. (2023) argue that the geometry and volume of low-loss regions can explain part of neural-network generalization without relying entirely on the implicit bias of gradient descent.”

• cites Chiang et al. (2023)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For MAJORITY in A_gen, the output changes only once, between counts (n-1)/2 and (n+1)/2, so the separating interval narrows as n increases, requiring increasingly precise parameter choice.

“For MAJORITY, the output changes only once, between counts (n -1)/2 and (n + 1)/2.These counts differ by just one, so the interval that separates them narrows as n increases.Consequently, placing the decision boundary between them requires increasingly precise parameter choice.”

May combine two claims.

On the INDEX task, the volume analysis isolates the length-sensitive components, suggesting a modification to the model itself that improves length extrapolation.

“On the INDEX task, our volume analysis isolates the length-sensitive components, suggesting a modification to the model itself that improves length extrapolation.”

May combine two claims.

The normalization introduces no new trainable parameters, so A_norm(h) has the same parameter dimension and uses the same B as A_pos(h).

“The normalization introduces no new trainable parameters, so A norm (h) has the same parameter dimension and uses the same B as A pos (h).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The experiments use 20 random seeds per task.

“We use 20 random seeds per task and draw a fresh batch at each update, sampling its length uniformly from 1, . . ., N ; for MAJORITY, we use only odd lengths.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Parameter-space volume depends on the distribution placed on the parameters.

“Parameter-space volume depends on the distribution placed on the parameters.”

May combine two claims.

The original volume proof identifies the parameter combinations whose effects grow with the index k, and those combinations are associated with the behavior of trained models.

“The original volume proof identifies the parameter combinations whose effects grow with the index k, and those combinations are associated with the behavior of trained models.”

• AssumptionThe authors state this as an assumption: Equation (1) assumes a uniform distribution over a box of size 2B in the raw coordinates.

Classified as probably not a substantial claim.

At each test length, all seeds and model variants for the same task use the same saved bank of 1,000 examples.

“At each test length, all seeds and model variants for the same task use the same saved bank of 1,000 examples.”

Classified as probably not a substantial claim.

No input longer than N is used in checkpoint selection.

“No input longer than N is used in checkpoint selection.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The same 20 seeds, 2000-2019, are used at each maximum training length N ∈ {11, 21, 31, 41, 51}.

“We use the same 20 seeds, 2000-2019, at each maximum training length N ∈ {11, 21, 31, 41, 51}.”

Classified as probably not a substantial claim. May combine two claims.

The two-sided permutation test uses 10,000 permutations and shuffles ranks only within the same value of N.

“The two-sided permutation test uses 10,000 permutations and shuffles ranks only within the same value of N ; both tests give p < 10 -3 .”

Classified as probably not a substantial claim.

To avoid treating differences among the five training maxima as evidence for a coefficient association, the analysis ranks each coefficient magnitude and the seed-level mean OOD accuracy separately within each value of N and then pools the within-N ranks.

“To avoid treating differences among the five training maxima as evidence for a coefficient association, we rank each coefficient magnitude and the seed-level mean OOD accuracy separately within each value of N , and then pool the within-N ranks.”

Classified as probably not a substantial claim. May combine two claims.

Training and qualification did not have access to the test banks, and every newly trained checkpoint was selected and frozen before its test evaluation.

“Training and qualification did not have access to the test banks, and every newly trained checkpoint was selected and frozen before its test evaluation.”