The Quantization Model of Neural Scaling

The measured slope ≈ -1.24 is steeper than the slope of -1.08 expected from the measured parameter-scaling exponent from Figure 3.

“which is a steeper than the slope of -1.08 expected from the measured parameter-scaling exponent from Figure 3, though within the margin of error given the uncertainty of our clustering methodology.”

Scaling with respect to parameters is noisier than data scaling.

“We see that scaling w.r.t.parameters is noisier than data scaling, possibly due to model initialization having some influence on which quanta are learned (for our data scaling experiments, we use the same seed and same model size for all runs).”

For the first six models in the Pythia sequence, the mean loss scales as a power law with exponent α_N = 0.083.

“First, we find that for the first six models in the Pythia sequence, the mean loss scales as a power law with exponent α N = 0.083, roughly in line with the parameter scaling exponent of 0.076 measured in [3].”

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

QDG finds clusters for more abstract prediction rules.

“We also find clusters for more abstract prediction rules.”

The quantum shown in the left column of Figure 1 is the skill of incrementing a numerical sequence.

“For instance, the quantum shown on the left column of Figure 1 is the skill of incrementing a numerical sequence, and the examples involve predicting a variety of different tokens representing numbers.”

QDG clusters that involve predicting the same token usually do so for a coherent reason, rather than merely because they share the same output.

“While most clusters involve the prediction of the same token, manually inspecting these clusters we find that they usually involve predicting the same token for a coherent reason, rather than being based merely on having the same output.”

For scaling on individual subtasks, there is a rough scale of data or parameters below which networks do not learn the task, and above which they do.

“We also see that for scaling on individual subtasks, there is a rough scale of data or parameters below which networks do not learn the task, and above which they do.”

The mechanism of neural scaling on the multitask sparse parity task coincides with the theory from Section 2.

“We observe power law neural scaling in data and parameters on this task, and find that the mechanism of neural scaling coincides with our theory from Section 2.”

We presented a toy dataset where neural scaling exhibits these properties.

“We presented a toy dataset where neural scaling exhibits these properties.”

The frequency at which the discovered quanta are used in the training distribution roughly follows a power law corresponding with the empirical scaling exponent for language models.

“We tentatively find that the frequency at which these quanta are used in the training distribution roughly follows a power law corresponding with the empirical scaling exponent for language models, a prediction of our theory.”

• AssumptionThe authors state this as an assumption: Many natural prediction problems decompose into an enumerable set of computations, pieces of knowledge, or skills, which models must learn to reduce loss.
• AssumptionThe authors state this as an assumption: Some quanta are more useful for reducing loss than others, leading to a natural ordering of the quanta; optimally trained networks should learn the quanta in that order.
• AssumptionThe authors state this as an assumption: The frequencies at which the quanta are used for prediction follow a power law.

The slope of the envelope of the rank-frequency curves from cluster rank 100-1000 is ≈ -1.24.

“We measure the slope of the envelope of the rank-frequency curves from cluster rank 100-1000 to be ≈ -1.24”

• Also stated in: § 5.1 The natural distribution over language modeling quanta · plain

  “Between ranks 100-1000, we measure a slope of ≈ -1.24, about 0.16 off our expected slope of -1.08, and so within the margin of error.”

Most QDG clusters involve the prediction of the same token.

“While most clusters involve the prediction of the same token, manually inspecting these clusters we find that they usually involve predicting the same token for a coherent reason, rather than being based merely on having the same output.”

Using language model gradients, the authors automatically decompose model behavior into a diverse set of skills (quanta).

“Using language model gradients, we automatically decompose model behavior into a diverse set of skills (quanta).”

Gradients contain enough information to allow automatic discovery of many coherent clusters of LLM behavior.

“While crude, we find that gradients contain enough information to allow us to automatically discover many coherent clusters of LLM behavior using the following algorithm:”

There exist tokens for which loss drops as a single phase transition in scale.

“In natural language, we observe that model performance on most tokens improves at multiple scales, suggesting that most tokens are polygenic, but we can find tokens for which loss drops as a single phase transition in scale.”

In QDG on multitask sparse parity, elements from the same subtask have on average higher angular similarity than elements between subtasks.

“We plot the full similarity matrix where samples are ordered according to their a priori known subtask, rather than their cluster from QDG, and see a clear pattern where elements from the same subtask have on average higher angular similarity than elements between subtasks.”

On each individual subtask of multitask sparse parity, loss follows a reverse-S curve, dropping after an initial plateau.

“Training dynamics on the multitask sparse parity problem are highly nontrivial -on each individual subtask, loss follows a reverse-S curve, dropping after an initial plateau.”

In natural language, model performance on most tokens improves at multiple scales.

“In natural language, we observe that model performance on most tokens improves at multiple scales, suggesting that most tokens are polygenic, but we can find tokens for which loss drops as a single phase transition in scale.”

Despite approximately-zero-loss tokens being by far the most common, they do not contribute much mass to the mean loss.

“We see that despite approximately-zero-loss tokens being by far the most common, they do not contribute much mass to the mean loss.”

The frequencies at which the quanta we discover are used for prediction in natural text seem to roughly track the power law our theory would predict, though this measurement is quite imprecise.

“The frequencies at which the quanta we discover are used for prediction in natural text seem to roughly track the power law our theory would predict, though this measurement is quite imprecise.”

Many, though not all, QDG clusters reveal some coherent model behavior.

“We find that many, though not all, QDG clusters reveal some coherent model behavior.”

In multitask sparse parity, all prediction problems are monogenic.

“In multitask sparse parity, all prediction problems are monogenic.”

In a toy model of the clustering process, similar-looking curves can be obtained when the dimension and noise are high.

“In Appendix E, Figure 16 we find that we can get similar-looking curves in a toy model of this clustering process when the dimension and noise is high.”

• AssumptionThe authors state this as an assumption: The dataset has N = 1000 subtasks, each subtask containing n_i = floor(A i^α) tokens (A = 1000).
• AssumptionThe authors state this as an assumption: Gradients within a subtask i are modeled with a Gaussian distribution N(m_i, σ^2 I_{d×d}), where d is the embedding dimension, σ is the noise level, and m_i is the Gaussian mean.
• AssumptionThe authors state this as an assumption: m_i itself is drawn from the standard Gaussian distribution m_i ∼ N(0, I_{d×d}).

From the rank-frequency plot of the clusters on multitask sparse parity, we do not recover a slope of -1.4, but rather a lower slope of ≈ -1.1.

“However, from the rank-frequency plot of the clusters, we do not recover a slope of -1.4,but rather a lower slope of ≈ -1.1.”

Power law neural scaling is observed in data and parameters on the multitask sparse parity task.

“We observe power law neural scaling in data and parameters on this task, and find that the mechanism of neural scaling coincides with our theory from Section 2.”

The parameter scaling exponent α_N is measured to be ≈ 0.083 from the first six points along the curve for Pythia models.

“Left: Scaling of mean test loss w.r.t.non-embedding parameters for the Pythia models[29].The parameter scaling exponent α N is measured to be ≈ 0.083 from the first six points along the curve (the seventh model appears to break the trend).”

• cites [29]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Most of the Rosenfeld et al. samples lie below the α_D = α_N line, as the model would predict.

“most of the Rosenfeld et al. [2] samples lie below the α D = α N line, as our model would predict”

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Larger models achieve approximately 0 loss on an increasing fraction of samples.

“larger models achieve ≈ 0 loss on an increasing fraction of samples”

Convergence time S_k on subtask k scales as S_k proportional to p^{-0.81}, where p is the frequency of that subtask.

“We see that convergence time S k on subtask k is S k ∝ p -0.81”

Despite sparse parity having a known decomposition into subtasks that are power law distributed in frequency, QDG does not recover this same power law from samples.

“Despite sparse parity having a known decomposition into subtasks which are power law distributed in frequency, we do not recover this same power law from samples.”

The gradient similarity matrix of natural language samples exhibits a block-diagonal structure when samples are ordered by QDG cluster.

“Samples are reordered according to their QDG cluster (with 400 clusters) to reveal the block-diagonal structure of the similarity matrix.”

The mean loss decreases smoothly, averaging over the phase transitions in the model's performance on subtasks.

“The mean loss decreases smoothly, averaging over these phase transitions in the model's performance on subtasks.”

The slope of the envelope of the rank-frequency curves from cluster rank 100-1000 is measured to be approximately -1.24.

“We measure the slope of the envelope of the rank-frequency curves from cluster rank 100-1000 to be ≈ -1.24”

For low α, the scaling curves in D also deviate from theory substantially.

“for low α the scaling curves in D also deviate from theory substantially”

α_D is substantially higher than the theoretical prediction α/(α + 1) for small α.

“We observe that α D is substantially higher than the theoretical prediction α/(α + 1) for small α.”

In Figure 9, the number of subtasks learned n, when including subtasks learned after early-stopping, seems to be in line with theory: n ∝ D^{1/(α+1)}.

“In Figure 9, we show how the number of subtasks learned n, when we include subtasks learned after early-stopping, seems to be in line with theory: n ∝ D 1/(α+1) .”

Scaling curves in steps S are typically steeper than the α_S = α/(α + 1) predicted from theory.

“scaling curves in steps S are typically steeper than the α S = α/(α + 1) predicted from theory”

Neural networks exhibit power law scaling in loss w.r.t. parameters N, training time S, and training samples D when trained on the multitask sparse parity dataset.

“Top: Neural networks exhibit power law scaling in loss w.r.t.parameters N , training time S, and training samples D (for multi-epoch training) when trained on the multitask sparse parity dataset.”

There is clear correlation between the estimated envelope slope and α.

“Although there is clear correlation between the estimated envelope and α”

Over time, models achieve approximately 0 loss on an increasing fraction of tokens, similar to scaling in model size.

“Over time, models achieve ≈ 0 loss on an increasing fraction of tokens, similar to scaling in model size.”

Low-loss samples contribute minimal mass to the mean loss.

“Low-loss samples contribute minimal mass to the mean loss”

Per-sample scaling curves can have diverse behavior, with individual samples exhibiting sharp versus smooth improvement.

“Per-sample scaling curves can have diverse behavior.”

• Also stated in: § E.2 A toy model of QDG uncertainty and bias · plain

  “Additional LLM scaling curves on individual samples which exhibit sharp vs smooth improvement.”

The mean loss is dominated by samples with much higher loss of 5-10 bits, depending on scale.

“which is instead dominated by samples with much higher loss of 5-10 bits (depending on scale)”

Losses approximately equal to 0 are by far the most common in the distribution over losses on individual samples.

“Losses ≈ 0 are by far the most common”

For Pythia models, the measured exponents α_S are between 0.037 and 0.06.

“We measure exponents α S between 0.037 and 0.06.”

Many clusters from QDG on TinyStories reflect simple recurring patterns in the TinyStories dataset, like predicting ' time' after 'Once upon a'.

“Many of these clusters reflect some simple recurring pattern in the TinyStories dataset, like predicting”

Using the envelope slope to estimate α gives an error on the order of 0.2.

“if we use the envelope slope to estimate α, the error is on the order of 0.2”

α_S is consistently higher than the theoretical prediction α/(α + 1).

“We observe that α S is consistently higher than the theoretical prediction α/(α + 1).”

Scaling behavior on individual subtasks exhibits emergence, where subtasks are suddenly learned above a particular scale.

“Scaling behavior on individual subtasks exhibits emergence, where subtasks are suddenly learned above a particular scale.”

Low-loss samples contribute minimal mass to the mean loss, which is instead dominated by samples with much higher loss of 5-10 bits (depending on scale).

“Right: the expected loss integrand Lp(L) for models of different sizes.Low-loss samples contribute minimal mass to the mean loss, which is instead dominated by samples with much higher loss of 5-10 bits (depending on scale).”

When data is distributed such that the use frequencies of quanta follow a power law, then power law neural scaling can arise as models learn more and more quanta, with smooth scaling curves averaging over many small cases of emergence.

“When data is distributed in such a way that the”

• AssumptionThe authors state this as an assumption: There is an underlying discreteness to everything that models learn.

Together QH1-QH3 can result in power law neural scaling.

“Together these can result in power law neural scaling.”

• AssumptionThe authors state this as an assumption: Many natural prediction problems decompose into an enumerable set of computations, pieces of knowledge, or skills, which models must learn to reduce loss.
• AssumptionThe authors state this as an assumption: Some quanta are more useful for reducing loss than others, leading to a natural ordering of the quanta; optimally trained networks should learn the quanta in that order.
• AssumptionThe authors state this as an assumption: The frequencies at which the quanta are used for prediction follow a power law.

The prediction that a power law in use frequencies explains power law scaling of loss is validated on toy datasets.

“We validate this prediction on toy datasets, then study how scaling curves decompose for large language models.”

The measured scaling exponent α_N = 0.083 from Figure 3 implies a power law distribution over quanta with exponent -1.083.

“The measured scaling exponent of α N = 0.083 from Figure 3 implies a power law distribution over quanta with exponent -1.083.”

The Quantization Hypothesis can produce power law scaling as a function of the number of quanta learned n.

“So far we have seen how the Quantization Hypothesis can produce power law scaling as a function of the number of quanta learned n.”

• AssumptionThe authors state this as an assumption: Many natural prediction problems decompose into an enumerable set of computations, pieces of knowledge, or skills, which models must learn to reduce loss.
• AssumptionThe authors state this as an assumption: Some quanta are more useful for reducing loss than others, leading to a natural ordering of the quanta; optimally trained networks should learn the quanta in that order.
• AssumptionThe authors state this as an assumption: The frequencies at which the quanta are used for prediction follow a power law.

From QH3, the kth quantum benefits prediction on a randomly chosen sample with probability p_k = 1/ζ(α+1) k^{-(α+1)} ∝ k^{-(α+1)} for a Zipf power law α > 0.

“From QH3, we have that the k th quantum benefits prediction on a randomly chosen sample with probability p k = 1 ζ(α + 1) k -(α+1) ∝ k -(α+1)(1) for a Zipf power law α > 0”

• AssumptionThe authors state this as an assumption: The frequencies at which the quanta are used for prediction follow a power law.

The difference between the measured slope ≈ -1.24 and the expected slope -1.08 is within the margin of error given the uncertainty of the clustering methodology.

“though within the margin of error given the uncertainty of our clustering methodology.”

• Also stated in: § 5.1 The natural distribution over language modeling quanta · plain

  “Between ranks 100-1000, we measure a slope of ≈ -1.24, about 0.16 off our expected slope of -1.08, and so within the margin of error.”

Mean test loss has been observed to decrease as a power law in both the number of network parameters and the number of training samples.

“Across a range of studies, mean test loss has been observed to decrease as a power law in both the number of network parameters (L ∝ N -α N ) and the number of training samples (L ∝ D -α D ) [1,2,3,4,5,6,7].”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [2]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [3]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [5]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [7]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

If a_k and b_k are k-independent, then a model that has learned the first n quanta will have expected loss L_n = a + (b-a)/(αζ(α+1)) n^{-α}.

“If a k and b k are k-independent (a k = a, b k = b), then a model that has learned the first n quanta will have expected loss L n = n k=1 ap k + ∞ k=n+1 bp k = ∞ k=1 ap k + ∞ k=n+1 (b -a)p k ≈ a + b -a ζ(α + 1) ∞ n k -(α+1) dk = a + b -a αζ(α + 1) n -α .”

• AssumptionThe authors state this as an assumption: All samples are monogenic, meaning that prediction relies on at most a single quantum.

The Quantization Model can be the mechanism of neural scaling for data with the right structure.

“Smooth power law scaling therefore averages over a large number of emergent changes in model performance when properly decomposed by subtask, a proof of concept that the Quantization Model can be the mechanism of neural scaling for data with the right structure.”

Under the single-epoch data scaling model, L(S)-L∞ ∝ S^{-α/(α+1)}, so α_S = α/(α+1).

“This gives scaling in training steps L(S) -L ∞ ∝ n -α ≈ (S/T ) -α/(α+1) ∝ S -α/(α+1) , and so α S = α/(α + 1).”

• AssumptionThe authors state this as an assumption: The magnitude of the gradients for learning these quanta follow a power law, so the convergence time for each quantum may follow a power law.

The size of the discovered clusters seems to decay at a rate (very roughly) compatible with observed neural scaling exponents, in line with the theory.

“We are encouraged that the size of our discovered clusters seem to decay at a rate (very roughly) compatible with observed neural scaling exponents, in line with our theory.”

A range of assumptions for a_k and b_k produce curves that are exact or approximate power laws, the latter including a small logarithmic correction.

“In Appendix A, we provide analogous derivations for other assumptions for a k and b k , and find that a range of assumptions produce curves that are exact or approximate power laws -the latter include a small logarithmic correction.”

When quanta are learned in order of decreasing use frequency, a power law in use frequencies explains observed power law scaling of loss.

“We show that when quanta are learned in order of decreasing use frequency, then a power law in use frequencies explains observed power law scaling of loss.”

• AssumptionThe authors state this as an assumption: Many natural prediction problems decompose into an enumerable set of computations, pieces of knowledge, or skills, which models must learn to reduce loss.
• AssumptionThe authors state this as an assumption: Some quanta are more useful for reducing loss than others, leading to a natural ordering of the quanta; optimally trained networks should learn the quanta in that order.
• AssumptionThe authors state this as an assumption: The frequencies at which the quanta are used for prediction follow a power law.

Extracting the power law exponent over quanta utilization frequency by measuring the slope of the rank-frequency curve should have uncertainty of at least 0.2.

“From our analysis in Appendix E, we think that extracting the power law exponent over quanta utilization frequency by measuring the slope of the rank-frequency curve should have uncertainty of at least 0.2.”

Under the model, multi-epoch and single-epoch data scaling exponents coincide: α_D = α_S.

“Under this model, multi-epoch and single-epoch data scaling exponents coincide: α D = α S .”

Smooth power law scaling averages over a large number of emergent changes in model performance when properly decomposed by subtask.

“Smooth power law scaling therefore averages over a large number of emergent changes in model performance when properly decomposed by subtask, a proof of concept that the Quantization Model can be the mechanism of neural scaling for data with the right structure.”

Most tokens are polygenic.

“In natural language, we observe that model performance on most tokens improves at multiple scales, suggesting that most tokens are polygenic, but we can find tokens for which loss drops as a single phase transition in scale.”

In our model, high linearity would result from a task's relevant quanta being widely spread along the Q Sequence.

“In our model, high linearly would result from a task's relevant quanta being widely spread along the Q Sequence, and high breakthroughness would result from a task being monogenic or from the relevant quanta being close together in the Q Sequence.”

• AssumptionThe authors state this as an assumption: The quanta are modeled as being independent, with learning order given just by the use frequencies.

It is possible for scaling laws to arise from the Quantization Model for data with the right structure.

“It is therefore possible for scaling laws to arise from the Quantization Model for data with the right structure.”

The multitask sparse parity dataset imposes a power law distribution over subtasks in data.

“This imposes a power law distribution over subtasks in data.”

When particular capabilities are examined, larger models often have emergent abilities, i.e., qualitatively different performance than smaller models.

“Although aggregate performance changes smoothly with scale, when particular capabilities are examined, larger models often have emergent abilities, i.e., qualitatively different performance than smaller models [8,9].”

• cites [8]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [9]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

From α_N = α, the model predicts α_D = α_N/(α_N+1).

“From our earlier result that α N = α, we would therefore predict that α D = α N /(α N + 1).”

Under the multi-epoch data scaling model, L(D)-L∞ ∝ D^{-α/(α+1)}, so α_D = α/(α+1).

“Under this model, we get scaling in data samples L(D) -L ∞ ∝ n -α ∝ (D/τ )) -α/(α+1) ∝ D -α/(α+1) , and so α D = α/(α + 1).”

• AssumptionThe authors state this as an assumption: For each quantum, a threshold of τ examples utilizing quantum k are needed in the training set for quantum k to be learned.

In our model, using a data threshold τ ≫ 1 lowers the variance in the scaling curve.

“In our model, using a data threshold τ ≫ 1 lowers the variance in the scaling curve”

• AssumptionThe authors state this as an assumption: The quanta are modeled as being independent, with learning order given just by the use frequencies.

For the case b_k = -log p_k and a_k = 0, the expected loss contains a power law term n^{-α} plus a log term n^{-α} log n.

“which contains a power law term n -α plus a log term n -α log n”

• AssumptionThe authors state this as an assumption: The first n quanta are learned, reducing the loss from b_k to a_k for k ≤ n, while the loss remains b_k for k > n.

Deviation from theory for the scaling exponent of loss L w.r.t. D likely originates from failure to regularize network training, leading to early-stopping ending training before some subtasks can be learned.

“Deviation from theory for the scaling exponent of loss L w.r.t.D therefore likely originates from our failure to regularize network training, leading to early-stopping ending training before some subtasks can be learned.”

The model of scaling predicts that α_D = α_N/(α_N + 1).

“Our model of scaling predicts that α D = α N /(α N + 1)”

• AssumptionThe authors state this as an assumption: The first n quanta are learned, reducing the loss from b_k to a_k for k ≤ n, while the loss remains b_k for k > n.

For the case b_k = -log p_k and a_k = -log(Cp_k), the expected loss is a power law n^{-α} plus a constant.

“which is a power law n -α plus a constant”

• AssumptionThe authors state this as an assumption: The first n quanta are learned, reducing the loss from b_k to a_k for k ≤ n, while the loss remains b_k for k > n.

For very large n, the log term can be ignored, so the expected loss is still approximately a power law of n with exponent -α.

“For very large n, the log term can be ignored, so L is still approximately a power law of n with exponent -α”

• AssumptionThe authors state this as an assumption: The first n quanta are learned, reducing the loss from b_k to a_k for k ≤ n, while the loss remains b_k for k > n.