In-context Learning and Induction Heads

May combine two claims.

During the phase change, the majority of in-context learning ability (as measured by difference in loss between tokens early and late in the sequence) is acquired, and simultaneously induction heads form within the model that are capable of implementing fairly abstract and fuzzy versions of pattern completion.

“During this phase change, the majority of incontext learning ability (as measured by difference in loss between tokens early and late in the sequence) is acquired, and simultaneously induction heads form within the model that are capable of implementing fairly abstract and fuzzy versions of pattern completion.”

May combine two claims.

Transformer language models undergo a phase change early in training during which induction heads form and in-context learning improves dramatically.

“Argument 1: Transformer language models undergo a 'phase change' during training, during which induction heads form and simultaneously in-context learning improves dramatically.”

• Also stated in: § In-context Learning and Induction Heads · p. 3 · plain

  “Argument 1 (Macroscopic co-occurence) : Transformer language models undergo a 'phase change' early in training, during which induction heads form and simultaneously in-context learning improves dramatically .”

May combine two claims.

For small models, we can explain mechanistically how induction heads work, and can show they contribute to in-context learning.

“Argument 5 ( Mechanistic plausibility of induction head generality): For small models, we can explain mechanistically how induction heads work, and can show they contribute to incontext learning.”

May combine two claims.

Our first line of evidence comes from correlating measures of in-context learning and measures of induction head presence over the course of training; we observe a tight co-variation between them across dozens of models, of different sizes, trained on different datasets.

“Our first line of evidence comes from correlating measures of in-context learning and measures of induction head presence over the course of training. Specifically, we observe a tight co-variation between them across dozens of models, of different sizes, trained on different datasets”

May combine two claims.

Before this window there is less than 0.15 nats of in-context learning, after it there is roughly 0.4 nats, an amount that remains constant for the rest of training and is also constant across many different model sizes (except for the one layer model where not much in-context learning ever forms).

“Before this window there is less than 0.15 nats of in-context learning, after it there is roughly 0.4 nats, an amount that remains constant for the rest of training and is also constant across many different model sizes (except for the one layer model where not much in-context learning ever forms).”

May combine two claims.

Before the phase change, loss largely stops improving around token 50, but after the phase change, loss continues to improve past that point.

“Before the phase change, loss largely stops improving around token 50, but after the phase change, loss continues to improve past that point.”

May combine two claims.

Varying the definition of in-context learning score (e.g., using different token indices) does not matter; all definitions show that the amount of in-context learning varies only slightly between models.

“all definitions appear to show that the amount of in-context learning varies only slightly between models.”

May combine two claims.

A one-layer smeared-key model experiences the phase change, while a regular one-layer model does not.

“The key observations are that (1) a one-layer smeared-key model experiences the phase change, while a regular onelayer model does not;”

May combine two claims.

The exception is the one-layer model, where induction heads never form, just as in-context learning never substantially develops for the one-layer model.

“The exception is the one-layer model, where induction heads never form - just as in-context learning never substantially develops for the one-layer model.”

May combine two claims.

The translation head in layer 7 of the 40-layer model showcases translation between English, French, and German, and is an induction head by the same definition.

“Here, we explore a head we found in layer 7 of our 40-layer model, showcasing translation between English, French, and German. (As with the others in this section, this head is also an 'induction head' by the same definition we've been using all along, because when shown repeated random tokens, it uses a 'prefix matching' attention pattern to copy the sequence verbatim.)”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Over 75% of final in-context learning forms in the phase change window.

“Since over 75% of final in-context learning forms in this window, one might naively believe this to be the amount of in-context learning that induction heads are responsible for.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The only other hyperparameter change that occurs within the range of the phase change is the learning rate warm-up, which ramps up over the first 1.5e9 tokens.

“The only other hyperparameter change that occurs within the range of the phase change is the learning rate warm-up, which ramps up over the first 1.5e9 tokens.”

The quote only partly states this claim.

All models form induction heads.

“All of them form induction heads.”

May combine two claims.

All models have the same sharp increase in in-context learning, with the same rough amounts before and after the transition.

“All of them have the same sharp increase in in-context learning, with the same rough amounts before and after the transition.”

May combine two claims.

The pointer-arithmetic mechanism is not available to the models studied here, because they do not add positional information into the residual stream.

“This mechanism isn't available to the models we study here, since they do not add positional information into the residual stream.”

May combine two claims.

Large models predict tokens at all indices better than small models, and they are best at predicting later tokens.

“First, it's important to be clear that large models still predict tokens at all indices better than small models, and they're best at predicting later tokens.”

May combine two claims.

Whether the model is a tiny two-layer model or a fairly large 13 billion parameter model, and whether it has just gone through the phase change or trained much longer, does not matter for the in-context learning score; only whether the model has gone through the phase change at all matters.

“It appears to not matter whether the model is a tiny two layer model or a fairly large 13 billion parameter model, nor whether the model has just gone through the phase change or trained much longer. The only thing that matters, seemingly, is whether the model has gone through the phase change at all.”

May combine two claims.

The example heads score highly on copying and prefix matching evaluations, and are thus induction heads by the strict empirical definition, at the same time as they perform more sophisticated tasks.

“We verify that these heads score highly on our 'copying' and 'prefix matching' evaluations (that is, they increase the probability of the token they attend to, and attend to tokens where the prefix matches the present token on random text), and are thus 'induction heads' by our strict empirical definition, at the same time as they also perform these more sophisticated tasks.”

May combine two claims.

The attention heads described above simultaneously implement both the abstract behaviors and satisfy the formal definition of induction head (literal copying of random sequences using prefix matching).

“We emphasize again that the attention heads that we described above simultaneously implement both the abstract behaviors that we described, and these very same attention heads (as in, the exact same head in the same layer) also satisfy the formal definition of induction head (literal copying of random sequences using prefix matching).”

May combine two claims.

Induction heads from larger transformers (40-layer, 13B parameters) exhibit literal copying, translation, and a specific type of abstract pattern matching.

“we show some anecdotal examples of induction heads from larger transformers (our 40-layer model with 13 billion parameters) that exhibit exactly such behaviors - namely literal copying, translation, and a specific type of abstract pattern matching.”

May combine two claims.

Almost all the in-context learning in small attention-only models appears to come from induction heads, beginning at the start of the phase change and remaining true through the end of training.

“In fact, almost all the in-context learning in small attention-only models appears to come from these induction heads! This begins at the start of the phase change, and remains true through the end of training.”

May combine two claims. The quote only partly states this claim.

The literal copying head predicts repeating names 'Dursley' and 'Potters', the phrase 'a small son', and entire repeated sentences by attending back to a previous instance where the phrase was present.

“If you explore the visualization, you'll see that the head predicts repeating the names”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The pattern-matching head (layer 26 of the 40-layer model) does more complex pattern matching, possibly learning a simple function in context.

“In this final example, we show an attention head (found at layer 26 of our 40-layer model) which does more complex pattern matching. One might even think of it as learning a simple function in context!”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 6-layer MLP model shows a 'loss spike'.

“The 6-layer MLP model shows a 'loss spike' .”

May combine two claims.

Basic copying heads also exist in one-layer models, which don't have the greatly increased in-context learning we see in two-layer models.

“However, basic copying heads also exist in one-layer models, which don't have the greatly increased incontext learning we see in two-layer models.”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

The 6-layer MLP model has one lone induction head whose ablation has the opposite effect on the in-context learning score.

“The 6-layer MLP model has one lone induction head whose ablation has the opposite effect on the in-context learning score.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The simplest induction heads match just one preceding token, but we also often observe induction heads that perform a fuzzy match over several preceding tokens.

“The simplest induction heads match just one preceding token. But we also often observe induction heads that perform a fuzzy match over several preceding tokens.”

May combine two claims.

The loss decreases more slowly for small models than large models before the phase change, but the opposite is true after.

“The key observation is that the loss decreases more slowly for small models than large models before the phase change, but the opposite is true after.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The 4-layer MLP model ablations are nowhere near as 'peaky' as those of any other model.

“The 4-layer MLP model ablations are nowhere near as 'peaky' as those of any other model.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The order of the derivative of loss curves of models of different sizes switches at the phase change.

“Another observation we find quite striking is that if one looks at the derivative of the loss curves of models of different sizes, it appears that their order switches at the phase change.”

May combine two claims.

Large models gain all their advantage over small models very early in the context; the majority of the difference forms in the first ten tokens.

“What's going on is that the large models gain all their advantage over small models very early in the context. In fact, the majority of the difference forms in the first ten tokens:”

Borderline on S3 (load-bearing): unclear from the text. May combine two claims.

In practice, induction heads don't exhibit these properties perfectly, and our measurements give us a continuum, but there is a clear subset of heads which exhibit these properties with much greater than random chance.

“In practice, induction heads don't exhibit these properties perfectly, and our measurements give us a continuum, but there is a clear subset of heads which exhibit these properties with much greater than random chance.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Around token 100, there's a regime where small models reduce their loss very slightly more per token than large models.

“Around token 100, there's a regime where small models reduce their loss very slightly more per token than large models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Induction heads seem to largely decouple A and B, meaning they don't have a fixed table of bigram statistics they can update on, but rather can abstract to new patterns.

“More specifically, induction heads seem to largely decouple A and B . While some induction heads may specialize on certain kinds of A s or B s, this significant decoupling of A and B means that they don't have a fixed table of bigram statistics they can update on, but rather can abstract to new patterns.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Full-scale models above 16 layers start to show a small number of heads that score well on 'prefix search' but get a negative score on copying, which means they are not induction heads.

“Full-scale models above 16 layers start to show a small number of heads that score well on 'prefix search' , but get a negative score on copying, which means they are not induction heads.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Ablating most other attention heads appears to increase in-context learning.

“In fact, ablating most other attention heads appears to increase in-context learning.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The 6-layer attention-only model has an unusual head that develops in the later half of training; this head is not an induction head, and ablating it has an effect similar to reversing the phase change.

“The 6-layer attention-only model has an unusual head that develops in the later half of training. This head is not an induction head, and yet ablating it has an effect similar to reversing the phase change (in the 'before-and-after vector' attribution plot).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Small models gain slightly more in the mid-context, catching up a tiny bit with large models, but it's a small effect.

“In fact, small models gain slightly more in the mid-context, catching up a tiny bit with large models, but it's a small effect.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The change in weight decay at approximately 5 billion tokens occurs after the phase change and is an unrelated phenomenon.

“Note that in the QK-circuit plot, the change in weight decay at approximately 5 billion tokens (described in Model Details) can be seen particularly prominently as a bend or hiccup in the plotted trajectories, but occurs after the phase change and is an unrelated phenomenon.”

Borderline on S3 (load-bearing), S4 (based): unclear from the text.

The prefix matching evaluator correlates well with the trace of QK eigenvalues for the previous-token QK-circuit term.

“Our prefix matching evaluator correlates well with the trace of QK eigenvalues for the previous-token QK-circuit term.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

In small models with MLPs, by the 5L and 6L size, induction heads form more in the second-to-last layer.

“In small models with MLPs, by the 5L and 6L size, the induction heads form more in the second-to-last layer.”

Probably not a substantial claim: may not pass S3 (load-bearing).

At least two importantly different ways of conceptualizing and measuring in-context learning exist in the literature: few-shot learning of specific tasks and observing loss at different token indices.

“At least two importantly different ways of conceptualizing and measuring in-context learning exist in the literature. The first conception, represented in Brown et al. , focuses on few-shot learning of specific tasks.”

• Also stated in: § In-context Learning · p. 4 · plain

  “The second conception of in-context learning, represented in Kaplan et al. , focuses on observing the loss at different token indices , in order to measure how much better the model gets at prediction as it receives more context.”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

In prior work, for 2-layer attention-only models, induction heads implement pattern copying behavior and appear to be the primary source of in-context learning.

“For 2-layer attention-only models, we were able to show precisely that induction heads implement this pattern copying behavior and appear to be the primary source of in-context learning.”

• Also stated in: § Induction Heads · p. 5 · plain

  “In our previous paper, we discovered a special kind of attention head - which we named induction heads - in two layer attention-only models.”

• cites [6]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [7]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

For all models, we can make a strong case that induction heads play some role in in-context learning, as we can demonstrate examples and show suggestive correlations.

“Conversely, for all models, we can make a strong case that induction heads play some role in incontext learning, as we can demonstrate examples and show suggestive correlations.”

Probably not a substantial claim: may not pass S4 (based).

These circuits implement most in-context learning in large models.

“and furthermore, that these circuits implement most in-context learning in large models.”

May combine two claims.

For small, attention-only models, we have strong evidence that attention heads are the mechanism for the majority of in-context learning, supported by ablations and mechanistic reverse engineering.

“For small, attention-only models, we believe we have strong evidence that attention heads are the mechanism for the majority of in-context learning, as we have evidence supported by ablations and mechanistic reverse engineering.”

Probably not a substantial claim: may not pass S1 (asserted), S4 (based).

There are circuits which have the same or similar mechanism to 2-layer induction heads and perform a 'fuzzy' or 'nearest neighbor' version of pattern completion, completing [A*][B*] ... [A] → [B], where A* ≈ A and B* ≈ B are similar in some space.

“Specifically, the thesis is that there are circuits which have the same or similar mechanism to the 2-layer induction heads and which perform a 'fuzzy' or 'nearest neighbor' version of pattern completion, completing [A*][B*] … [A] → [B] , where A* ≈ A and B* ≈ B are similar in some space”

Classified as probably not a substantial claim. May combine two claims.

For large models with MLPs, we must rely on mainly correlational evidence, which could be confounded.

“For large models with MLP's, we must therefore rely on mainly correlational evidence, which could be confounded.”

May combine two claims.

The first attention head copies information from the previous token into each token; the second head searches for a previous place where the present token A occurred and attends to the next token B, copying it and causing the model to be more likely to output B as the next token.

“The first attention head copies information from the previous token into each token. This makes it possible for the second attention head to attend to tokens based on what happened before them, rather than their own content. Specifically, the second head (which we call the”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Emergent in-context learning was noted in GPT-2 and gained significant attention in GPT-3.

“Emergent in-context learning was noted in GPT-2 and gained significant attention in GPT-3 .”

• cites [10]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Induction heads are implementing a simple algorithm, and are not memorizing a fixed table of n-gram statistics; the rule [A][B] ... [A] → [B] applies regardless of what A and B are.

“Notice that induction heads are implementing a simple algorithm, and are not memorizing a fixed table of n-gram statistics. The rule [A][B] … [A] → [B] applies regardless of what A and B are.”

May combine two claims.

The translation head's attention pattern is more or less off-diagonal but meanders because different languages have different word order and token lengths.

“Note that the overall attention pattern (in red, top left) is more or less an 'off-diagonal' , but it meanders a little bit away from a sharp diagonal. The meandering is because different languages have different word order and token lengths.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

If we trust the previous paper's analysis, we can know how induction heads behave without actually running them, and the argument that they contribute to in-context learning goes through.

“If we trust this analysis, we can know how induction heads behave without actually running them, and the argument we made in the previous paragraph goes through.”

• AssumptionThe authors state this as an assumption: The previous paper's reverse engineering analysis of induction heads is trustworthy/correct.

Probably not a substantial claim: may not pass S3 (load-bearing).

In large models, the analysis over training has low time resolution (only 15 points in time).

“In large models, we have low time resolution on our analysis over training. Co-occurrence when one only has 15 points in time is less surprising and weaker evidence.”

May combine two claims.

The argument that some in-context learning is performed by induction heads is very strong, both in large and small models, though it does not say what fraction.

“This argument doesn't say anything about what fraction of in-context learning is performed by induction heads, but it seems like a very strong argument that some is, both in large and small models.”

May combine two claims.

The ablations measure the marginal effects of removing attention heads; if two heads do something similar and layer norm before logits rescales, the importance of individual heads may be masked.

“Our ablations measure the marginal effects of removing attention heads from the model. To the extent two heads do something similar and the layer norm before the logits rescales things, the importance of individual heads may be masked.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The definition of induction heads (in terms of their behavior on repeated random sequences) suggests they behave this way quite generally.

“And more generally, our definition of induction heads (in terms of their behavior on repeated random sequences) suggests they behave this way quite generally.”

Probably not a substantial claim: may not pass S4 (based).

The smeared-key architecture modification should make it easy for transformers of any depth to express induction heads.

“We define a 'smeared key”

Probably not a substantial claim: may not pass S3 (load-bearing).

We do not have ablations for full-scale models.

“Unfortunately, we do not have ablations for our full-scale models.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Extrapolation from small models suggests induction heads are responsible for the majority of in-context learning in large models.

“Argument 6: Extrapolation from small models suggests induction heads are responsible for the majority of in-context learning in large models.”

Classified as probably not a substantial claim. May combine two claims.

In the previous paper, the authors reverse engineered induction heads, showing from the parameter level how they implement induction behavior (and that they should).

“In our previous paper, we were able to reverse engineer induction heads, showing from the parameter level how they implement induction behavior (and that they should).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the small models studied in the previous paper, all induction heads have the described QK term driving their attention pattern and a positive eigenvalue circuit performing copying.

“In the small models we studied in our previous paper, all induction heads have the described QK term driving their attention pattern and a positive eigenvalue circuit performing copying.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

It is natural for these more abstract induction heads to also exhibit the basic copying behaviors underlying the definition.

“Our point in this section is just that it's actually quite natural for these more abstract induction heads to also exhibit the basic copying behaviors underlying our definition.”

Probably not a substantial claim: may not pass S4 (based).

We are currently unable to reverse engineer induction heads in models with MLP layers.

“A bigger issue is that right now we're unable to reverse engineer induction heads in models with MLP layers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When induction heads were first introduced, they could be seen as a kind of 'in-context nearest neighbor' algorithm.

“When we first introduced induction heads, we observed that they could be seen as a kind of”

May combine two claims.

Induction heads require a composition of attention heads, which is only possible with two or more layers.

“induction heads require a composition of attention heads, which is only possible with two or more layers.”

• cites [15]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The phase change is an abrupt change that alters the model's behavior and has both macroscopic (loss and in-context learning curves) and microscopic (induction heads) manifestations.

“We call this transition 'the phase change' , in that it's an abrupt change that alters the model's behavior and has both macroscopic (loss and in-context learning curves) and microscopic (induction heads) manifestations, perhaps analogous to e.g. ice melting.”

May combine two claims.

In models with MLPs, in-context learning could also come from interactions between MLP and attention layers, so head ablations in MLP models may not give the full picture.

“But in models with MLPs, in-context learning could also come from interactions between MLP and attention layers. While ablating attention heads would affect such mechanisms, the relationship between the effect of the ablation on in-context learning and its true importance becomes more complicated. As a result, we can't be fully confident that head ablations in MLP models give us the full picture.”

May combine two claims.

Induction heads are the primary mechanism for in-context learning in small attention-only models, but the evidence is only suggestive for the MLP case.

“All things considered, we feel comfortable concluding from this that induction heads are the primary mechanism for in-context learning in small attention-only models, but see this evidence as only suggestive for the MLP case.”

May combine two claims.

The translation head's logit attribution patterns are not perfectly sharp; it does not always directly increase the logit of the corresponding prediction, likely because its output needs further processing by later layers.

“The logit attribution patterns for this head are not perfectly sharp; that is, even in cases where the attention head is attending to the matching word in an earlier language, it does not always directly increase the logit of the corresponding prediction. We would guess that this is because this head's output needs to be further processed by later layers.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

It is highly improbable that the two sharp transitions (induction head formation and in-context learning improvement) would co-occur across so many models by pure chance, without any causal connection.

“It's highly improbable these two sharp transitions would co-occur across so many models by pure chance, without any causal connection.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the previous paper, only a single induction head in a small attention-only model was reverse engineered, although others can be and have been reverse engineered.

“In the previous paper, we only reverse engineered a single induction head in a small attention-only model, although we can reverse engineered others (and have done so).”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Using previous context as data points for a nearest neighbor algorithm, and nearest neighbors improves as one gives it more data points.

“We're essentially using the previous context as data points for a nearest neighbor algorithm, and nearest neighbors improves as one gives it more data points.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Aside from the experiments varying the dataset, all these models were trained on the same tokens in the same order.

“That said, it's also the case that aside from the experiments varying the dataset, all these models were trained on the same tokens in the same order”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Some bits of information are harder to learn than others.

“Some bits of information are harder to learn than others.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

The model must learn more sophisticated algorithms such as induction heads, which is likely harder to learn yet results in fewer bits of information.

“But later in training all the lowhanging fruit has been plucked, and the model must learn more sophisticated algorithms such as induction heads, which is likely harder to learn yet results in fewer bits of information.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Induction heads don't occur in 1 layer models, because they require a composition of attention heads in different layers.

“Note that induction heads don't occur in 1 layer models, because they require a composition of attention heads in different layers.”

Classified as probably not a substantial claim.

Kaplan et al. found that 1-layer transformers do not follow the same scaling laws as larger transformers.

“This was previously observed by Kaplan et al. who found that 1-layer transformers do not follow the same scaling laws as larger transformers.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Emergent in-context learning was compellingly demonstrated in GPT-3.

“Emergent in-context learning was compellingly demonstrated in GPT-3 .”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Research on universality began with Li et al. who showed that many neurons are highly correlated with neurons in retrained versions of the same model.

“Research on universality began with Li et al. who showed that many neurons are highly correlated with neurons in retrained versions of the same model.”

• cites [42]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Brown et al. find that, while aggregate loss scales predictably with model size, models' ability to perform specific tasks like arithmetic can change abruptly.

“Brown et al. find that, while aggregate loss scales predictably with model size, models' ability to perform specific tasks like arithmetic can change abruptly.”

• cites [1]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Saxe et al.'s framework finds that linear neural networks can be understood in terms of the evolution of independent paths through the network, with each path corresponding to a principal component of the data.

“Very roughly, they find that linear neural networks can be understood in terms of the evolution of independent paths through the network, with each path corresponding to a principal component of the data.”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

A number of papers have shown that in aggregate, neural networks develop representations with a lot of shared information.

“More recently, a number of papers have shown that in aggregate, neural networks develop representations with a lot of shared information ( e.g. ).”

• cites [43]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [44]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based).

The calculation of each attention head's attention pattern and output vector is separable, implemented by different circuits consisting of different sets of parameters which may be run and studied in isolation.

“the insight in our previous work that the calculation of each attention head's attention pattern and output vector is separable , that is, implemented by different circuits consisting of different sets of parameters which may be run and studied in isolation”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Induction heads just seem conceptually more powerful.

“Further, induction heads just seem conceptually more powerful.”

Probably not a substantial claim: may not pass S3 (load-bearing).

O'Connor & Andreas have experiments suggesting that removing all words except nouns can improve loss.

“O'Connor & Andreas have some experiments suggesting that removing all words except nouns can improve loss.”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Without the 'start of sequence' token, some heads that were doing prefix matching on real data would get anomalously low scores on our test sequences.

“Without the 'start of sequence' token, some heads that were doing prefix matching on real data would get anomalously low scores on our test sequences”

Probably not a substantial claim: may not pass S3 (load-bearing).

The only other potential contender for driving in-context learning in two-layer attention only models would be basic copying heads.

“The only other potential contender for driving in-context learning in two-layer attention only models would be basic copying heads.”

Probably not a substantial claim: may not pass S3 (load-bearing).

O'Connor & Andreas's finding that removing all words except nouns can improve loss seems inconsistent with the induction head hypothesis.

“This seems inconsistent with the induction head hypothesis.”

• cites [30]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In the Xie et al. experiments, the structure of the synthetic data does not benefit from Transformers, and LSTMs are perhaps better at simulating HMMs.

“we suspect that the structure of the synthetic data doesn't benefit from Transformers, and that LSTMs are perhaps better at simulating HMMs.”

• cites [31]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

By defining induction heads in terms of their behavior on repeated copies of random sequences, we can be confident that it's actually relying on induction rather than a simple copying head.

“By defining induction heads in terms of their behavior on repeated copies of random sequences, we can be confident that it's actually relying on induction rather than, say, a simple copying head that heuristically attends to previous tokens which could plausibly slot in well after the next token, even if that hasn't occurred yet in the current context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The relative loss between token 50 and token 500 likely represents harder and harder bits over the course of training.

“So the relative loss between token 50 and token 500 likely represents harder and harder bits over the course of training.”

May combine two claims.

We generally expect Transformers to outperform LSTMs at in-context learning on natural text, with induction heads as a major explanation.

“We generally expect Transformers to outperform LSTMs at in-context learning on natural text (as seen in Kaplan et al. ), with induction heads as a major explanation.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

We interpret the regime around token 100 as the small models picking up low-hanging fruit that the large models already got in the very early context.

“We interpret this as the small models picking up low-hanging fruit that the large models already got in the very early context.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Machine learning models change in smooth, predictable ways described by scaling laws.

“the observation that machine learning models change in smooth, predictable ways described by scaling laws has emerged as a useful tool for modelling the properties of models before training them.”

• cites [13]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

The fixed amount of further loss decrease is in some sense 'more difficult in-context learning' for large models, since they're starting from a lower loss baseline.

“It seems likely this fixed amount is in some sense”

May combine two claims.

In an induction head, where the head attends must be a function of the token before the token it attends to, but an individual attention head computes attention scores only from the source and destination token; without using information written by a second earlier attention head, the score from the attended token can't be a function of the token that precedes it.

“In an induction head, where the head attends must be a function of the token before the token it attends to. But an individual attention head computes attention scores only from the source and destination token. Without using information written by a second earlier attention head, the score from the attended token can't be a function of the token that precedes it.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Increasingly, research suggests that biological and artificial neural networks learn similar representations.

“Increasingly, research suggests that biological and artificial neural networks learn similar representations ( e.g. ).”

• cites [46]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [47]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [48]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

With the attention pattern frozen, the action of each head is purely linear, which means that in an attention-only model, a pattern-preserving ablation is equivalent to subtracting all terms containing the ablated head from the output logits.

“As an interesting aside, we note that with the attention pattern frozen the action of each head is purely linear, which means that in an attention-only model, a pattern-preserving ablation is equivalent to subtracting all terms containing the ablated head from the output logits.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Saxe et al. discovered closed form solutions to learning dynamics for linear neural networks without activation functions.

“One remarkable result in learning dynamics, by Saxe et al , has been the discovery of closed form solutions to learning dynamics for linear neural networks without activation functions.”

• cites [23]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

A same-matching matrix associated with a previous token term in the QK circuit is a simple way to implement basic prefix matching behavior.

“A 'same-matching' matrix associated with a 'previous token' term in the QK circuit is a simple way to implement basic prefix matching behavior.”

Classified as probably not a substantial claim. May combine two claims.

The set of 10,000 randomly-selected examples is held the same between all snapshots and between all models in all three families.

“The set is held the same between all snapshots and between all models in all three families.”

Classified as probably not a substantial claim. May combine two claims.

Copying, prefix matching, and previous token attention are relevant to induction heads' function of predicting repeated sequences of random tokens.

“We are interested in these properties in particular - copying, prefix matching, and previous token attention - because they are relevant to induction heads' function of predicting repeated sequences of random tokens.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The random index used for each example is consistent between each model.

“(The random index used for each example is consistent between each model.)”

Classified as probably not a substantial claim.

With the exception of attention head ablations, the properties are measured on all snapshots of all models across training (200 snapshots each for twelve small models, and 14 or 15 snapshots each for six full-scale models).

“With the exception of the attention head ablations (which are the most computationally-intensive, and so were run only on the small models), we measure these properties on all snapshots of all models across training (that is, 200 snapshots each for the twelve small models, and 14 or 15 snapshots each for the six full-scale models).”