Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models

“One of the impressive capabilities demonstrated by large language models is their ability to do few-shot learning by providing examples in-context and asking the model to generate a response to follow the final input provided [3].”

“Li et al. [5] studies generalization properties of transformers in this setting and demonstrates similar results for linear dynamical systems.”

“Ravent´os et al. [6] investigates the role of pretraining function diversity for in-context learning in the setting of pure linear regression – arguing that a sufficiently diverse distribution over linear tasks in pretraining is needed for ICL at test-time.”

“Like Garg et al. [4] and Aky¨urek et al. [1], we frame the ICL problem as providing a single prompt sequence ) to the model (i.e. a transformer) and generating a prediction for f ( x n +1 ): ˜ f ( x n +1 ).”

“Like Garg et al. [4] and Aky¨urek et al. [1], we frame the ICL problem as providing a single prompt sequence ) to the model (i.e. a transformer) and generating a prediction for f ( x n +1 ): ˜ f ( x n +1 ).”

“A direct estimation method (Direct) adopts the conditional probability of candidate answers that can be represented by tokens in the vocabulary of language models (Brown et al., 2020).”

“Evidence exists for transformer models identifying training data patterns very robustly, while performance degrading outside of distribution [82].”

“While LLMs selectively show emergent abilities on tasks drawn from cognitive psychology, their performance can deteriorate when stimuli are edited to be less familiar ( 16 ), and indeed transformers are more generally less successful in solving problems external to their training data ( 17 ).”

“…success remain an active area of research (Aky¨urek et al., 2023; Dai et al., 2023; Olsson et al., 2022; Von Oswald et al., 2023; Li et al., 2023b; Yadlowsky et al., 2023; von Oswald et al., 2023; Bai et al., 2023; Peyrard & Cho, 2025), increasing model size and training data have consistently…”

“Research indicates that the diversity and structure of training data impact ICL more than dataset size alone [13, 73, 89, 95].”

“…language and a set of in-context examples to teach models new distributions, but such adaptation is limited to settings that are well expressed using a set of language instructions that are further roughly similar to the distributions already seen during training (Yadlowsky et al., 2023).”

“…delving into the inner mechanism (Olah, 2022; Olsson et al., 2022) of ICL through different theoretical conceptual lenses such as linear regression formulation (Garg et al., 2022; Li et al., 2023b), meta-learning (Dai et al., 2023; Irie et al., 2022), and Bayesian inference (Xie et al., 2022).”

“Mixed Linear Models Several works observed that trans-formers can achieve good performance on a mixture of linear models (Bai et al., 2023; Pathak et al., 2023; Yadlowsky et al., 2023).”

“As highlighted by Yadlowsky et al. (2023), when confronted with problems beyond their pretraining data, transformer models exhibit various failure modes and their generalization abilities deteriorate, even for simple problems.”

“We hypothesize that Code LLMs have not seen enough task-specific training data and cannot generalize to those unseen tasks (Yadlowsky et al., 2023).”

“However, another line of research (Yadlowsky et al., 2023; Ahuja & Lopez-Paz, 2023) has raised a contrasting view, showing that ICL struggles to generalize to OOD tasks where there are distributional shifts in either the input distribution P ( X ) or the input-label mapping P ( Y | X ) .”

“In the realm of in-context learning, where input-target pairs are part of the prompt (Brown et al., 2020), Xie et al. (2021) and Yadlowsky et al. (2023) show that the ability to generalize depends on the choice of pre-training tasks.”

“Skill learning fails if the ground-truth data function is out of the pre-training function class (Yadlowsky et al., 2023); ICL degrades to skill recognition with Bayesian optimal estimator.”

“For the questions that are completely out-of-domain of the pre-trained and injected knowledge, the model’s generalization ability on new tasks degrades due to extrapolation [Yadlowsky et al., 2023].”

“Similarly, Yadlowsky et al. [2023] studied the ability of transformers to perform model selection between different function class families.”

“It is especially important to find a suitable training data recipe as decisions made in the pre-training stage can propagate downstream to domain-and task-specific settings (Yadlowsky et al., 2023).”

“Yadlowsky et al. (2023) exhibit vulnerabilities and suffer generalization limitations for functions outside training domains, suggesting ICL capabilities are heavily reliant on training data coverage rather than generalizable inductive biases.”

“Yadlowsky et al. (2023) previously discussed the generalization behavior of ICL models.”

“Other hypotheses are that ICL interpolates among training samples 65 , utilizes compositional structure 66 or recom-bines tasks learned in pre-training 63 .”

“Yadlowskey et al. [4] showed when Transformers are trained for ICL on a mixture of function classes, ICL cannot generalize well to function classes not present in the training mixture.”

“LLMs, or more generally decoder Transformer models, make use of prompting as a way to align their predictions to a particular data mixture seen in pre-training, and thus fail to solve tasks not present in their pre-training or instruction-tuning data [97, 55].”

“Similarly, while transformers excel at learning tasks similar to their pre-training data, their ability to generalise beyond that scope remains limited (Yadlowsky et al., 2023).”

“…for their emergent abilities (Lu et al., 2023) – yet, even with in-context learning, Transformers cannot generalize beyond their pre-training data (Yadlowsky et al., 2023), and their performance varies considerably across languages and is particularly low in languages underrepresented in the…”

“Properties that have been identified as beneficial for the emergence of ICL include burstiness, a large number of rarely occurring classes (Chan et al., 2022), and diverse tasks (Kirsch et al., 2022; Yadlowsky et al., 2023; Raventós et al., 2024).”

“Yadlowsky et al. [54] further showed that pretraining data mixtures enable narrow model selection capabilities: Transformers trained on diverse function classes can implicitly select the appropriate model class from context, although this capability remains limited to classes seen during training.”

“…settings (Min et al., 2022; Aky ¨ urek et al., 2022; Wei et al., 2023; Von Oswald et al., 2023; Singh et al., 2023; Dong et al., 2024), with work probing the roles of training data and demonstrations (Chan et al., 2022; Shin et al., 2022; Yadlowsky et al., 2023; An et al., 2023; Liu et al., 2024).”

“For example, a Meta-ICL model trained on linear and cosine functions fails on their composition (Yadlowsky et al., 2023)—and does not capture higher-order abilities like in-context reasoning (Wei et al., 2022; Kojima et al., 2022) or adaptive self-refinement (Madaan et al., 2023) – unless…”

“Additionally, Razeghi et al. [28] and Yadlowsky et al. [38] linked lower-order co-occurrences to improved numerical reasoning, while McCoy et al. [23] associated task prevalence in pre-training with better performance on tasks like ROT13.”

“…• It is widely believed in ICL researches that during pre-training, LLM models acquire a broad range of semantic prior knowledge from the training data, which later aids task-specific learning representations (Chan et al., 2022; Shin et al., 2022; Yadlowsky et al., 2023; Yang et al., 2024).”

“Yadlowsky et al. [2023] show that when trained on ICL instances where the regression function belongs to a union of distinct function classes, the learned algorithm fails to generalize beyond the pretraining function classes.”

“The extent and nature of general capabilities have also sought to divide the AI community [14, 15, 8], particularly concerning i) the level of understanding that LLMs possess, and ii) the extent of their creative abilities beyond their auto-regressive training [2, 23, 12, 30, 5, 28, 3].”

“Yadlowsky et al. (2023) investigated a framework similar to Garg et al. (2022), where they explored training models on a mixture of function classes; however, they did not delve into curriculum learning strategies.”

“…that a single transformer can adaptively choose between different base algorithms or even qualitatively different tasks (e.g., regression vs. classification) based on the prompt (Bai et al., 2023a), effectively selecting between different function classes in-context (Yadlowsky et al., 2023).”

“The emergence and form of these algorithms can depend on the diversity of pretraining tasks, and transformers can also select among candidate algorithms or task families from the prompt (Raventos et al., 2023; Bai et al., 2023; Yadlowsky et al., 2023).”

“…at filtering documents (Penedo et al., 2023; 2024; Li et al., 2024b; Su et al., 2024) with LLM judgments of quality or re-weighting and re-sampling the mixture of documents (Yadlowsky et al., 2023; Albalak et al., 2023; Xie et al., 2023; Li et al., 2024b; Ye et al., 2025; Wettig et al., 2025).”

“51 Other hypotheses are that ICL interpolates among training samples, 52 utilizes compositional structure, 53 or recombines tasks learned in pre-training.”

“Transformer-based Model: Our final approach treats speaker adaptation as a sequence modelling problem, leveraging trans-formers’ ability to learn from in-context examples [26, 16].”

“Yadlowsky et al. [63] attributed the impressive ICL capabilities of transformers to the diversity and range of data mixtures in their pre-training, rather than their inductive biases for generalizing to new tasks.”

“Specifically, a model trained on trajectories from a random or fixed policy will degrade when evaluated on informative trajectories collected by an active, reward-maximizing deployment policy (Yadlowsky et al., 2023).”

“In particular, properties such as task-family mixtures, long-range coherence, or “burstiness” can shape the implicit meta-learning behavior that appears at inference time (Chan et al., 2022; Xie et al., 2022; Yadlowsky et al., 2023).”

“The basic setup for analyzing ICL was first introduced by Garg et al. (2022) and has since been widely adopted (Yadlowsky et al., 2023; Pan et al., 2023).”

“…solved in a practical sense, and some are open discussion topics, such as their poor generalization under the Out-of-Distribution (OOD) conditions [70], catastrophic loss of dimensionality, i.e., degradation to a rank-1 matrix over multiple layers [21], loss of plasticity and forgetting [49], to…”

“…solved in a practical sense, and some are open discussion topics, such as their poor generalization under the Out-of-Distribution (OOD) conditions (Yadlowsky et al. 2023), catastrophic loss of dimensionality, i.e. degradation to a rank-1 matrix over multiple layers (Dong et al. 2021), loss of…”

“If this generalizability holds true for LLMs is of ongoing debate [9, 10] and could be represented as what we claim as qualitative intelligence.”

“Yadlowsky et al. [63] attributed the impressive ICL capabilities of transformers to the diversity and range of data mixtures in their pre-training, rather than their inductive biases for generalizing to new tasks.”