Language Models are Few-Shot Learners

Existing work on dataset processing varies in the level of detail provided, from no details to detailed filtering instructions with or without code release.

“Existing work tends to fall on a spectrum from no details at all (Brown et al., 2020) to detailed filtering instructions, with (Raffel et al., 2020) or without the dataset release (Rae et al., 2021) to detailed filtering instructions with the accompanying code (Gao et al., 2020; Conneau et al., 2020; Ortiz Suárez et al., 2019).”

The number of reported contributors increased from 25 for GPT-3 to 284 for GPT-4.

“The number of reported contributors increased from 25 for GPT-3 [14] to 284 for GPT-4 [12]”

Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance.

“Few-shot learning works so well with LLMs that it is unclear whether sufficient task-specific data can ever be gathered to train a customized model to rival their performance [3,12].”

ASAP defaults to three shots because related work and the authors' own experiments confirmed that more shots did not significantly improve performance.

“A𝑆𝐴𝑃 defaults to three shots because related work [3,12] has shown, and our own experiments with A𝑆𝐴𝑃 confirmed, that more shots did not significantly improve performance.”

Large language models are recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance.

“LLMs refer to the recent large-scale models with over a billion parameters that exhibit strong few-shot and zero-shot performance [9,16].”

ChatGPT-3 can solve few-shot tasks through in-context learning or sequential reasoning in complex tasks, while GPT-2 or BERT cannot do that well.

“For example, ChatGPT-3 can solve few-shot tasks through in-context learning or sequential reasoning in complex tasks, while GPT-2 or BERT can not do that well.”

The language modeling objective used for many recent large LMs-predicting the next token on a webpage from the internet-is different from the objective 'follow the user's instructions helpfully and safely'.

“This is because the language modeling objective used for many recent large LMs-predicting the next token on a webpage from the internet-is different from the objective 'follow the user's instructions helpfully and safely' (Radford et al., 2019; Brown et al., 2020; Fedus et al., 2021; Rae et al., 2021; Thoppilan et al., 2022).”

Large language models have achieved notable successes, demonstrating significant potential in attaining human-like intelligence.

“In recent years, large language models (LLMs) have achieved notable successes, demonstrating significant potential in attaining human-like intelligence [5][6][7][8][9][10].”

GPT-3 is trained on 175 billion parameters from 45 terabytes of human-created text collected from web pages, Wikipedia, books, and social media data.

“GPT-3 is trained on 175 billion parameters from 45 terabytes of human-created text collected from web pages, Wikipedia, books, and social media data [13].”

LLMs process information within a finite attention window, and their internal representations do not persist across interactions, causing earlier details to be forgotten once they fall outside the active context.

“These models process information within a finite attention window, and their internal representations do not persist across interactions, causing earlier details to be forgotten once they fall outside the active context (Brown et al., 2020; Beltagy et al., 2020a).”

In-context learning is constrained by context window length and lacks persistent memory across sessions.

“ICL is constrained by context window length and lacks persistent memory across sessions”

Although AI outperforms humans in an ever-broadening range of tasks, it trails far behind when it comes to ToM.

“Yet, although AI outperforms humans in an ever-broadening range of tasks, from playing poker (20) and Go (21) to translating languages (22) and diagnosing skin cancer (23), it trails far behind when it comes to ToM.”

GPT-3 has 175 billion parameters and was trained on hundreds of billions of words from the Internet and books.

“The model itself is large (it has 175 billion parameters), and it was trained on a vast amount of data: hundreds of billions of words from the Internet and books.”

Large pre-trained language models struggle with tasks that require unbounded multi-step computation, such as adding integers or executing programs.

“However, they struggle with tasks that require unbounded multi-step computation, such as adding integers (Brown et al., 2020) or executing programs (Austin et al., 2021).”

Using a broad data distribution is the best way to advance general AI.

“In SIMA, however, we are drawing inspiration from the lesson of large language models that training on a broad distribution of data is the most effective way to make progress in general AI (e.g., Brown et al., 2020;Hoffmann et al., 2022;OpenAI, 2023;Anil et al., 2023;Gemini Team et al., 2023).”

Pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks.

“Studies demonstrate that pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks [23,150].”

GPT-3 was trained on a vast corpus of 45 terabytes of text sourced from internet-based entities like Common Crawl and Wikipedia.

“GPT-3's training on a vast corpus of 45 terabytes of text, sourced from a variety of internet-based entities like Common Crawl and Wikipedia, set a precedent for the scale of data utilised in AI development (Brown et al., 2020;Rudolph et al., 2023a).”

Recent pre-trained LLMs such as GPT-3 and ChatGPT can engage in fluent, multi-turn conversations out-of-the-box.

“Recent pre-trained LLMs (e.g., GPT-3 [8], ChatGPT [1]) can engage in fuent, multi-turn conversations outof-the-box”

The only adaptable component of LLMs is their in-context learning ability.

“The only adaptable component of LLMs is their in-context learning ability-a (known to be emergent) characteristic of LLMs that enables fast adaption to the context and so perform zero- or few-shot tasks (Brown et al. 2020).”

LLMs such as GPT-4 and LLaMA rely on massive pre-training datasets sourced from the open Internet.

“LLMs such as GPT-4 (refs. 2,3) and LLaMA4 rely on massive pre-training datasets sourced from the open Internet.”

Language models can potentially suffer from biases because their training data is based on text crawled from the internet that has not specifically been checked for bias.

“Just as with any other type of AI that is based on large amounts of data, language models can potentially suffer from biases. Most models use training data based on text crawled from the internet. Such text generally has not specifically been checked for bias.”

Current mainstream VLMs are pre-trained on large-scale Transformer architectures with billions or tens of billions of parameters, requiring large computational resources and time, limiting training efficiency and scale-up and causing deployment bottlenecks.

“Current mainstream VLMs [209,206,204,208] are usually pre-trained based on large-scale Transformer architectures, which usually have billions or even tens of billions of parameters, and pre-training on large-scale datasets requires a large amount of computational resources and time costs, which limits the training efficiency and scale-up ability of the models, and makes the model deployment face”

Transformer models like Copilot can best be seen as continuing the prompt they were given, so Codex will generate code of similar sophistication to the prompt, including more bugs if prompted with buggy code.

“However, transformer models like Copilot can best be seen as continuing the prompt they were given. This has the effect that Codex will generate code of a similar level of sophistication to the code which it was prompted with, including the possibility of introducing more bugs if prompted with buggy code [22].”

Within the past five years, NLP advancements have achieved huge gains in accuracy on standard datasets, with state-of-the-art accuracy as high as 99% for certain tasks.

“Within the past five years, advancements in NLP have achieved huge gains in accuracy when tested against standard datasets [8, 18, 34, 56, 58, 60, 63], with state-ofthe-art accuracy in natural language processing models as high as 99% for certain tasks [8, 34].”

In-context learning facilitates adaptation to novel tasks without parameter updates by leveraging demonstration examples within prompts, with performance significantly influenced by example selection and ordering strategies.

“Incontext learning facilitates adaptation to novel tasks without parameter updates by leveraging demonstration examples within prompts, with performance significantly influenced by example selection and ordering strategies [369, 103, 1296, 1024, 928, 852, 1148, 352, 582].”

Few-shot prompting extends zero-shot capability by incorporating limited exemplars to guide model responses, demonstrating task execution through strategic example selection.

“Few-shot prompting extends this capability by incorporating limited exemplars to guide model responses, demonstrating task execution through strategic example selection [1371, 405, 103, 552, 794, 1381].”

Modern LLM short-term memory frequently manifests as in-context learning, reflecting models' ability to acquire and process information temporarily within context windows.

“Modern LLM short-term memory frequently manifests as in-context learning, reflecting models' ability to acquire and process information temporarily within context windows [1189, 103].”

The scaling up of network and training data leads to leap-forward capabilities in LLMs, including emergent abilities such as in-context learning, instruction following, and step-by-step reasoning.

“The scaling up of network and training data lead to the leap-forward of large language models' capability.”

Many works trace their methodologies for prompted evaluation to Brown et al. (2020), who thoroughly document the majority of their evaluation setups, despite not releasing code.

“Many works, for example, trace their methodologies for prompted evaluation to Brown et al. (2020), who rather thoroughly document the majority of their evaluation setups, despite not releasing code.”

Token-length normalization requires no additional LM calls and is used alternately with raw loglikelihoods for most tasks by Brown et al. (2020).

“Token-length normalization : each a i 's loglikelihood is divided by m i , its length in tokens, to gain the per-token loglikelihood of each answer. This approach requires no additional LM calls, and is used alternately with raw loglikelihoods for most tasks by Brown et al. (2020).”

Prompt learning methods can adapt pre-trained transformers to target domains by training very few parameters and achieve comparable performance with finetuning the whole heavy model.

“prompt learning methods [2,5,8,16,18,32,42] emerge recently as efficient and effective solutions for adapting pre-trained transformers to the target domain via only training very few parameters (i.e., prompts), and achieve comparable performance with finetuning the whole heavy model.”

The choice of examples can drastically alter the output of large language models.

“Brown et al. [5] highlighted that the choice of examples can drastically alter the output of large language models.”

LLMs' offline training paradigm often limits access to customized and up-to-date information.

“their offline training paradigm often limits access to customized and up-to-date information.”

Transformers' computational efficiency and scalability have made it possible to train models of unprecedented size, with over 100B parameters.

“Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al., 2020; Lepikhin et al., 2020).”

In-context learning learns a new task via inference alone by conditioning on a concatenation of training data as demonstrations, without gradient updates.

“the LM learns a new task via inference alone by conditioning on a concatenation of the training data as demonstrations, without any gradient updates.”

A limiting factor for in-context learning (ICL) in LLMs is the context window, restricting prior research to the few-shot ICL regime.

“A limiting factor for in-context learning (ICL) in LLMs is the context window, restricting prior research to the few-shot ICL regime.”

Brown et al. reported improved performance as you increase the number of examples up to 64 for in-context learning in LLMs, and later works corroborated this finding.

“Brown et al. [8] reported improved performance as you increase the number of examples (up to 64) for in-context learning in LLMs , and later works corroborated this finding [39].”

ACE offers a flexible and efficient alternative to conventional model finetuning, as adapting contexts is generally cheaper than updating model weights.

“ACE offers a flexible and efficient alternative to conventional model finetuning, as adapting contexts is generally cheaper than updating model weights”

ChatGPT's superiority over its GPT-based predecessors can be linked to its ability to respond to multiple languages and generate refined and highly sophisticated responses based on advanced modeling.

“The superiority of ChatGPT compared to its GPT-based predecessors can be linked to its ability to respond to multiple languages generating refined and highly sophisticated responses based on advanced modeling [6,7].”

ChatGPT is based on the generative pre-trained transformer (GPT) architecture.

“Developed by OpenAI (OpenAI, L.L.C., San Francisco, CA, USA), ChatGPT etymology is related to being a chatbot (a program able to understand and generate responses using a text-based interface) and is based on the generative pre-trained transformer (GPT) architecture [6,7].”

Prompting methods are particularly useful in few-shot learning because there are generally not enough training examples to fully specify the desired behavior.

“Prompting methods are particularly useful in the latter case [9,32,117], as there are generally not enough training examples to fully specify the desired behavior, and thus using a prompt to push the model in the right direction is particularly effective.”

Large-scale pre-training allows LMs to handle linguistic phenomena such as analogies, negations, semantic role sensitivity, semantic similarity, cant understanding, and rare word understanding.

“Besides factual knowledge, large-scale pre-training also allows LMs to handle linguistic phenomena such as analogies [9], negations [25], semantic role sensitivity [25], semantic similarity [131], cant understanding [131], and rare word understanding [116].”

Pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger.

“Within the context of pre-trained LMs, researchers have found that pre-trained embeddings and LMs can perform simple operations such as addition and subtraction when the number of digits is small but fail when the numbers are larger [9,88,139].”

The increasingly large context length of LLMs enables in-context learning (ICL) by incorporating more documents, enhancing the model's ability to capture intricate writing patterns.

“The increasingly large context length of LLMs enables in-context learning (ICL) [Brown et al., 2020] by incorporating more documents, enhancing the model's ability to capture intricate writing patterns.”

Generative AI models such as GPT-3 and DALL·E remain reactive generating content solely in response to prompts without maintaining a persistent state or engaging in iterative reasoning.

“Generative AI models such as GPT-3 [120] and DALL·E https://openai.com/index/dall-e-3/, remain reactive generating content solely in response to prompts without maintaining a persistent state or engaging in iterative reasoning.”

These networks are trained through unsupervised machine learning on enormous amounts of text from various sources, predominantly open-domain web resources, with academic content constituting only a small fraction.

“These networks are trained through unsupervised machine learning methods on enormous amounts of text from various sources, predominantly open-domain web resources, with academic content constituting only a small fraction (Brown et al., 2020).”

Many current coding benchmarks are susceptible to contamination because widely used open-source repositories are prime candidates for inclusion in large-scale web-crawled pre-training corpora.

“First, many benchmarks are susceptible to contamination [7, 13, 15, 20], as exemplified by recent works [5, 7, 15] and social media posts [1, 19]. This risk arises because widely used open-source repositories-particularly those distributed under permissive licenses (e.g., MIT, Apache 2.0, BSD)-are prime candidates for inclusion in the large-scale web-crawled corpora used to pre-train LLMs [3].”

Prompt-tuned T5-Small matches GPT-3 XL (over 16 times larger), and prompt-tuned T5-Large beats GPT-3 175B (over 220 times larger).

“Figure 1 shows that prompt tuning beats GPT-3 prompt design by a large margin, with prompttuned T5-Small matching GPT-3 XL (over 16 times larger), and prompt-tuned T5-Large beating GPT-3 175B (over 220 times larger).”

Fine-tuning improves model performance drastically compared to few-shot learning on datasets large and small.

“As shown in Table 8, fine-tuning improves the model performance drastically compared to few-shot learning on datasets large and small.”

GPT-3 175B can adapt its behavior with just a few additional training examples, but the result depends heavily on the input prompt.

“While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily on the input prompt (Brown et al., 2020).”

Pretrained LLMs cannot solve addition and parity robustly without careful prompting.

“even pretrained LLMs cannot solve these tasks robustly (Brown et al., 2020;Chowdhery et al., 2022;Anil et al., 2022) without careful prompting (Zhou et al., 2022b).”

Agentic evaluations are inherently stochastic processes.

“Agentic evaluations are inherently stochastic processes.”

Underlying LLM models sample from probability distributions rather than deterministically computing outputs, introducing trial-to-trial variance.

“The underlying LLM models sample from probability distributions rather than deterministically computing outputs (Brown et al. 2020; Holtzman et al. 2020; He and Lab 2025), introducing trial-to-trial variance..”

Modern language models often have hundreds of billions of parameters.

“Following this trend, modern language models often have hundreds of billions of parameters (Brown et al., 2020; Rae et al., 2021; Zeng et al., 2021; Kim et al., 2021).”

Using demonstrations with gold labels significantly improves performance over no demonstrations.

“using the demonstrations with gold labels significantly improves the performance over no demonstrations”

Copilot is powered by the OpenAI Codex family of models, derived by fine-tuning GPT-3 on public GitHub repositories.

“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”

GPT models have shown exceptional performance in text generation-related tasks and significant few-shot learning capabilities.

“These models have shown exceptional performance in text generation-related tasks, and significant few-shot learning capabilities.”

Training the 175B SFT model requires 4.9 petaflops/s-days, training the 175B PPO-ptx model requires 60 petaflops/s-days, and GPT-3 required 3,640 petaflops/s-days.

“training our 175B SFT model requires 4.9 petaflops/s-days and training our 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for GPT-3 (Brown et al., 2020).”

Copilot is powered by the OpenAI Codex family of models, which are derived by fine-tuning GPT-3 on publicly available GitHub repositories.

“Copilot is powered by the OpenAI Codex family of models [Chen et al. 2021], which are derived by fine-tuning GPT-3 [Brown et al. 2020] on publicly available Github repositories.”

The LLM used in this work (GPT-3) is open to public use with easy registration and no user education.

“The LLM used in this work (i.e., GPT-3 [13]) is open to public use with easy registration and no user education.”

Large language models have limitations such as lack of interpretability and potential for bias, and unexpected brittleness in relatively simple tasks.

“However, as previously mentioned, the use of these models should be done with caution, as they also have limitations such as lack of interpretability and potential for bias, unexpected brittleness in relatively simple tasks [9] which need to be addressed.”

LLMs are limited in their ability to maintain and reason over long-term context.

“Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks (Brown et al., 2020; Achiam et al., 2023; Wei et al., 2022), yet they remain limited in their ability to maintain and reason over long-term context.”

GPT-3 is an autoregressive language model that uses the transformer architecture.

“GPT-3 is an autoregressive language model (2), which utilizes the transformer architecture (9)—a deep learning model that heavily relies on the mechanism of self-attention—to produce human-like text.”

Very large models may be able to perform additions and multiplications with 3 or fewer digits few-shot.

“In this experiment, we evaluate in the few-shot regime using a 137B parameter pre-trained decoder-only model, as previous work indicates that very large models may be able to perform additions and multiplications with 3 or fewer digits few-shot (Brown et al., 2020).”

GPT-3 struggles to perform few-shot addition on numbers with greater than three digits.

“For example, GPT-3 struggles to perform few-shot addition on numbers with greater than three digits (Brown et al., 2020).”

Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text or synthesizing computer programs.

“Large pre-trained language models perform remarkably well on tasks that can be done 'in one pass', such as generating realistic text (Brown et al., 2020) or synthesizing computer programs (Chen et al., 2021; Austin et al., 2021).”

LLMs have evolved to exhibit graduate-level STEM proficiency, generate hypotheses, synthesize literature, and draft manuscripts.

“From simple next-token predictors [1, 2], large language models (LLMs) have evolved to exhibit graduate-level STEM proficiency [3-5], generate hypotheses [6, 7], synthesize literature [8], and draft manuscripts [9].”

Min et al. has shown that in-context learning works by activating patterns from the training data.

“Min et al. [44] has shown that rather than learning new tasks, in-context learning [42] works by activating patterns from the training data.”

Pretrained language models, especially large ones, appear to have significantly improved sample efficiency when learning new tasks.

“However, pretrained language models, especially large ones, appear to have significantly improved sample efficiency when learning new tasks [1,59,60]”

Early work on few-shot learning and prompt engineering found that the LLM could be further biased towards high-quality productions by pre-processing the input string.

“Early work on few-shot learning (Brown et al., 2020) and prompt engineering (Wei et al., 2022b; Kojima et al., 2022; Xu et al., 2023c) found that the LLM could be further biased towards high-quality productions by pre-processing the input string.”

GPT-3 unlocked flexible few-shot and zero-shot reasoning for NLP tasks.

“GPT-3 (Brown et al., 2020) unlocked flexible few-shot and zero-shot reasoning for NLP tasks”

LLM workloads contain massive prefix redundancy: system prompts repeated across thousands of requests, few-shot examples, retrieval-augmented contexts.

“LLM workloads contain massive prefix redundancy: system prompts repeated across thousands of requests, few-shot examples [32], retrieval-augmented contexts.”

In a different but plausible world in which MMLU-style prompting was the standard introduced in Brown et al. (2020), the perceived performance of GPT-NeoX-20B and other models trained on the Pile would be extremely low, despite strong results that would be observed in the 'cloze' setting.

“in a different but plausible world in which MMLU-style prompting was the standard introduced in Brown et al. (2020), the perceived performance of GPT-NeoX-20B and other models trained on the Pile (Gao et al., 2020) would be extremely low, despite strong results that would be observed in the 'cloze' setting.”

Each object is associated with fully simulatable object states (e.g., apple with cooked and sliced, but not toggledOn).

“Through crowdworkers, students, and GPT-3 [58], we also associate each object with our fully simulatable object states: for example, apple is associated with cooked and sliced , but not toggledOn .”

All properties that apply to many of the 2,964 leaf synsets and therefore require large-scale annotation are done by either GPT-3 or crowdworkers.

“All properties that apply to many of the 2,964 leaf synsets and therefore require a large-scale annotation are done by either GPT-3 [58] or crowdworkers.”

LLMs consist of neural networks trained on extensive text corpora to identify probable word and language patterns, which are then stored as weights within the neural network.

“LLMs consist of neural networks trained on extensive text corpora to identify probable word and language patterns, which are then stored as weights within the neural network (Brown et al., 2020; Vaswani et al., 2017).”

LLMs are designed to predict probable language outputs in response to specific prompts, such as predicting the most likely word following a sequence of words or generating sentences or paragraphs as plausible responses to a given textual prompt.

“Large language models (LLMs) are designed to predict probable language outputs in response to specific prompts, such as predicting the most likely word following a sequence of words or generating sentences or paragraphs as plausible responses to a given textual prompt.”

Language models reproduce stereotypes and biases.

“Initial audits also highlight that language models reproduce stereotypes and biases [23, 54, 83]”

ChatGPT is a variant of the GPT-3 artificial intelligence language model developed by OpenAI.

“ChatGPT is a variant of the GPT-3 (Generative Pre-trained Transformer 3, Brown et al., 2020) artificial intelligence language model developed by OpenAI.”

In-context learning is a key ability that enables LLMs to adapt to new tasks on the fly.

“In-context learning (ICL) is a key ability that enables LLMs to adapt to new tasks on the fly (Brown et al., 2020).”

Providing examples of desired interactions in prompts substantially improved GPT-3's performance on a battery of tasks such as question answering and language translation.

“The original GPT-3 paper demonstrated that examples substantially improved the performance of GPT-3 on a battery of tasks such as question answering and language translation [9].”

Large language models have demonstrated exceptional language generation capabilities across various domains.

“Recently, large language models (LLMs) (Brown et al., 2020;Ouyang et al., 2022;OpenAI, 2023) have demonstrated exceptional language generation capabilities across various domains.”

LLM approaches rely on training on as large and varied a collection of language data as possible to extract general properties of a language.

“These practical applications of language technology are increasingly reliant on approaches based on trained Large Language Models (LLMs) [23, 41, 104, 111], whereby models are first exposed to as large and varied a collection of language data as possible with the aim of extracting 'general' properties of a language of interest.”

Pretrained language models can recall and use stored information more accurately as their scale increases.

“they can recall and use this information more accurately with increasing scale (Brown et al., 2020).”

Brown et al. (2020) report that the GPT-3 model with 175 billion parameters used 3.14·10^23 floating point operations of compute to train using NVIDIA V100 GPUs on a cluster provided by Microsoft.

“Brown et al. (2020) report that the GPT-3 model with 175 billion parameters used 3 . 14 · 10 23 floating point operations (FPOs) of compute to train using NVIDIA V100 GPUs on a cluster provided by Microsoft.”

Foundation models aim to supply broad knowledge and skills that can be adapted through fine-tuning or prompting.

“These models are designed to provide a strong foundation of general knowledge and skills that can be built upon and adapted to new situations via fine-tuning or prompting with demonstrations (Brown et al., 2020).”

Foundation models usually need large and varied datasets to reach their generality.

“Foundation models typically require large, diverse datasets to achieve their generality (Brown et al., 2020; Mahajan et al., 2018; Schuhmann et al., 2022; Sun et al., 2017; Zhai et al., 2022).”

Few-shot prompting provides a model with a prompt consisting of k in-context examples of the target task, each in the form of input-output pairs.

“Few-shot prompting (also referred to as”

API calls to a publicly available LLM expose the study to variable inference time calls and unforeseen model updates.

“We utilize API calls to a publicly available LLM (Brown et al., 2020) which, whilst sufficient in the context of our investigation, exposes us to variable inference time calls and unforeseen model updates.”

Few-shot prompting instructs the language model to solve a task with several input-output demonstrations.

“Few-shot prompting instructs the language model to solve a task with several input-output demonstrations (Brown et al., 2020).”

The capabilities of transformer-based language models significantly improve with model size.

“Because the capabilities of these models significantly improve with model size (Brown et al., 2020; Radford et al., 2019; Tamkin et al., 2021)”

Developers of models like GPT-3 seem to be aware of the bias problem and discuss gender, race, and religious biases in their article.

“Developers of models like GPT-3 seem to be aware of this problem and discuss gender, race, and religious biases in their article (Brown et al., 2020).”

The language module uses BioGPT, an autoregressive generative model based on the Transformer decoder architecture of GPT-3, trained on 15 million biomedical articles from PubMed with a vocabulary size of 42,384.

“Our language module uses BioGPT23, an autoregressive generative model based on the Transformer24 decoder architecture of GPT-325, trained on 15 million biomedical articles from PubMed with a vocabulary size of 42,384.”

State-of-the-art LLMs are generally implemented using the Transformer architecture, a multi-layer feedforward network with attention mechanism.

“State-of-the-art LLMs [66, 67, 7, 52] are generally implemented using the Transformer architecture, which is a multi-layer feedforward network equipped with attention mechanism [71].”

LLMs store extensive knowledge during pre-training and recall it during inference.

“Large language models (LLMs) have demonstrated the capability to store extensive knowledge during pre-training and recall it during inference”

The emergence of large language models, represented by GPT-3, marks an important milestone for the AI community.

“There has been tremendous progress in language models in recent years, where the emergence of large language models, represented by GPT-3 [3], marks an important milestone for the entire AI community.”

Given a natural language instruction and task demonstrations, LLMs generate output by completing the word sequence without training or tuning.

“In-context learning [167-173] is one of the emergent abilities of LLMs that differentiate LLMsfrompreviouspre-trainedlanguagemodels,where,givenanaturallanguageinstruction and task demonstrations, LLMs would generate the output by completing the word sequence without training or tuning [3].”

Large language models are widely known to leverage extensive amounts of open-domain knowledge during their training and fine-tuning processes, including Wikipedia, books, and various websites.

“Large language models are widely known to leverage extensive amounts of open-domain knowledge during their training and fine-tuning processes. These knowledge sources include well-known references such as Wikipedia, books, and various websites [3].”

The mutual information approach is nonstandard, provided in lm-eval under acc mutual info, and used selectively by Brown et al. (2020) and Askell et al. (2021) for certain tasks.

“Although this approach is nonstandard, it is provided in lm-eval under the option acc mutual info , and used selectively by Brown et al. (2020) and Askell et al. (2021) for certain tasks.”

The three primitive operations implemented in lm-eval are sufficient to implement the major ways in the literature that have been used to evaluate LMs.

“Provided with these three primitive operations, we are able to implement the major ways in the literature that have been used to evaluate LMs (Gao et al. (2020), Brown et al. (2020), inter alia ).”

AI-generated content (from GPT-3 and similar models) can be difficult to distinguish from human-written content.

“As these models are trained on human-written texts, the content generated by these tools can be quite difficult to distinguish from human-written content.”

Brown et al. first showed that GPT-3 has the ability to learn from a few examples in the context without parameter update.

“Brown et al. [11] first show that GPT-3 has the ability to learn from a few examples in the context without parameter update.”

The study relied on a zero-shot prompting strategy.

“While we relied on a zero-shot strategy (see Brown et al., 2020)”

This inverse scaling contrasts with other NLP tasks, where performance improves with model size.

“This contrasts with other NLP tasks, where performance improves with model size.”

Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100×.

“Exploring many-shot ICL is now feasible, given the recent increase in context windows of publicly available LLMs by at least 100 × : from only a few thousand tokens in GPT-3 [8] and Llama 2 [57] to 1M tokens in Gemini 1.5 Pro [16].”

The paper's conclusions, especially regarding consciousness, apply to other models such as BERT, GPT-3, T5 and future models using the same or similar architectures.

“This means that our conclusions, especially regarding consciousness, apply to other models such as BERT [1], GPT-3 [2], T5 [4] and future models using the same, or similar, architectures.”

Larger models are more capable.

“Behaving in an aligned fashion is just another capability, and many works have shown that larger models are more capable [Kaplan et al., 2020, Rosenfeld et al., 2019, Brown et al., 2020], finetune with greater sample efficiency [Henighan et al., 2020, Askell et al., 2021], and do not suffer significantly from forgetting [Ramasesh et al., 2022].”

The large-scale pretraining of LLMs equips them with vast knowledge across a wide range of domains.

“First, the large-scale pretraining of LLMs equips them with vast knowledge across a wide range of domains (Brown et al. 2020, Touvron et al. 2023).”

The GPT architecture utilizes a neural network to process natural language, generating responses based on the context of input text.

“The GPT architecture utilizes a neural network to process natural language, thus generating responses based on the context of input text [7].”

In-context learning is a key mechanism for foundation model programming.

“In-context learning (McCann et al. 2018; Radford et al. 2018; Brown et al. 2020) is a key mechanism for foundation model programming.”

Chinchilla, a 70B-parameter model trained on 1.4 trillion tokens, outperformed Gopher, GPT-3, and others across a broad benchmark suite under a matched compute budget.

“Chinchilla outperformed the much larger Gopher [Rae et al., 2021], GPT-3 [Brown et al., 2020], and others [Smith et al., 2022] across a broad benchmark suite”

Few-shot demonstrations take advantage of the ability of strong language models to learn repetitive patterns.

“These few-shot demonstrations take advantage of the ability of strong language models to learn repetitive patterns [9].”

Qwen3 models utilize Qwen's tokenizer, which implements byte-level byte-pair encoding (BBPE) with a vocabulary size of 151,669.

“Qwen3 models utilize Qwen's tokenizer (Bai et al., 2023), which implements byte-level byte-pair encoding (BBPE, Brown et al., 2020;Wang et al., 2020;Sennrich et al., 2016) with a vocabulary size of 151,669.”

LLMs have shown promising performances in various classification tasks.

“LLMs have shown promising performances in various classification tasks (Brown et al., 2020), and prior work (Don-Yehiya et al., 2024; Shaikh et al., 2025) has also explored prompting LLMs (specifically GPT-4o-mini) to classify user feedback in multi-turn user-LLM interactions.”

In-context learning allows the model to learn from input examples, enabling it to understand and manage each new task.

“In-context learning (Brown et al., 2020) allows the model to learn from input examples, enabling it to understand and manage each new task.”

Increasing the number of parameters improved model performance.

“We note that increasing the number of parameters improved model performance, consistent with recent advances in NLP7.”

Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing and were also recently applied to model short DNA sequences.

“Transformers are a class of deep learning models that have achieved substantial breakthroughs in natural language processing (NLP)6,7 and were also recently applied to model short DNA sequences8.”

Prompt tuning, which learns soft prompts through backpropagation to condition a frozen language model, outperforms GPT-3's few-shot learning by a large margin.

“Our end-to-end learned approach outperforms GPT-3's few-shot learning by a large margin.”

Many recently trained large models have been trained for approximately 300 billion tokens.

“Following Kaplan et al. (2020) and the training setup of GPT-3 (Brown et al., 2020), many of the recently trained large models have been trained for approximately 300 billion tokens (Table 1), in line with the approach of predominantly increasing model size when increasing compute.”

Pre-training predominantly utilizes a transformer-based architecture following the paradigm established by GPT, and further developed by OPT, Falcon, and Llama-2.

“Pre-training constitutes the foundational stage for LLMs, predominantly utilizing a transformer-based architecture following the paradigm established by GPT [29, 251, 252], and further developed by OPT[372], Falcon [243], and Llama-2 [300].”

The hypothesis that episodic memories could lead to unpredictable and undesirable behavior has a strong parallel in existing work on the effects of examples given to large language models.

“Our hypothesis that episodic memories could be recalled and used in ways which lead to unpredictable and potentially undesirable agent behavior has a strong parallel in existing work on the effects of examples given to large language models.”

AI agents are able to both search the internet and learn new skills in this manner today.

“AI agents are able to both search the internet [119], [120] and learn new skills in this manner today [97], [121].”

LLMs generate outputs by predicting likely token sequences based on patterns learned from large-scale training data.

“At the core of these systems are large language models (LLMs), which generate outputs by predicting likely token sequences based on patterns learned from large-scale training data (Brown et al. 2020;Radford et al. 2018;Vaswani et al. 2017).”

The official estimate indicates that GPT-3 consumes an order of 0.4 kWh of electricity to generate 100 pages of content, equivalent to roughly 0.004 kWh per page.

“The official estimate indicates that GPT-3 consumes an order of 0.4kWh of electricity to generate 100 pages of content, equivalent to roughly 0.004kWh per page.4”

LLMs often need training to achieve the best results.

“While LLMs achieve high quality on many problems with simple prompt engineering (Brown et al., 2020), they often need training to achieve the best results.”

Reaching a point in 3D space is feasible using motion planning because knowledge of a robot body is often known.

“Knowledge of a robot body is often known, hence reaching a point in the 3D space is feasible using motion planning [53, 59, 60].”

GPT-3 is used for all properties (total of eight) for which the Hamming distance and false-positive rate for a sample compared to a human-annotated ground-truth are both less than 10%.

“GPT-3 is used for all properties (total of eight) for which the Hamming distance and false-positive rate for a sample compared to a human-annotated ground-truth are both less than 10%.”

NLP modeling (including Transformer) does not require a deep structure.

“empirical observations indicate NLP modeling (including Transformer) does not require a deep structure (Vaswani et al., 2017; Brown et al., 2020)”

MoE-LLaVA uses a Mixture of Experts architecture that activates only relevant expert networks, reducing computational burden and improving reasoning efficiency on large datasets.

“MoE-LLaVA [208] uses a Mixture of Experts (MoE) architecture, activating only the relevant expert networks to reduce computational burden and improve reasoning efficiency, especially on large datasets.”

All HistoGPT models consistently produced text with high cosine similarity to the ground truth, according to the sentence embeddings provided by BioBERT and GPT-3-ADA.

“All HistoGPT models consistently produced text with high cosine similarity to the ground truth, according to the sentence embeddings provided by BioBERT33 and GPT-3-ADA25 (Fig. 3d).”

Codex can be prompted to generate racist, denigratory, and otherwise harmful outputs as code comments, and can generate code with structure reflecting stereotypes.

“The developers of Codex note that it can be prompted in ways that 'generate racist, denigratory, and otherwise harmful outputs as code comments' [22], and that it can generate code with structure that reflects stereotypes about gender, race, emotion, class, the structure of names, and other characteristics [22].”

These components are fundamental for task decomposition, logical inference, action execution, and iterative improvement, following established patterns in agentic systems literature.

“These components are fundamental for task decomposition, logical inference, action execution, and iterative improvement, following established patterns in agentic systems literature.”

Codex is more powerful than CodeGPT because it is a large-scale pretrained model trained on all source codes in GitHub based on GPT-3.

“It worth noting that Codex is more powerful than CodeGPT since it is a large-scale pretrained model that is trained on all source codes in GitHub based on GPT-3 (Brown et al., 2020).”

Virtually all existing architectures for dense prediction are based on convolutional networks.

“Virtually all existing architectures for dense prediction are based on convolutional networks [6,31,34,42,49,50,53].”

Most prior large language models have been trained for a single epoch.

“however, most prior large language models have been trained for a single epoch (Komatsuzaki, 2019;Brown et al., 2020)”

Using in-context learning, these methods generate reasonable plans in structured language, but require pre-defined action primitives to execute.

“Using in-context learning [42], these methods generate reasonable plans in structured language, but requires pre-defined action primitives to execute.”

Models seem to often be able to use novel words appropriately.

“Models also seem to (often) be able to use novel words appropriately [69, 73].”

GPT-3 performs well on two-digit addition and subtraction but not on three-digit addition or two-digit multiplication.

“GPT-3 performs well on two-digit addition and subtraction but not on more complex tasks, such as three-digit addition or two-digit multiplication [69].”

Large language models have demonstrated remarkable progress in understanding human expression, reasoning, few-shot learning, and incorporating extensive world knowledge.

“With the unprecedented scale of training and model parameters, the capability of large language models has been dramatically improved, leading to human-like performances in understanding, language synthesizing, common-sense reasoning, etc.”

Language models are mostly built upon deep neural networks, with the majority based on transformers or close variations, and are pre-trained at scale with massive unlabeled corpus then fine-tuned with task-oriented data.

“Nowadays, language models are mostly built upon deep neural networks, where two features need to be emphasized. First of all, the majority of language models are based on transformers or their close variations [28, 29].”

OpenAI has been expanding model parameters and training data while employing RLHF and Instruction Tuning to further fine-tune GPT-3, leading to emergent abilities of models like InstructGPT and ChatGPT.

“OpenAI has been expanding the model parameters and training data while employing techniques such as RLHF (Reinforcement Learning from Human Feedback) and Instruction Tuning to further fine-tune GPT-3 [3]. This has led to the emergent abilities of models like InstructGPT [37] and subsequent models like ChatGPT”