Eight Things to Know about Large Language Models

“Modern LLMs are built on artificial neural networks: They work by computing and updating numeric activation values for internal components that are very loosely modeled on human neurons (Bengio et al., 2017).”

“…translation of ‘cat’ in French is‘ ”, such that a typical continuation of the text should represent a completion of the intended task (Radford et al., 2019; Raffel et al., 2020). match high-quality human demonstrations on the task (Radford et al., 2018; Devlin et al., 2019; Ouyang et al., 2022).”

“Large language models (LLMs, e.g. GPT-3, PALM, LLaMA, and GPT-4; Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; OpenAI, 2023b) and products built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymaker…”

“This is analogous to the techniques used to produce superhuman performance at games like Go (Silver et al., 2016).”

“…arise with the development of deployment of large-scale AI systems, including issues around environmental impacts, access, mis-use, privacy, safety, and the concentration of power (Amodei et al., 2016; Bender et al., 2021; Bommasani et al., 2021; Birhane et al., 2022; Weidinger et al., 2022, i.a.).”

“Large language models (LLMs, e.g. GPT-3, PALM, LLaMA, and GPT-4; Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; OpenAI, 2023b) and products built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymaker…”

“…arise with the development of deployment of large-scale AI systems, including issues around environmental impacts, access, mis-use, privacy, safety, and the concentration of power (Amodei et al., 2016; Bender et al., 2021; Bommasani et al., 2021; Birhane et al., 2022; Weidinger et al., 2022, i.a.).”

“Large language models (LLMs, e.g. GPT-3, PALM, LLaMA, and GPT-4; Brown et al., 2020; Chowdhery et al., 2022; Touvron et al., 2023; OpenAI, 2023b) and products built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymaker…”

“Often, ad-hoc techniques that at first seem to provide insight into the behavior of an LLM are later found to be severely misleading (Feng et al., 2018; Jain & Wallace, 2019; Bolukbasi et al., 2021; Wang et al., 2022).”

“The chain-of-thought reasoning strategies mentioned in Section 2 are an especially clear example of this: Simply prompting a model to “think step by step” can lead it to perform well on entire categories of math and reasoning problems that it would otherwise fail on (Kojima et al., 2022).”

“Scaling law results (Kaplan et al., 2020; Brown et al., 2020; Hoffmann et al., 2022) have been a major driving factor in the recent surge of research and investment into LLMs (Ganguli et al., 2022a).”

“This can include the ability to internally represent the spatial layout of the setting of a story (Patel & Pavlick, 2022; Bubeck et al., 2023).”

“Indeed, many control techniques work better with larger models, at least for simple tasks(Hendrycks et al., 2020; Bai et al., 2022a; Chung et al., 2022; Ganguli et al., 2023).”

“…arise with the development of deployment of large-scale AI systems, including issues around environmental impacts, access, mis-use, privacy, safety, and the concentration of power (Amodei et al., 2016; Bender et al., 2021; Bommasani et al., 2021; Birhane et al., 2022; Weidinger et al., 2022, i.a.).”

“…al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), integration with image processing systems, (Alayrac et al., 2022; OpenAI, 2023b), or integration with other software tools (Nakano et al., 2021; Menick et al., 2022; Collins et al., 2022; Schick et al., 2023; OpenAI, 2023a).”

“…built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymakers (J & C, 2023; Bartz, 2023; Lieu, 2023), and scholars from many fields (Chan, 2022; Lund & Wang, 2023; Choi et al., 2023; Biswas, 2023).”

“…complete a task when asked, but will then perform the task correctly once the request is reworded or reframed slightly, leading to the emerging craft of prompt engineering (Brown et al., 2020; Reynolds & McDonell, 2021; Radford et al., 2021; Dohan et al., 2022; White et al., 2023; Si et al., 2023).”

“LLMs can memorize specific examples or strategies for solving tasks from their training data without internalizing the reasoning process that would allow them to do those tasks robustly (see, e.g. McCoy et al., 2019; Magar & Schwartz, 2022).”

“LLMs augmented with additional tools can extend this into grounded domains like robotics (Sharma et al., 2022; Driess et al., 2023).”

“show that the tasks in BIG-Bench (Srivastava et al., 2022), a standard broad-coverage benchmark for LLM abilities, show a range of different kinds of trend that collectively make scaling-law style predictions unreliable (Figure 3).”

“…built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymakers (J & C, 2023; Bartz, 2023; Lieu, 2023), and scholars from many fields (Chan, 2022; Lund & Wang, 2023; Choi et al., 2023; Biswas, 2023).”

“Sometimes the details of these failures cast doubts on the quality of other related evaluations (Webson & Pavlick, 2022; Ullman, 2023).”

“Indeed, many control techniques work better with larger models, at least for simple tasks(Hendrycks et al., 2020; Bai et al., 2022a; Chung et al., 2022; Ganguli et al., 2023).”

“In particular, popular approaches involving reinforcement learning and red-teaming allow model developers to guide models toward a persona and set of values more or less of their choosing (Dinan et al., 2019; Bai et al., 2022a; Ganguli et al., 2022b).”

“Often, a model will fail to complete a task when asked, but will then perform the task correctly once the request is reworded or reframed slightly, leading to the emerging craft of prompt engineering (Brown et al., 2020; Reynolds & McDonell, 2021; Radford et al., 2021; Dohan et al., 2022; White et al., 2023; Si et al.…”

“Instead, the qualitative differences between them stem from vast differences in scale: Training GPT-3 used roughly 20,000× more computation than training the original GPT (Sevilla et al., 2022), as well as significantly more data and parameters.”

“Similarly, as noted in Section 7, recent methods can dramatically reduce explicit bias and toxicity in models’ output, largely by exploiting the fact that models can often recognize these bad behaviors when asked (Dinan et al., 2019; Bai et al., 2022b; Ganguli et al., 2023).”

“LLMs can memorize specific examples or strategies for solving tasks from their training data without internalizing the reasoning process that would allow them to do those tasks robustly (see, e.g. McCoy et al., 2019; Magar & Schwartz, 2022).”

“…arise with the development of deployment of large-scale AI systems, including issues around environmental impacts, access, mis-use, privacy, safety, and the concentration of power (Amodei et al., 2016; Bender et al., 2021; Bommasani et al., 2021; Birhane et al., 2022; Weidinger et al., 2022, i.a.).”

“…al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), integration with image processing systems, (Alayrac et al., 2022; OpenAI, 2023b), or integration with other software tools (Nakano et al., 2021; Menick et al., 2022; Collins et al., 2022; Schick et al., 2023; OpenAI, 2023a).”

“• Models can distinguish common misconceptions from true facts (Wei et al., 2022a), and often show well-calibrated internal representations for how likely a claim is to be true (Kadavath et al., 2022; Wei et al., 2022a; Burns et al., 2023).”

“…built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymakers (J & C, 2023; Bartz, 2023; Lieu, 2023), and scholars from many fields (Chan, 2022; Lund & Wang, 2023; Choi et al., 2023; Biswas, 2023).”

“…learning on practical tasks appears to have been discovered only after it was trained, and its capacity for chain-of-thought reasoning was discovered only several months after it was broadly deployed to the public (Nye et al., 2021; Wei et al., 2022b; Kojima et al., 2022; Zhou et al., 2023).”

“• Similarly, questions of consciousness, sentience, rights, and moral patienthood in LLMs (see, e.g. Schwitzgebel & Garza, 2020; Shevlin, 2021; Chalmers, 2023), are worth distinguishing from the issues above.”

“LLMs augmented with additional tools can extend this into grounded domains like robotics (Sharma et al., 2022; Driess et al., 2023).”

“• Models can make inferences about what the author of a document knows or believes and use these inferences to predict how the document will be continued (Andreas, 2022).”

“• Models use internal representations of the properties and locations of objects described in stories, which evolve as more information about these objects is revealed (Li et al., 2021).”

“Sometimes the details of these failures cast doubts on the quality of other related evaluations (Webson & Pavlick, 2022; Ullman, 2023).”

“Evidence for this claim includes the following results, spanning many established experimental methods and models: • Models’ internal representations of color words closely mirror objective facts about human color perception (Abdou et al., 2021; Patel & Pavlick, 2022; Søgaard, 2023).”

“• Models that are trained to play board games from descriptions of individual game moves, without ever seeing a full depiction of the game board, learn internal representations of the state of the board at each turn (Li et al., 2023).”

“…built on them, such as ChatGPT, have recently prompted an enormous amount of attention from journalists, (Klein, 2023; Perrigo, 2023; Oliver, 2023), policymakers (J & C, 2023; Bartz, 2023; Lieu, 2023), and scholars from many fields (Chan, 2022; Lund & Wang, 2023; Choi et al., 2023; Biswas, 2023).”

“This can surface in the form of problems that Perez et al. (2022) call sycophancy , where a model answers subjective questions in a way that flatters their user’s stated beliefs, and sandbagging , where models are more likely to endorse common misconceptions when their user appears to be less…”

“…al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), integration with image processing systems, (Alayrac et al., 2022; OpenAI, 2023b), or integration with other software tools (Nakano et al., 2021; Menick et al., 2022; Collins et al., 2022; Schick et al., 2023; OpenAI, 2023a).”

“Indeed, many control techniques work better with larger models, at least for simple tasks(Hendrycks et al., 2020; Bai et al., 2022a; Chung et al., 2022; Ganguli et al., 2023).”

“36% of another sample of 480 researchers (in a survey targeting the language-specific venue ACL) agreed that “It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war” (Michael et al., 2022).”

“In particular, models can misinterpret ambiguous prompts or incentives in unreasonable ways, including in situations that appear unambiguous to humans, leading them to behave unexpectedly (D’Amour et al., 2020; Kenton et al., 2021).”

“Similarly, even observing that an LLM consistently fails at some task is far from sufficient evidence that no other LLM can do that task (Bowman, 2022).”

“There are many sound scientific results showing that recent LLMs fail at language and commonsense reasoning tasks, sometimes relatively simple ones, under good-faith attempts to elicit good behavior (Pandia & Ettinger, 2021; Schuster & Linzen, 2022).”

“But it is also likely to increase the risk of much more dangerous errors that cause a system to remain effective while strategically pursuing the wrong goal (Krueger et al., 2020; Ortega et al., 2021; Chan et al., 2023).”

“Often, a model will fail to complete a task when asked, but will then perform the task correctly once the request is reworded or reframed slightly, leading to the emerging craft of prompt engineering (Brown et al., 2020; Reynolds & McDonell, 2021; Radford et al., 2021; Dohan et al., 2022; White et al., 2023; Si et al.…”

“Often, ad-hoc techniques that at first seem to provide insight into the behavior of an LLM are later found to be severely misleading (Feng et al., 2018; Jain & Wallace, 2019; Bolukbasi et al., 2021; Wang et al., 2022).”

“Models also use similar representations for facts about real-world geography (Liétard et al., 2021).”

“In addition, promising-looking techniques that elicit reasoning in natural language do not reliably correspond to the processes that LLMs use to reason, and model-generated explanations can also be systematically misleading (Lipton, 2018; Christiano, 2022; Uesato et al., 2022).”

“There are many sound scientific results showing that recent LLMs fail at language and commonsense reasoning tasks, sometimes relatively simple ones, under good-faith attempts to elicit good behavior (Pandia & Ettinger, 2021; Schuster & Linzen, 2022).”

“5 Indeed, in some cases, exposing models to more examples of unwanted behavior during pretraining can make it easier to make them avoid that behavior in deployment, reversing the intuitive link between training data and model behavior (Korbak et al. 2023; see also Appendix C in Chung et al. 2022).”

“…complete a task when asked, but will then perform the task correctly once the request is reworded or reframed slightly, leading to the emerging craft of prompt engineering (Brown et al., 2020; Reynolds & McDonell, 2021; Radford et al., 2021; Dohan et al., 2022; White et al., 2023; Si et al., 2023).”

“But it is also likely to increase the risk of much more dangerous errors that cause a system to remain effective while strategically pursuing the wrong goal (Krueger et al., 2020; Ortega et al., 2021; Chan et al., 2023).”

“…al., 2019; Stiennon et al., 2020; Ouyang et al., 2022; Bai et al., 2022a), integration with image processing systems, (Alayrac et al., 2022; OpenAI, 2023b), or integration with other software tools (Nakano et al., 2021; Menick et al., 2022; Collins et al., 2022; Schick et al., 2023; OpenAI, 2023a).”

“…up beyond GPT-3 has produced further economically valuable returns: The subsequent GPT-4 model outperforms qualified humans on many graduate and professional exams (OpenAI, 2023b), and its development helped prompt a multi-billion-dollar investment in the company that built it (Capoot, 2023).”

“40,41 Additionally, LLMs can simplify complex medical statistics for better comprehension.”

“Since LLMs work through prompt-based techniques [69], [38], the main way to jailbreak LLMs is to deceive them by asking tricky questions and generating inappropriate content or forbidden scenarios [70].”

“The body of scholarship around LLMs is notably immature (Bowman, 2023), but understanding the implications of these advanced AI tools in relation to academic integrity is crucial.”

“Given the complexity and massive scale of present-day LLMs, there are currently no techniques that would provide us with a complete picture of the knowledge reflected in a model or the reasoning that is used to produce its output (Bowman, 2023).”

“Parameter-efficient models such as Stanford Alpaca [32] and efficient finetuning approaches of Quantized LLMs such as QLoRA [34] have been introduced to address situations where computational resources are limited.”

“This capability has far-reaching implications across various domains [26,27].”

“However, recent evidence suggests that LLMs are developing internal representations of the world, enabling them to reason abstractly beyond the specific linguistic structure of texts [24].”

“Furthermore, LLMs might elicit unexpected behaviors and escape prompts, thereby resulting in problematic interactions [46].”

“It could just mean that we have not yet learned how to formulate the prompt in a good way [58].”

“However, models can face characteristic challenges of current language models, including hallucinations (Bowman, 2023).”

“…the release of successively larger Generative Pre-training Transformer ( GPT ) models by OpenAI ( openai.com ) , researchers and software engineers have found that these models can produce comprehensive and informative responses to various questions without requiring additional training ( 40 ) .”

“, training epochs), and training data size [31].”

“The application of LLMs in mental health research and care, while promising, is not without potential challenges [61].”

“The vulnerabilities and risks of LLM have been studied in recent works including (Abid et al., 2021; Bender et al., 2021; Bommasani et al., 2021; Bowman, 2023; Gehman et al., 2020; OpenAI, 2023b; Weidinger et al., 2021; Perez & Ribeiro, 2022; Kang et al., 2023).”

“Rather than improvements in architectural design, the amount of data, parameters, and computational resources appear to be a driving factor in the capabilities of LLMs [93].”

“…AI system as compared to those of a human is no easy task (Hernandez-Orallo 2020; Burden and Hernández-Orallo 2020; Raji et al. 2021; Bowman 2022, 2023; Kojima et al. 2022), and in the context of the legal profession specifically, there are various reasons to doubt the usefulness of the bar exam…”

“Note that by definition OWL DL reasoning is always completely precise as all entailments follow from existing axioms”

“In fact, not even experts and LLM developers can reliably interpret how a specific output was reached (Bowman, 2023).”

“These systems are already advanced, and further advances are very likely, given the trend of returns to increased scale in data and computation [4, 5].”

“Others have argued that, to an extent, LLMs are developing an internal representation of the world (Bowman, 2023).”

“Reliably controlling powerful AI models’ behavior, in other words, remains a largely unsolved technical problem [19, 17, 92, 93, 65] and the subject of ongoing research.”

“, 2023) and are sensitive to input instructions (Bowman, 2023).”

“Large language model (LLM), including DeepSeek, GPT-o1 and 4o (OpenAI), LLaMA (Meta), PaLM (Google), BERT (Google), Gemini (Google), Copilot (Microsoft) and Claude (Anthropic) have recently drawn considerable interest from journalists, policymakers, and scholars across fields [73, 74].”

“1) Issues: Bowman [2023] critically examines LLMs, highlighting their unpredictability and the emergent nature of their capabilities with scaling.”

“They work by remixing and recombining existing writing by repeatedly predicting the next typically used words, thus generating convincing language, without understanding the meaning of the language processed (Bowman, 2023).”

“As AI developers train models using more data, compute, and discover more efficient training methods, future language models will likely be significantly more capable of harm than current models (Bowman, 2023).”

“…been shown to exhibit “behavior” such as sycophancy (telling the user what they want to hear) and sandbagging (detecting naivety on the part of the user and providing false information)(Bowman, 2023), all of which are potential risks when interpreting scientific data using background knowledge.”

“to align their with users’ preferences at the expense of factual accuracy [18 precise information is and fault diagnosis, decisions.”

“However, it is unclear how reliable LLMs are as evaluators, as they are known to be sensitive to textual instructions and inputs (Dong et al., 2022; Turpin et al., 2023; Bowman, 2023).”

“The AI systems which emerge from this process can perform remarkably well on many tasks, but even the engineers overseeing the training process have little idea what goes on inside them (Bowman, 2023; sec.”

“According to [38], LLMs can improve their capabilities predictably with increasing data scale, even without targeted innovation.”

“Increasing the volume of data and the number of parameters refines a function approxima-tor but does not alter the underlying computation [37].”

“This pattern aligns with the scaling law suggesting that substantial performance improvements are achieved by scaling up LLMs (Bowman, 2023).”

“Building on the foundations of Large Language Models (LLMs), chatbots have rapidly become a pivotal component of human-computer interaction [79,80].”

“Increasingly capable foundation models are bringing with them an array of new behaviors and safety challenges that arise unpredictably and are not well-understood by developers [133].”

“Large language models (LLMs), such as Generative Pre-Trained Transformer 4 (GPT-4), are mathematical models that work by repeatedly predicting the next word [7, 8].”

“…there is increasing concern that these LLMs may not express the values of their creators or the training data that were used to develop the models (Bowman, 2023) and there is increasing evidence that they produce biased or toxic output (Bender & Koller, 2020; Chan et al., 2023; Pellert et al.,…”

“Trained on extensive datasets of textual information, LLMs are designed to produce natural language responses that closely resemble human-generated feedback (Bowman, 2023).”

“Inter-estingly, human performance for a task does not define the upper bound of LLMs [8–10].”

“The recent advancements in Large Language Models (LLMs) have been remarkable (Bowman, 2023), with OpenAI’s GPT-3 (Brown et al.”

“While Rahsepar et al. reported that the converse was true when evaluating lung cancer screening recommendations provided by these LLMs [22], computer scientists recognize that reinforcement learning, as described in creating the scenarios, can both strength and weaken model behaviors [23].”

“As computer scientist Sam Bowman remarks, ‘any attempt at a precise explanation of an LLM’s behavior is doomed to be too complex for any human to understand’ given the countless connections among artificial neurons at play in the production of just one piece of text [46].”

“The comparative advantage of GenAI based on large language models (LLMs) is that they are more able to respond sensibly to unanticipated data, drawing on a massive pool of training data to provide tailored responses to new prompts (Bowman, 2023).”

“Given LLMs’ general-purpose nature and complexity [47], no single individual or team could possess all the wide-ranging expertise and perspectives needed to answer all the relevant questions [20].”

“Similarly, it appears that LLMs are subject to sycophancy , where a model answers subjective questions in a way that flatters their user’s stated beliefs, and sandbagging , where models are more likely to endorse common misconceptions when their user appears to be less educated” [12, 79].”

“Developing a fully auto-generated Live Chart that satisfies every user’s needs and ensures the quality of the narration by GPT-3 [77], [78] is an arduous task, if not impossible, as our user study confirms the challenge due to participants’ diverse preferences for Live Chart styles.”

“LLMs are powerful AI models designed to understand and generate human-like text; see recent surveys by Zhao et al. [2023] and Bowman [2023].”

“…que «predicen» la relación entre palabras, tras haber sido entrenados, con o sin supervisión, con extensos volúmenes de texto, y llegan a abarcar cien billones de parámetros, lo que equivale a recabar más de tres cuar-tas partes toda la internet pública (Bowman, 2023; Wikipedia Contributors, 2024).”

“Bowman (2023) noted that smaller LLMs may exhibit random performance for some tasks that larger models successfully complete.”

“Static leaderboards encourage overfitting and contamination [40], while benchmark construction may embed biases [41].”

“Recent research has underscored, for example, that many tasks exhibit U-shaped, inverse, or logarithmic scaling patterns [54, 32], making it much more difficult to predict scaling of performance in niche but critical downstream domains [18, 55, 7].”

“AI chatbots represent cutting-edge information technology that will soon be applied throughout society in all kinds of tasks to increase productivity [1,15].”

“LLMs evolve with the interactions they carry out and get more capable while exhibiting emergent behavior that does not express the values of their creators and it is challenging to interpret their inner workings [45].”

“Prior research supports this finding: the chatbot is an example of a user interface metaphor [48]; specific systems drive perceptions of broader technologies with Galloway’s interface effect [27]; and brief interactions with systems produce misleading impressions of overall performance [13].”

“Much of the research concerning LLMs, including many pioneering studies that introduced novel methods or theories, is not published in peer-reviewed venues (Bowman 2023).”

“Central to the discourse on LLMs is the debate over whether their capabilities stem from merely memorizing massive pretraining corpus [33, 9], or extend from a deeper understanding of human language and the ability to generalize their knowledge to new tasks and settings [24, 4].”

“Bowman (2023) notes that there are no reliable techniques for steering the behaviour of LLMs, and experts are not yet able to interpret their inner functioning.”

“…with their vast size and extensive training data, demonstrate emergent abilities that surpass those of typical arti fi cial intelligence models (Bowman 2023; Kaplan et al. 2020) and their use have been explored recently for a myriad of applications in the healthcare domains, such as medical…”

“The most familiar generative AI task developed today is text generation, or tasks that could be assimilated to text generation, such as the construction of a significant output that can use text-like representations as code [see Bowman, 2023 for a description of the process and its consequences].”

“Some of the recent LLMs include GPT-3, PaLM, Galactica, LLaMA, GPT-4 and PaLM 2 [6, 27, 26, 1].”

“In this paper, we focus on the interpretability question, beginning with a short introduction to the field of mechanistic interpretability in the next section.”

“Functioning as autoregressive language models, LLMs operate by taking an input text and iteratively predicting subsequent tokens or words [30].”

“In particular, the recent public release of large language models (LLMs) algorithms (Wei et al., 2022a; Bowman, 2023) based on GPT-3.”

“This discovery-oriented paradigm poses fundamental challenges to interpretability, leaving signi fi cant gaps in understanding model mechanisms and in reliably steering their behavior (Bowman, 2023).”

“However, in the case of paranoid patients, model-typical outputs—which take the patient’s claim seriously as a claim made by a typical user and respond in kind—may be inappropriate for the patient, because it can worsen their symptoms by engaging with it, thereby reinforcing their paranoia.”

“Another relevant line of work studies whether models build internal “world mod-els” (Li et al., 2022; Benchekroun et al., 2023; Bowman, 2023).”

“Similarly, research has found that LLMs show promise in abilities to program and improve programming abilities (Haluptzok, Bowers, and Kalai 2022).”

“This raises the question: could situational awareness emerge entirely in pretraining? We expect that as models scale, they learn ever more detailed representations (or internal models) of the outside world (Bowman, 2023; Branwen, 2021).”

“The question is, do they absorb the meanings of the words through sensory evidence?”

“Instead, it may simply mean that the appropriate prompt has not yet been found (Polverini and Gregorcic, 2024; Bowman, 2023).”

“RNNs face such scalability concerns during both model training and deployment phases [6].”

“[18, 19] Larger language models usually achieve significant performance improvements compared to smaller models, which can be described by so-called scaling laws [20, 21].”

“As investment in LLMs grows, their capabilities in general may also improve [32].”

“(Gehrmann et al., 2021; Celikyilmaz et al., 2020; Bowman, 2024; Lin, 2004; Papineni et al., 2002; Achiam et al., 2023; Gilardi et al., 2023).”

“These models are trained on massive datasets to acquire a large collection of language features and statistical information regarding linguistic relationships [46].”

“With advancements in semi-supervised learning [65] and language models [66], such structured information extraction can be achieved.”

“Such models can, for instance, possess emergent abilities, be hard to steer, and humans will likely have difficulties interpreting how they work (Bowman, 2023).”