Machine Psychology

While the previous generation of models frequently exhibited human-like heuristics and biases, they have largely disappeared in the latest generation of LLMs.

“Interestingly, there is evidence from several studies showing that, while the previous generation of models frequently exhibited human-like heuristics and biases, they have largely disappeared in the latest generation of LLMs (Chen, Liu, et al., 2023;Hagendorff et al., 2023).”

• cites Chen, Liu, et al., 2023 sourceDisputed

  The Discussion (page unavailable) says GPT remains sensitive to framing and can use corpus-based heuristics in unfamiliar decision tasks. The Abstract likewise reports reduced rationality under alternative framing and discrete choice. The paper does not compare model generations, and its findings run contrary to the claim that such biases have largely disappeared. (LLM, conf=0.86)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites Hagendorff et al., 2023 sourcePartly supported

  The Discussion reports that pre-ChatGPT models often gave intuitive responses, whereas ChatGPT models answered a great majority of the tested tasks correctly. But the Discussion also limits the study to OpenAI’s GPT family and two types of tasks, so it does not establish that these biases largely disappeared across the latest generation of LLMs generally. Page numbers were not provided in our copy. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Many interesting behaviors are only displayed by large models with billions of parameters.

“Many interesting behaviors are only displayed by large models with billions of parameters (Kaplan et al., 2020;Wei, Tay, et al., 2022)”

• cites Kaplan et al., 2020Partly supported

  The Discussion (p. 19) says that larger models perform better and that smooth improvements in loss may hide qualitative changes in capability. It does not report that interesting behaviors have been observed only in models with billions of parameters; that threshold and exclusivity are unsupported. (LLM, conf=0.81)

  Experimental — accuracy not yet measured
• cites Wei, Tay, et al., 2022 sourceSupported

Similar performance-competence issues apply when assessing the capabilities of machine learning systems and particularly LLMs.

“More recent work has suggested that similar issues apply when assessing the capabilities of machine learning systems (Firestone, 2020), and particularly LLMs (Lampinen, 2022).”

• cites Firestone, 2020 sourceSupported
• cites Lampinen, 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Properties that self-report measures usually consider, such as personality, morality, or clinical disorders, are sensitive to prompting in LLMs.

“Properties that such measures usually consider, such as personality, morality, or clinical disorders, are famously sensitive to prompting (Dominguez-Olmedo et al., 2023;Röttger et al., 2024)”

• cites Dominguez-Olmedo et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Röttger et al., 2024 sourcePartly supported

  The Discussion reports that minimal changes in prompt phrasing or context can lead LLMs to express opposing views, supporting prompt sensitivity for values and opinions. The paper uses the Political Compass Test as its case study; our copy does not establish the same sensitivity for personality or clinical-disorder measures, and its discussion of moral evaluations does not establish it for morality measures specifically. The cited passages provide section labels but no page numbers. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Early experiments with models such as GPT-3 showed that they struggle to solve theory of mind tasks.

“While early experiments with models such as GPT-3 showed that they struggle to solve theory of mind tasks (Sap et al., 2022)”

• cites Sap et al., 2022 sourceSupported

Later models demonstrate an increasing ability to reliably infer unobservable mental states in others.

“later models demonstrate an increasing ability to reliably infer unobservable mental states in others (Strachan et al., 2024;Holterman and Deemter, 2023;Moghaddam and Honey, 2023).”

• cites Strachan et al., 2024 sourcePartly supported

  The Results (Performance across theory of mind tests; page unavailable) show GPT-4 outperforming GPT-3.5 on several tasks involving mental-state interpretation, consistent with improvement in a later model. But GPT-4 still struggled with faux pas, and the Discussion cautions that successful responses may not reflect human-like reasoning. The source therefore does not establish that later models reliably infer others’ unobservable mental states in general. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites Holterman and Deemter, 2023 sourcePartly supported

  The Results and Conclusion report that ChatGPT-4 outperformed ChatGPT-3 on questions about human reasoning biases, answering about 80% correctly. But the Conclusion describes this cautiously as possible comprehension of one facet of theory of mind, not reliable inference of others’ unobservable mental states generally. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Moghaddam and Honey, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Binz and Schulz found that GPT-3 displays some of the same cognitive biases observed in people.

“They found that GPT-3 (Brown et al., 2020) displays some of the same cognitive biases observed in people.”

• cites Binz and Schulz (2023) sourceSupported
• cites Brown et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Like people, LLMs reason more accurately about familiar, believable, or grounded situations compared to unfamiliar, unbelievable, or abstract problems.

“They found that, like people, LLMs reason more accurately about familiar, believable, or grounded situations, compared to unfamiliar, unbelievable, or abstract problems.”

• cites Dasgupta et al. (2022) sourcePartly supported

  The Discussion (p. 22) says models, like humans, often struggle when problems are abstract or conflict with prior expectations. But the syllogism results (pp. 10–11) qualify the believability comparison: believable conclusions improve accuracy when the argument is valid, while unbelievable conclusions improve accuracy when it is invalid. The paper therefore does not support a general claim that believable problems yield more accurate reasoning than unbelievable ones. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Machine psychology can reveal patterns of social behavior and interaction among LLMs, individually and collectively, for problem solving or world simulation.

“In sum, machine psychology can reveal patterns of social behavior and interaction among LLMs, individually and collectively, be it for problem solving or world simulation (Guo et al., 2024).”

• cites Guo et al., 2024 sourcePartly supported

  The Psychology passage (Sandbox section; page unavailable) describes using psychological theories and experiments to analyze individual agents’ behavior and emergent collective patterns in simulations. Section 4.1 describes agents interacting to solve problems, but does not say that psychological analysis reveals their social behavior in that setting. The claim extends the psychology point from world simulation to problem solving. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

The presence of in-context learning in LLMs can be traced back to data distributional properties such as burstiness and the presence of large numbers of rarely occurring classes.

“For example, Chan et al., 2022 have demonstrated that the presence of in-context learning in LLMs can be traced back to data distributional properties such as burstiness, where items appear in clusters rather than being uniformly distributed over time, and the presence of large numbers of rarely occurring classes.”

• cites Chan et al., 2022 sourcePartly supported

  The Introduction (p. 3) reports that in-context learning emerged in a transformer trained on Omniglot data when the data included burstiness and many rarely occurring classes. But the Discussion (pp. 8–9) says these properties *may explain* in-context learning in large language models; the paper did not directly demonstrate that they cause it in LLMs. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Diagnostic datasets were used to expose the ways in which learned systems solved tasks, often counter to human intuitions.

“Such diagnostic datasets were used to expose the ways in which learned systems solved tasks -often counter to human intuitions (Geirhos et al., 2020;McCoy, Pavlick, et al., 2019;Hermann and Lampinen, 2020;Dasgupta et al., 2022;Singla and Feizi, 2021).”

• cites Geirhos et al., 2020 sourceSupported
• cites McCoy, Pavlick, et al., 2019 sourceSupported
• cites Hermann and Lampinen, 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

As LLMs expand in the kinds of stimuli they can interpret and the ways they can interact with the world, the space of transferable paradigms increases.

“As LLMs expand in the kinds of stimuli they can interpret -e.g.visual (OpenAI, 2023b;Zhang, Huang, et al., 2024;Gemini Team et al., 2024) -and the ways in which they can interact with the world -e.g.embodiment and tool use (Mialon et al., 2023) -, the space of transferable paradigms increases.”

• cites OpenAI, 2023bNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhang, Huang, et al., 2024 sourceSupported
• cites Gemini Team et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Some experiments may be procedurally generated, which makes them inherently less susceptible to data contamination issues.

“some experiments may be procedurally generated (instead of consisting of a static dataset), which makes them inherently less susceptible to data contamination issues (Coda-Forno, Binz, Wang, et al., 2024).”

• cites Coda-Forno, Binz, Wang, et al., 2024 sourcePartly supported

  Section B.6.2 (page unavailable) says the temporal-discounting task is the only one that is not procedurally generated, supporting the claim that some experiments are generated rather than fixed. The passages found do not say that procedural generation makes experiments less susceptible to data contamination; searches for contamination and related terms did not establish that link. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Self-reflection (automated recursive criticizing and subsequent self-improvement of LLM outputs by the LLM itself) can improve reasoning abilities.

“Moreover, self-reflection, meaning the automated, recursive criticizing and subsequent self-improvement of LLM outputs by the LLM itself, is a further technique that can improve reasoning abilities (Nair et al., 2023;Kim et al., 2023).”

• cites Nair et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Kim et al., 2023 sourceSupported

Different expressions of the same concept compete for probability, which can lower the chances of selecting the correct answer.

“Here, one must keep in mind that different expressions of the same concept compete for probability, which can lower the chances of selecting the correct answer (Holtzman et al., 2021).”

• cites Holtzman et al., 2021 sourceSupported

Multiple studies have compared the outputs of transformer in-context learning with the outputs of hypothesized learning algorithms.

“multiple studies have compared the outputs of transformer in-context learning with the outputs of hypothesized learning algorithms (Oswald et al., 2023;Akyürek et al., 2022).”

• cites Oswald et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Akyürek et al., 2022 sourceSupported

LLMs can succumb to various biases influencing the processing of prompts.

“Furthermore, LLMs can succumb to various biases influencing the processing of prompts (Zhao, Wallace, et al., 2021;Chan et al., 2022).”

• cites Zhao, Wallace, et al., 2021Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Chan et al., 2022 sourcePartly supported

  The Discussion (p. 9) reports a bias in experimental transformers toward using either labels learned during training or labels supplied in the current context. That supports the general idea that prior learning can affect how a model uses contextual input, but the paper does not establish that LLMs succumb to various biases when processing prompts; its experiments use image-label sequences rather than LLM prompts. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

In-context learning can enable overcoming safety fine-tuning.

“or the ability to overcome safety fine-tuning (Anil et al., 2024;Zheng et al., 2024).”

• cites Anil et al., 2024 sourceNot checked — no full text of the source
• cites Zheng et al., 2024 sourceSupported

Scoring answers so that the model responds directly inline, rather than selecting a choice, can yield more signal.

“though even in this case, scoring the answers so that the model responds directly inline, rather than selecting a choice, can yield more signal (Hu and Levy, 2023).”

• cites Hu and Levy, 2023 sourceSupported

Prompting LLMs to use code for solving tasks can improve symbolic or numeric reasoning.

“Regarding improvements in symbolic or numeric reasoning, another technique is to prompt LLMs to use code for solving tasks (Zhang, Ge, et al., 2024).”

• cites Zhang, Ge, et al., 2024 sourceCould not verify

  The survey’s Section 9 discusses using LLMs to generate text prompts for vision-language tasks, not prompting LLMs to use code. Searches for symbolic or numeric reasoning and code-based task solving found no passage addressing the attributed technique. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

AI-generated analogies lack some human-like psycholinguistic properties, particularly in text cohesion, language, and readability.

“In another study, researchers examined long-form analogies generated by ChatGPT, finding that AI-generated analogies lack some human-like psycholinguistic properties (Seals and Shalin, 2023), particularly in text cohesion, language, and readability.”

• cites Seals and Shalin, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

How LLMs learn in-context depends on the problem formulation.

“Likewise, Schubert et al., 2024 have shown that how LLMs learn in-context depends on the problem formulation.”

• cites Schubert et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLMs respond similarly to humans on garden path sentences.

“Furthermore, researchers applied garden path sentences -sentences that lead the reader to initially interpret them incorrectly due to their ambiguous structure -to LLMs, showing that the models respond similarly to humans (Aher et al., 2023;Christianson et al., 2001).”

• cites Aher et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Christianson et al., 2001 sourceNot checked — no full text of the source

Researchers found that cooperative and coordinative behavior in LLMs playing games reveals persistent behavioral signatures in the models.

“For example, researchers looked at cooperative and coordinative behavior in LLMs playing games, revealing persistent behavioral signatures in the models (Akata et al., 2023).”

• cites Akata et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Behavioral methods in psychology that treat behavior directly as the experimental variable of interest scale gracefully with model size.

“behavioral methods in psychology that treat behavior directly as the experimental variable of interest scale gracefully with model size.”

LLM behavior is often too complex to predict purely from current mechanistic understanding of model weights and activation patterns.

“Such behavior is often too complex to predict purely from our current mechanistic understanding of model weights and activation patterns (Grön et al., 2003).”

• cites Grön et al., 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Machine psychology research is not primarily interested in increasing or measuring an LLM's performance, but rather in understanding behavioral patterns.

“In contrast, machine psychology research is not primarily interested in increasing (or measuring) an LLM's performance, but rather in understanding behavioral patterns.”

The traditional train/test framework does not generalize well to large-scale foundation models.

“This framework does not generalize well to large-scale foundation models for two reasons.”

Behavioral approaches can easily be applied by the broader academic community to closed-source state-of-the-art models whose internal workings are not disclosed to the public.

“Another practical advantage is that these behavioral approaches can easily be applied by the broader academic community to closed-source state-of-the-art models whose internal workings are not disclosed to the public.”

When using Internet-scale training data for models, the train/test split has become harder to maintain.

“First, when using Internet-scale training data for models, this split has become harder to maintain (Li and Flanigan, 2023;Khan et al., 2023).”

• cites Li and Flanigan, 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Khan et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Results drawn from self-report measures should be taken contextually as a property of a specific system prompt on a model instead of as a fundamental or general property of the LLM itself.

“But results drawn from these measures should be taken contextually (e.g. as a property of a specific system prompt on a model) instead of as a fundamental or general property of the LLM itself.”

Emergent behaviors can be difficult to study through the lens of the components that gave rise to them.

“Emergent behaviors can be difficult to study through the lens of the components that gave rise to it (Anderson, 1972)”

• cites Anderson, 1972 sourceNot checked — no full text of the source

Test-only benchmarks resemble evaluation frameworks from the behavioral sciences.

“In several cases, these benchmarks already resemble evaluation frameworks from the behavioral sciences (Bubeck et al., 2023)”

• cites Bubeck et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The space of relevant paradigms increases as LLMs are allowed to interact through self-reflection, self-instruction, or in swarms.

“The space of relevant paradigms increases as LLMs are allowed to interact through self-reflection (Nair et al., 2023), self-instruction (Wang, Wei, et al., 2022), or in swarms (Zhuge et al., 2023).”

• cites Nair et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Wang, Wei, et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Zhuge et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

LLMs are susceptible to manipulations of problem phrasing similar to those that affect people.

“It is well-known that LLMs are also susceptible to similar manipulations.”

The empirical tradition in psychology is the tradition most amenable for transferring insights to machine psychology.

“and is the tradition we argue is the most amenable for transferring insights to machine psychology.”

Investigations of LLM false belief induction and tradeoffs between communicative values like honesty and helpfulness contribute to understanding and improving alignment with human values for AI safety.

“these investigations also contribute to understanding and improving alignment with human values for AI safety (Ji et al., 2023).”

• cites Ji et al., 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The properties of the training data shape LLM behavior.

“In the context of LLMs, one can look at how the properties of the training data shape their behavior.”

People use heuristics that are adapted to the problems they encounter during their everyday interactions with the world.

“Instead, they use heuristics that are adapted to the problems they encounter during their everyday interactions with the world (Todd and Gigerenzer, 2012).”

• cites Todd and Gigerenzer, 2012Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Methods and ideas from psychology and psycholinguistics provide guidance on how to assess processes through language behaviors in LLMs, potentially by drawing comparisons between LLMs and humans.

“In each of these cases, methods and ideas from psychology and psycholinguistics provide guidance on how to assess processes through language behaviors in LLMs, potentially by drawing comparisons between LLMs and humans.”

The absence of behavior Y does not necessarily indicate the absence of the construct X.

“These should not be taken to mean that they lack the abstract capability for math or language.”

Applying paradigms from developmental psychology to LLMs can give deeper insights into how these models manage complex social interactions.

“By applying paradigms from this area of developmental psychology to LLMs, researchers can gain deeper insights into how these models manage complex social interactions.”

The empirical tradition in psychology has yielded lasting understanding of natural intelligence.

“This tradition has yielded lasting understanding of natural intelligence (Frank et al., 2024)”

• cites Frank et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Sampling biases in benchmarks or task datasets, especially prevalent in small sample sizes, can diminish the quality of machine psychology studies.

“Sampling biases in the used benchmarks or task datasets, which are especially prevalent in small sample sizes, can diminish the quality of machine psychology studies.”

This holds for humans, too, where mental terms used to explain behavior do not directly correlate with specific sets of neural activations.

“This holds for humans, too, where mental terms used to explain behavior do not directly correlate with specific sets of neural activations.”

Studies that are not careful about testing the robustness of their conclusions risk being spurious and non-generalizable.

“studies that are not careful about testing the robustness of their conclusions risk being spurious and non-generalizable.”

LLMs exhibit learning abilities that were not explicitly designed into the models (they are emergent).

“LLMs exhibit learning abilities that were not explicitly designed into the models (they are emergent), and thus one does not understand the underlying learning algorithm.”

• cites Brown et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The standard prompt design (vignette plus open- or close-ended question or task) can be enhanced by prefixes or suffixes eliciting improved reasoning capabilities in LLMs.

“The standard prompt design, comprising a vignette plus an open-or close-ended question or task, can be enhanced by prefixes or suffixes eliciting improved reasoning capabilities in LLMs.”

Interpreting LLM responses comes with a challenge: a strong tendency exists to confer mental concepts or psychological terms to LLMs that were hitherto reserved for human and animal minds.

“interpreting LLM responses comes with a challenge.A strong tendency exists to confer mental concepts or psychological terms to LLMs that were hitherto reserved for human and animal minds.”

Recency biases in LLMs lead to a tendency to rely more heavily on information appearing toward the end of prompts.

“Recency biases in LLMs, for instance, lead to a tendency to rely more heavily on information appearing toward the end of prompts.”

The settings under which machine learning protocols were developed (a test set fixed for all practitioners and effectively infinitely large) no longer hold in the small test-only behavioral experiments setting.

“the settings under which those protocols were developed -a test set that is fixed for all practitioners and is effectively infinitely large -no longer hold in the small test-only behavioral experiments setting.”

LLMs exhibit emergent in-context learning: the ability to learn from context (the prompt) without requiring any gradient-based updates in weights.

“In particular, LLMs exhibit emergent in-context learning -the ability to learn from context (the prompt) without requiring any gradient-based updates in weights (Brown et al., 2020).”

• cites Brown et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

A few prompt design approaches have been found to confer broader performance benefits.

“a few prompt design approaches have nonetheless been found to confer broader performance benefits.”

By postulating (mental) unobservable states, be it with regard to brains or artificial neural networks, one increases explanatory resources.

“By postulating (mental) unobservable states, be it with regard to brains or artificial neural networks, one increases explanatory resources (Sellars, 1997).”

• cites Sellars, 1997Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Multiple-choice format can limit reasoning capabilities because LLMs might be prompted to provide brief responses, thereby circumventing reasoning.

“but can also limit them because LLMs might be prompted to provide brief responses, thereby circumventing reasoning in the process of prompt completion.”

Majority label biases can cause LLMs to be skewed towards labels, classes, or examples that are frequent in a few-shot learning setting.

“Moreover, majority label biases can cause LLMs to be skewed towards labels, classes, or examples that are frequent in a few-shot learning setting.”

By applying psychological terms to LLMs, the explanatory power increases, even if no direct neural correlates to these terms exist.

“By applying psychological terms to them, the explanatory power increases, even if no direct neural correlates to these terms exist.”

Generating multiple chain-of-thought reasoning paths and taking the majority response as the final one can extend chain-of-thought prompting.

“This can be extended even further by generating multiple chain-of-thought reasoning paths and taking the majority response as the final one (Wang, Wei, et al., 2022).”

• cites Wang, Wei, et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Few-shot learning in LLMs improves performance after repeated exposure to a given task.

“Another method to increase reasoning is to utilize the ability for few-shot learning in LLMs (Brown et al., 2020), where the LLM's performance improves after repeated exposure to a given task.”

• cites Brown et al., 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Several existing machine psychology studies rely on small sample sizes or convenience samples (non-systematic sequences of prompts).

“Another common shortcoming of several existing machine psychology studies is that they rely on small sample sizes or convenience samples, meaning non-systematic sequences of prompts.”

Many studies in machine psychology do not avoid training data contamination.

“Many studies conducted in the field of machine psychology have a significant shortcoming in common, namely that they do not avoid training data contamination.”

LLMs are likely to have already experienced identical or similar tasks during training, causing them to simply reproduce known token patterns.

“In this way, LLMs are likely to have already experienced identical or similar tasks during training, thus causing LLMs to simply reproduce known token patterns.”

Framing questions in a multiple-choice format was shown to improve reasoning capabilities in some cases.

“This was shown to improve reasoning capabilities in some cases (Kadavath et al., 2022)”

• cites Kadavath et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Slight changes in prompts can change model outputs significantly.

“This is because slight changes in prompts can change model outputs significantly.”

One does not understand the underlying learning algorithm for these emergent learning abilities.

“LLMs exhibit learning abilities that were not explicitly designed into the models (they are emergent), and thus one does not understand the underlying learning algorithm.”

Least-to-most prompting decomposes problems into subproblems to increase accuracy in LLMs.

“Similar to chain-of-thought prompting is least-to-most prompting, which also decomposes problems into a set of subproblems to increase accuracy in LLMs (Zhou et al., 2022).”

• cites Zhou et al., 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

These studies use prompts from existing psychology studies and apply them to LLMs without changing their wording, task orders, etc.

“They use prompts from existing psychology studies and apply them to LLMs without changing their wording, task orders, etc.”

Omitting such prefixes and suffixes can lead to underestimations of the model's capabilities.

“On the other hand, omitting such prefixes and suffixes can lead to underestimations of the model's capabilities.”

In-context learning can enable learning an entire language from context alone.

“the capability to learn an entire language from context alone (Munkhdalai et al., 2024;Gemini Team et al., 2024)”

• cites Munkhdalai et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Gemini Team et al., 2024Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Thick descriptions help in making sense of LLMs when thin descriptions are insufficient to explain behavioral patterns.

“Thick descriptions help in making sense of LLMs when thin descriptions are insufficient to explain behavioral patterns.”