CoChat
Sign Up
Cited by 112Computational and Text Analysis MethodsEthics and Social Impacts of AITopic Modeling

Perils and opportunities in using large language models in psychological research

Open access
Abdurahman, Suhaib, Atari, Mohammad, Karimi-Malekabadi, Farzan, Xue, Mona J, Trager, Jackson, Park, Peter S, Golazizian, Preni, Omrani, Ali +1 more·2024·PNAS Nexus
DOI 10.1093/pnasnexus/pgae245PMC PMC11249969
Open this paper in CoChat — free accountOpen PDF

Abstract

The emergence of large language models (LLMs) has sparked considerable interest in their potential application in psychological research, mainly as a model of the human psyche or as a general text-analysis tool. However, the trend of using LLMs without sufficient attention to their limitations and risks, which we rhetorically refer to as “GPTology”, can be detrimental given the easy access to models such as ChatGPT. Beyond existing general guidelines, we investigate the current limitations, ethical implications, and potential of LLMs specifically for psychological research, and show their concrete impact in various empirical studies. Our results highlight the importance of recognizing global psychological diversity, cautioning against treating LLMs (especially in zero-shot settings) as universal solutions for text analysis, and developing transparent, open methods to address LLMs’ opaque nature for reliable, reproducible, and robust inference from AI-generated data. Acknowledging LLMs’ utility for task automation, such as text annotation, or to expand our understanding of human psychology, we argue for diversifying human samples and expanding psychology’s methodological toolbox to promote an inclusive, generalizable science, countering homogenization, and over-reliance on LLMs.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper warns against “GPTology”—the hurried, unjustified use of large language models as replacements for human participants or as universal text-analysis tools in psychology. It combines review with empirical demonstrations showing LLM biases, reduced response variance, weak zero-shot annotation, and reproducibility problems, while arguing LLMs can still aid research if validated, fine-tuned, and used cautiously.

Key findings

  • GPT-3.5’s responses to the Moral Foundations Questionnaire-2 across 1,000 queries had 43–121 times smaller variance than human data across six moral domains (all *P*s < 0.001), and its moral-value network differed substantially from a diverse human sample (*N* = 3,902 from 19 populations).
  • Across self-report measures from over 43 countries, ChatGPT showed significantly less variance and demographic differences; for example, it was more agreeable than politically liberal (*d* = −0.230), conservative (*d* = −0.403), and moderate (*d* = −0.285) individuals, and endorsed less right-wing authoritarianism than male (*d* = 0.44), White (*d* = 0.35), and younger 18–24 (*d* = 0.49) participants.
  • For moral-language annotation of 2,983 social media posts, fine-tuned BERT achieved F1 = 0.48 and fine-tuned ChatGPT-3.5 achieved F1 = 0.53, outperforming zero-shot ChatGPT (F1 = 0.22), LIWC (F1 = 0.27), and few-shot ChatGPT (F1 = 0.32); zero-shot ChatGPT was over 10 times more likely to predict a text as moral compared with human raters (Δodds = +1,240%, *P* < 0.001).
  • Minor prompt changes significantly altered outputs: adding a study introduction to the Big Five Inventory reduced Openness (*d* = −0.30), Extraversion (*d* = −0.54), and Agreeableness (*d* = −0.14); a modified moral prompt changed care (−56%), equality (−62%), loyalty (−33%), authority (+71%), and purity (+164%) annotations.

Methods

  • The paper combines a critical review with original empirical analyses. It administered the MFQ-2 to GPT-3.5 1,000 times and compared responses with a culturally diverse human sample from 19 populations, then extended comparisons to self-report measures from over 43 countries, including Big Five Inventory (*N* = 3,924), Need for Closure (*N* = 315), Need for Cognition (*N* = 900), Right-wing authoritarianism (*N* = 1,020), Empathizing–Systemizing (*N* = 3,141), and Rational–Experiential Inventory (*N* = 1,456).
  • For text analysis, ChatGPT annotated 2,983 social media posts for moral language in zero-shot, few-shot, and fine-tuned settings; results were compared with fine-tuned BERT, LIWC, human raters, and Contextualized Construct Representation (CCR). The authors also tested prompt modifications and an open-source LLaMA-based pipeline.
  • Main analyses included variance comparisons, effect sizes (*d*), F1 scores, odds changes, inter-correlations and partial-correlation networks, and Dunnett’s tests.

Limitations

  • Stated: the authors note that LLMs are rapidly evolving and that the concerns raised “could already be mitigated in the near future”; they also caution that open-source models do not fully solve the black-box problem, prompting may be insufficient to close demographic bias gaps, fine-tuning is resource-intensive, and few-shot learning depends on example quality and stability.
  • Stated: closed-source models lack freely available past versions, and even added parameters such as “seed” do not guarantee reproducibility; “system_fingerprint” increases transparency but not reproducibility itself.
  • Evident from the text: the empirical demonstrations focus on GPT-3.5/ChatGPT and selected psychological measures and annotation tasks, so the specific numerical findings may not generalize to all LLMs, model versions, or research domains.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/pnasnexus/pgae245

Cites

111 of 111 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 100 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GPT-3.5's variance was 43-121 times smaller than human data in different moral domains, even when using parameter settings for maximum variability in generated responses.

    “GPT-3.5’s variance was 43–121 times smaller than human data in different moral domains, even when using parameter settings for maximum variability in generated responses.”
    Quote in the paperSubstantial claim
  • Findingown data

    Differences of this kind and strength were observed across all surveys and demographic groups.

    “Differences of this kind and strength were observed across all surveys and demographic groups.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The networks of moral values based on GPT queries and a diverse human sample are substantially different.

    “As shown in Fig. 1(b) and (c), the networks of moral values are substantially different, indicating that GPT struggles to produce previously established nomological networks.”
    Quote in the paperSubstantial claim
  • Findingown data

    When responding to personality surveys, ChatGPT was significantly more agreeable than politically liberal individuals, conservatives, and moderates.

    “For example, when responding to personality surveys, ChatGPT was significantly more agreeable than politically liberal individuals (d=−0.230, P<0.001, 95% CI [−0.369, −0.091]), conservatives (d=−0.403, P<0.001, 95% CI [−0.550, −0.257]), and moderates (d=−0.285, P<0.001, 95% CI [−0.424, −0.145]) as shown in Fig. 2.”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-3.5 produces substantially smaller variance in six moral domains (care, equality, proportionality, loyalty, authority, and purity) compared with actual human populations, with all Ps < 0.001.

    “As can be seen in Fig. 1(a), GPT produces substantially smaller variance in six moral domains (i.e. care, equality, proportionality, loyalty, authority, and purity) compared with actual human populations (all Ps<0.001).”
    Quote in the paperSubstantial claim
  • May combine two claims. 38
  • Classified as probably not a substantial claim. 17

  • May combine two claims.

    Findingown data

    Across a broad range of self-report measures from participants in over 43 countries, ChatGPT responses generally showed significantly less variance across all measures and differed significantly between various demographics.

    “Across these psychological constructs, we consistently found that ChatGPT responses generally showed significantly less variance across all measures (see Table S20 for an overview of variance differences across all surveys) and differed significantly between various demographics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingcitation

    On the MFTC dataset, BERT, despite being applied cross-context, outperformed GPT-4 on the Authority and Purity foundations, and GPT-4-Turbo also on the Loyalty foundation.

    “Further comparisons (Tables S7 and S8 in Rathje et al. (75)) on another moral foundations dataset, on which the BERT model was not fine-tuned (MFTC; (81)), showed that BERT, despite being applied cross-context, outperformed GPT-4 on the Authority and Purity foundations, and GPT-4-Turbo also on the Loyalty foundation.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Rathje et al. (75) Rathje, 2023

      Not checked in the free run.

    • cites [81] Hoover, 2020, Moral foundations twitter corpus: a collection of 35k tweets annotated for moral sentiment, Soc Psychol Personal Sci source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The LIWC-based approach outperformed ChatGPT zero-shot with an F1 score of 0.27 and was significantly less likely and less extreme in deviating from trained human annotators.

    “Notably, even the LIWC-based approach outperformed ChatGPT (zero-shot) with an F1 score of 0.27 and was significantly less likely and less extreme in deviating from trained human annotators, despite being a magnitudes smaller, less complex, and cheaper model.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Zero-shot GPT-3.5 outperformed the reported GPT-4 results on three foundations (Care, Proportionality, Authority), indicating inconsistent advancements between newer and older models.

    “Interestingly, even our zero-shot application of GPT-3.5 outperformed the reported GPT-4 results on three foundations (Care, Proportionality, Authority), indicating inconsistent advancements between newer and older models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rathje et al. (75) Rathje, 2023

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    Fine-tuned BERT achieved an average F1 score of 0.48 and fine-tuned ChatGPT-3.5 an average score of 0.53, compared with reported F1 scores of 0.34 for GPT-4 and 0.29 for GPT-4-Turbo.

    “Our fine-tuned models also performed substantially better on average, with fine-tuned BERT achieving an average F1 score of 0.48 and fine-tuned ChatGPT-3.5 an average score of 0.53, compared with reported F1 scores of 0.34 for GPT-4 and 0.29 for GPT-4-Turbo.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rathje et al. (75) Rathje, 2023

      Not checked in the free run.

  • May combine two claims.

    Findingcitation

    GPT-4 performed slightly better than BERT on the Loyalty foundation and significantly better only on the Care foundation.

    “GPT-4 performed slightly better than BERT on the Loyalty foundation and significantly better only on the Care foundation.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rathje et al. (75) Rathje, 2023

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    ChatGPT was more extreme in over- or under-predicting a moral sentiment, while BERT did in all but one case not significantly deviate from trained human annotators.

    “Additionally, ChatGPT was more extreme in over- or under-predicting a moral sentiment, while BERT did in all but one case not significantly deviate from trained human annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGPT was still more extreme in over- or under-predicting individual moral sentiments compared with BERT, which replicated the distribution of moral foundations most similar to human annotators.

    “However, note that in our experiments, ChatGPT was still more extreme in over- or under-predicting individual moral sentiments compared with BERT, which replicated the distribution of moral foundations most similar to human annotators.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Both few-shot learning and fine-tuning increased the performance of ChatGPT in the experiments.

    “As anticipated, both few-shot learning and fine-tuning increased the performance of ChatGPT in our experiments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Additional analyses show increased performance for ChatGPT when predicting at the item level instead of the construct level, highlighting that LLMs can benefit from integrating theory-driven approaches.

    “Similarly, our additional analyses, showing increased performance for ChatGPT when predicting at the item level instead of the construct level, highlight that LLMs can benefit from integrating theory-driven approaches.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Despite the LLaMA-based model's significantly smaller size (7 billion vs. 175 billion parameters) and being self-hosted on consumer-grade hardware, it achieved comparable performance in a text annotation task (F1 of 0.23 vs. 0.22) and a similar response pattern when answering self-report psychological measures.

    “Despite the model’s significantly smaller size (7 billion vs. 175 billion parameters) and while being self-hosted on consumer-grade hardware, it achieved a comparable performance in a text annotation task discussed above (F1 of 0.23 vs. 0.22) and a similar response pattern when answering self-report psychological measures.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The model was significantly less likely to annotate the care foundation (−56%), the equality foundation (−62%), and the loyalty foundation (−33%), and significantly more likely to annotate the authority foundation (+71%) and the purity foundation (+164%).

    “was significantly less likely to annotate the care foundation (−56%, P<0.001), the equality foundation (−62%, P<0.001), the loyalty foundation (−33%, P=0.009), and significantly more likely to annotate the authority foundation (+71%, P<0.001), and purity foundation (+164%, P=0.020).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    ChatGPT has shown cultural biases toward American perspectives or majority populations in general.

    “For example, ChatGPT has shown gender biases favoring male perspectives and narratives (35, 36), cultural biases toward American perspectives (37) or majority populations in general (38), and political biases favoring liberal, environmental, and left-libertarian viewpoints (11, 39, 40).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [37] Cao, 2023

      Not checked in the free run.

    • cites [38] Chen, 2022

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Models like GPT have been trained chiefly on WEIRD (Western, Educated, Industrialized, Rich, Democratic) people's textual data primarily in English, perpetuating the English-centricity of psychology.

    “Models like GPT have been trained chiefly on WEIRD (Western, Educated, Industrialized, Rich, Democratic; (32)) people’s textual data primarily in English, perpetuating the English-centricity of psychology (33), hindering efforts to take linguistic diversity seriously.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [32] Henrich, 2010, The weirdest people in the world?, Behav Brain Sci source

      Not checked in the free run.

    • cites [33] Blasi, 2022, Over-reliance on English hinders cognitive science, Trends Cogn Sci (Regul Ed) source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The difference in convenience between zero-shot applications of LLMs and fine-tuning of models may not be as stark as commonly perceived.

    “Furthermore, the difference in convenience between zero-shot applications of LLMs and fine-tuning of models may not be as stark as commonly perceived.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Fine-tuned small(er) language models for various tasks are now increasingly publicly available.
    • Rests on an assumption: There are more and more high-quality and specialized datasets available to researchers for fine-tuning language models themselves.

  • May combine two claims.

    Premisecitation

    Training and deploying LLMs is energy intensive and therefore connected to significant carbon emissions.

    “See also the discussion of environmental costs of LLMs in the Supplementary Materials, as training and deploying LLMs is energy intensive and therefore connected to significant carbon emissions (92, 93).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [92] Bender, 2021

      Not checked in the free run.

    • cites [93] Strubell, 2019

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The fine-tuned model still outperformed GPT-4-Turbo on three out of four foundations (and two out of four for GPT-4) and performed better on average than both.

    “We suggest a slightly different narrative, given that the fine-tuned model still outperformed GPT-4-Turbo on three out of four foundations (and two out of four for GPT-4) and performed better on average than both.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rathje et al. (75) Rathje, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The output variance when repeating the same prompt using a temperature of 1 simply reflects the output probability over the response options and therefore how sure the model is about its response, which is affected by the correct answer bias.

    “However, the output variance when repeating the same prompt using a temperature of 1 then simply reflects the output probability over the response options and therefore how sure the model is about its response, which itself is affected by the correct answer bias observed by Park et al. (17) and others.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17] Park, 2024, Diminished diversity-of-thought in a standard large language model, Behav Res source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.

    “Kocoń et al. (76) found that LLMs may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. (18) found that LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18] Huang, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Recent critiques emphasize limitations of LLMs in text analysis, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas.

    “However, this perspective of LLMs, as a convenient and comprehensive tool in psychological text analysis, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas (18, 76, 77).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [18] Huang, 2023

      Not checked in the free run.

    • cites [76] Kocoń, 2023, ChatGPT: Jack of all trades, master of none, Inf Fusion source

      Not checked in the free run.

    • cites [77] Reiss, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    LLMs may produce inconsistent outputs when subjected to minor prompt variations and can fall short of scientific thresholds for reliability, sometimes even when pooling outputs from multiple repetitions and prompts.

    “Furthermore, Reiss (77) demonstrated that LLMs may produce inconsistent outputs when subjected to minor prompt variations and can fall short of scientific thresholds for reliability, sometimes even when pooling outputs from multiple repetitions and prompts.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [77] Reiss, 2023

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Human morality has been argued to be a particularly difficult concept to grasp for language models.

    “Human morality has also been argued to be a particularly difficult concept to grasp for language models, making it an interesting test case for comparing powerful LLMs against less complex smaller language models (90), as well as testing the efficacy of different LLM application strategies.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [90] Russell, 2019

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Webb et al. examined the analogical reasoning abilities of ChatGPT and found that it had developed an emergent capacity for zero-shot reasoning, allowing it to solve a wide range of analogy problems without explicit training.

    “For instance, Webb et al. (26) examined the analogical reasoning abilities of ChatGPT and found that it had developed an emergent capacity for zero-shot reasoning, allowing it to solve a wide range of analogy problems without explicit training.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] Webb, 2023, Emergent analogical reasoning in large language models, Nat Hum Behav source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Using LLMs conveniently out-of-the-box can sometimes drastically fail, while fine-tuning can mitigate the very same issues.

    “These findings show that, first, LLMs’ supposed cross-context and flexibility advantages may not always hold, and second, that using LLMs conveniently “out-of-the-box” can sometimes drastically fail while highlighting that fine-tuning can mitigate the very same issues.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Online posts often contain little contextual information due to length constraints.

    “This is because moral values are often only implicitly signaled in language, and online posts often contain little contextual information due to length constraints.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    The MFQ-2 is a new measure published in 2023, and its items are unlikely to exist verbatim in the corpora on which GPT was trained, so it does not have the problem of extensively appearing in the training corpora.

    “The MFQ-2 is a new measure published in 2023, and its items are unlikely to exist verbatim in the corpora on which GPT was trained; hence, it does not have the problem of extensively appearing in the training corpora.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Fine-tuning ChatGPT or applying a few-shot paradigm can yield better results but such processes are significantly more resource-intensive and expensive compared with LIWC or even BERT.

    “It should be noted here that while fine-tuning ChatGPT or applying a few-shot paradigm can yield better results, such processes are significantly more resource-intensive and expensive compared with LIWC or even BERT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    Moral values are often only implicitly signaled in language.

    “This is because moral values are often only implicitly signaled in language, and online posts often contain little contextual information due to length constraints.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    The authors caution against treating the use of open-source rather than closed-source LLMs as sufficient to solve the black-box problem: the problem remains largely unsolved, even for open-source LLMs.

    “We caution against mistakenly treating the use of open-source rather than closed-source LLMs as sufficient to solve the black-box problem: a problem that remains largely unsolved, even for open-source LLMs.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Compared with closed-source LLMs, open-source LLMs may come with a replicability benefit and increased control over the model in computational-psychological research.

    “Compared with closed-source LLMs, open-source LLMs may thus come with a replicability benefit and increased control over the model when it comes to computational–psychological research.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisereasoning

    One of the main advantages of computational models is that they allow us to look under the hood and thus make inferences about psychological processes that may be difficult to test otherwise.

    “One of the main advantages of computational models is that they allow us to “look under the hood” and thus make inferences about psychological processes that may be difficult to test otherwise.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Across the experiments, LLMs can achieve high performances, but this cannot always be achieved using the model's zero-shot capabilities and instead requires exploring techniques such as few-shot prompting or outright fine-tuning.

    “Across our experiments, we found that LLMs can achieve high performances but stress that this cannot always be achieved using the model’s zero-shot capabilities and instead requires exploring techniques, such as few-shot prompting or outright fine-tuning of the LLM”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationnone given

    In cases where data for fine-tuning is scarce, such as with new constructs or understudied populations, LLMs' zero-shot capabilities may still offer acceptable performances and allow researchers to investigate urgent research questions.

    “Particularly in cases where data for fine-tuning is scarce, such as with new constructs or understudied populations, LLMs’ zero-shot capabilities may still offer acceptable performances and allow researchers to investigate urgent research questions.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Such lack of transparency is antithetical to the scientific principles of openness and replicability that should be central to computational-psychological research.

    “Such lack of transparency is antithetical to the scientific principles of openness and replicability that should be central to computational–psychological research (106).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [106] Hofman, 2021, Integrating explanation and prediction in computational social science, Nature source

      Not checked in the free run.

  • May combine two claims.

    Premisenone given

    ChatGPT and other closed-source models require, if possible at all, fine-tuning on a corporate platform, which introduces constraints compared with models like BERT that provide full control about the fine-tuning process and can be trained offline without additional costs.

    “Additionally, ChatGPT and other closed-source models require (if possible at all) fine-tuning on a corporate platform, which introduces constraints compared with models like BERT which provide full control about the fine-tuning process and can be trained offline without additional costs”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Overall results show that LLMs can be a superior tool but achieving this level of performance might not always be trivial and convenient.

    “Overall, our results show that LLMs can be a superior tool but they also show that achieving this level of performance might not always be trivial and convenient.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Fine-tuning can be costly, might not always work, and negates many of the conveniences of using LLMs in a zero-shot setting by requiring a dataset for training and expertise with training models.

    “For example, fine-tuning can be costly, might not always work, and negates many of the conveniences of using LLMs in a zero-shot setting by requiring a dataset for training and expertise with training models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning

    The system_fingerprint value increases transparency, allowing researchers to identify potential reasons for failure to replicate, but does not increase reproducibility itself.

    “However, the “seed” does not guarantee reproducibility, and while the “system_fingerprint” increases transparency, allowing researchers to identify potential reasons for failure to replicate, it does not increase reproducibility itself.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Openness is a spectrum requiring researchers to closely monitor to what extent LLMs fulfill relevant openness criteria for their projects.

    “Additionally, openness is a spectrum requiring researchers to closely monitor to what extent LLMs fulfill relevant openness criteria for their projects.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [110] Liesenfeld, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Even when updates address and mitigate detected biases, they also introduce the potential for process reproducibility failure in the generated data and impede reproducibility.

    “This means that even when updates address and mitigate detected biases, they also introduce the potential for “process reproducibility failure” (98) in the generated data and impede reproducibility, critical for scientific rigor (15, 100, 101).”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [98] Nosek, 2022, Replicability, robustness, and reproducibility in psychological science, Annu Rev Psychol source

      Not checked in the free run.

    • cites [15] Bail, 2024, Can generative AI improve social science?, Proc Natl Acad Sci U S A source

      Not checked in the free run.

    • cites [100] Spirling, 2023, Why open-source generative AI models are an ethical way forward for science, Nature source

      Not checked in the free run.

    • cites [101] Zhai, 2023, Chatgpt for next generation science learning, XRDS: Crossroads ACM Mag Stud source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Past work and the present findings consistently demonstrate that for many use cases, smaller (fine-tuned) models can be more powerful and less biased than the current large (generative) language models, particularly in zero-shot and few-shot settings.

    “Taken together, past work and our present findings consistently demonstrate that for many use cases, smaller (fine-tuned) models can be more powerful and less biased than the current large (generative) language models, particularly in zero-shot and few-shot settings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Overconfidence with which researchers may trust open-source LLMs' findings as replicable and transparent can systematically cause such researchers to over-rely on these findings, even in out-of-distribution settings where the findings may not generalize.

    “The overconfidence with which researchers may trust open-source LLMs’ findings as replicable and transparent can systematically cause such researchers to over-rely on these findings, even in out-of-distribution settings where the findings may not generalize.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Wang et al. (34) argued that due to their training processes, LLMs inherently struggle with accurately representing identity groups, often leading to distorted, simplified, and stereotypical portrayals of these groups.

    “Wang et al. (34) argued that due to their training processes, LLMs inherently struggle with accurately representing identity groups, often leading to distorted, simplified, and stereotypical portrayals of these groups.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Wang et al. (34) Wang, 2024

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Considering context and specific prompts is important when using ChatGPT to emulate diverse human behaviors.

    “Our research emphasizes the importance of considering context and specific prompts when using ChatGPT to emulate diverse human behaviors (125).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [125] Rao, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    OpenAI offers embedding models, such as "text-embedding-ada-3," that can be used in a similar manner to BERT for downstream tasks.

    “Additionally, OpenAI offers embedding models, such as “text-embedding-ada-3,” that can be used in a similar manner to BERT for downstream tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    GPTology-the hurried and unjustified application of LLMs either as replacements for human participants or as an off-the-shelf one-size-fits-all method in psychological text analysis-can lead to a proliferation of low-quality research, especially if the convenience of using LLMs such as ChatGPT leads researchers to rely too heavily on them.

    “GPTology—which we define as the hurried and unjustified application of LLMs either as “replacements” for human participants, or as an off-the-shelf “one-size-fits-all” method in psychological text analysis—can lead to a proliferation of low-quality research, especially if the convenience of using LLMs such as ChatGPT leads researchers to rely too heavily on them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    Continuous algorithmic changes necessitate ongoing validation of prompting strategies, a task made challenging by the complex and opaque nature of LLMs.

    “Continuous algorithmic changes necessitate ongoing validation of prompting strategies, a task made challenging by the complex and opaque nature of LLMs (72, 123, 126).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [72] Liu, 2023, Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing, ACM Comput Surv source

      Not checked in the free run.

    • cites [123] Mishra, 2023

      Not checked in the free run.

    • cites [126] Jiang, 2022

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The presence of a WEIRD bias, along with the opaque and often irreproducible nature of these models, particularly the proprietary ones, makes LLMs a double-edged sword for psychological research.

    “While LLMs, especially fine-tuned ones, can achieve impressive performances on many tasks, the presence of a WEIRD bias, along with the opaque and often irreproducible nature of these models, particularly the proprietary ones, makes them a double-edged sword for psychological research.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Many advantages attributed to LLMs are also present in other models.

    “It is further crucial to recognize that many advantages attributed to LLMs are also present in other models.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The newest generation of LLMs offered as an online service (e.g. ChatGPT, Gemini, Claude) provides many researchers with an accessible yet powerful tool.

    “The newest generation of LLMs offered as an online service (e.g. ChatGPT, Gemini, Claude) provide many researchers with an accessible yet powerful tool—representing a pivotal advancement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Models such as BERT or open-source LLMs can be accessed via APIs, providing a convenient and low-cost option for researchers who cannot self-host these technologies.

    “For instance, models such as BERT or open-source LLMs can be accessed via APIs, providing a convenient and low-cost option for researchers who cannot self-host these technologies.”
    Quote in the paperNot a substantial claim
51 of 51

  • The paper relies on this without support.

    LLMs are trained on vast amounts of data that can contain many of the items and tasks used in psychological experiments, leading the model to rely on its memory instead of making inferences.

    “Furthermore, LLMs are trained on vast amounts of data, which can contain many of the items and tasks used in psychological experiments, thus leading the model to rely on its memory instead of making inferences, exacerbating the problem above.”

  • The paper relies on this without support.

    LLMs largely operate as black-boxes.

    “However, unlike traditional agent-based models, which define the environment, agents, interactions, and their consequences to allow for inferences based on these parameters, LLMs largely operate as black-boxes.”

  • The paper relies on this without support.

    Fine-tuned small(er) language models for various tasks are now increasingly publicly available.

    “Fine-tuned small(er) language models for various tasks are now increasingly publicly available.”

  • The paper relies on this without support.

    There are more and more high-quality and specialized datasets available to researchers for fine-tuning language models themselves.

    “Similarly, there are more and more high-quality and specialized datasets available to researchers for fine-tuning language models themselves.”

  • The paper relies on this without support.

    Human annotations provide a valid ground truth for moral language classification.

    “We compared the annotations of all models against the “ground truth” of human raters.”

  • The paper relies on this without support.

    ChatGPT's behavior is representative of large language models generally.

    “Building on this perspective, we extend these recent studies by investigating ChatGPT’s ability to annotate moral language across three distinct settings: zero-shot, few-shot, and fine-tuned.”

  • The paper relies on this without support.

    The human annotator demographics and psychometrics provided for the test data are accurate and valid for assessing annotation bias.

    “We did so by using human annotator demographics and psychometrics provided for our test data (see Supplementary Materials for a detailed description of the experimental procedures).”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT itself can be fine-tuned to further increase its performance.

    “Additionally, GPT itself can be fine-tuned to further increase its performance. For instance, one paper found that fine-tuned GPT outperformed even fine-tuned BERT at detecting moral foundations (73).”

    From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [73]

  • The output of human annotators, like GPT, is also not reproducible and usually has far lower test-retest reliability.

    “the output of human annotators, like GPT, is also not reproducible and usually has far lower test–retest reliability (73).”

    From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [73]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Conflicting direction of equality-annotator bias

    Minor

    The paper says ChatGPT is biased toward annotators with a lower preference for equality, then describes its agreement as being with equality-endorsing annotators. Those statements give opposite directions for the reported equality bias.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Rumelhart, 1987, Parallel distributed processing: explorations in the microstructure of cognition: foundations, 45
  2. Radford, 2019, Language models are unsupervised multitask learners, OpenAI Blog, 1, 9

Cited by

Sorted by citations

Related

Authors
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M.
Affiliations
  • University of Southern California (US)
  • Harvard University (US)
  • Massachusetts Institute of Technology (US)
Corresponding author
Atari, Mohammad
ORCID
Venue
PNAS Nexus
Publisher
Oxford University Press (OUP)
Published
2024-06-28
Volume
3
Issue
7
Pages
pgae245-pgae245
ISSN
2752-6542 (ISSN-L)
DOI
10.1093/pnasnexus/pgae245
PMC
PMC11249969
OpenAlex
W4400678448
Semantic Scholar
9e96c4a7dd4cbadd6d03f4a30325466641839831
Cited by
111
References
62
Citations by year
2026: 51 · 2025: 54 · 2024: 6
FWCI
96.27
Type
journal article
Language
en
Open access
Licence
Locations
  • PNAS Nexus (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
  • DARPA INCAS (HR001121C0165)
  • AFOSR (A9550-23-1-0463)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, psychological research methods, AI-generated data, text analysis, zero-shot learning, psychological diversity, sample diversity, reproducibility, AI ethics, ChatGPT, task automation, textual annotation, overreliance on AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review