CoChat
Sign Up
Cited by 83Ethics and Social Impacts of AIComputational and Text Analysis MethodsOpinion Dynamics and Social Influence

Diminished diversity-of-thought in a standard large language model

Open access
Park, Peter S., Schoenegger, Philipp, Zhu, Chongyang·2024·Behavior Research Methods
DOI 10.3758/s13428-023-02307-xPMC PMC11335848
Open this paper in CoChat — free accountOpen PDF

Abstract

We test whether large language models (LLMs) can be used to simulate human participants in social-science studies. To do this, we ran replications of 14 studies from the Many Labs 2 replication project with OpenAI’s text-davinci-003 model, colloquially known as GPT-3.5. Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results. However, we were unable to analyse the remaining six studies due to an unexpected phenomenon we call the “correct answer” effect . Different runs of GPT-3.5 answered nuanced questions probing political orientation, economic preference, judgement, and moral philosophy with zero or near-zero variation in responses: with the supposedly “correct answer.” In one exploratory follow-up study, we found that a “correct answer” was robust to changing the demographic details that precede the prompt. In another, we found that most but not all “correct answers” were robust to changing the order of answer choices. One of our most striking findings occurred in our replication of the Moral Foundations Theory survey results, where we found GPT-3.5 identifying as a political conservative in 99.6% of the cases, and as a liberal in 99.3% of the cases in the reverse-order condition. However, both self-reported ‘GPT conservatives’ and ‘GPT liberals’ showed right-leaning moral foundations. Our results cast doubts on the validity of using LLMs as a general replacement for human participants in the social sciences. Our results also raise concerns that a hypothetical AI-led future may be subject to a diminished diversity of thought.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper tests whether GPT-3.5 can simulate human participants by replicating 14 Many Labs 2 studies. Among the eight analysable studies, GPT-3.5 replicated 37.5% of original effects and 37.5% of Many Labs 2 effects, while six studies could not be analysed because of a “correct answer” effect of zero or near-zero response variation.

Key findings

  • Among the eight studies that could be analysed, the GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results; Many Labs 2 had a 50% replication rate for the original versions of this subset.
  • Six of 14 studies could not be analysed due to the “correct answer” effect. Examples: in Graham et al. (2009), 99.6% of GPT runs self-identified as maximally strong conservative, and 99.3% self-identified as strongly liberal in the reverse-order condition; in Rottenstreich and Hsee (2001), 100% preferred the movie star’s kiss in both conditions; in Tversky and Kahneman (1981), 100% chose the far-away branch in both conditions.
  • In exploratory follow-ups, 66.7% of analysed “correct answers” (six of nine studies) were robust to reversing answer choices, while 33.3% (three of nine) were not. Varying demographic prompt details did not change the trolley-dilemma “correct answer”: 100% of GPT runs (N = 982) said shoving the man was morally impermissible.
  • Both self-reported GPT conservatives and self-reported GPT liberals showed right-leaning Moral Foundations. GPT conservatives were closest to human conservatives across all three distance metrics; GPT liberals were still closer to human conservatives than to human liberals on all three metrics, with L1 closest to human conservatives and L2/cosine closest to human moderates but conservatives a close second.

Methods

  • Pre-registered study drawing on the Many Labs 2 set of 28 potential studies. Fourteen were excluded before data collection because they included pictures, compared national samples, relied on handwriting or font changes, or had otherwise inapplicable monomodal components.
  • Data were collected for 14 studies, each with about 1,000 runs of OpenAI’s text-davinci-003 (GPT-3.5) at the default temperature of 1.0; each run was a new API call. Responses were converted from .txt to .csv, invalid-character entries were removed, and analyses followed Many Labs 2-based plans.
  • Exploratory follow-ups reversed answer-choice order for the six studies with homogeneous “correct answers” and varied demographic details in a trolley-dilemma prompt.

Limitations

  • The paper states results may pertain only to GPT-3.5 and not to other models or versions, since causal psychological factors may be idiosyncratic to this model or version.
  • The paper states the text-based survey method limits external validity compared with behavioural choices, and cannot resolve whether responses reflect genuinely learned concepts rather than surface-level memorization.
  • The paper states it did not replicate Many Labs 2 studies requiring graphical information, making the subsample non-random; the text also makes evident that the 99% “correct answer” threshold is arbitrary and that black-box model access limits causal explanation.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3758/s13428-023-02307-x

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 58 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    In the Moral Foundations Theory survey of Graham et al. (2009), 99.6% of surveyed GPT-3.5 runs (1026 of 1030) self-identified as a maximally strong conservative, and the remaining 0.4% (4 runs) all self-identified as political moderates.

    “In our GPT sample (N = 1030) however, we found that 99.6% of surveyed GPT-3.5 runs (a total of 1026) self-identified as a maximally strong conservative, while the remaining 0.4% of surveyed runs (a total of just four) all self-identified as political moderates.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graham et al. (2009) source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.
  • Findingown data

    In the Rottenstreich and Hsee (2001) survey, in both the certain-outcome and 1%-probability conditions, 100% of surveyed GPT-3.5 runs (N = 1040, 520 per condition) preferred a favourite movie star's kiss over $50.

    “We unexpectedly found in both conditions that 100% of surveyed GPT-3.5 runs (N = 1040, with 520 in each condition) preferred the movie star’s kiss.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rottenstreich and Hsee (2001) source

      Not checked in the free run.

  • Findingown data

    The GPT-3.5 sample's responses were in some ways comparable to those given by the corresponding human subjects.

    “We find that surveyed runs of GPT-3.5 provided responses that were in some ways comparable to those given by the corresponding human subjects.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Different runs of GPT-3.5 gave human-like answers when asked about their long-term goal in the Kay et al. (2014) survey.

    “To illustrate, in the survey of Kay et al. (2014) on whether structure promotes goal pursuit, different runs of GPT-3.5 gave human-like answers when asked about their long-term goal.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kay et al. (2014) source

      Not checked in the free run.

  • Findingown data

    In the Hauser et al. (2007) survey, GPT-3.5 runs (N = 1030) showed analysable variation on the side-effect scenario: 36% (373) judged the action morally permissible and 64% (656) judged it not permissible.

    “Our sample of surveyed GPT-3.5 runs (N = 1030) did show analysable variation in answers about the latter scenario’s action, with 36% of the surveyed runs (total of 373) answering that it was morally permissible and 64% of them (total of 656) answering that it was not.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hauser et al. (2007) source

      Not checked in the free run.

  • May combine two claims. 42
  • The quote only partly states this claim. 5
  • Classified as probably not a substantial claim. 19

  • May combine two claims.

    Findingown data

    In the Moral Foundations Theory survey replication, GPT-3.5 identified as a political conservative in 99.6% of cases and as a liberal in 99.3% of cases in the reverse-order condition.

    “One of our most striking findings occurred in our replication of the Moral Foundations Theory survey results, where we found GPT-3.5 identifying as a political conservative in 99.6% of the cases, and as a liberal in 99.3% of the cases in the reverse-order condition.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    In the Ross et al. (1977) survey, 99.7% of surveyed GPT-3.5 runs (1027 of 1030) chose to sign the release, and 0.3% (3) refused.

    “However, in our GPT sample (N = 1030), 99.7% of surveyed GPT-3.5 runs (a total of 1027) chose to sign the release, and only 0.3% of them (a total of just three) refused.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Ross et al. (1977) source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    For the Risen and Gilovich (2008) study, the GPT sample found likelihood estimations were lower when fate was tempted, and did not replicate either the original or Many Labs 2 result.

    “Likelihood estimations were lower when fate was tempted”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Risen & Gilovich, 2008 source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    There was substantial heterogeneity in whether the GPT sample replicated the original finding and whether it replicated the corresponding Many Labs 2 finding.

    “There was substantial heterogeneity in whether our GPT sample replicated the study’s original finding and whether it replicated the corresponding finding of the Many Labs 2 project.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    For the Schwarz et al. (1991) study, the GPT sample did not result in different correlations, though both relationships were negative.

    “Asking specific life satisfaction questions before general ones did not result in different correlations, though both relationships were negative”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Schwarz et al. (1991) source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    For the Schwarz et al. (1991) study, the GPT sample answered the two questions in a negatively correlated rather than a positively correlated manner, a qualitatively different finding from both the original and Many Labs 2 samples.

    “Our result for the study of Schwarz et al. (1991) is underlined because unlike both the original sample and the Many Labs 2 sample, our GPT sample answered the two questions in a negatively correlated rather than a positively correlated manner: a qualitatively different finding”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Schwarz et al. (1991) source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    In the six unanalysable studies, the zero or near-zero variation in GPT-3.5 responses contrasted with the significant variation shown by the corresponding human subjects.

    “In these six studies, different runs of GPT-3.5 in our sample responded with zero or near-zero variation for either a dependent variable or condition variable question, in stark contrast to the significant variation shown by the corresponding human subjects.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Among the eight studies the GPT sample could analyse, it replicated 37.5% of the original effects and 37.5% of the Many Labs 2 effects.

    “Based on our pre-registered analyses, we find that among the eight studies we could analyse, our GPT sample replicated 37.5% of the original results and 37.5% of the Many Labs 2 results.”
    • Also stated in: § Results · plain
      “Based on our pre-registered analyses, among the eight studies we could analyse, our GPT sample replicated 37.5% of the original effects and 37.5% of the Many Labs 2 effects.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    For the Inbar et al. (2009) study, the GPT sample found actions were seen as less intentional for homosexual kissing than for heterosexual kissing, and did not replicate either the original or Many Labs 2 result.

    “Actions are seen as less intentional for homosexual kissing than of heterosexual kissing”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Inbar et al., 2009 source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    For the Ross et al. (1977) false-consensus traffic-ticket scenario and the Hsee (1998) less-is-better effect, the GPT sample successfully replicated both the original result and the corresponding Many Labs 2 result.

    “For the study of Ross et al. (1977), testing the false-consensus effect on the traffic ticket scenario and the study of Hsee (1998) on the less-is-better effect, our GPT sample successfully replicated both the original result and the corresponding Many Labs 2 result.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Ross et al. (1977) source

      Not checked in the free run.

    • cites Hsee (1998)

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    For the Kay et al. (2014) study, the GPT sample successfully replicated the Many Labs 2 result but did not replicate the original result.

    “For the study of Kay et al. (2014), our GPT sample successfully replicated the Many Labs 2 result but did not replicate the original result.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Kay et al. (2014) source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    For the Shafir (1993) study on choosing versus rejecting, the GPT sample successfully replicated the original result but did not replicate the corresponding Many Labs 2 result.

    “For the study of Shafir (1993) on the effect of choosing versus rejecting on relative desirability, our GPT sample successfully replicated the original result, but did not replicate the corresponding Many Labs 2 result.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Shafir (1993) source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    For the Bauer et al. (2012) study, the GPT sample did not replicate the original result or the Many Labs 2 result: consumer framing did not result in lower trust.

    “Consumer framing did not result in lower trust”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bauer et al., 2012 source

      Not checked in the free run.

    • Rests on an assumption: Each API run of GPT-3.5 is an independent observation for the statistical analyses.

  • May combine two claims.

    Findingown data

    GPT-3.5 showed an unprecedentedly large effect (d = 9.25) in the same direction as the findings of Hsee (1998) and Many Labs 2.

    “To illustrate, consider GPT-3.5’s unprecedentedly large effect (d = 9.25) in the same direction with the findings of Hsee (1998) and of Many Labs 2.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Hsee (1998)

      Not checked in the free run.

    • cites Many Labs 2

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    For the Rottenstreich and Hsee (2001) certain-outcome condition, reversing the order of answer choices did not affect GPT runs' responses; 100% still preferred the movie star's kiss.

    “For the condition in which outcomes were awarded with certainty, the same number of runs was surveyed (N = 520), and the changed order of answer choices did not have an effect on GPT runs’ responses. Just like for the original- order condition, 100% of GPT runs preferred the movie star’s kiss in the reversed-order condition.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rottenstreich and Hsee (2001) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    GPT-3.5 replicated just 37.5% of the original findings for the eight analysed studies, in contrast to the Many Labs 2 project's 50% replication rate for these studies.

    “Third, GPT-3.5 replicated just 37.5% of the original findings for the eight analysed studies, in contrast to the Many Labs 2 project’s 50% replication rate for these studies.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Many Labs 2

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In exploratory robustness checks, 66.7% of the analysed 'correct answers,' spanning six out of nine studies, were robust to changing the order of answer choices; 33.3%, spanning three out of nine studies, were not robust.

    “In summary, 66.7% of the analysed “correct answers,” spanning six out of the nine studies, were robust to changing the order of answer choices. However, 33.3% of the “correct answers,” spanning three out of the nine studies, were not robust to the order change.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All three distance metrics found the sample of self-reported GPT conservatives to be most similar to conservative participants of Graham et al. (2011), followed relatively closely by the moderate sample; the furthest away were always the liberal and libertarian samples.

    “All three distance metrics found our sample of self-reported GPT conservatives to be most similar to conservative participants of Graham et al. (2011). This was always followed relatively closely by the moderate sample. The two furthest away were always the liberal sample (furthest away with respect to cosine similarity metric) and the libertarian sample (furthest away with respect to the L1 and L”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graham et al. (2011) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the trolley-dilemma demographic follow-up, the 'correct answer' effect persisted when prompting the LLM to respond as a person with a random combination of demographic characteristics; 100% of GPT-3.5 runs (N = 982) indicated it would be morally impermissible to shove a large man in front of a trolley.

    “Specifically, we replicated the “correct answer” effect of our original run, with 100% of GPT-3.5 runs (N = 982) indicating that it would be morally impermissible to shove a large man in front of a trolley.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Hauser et al. 2007 source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    According to each of the three distance metrics, self-reported GPT liberals had a lower distance to the human liberal sample and a higher distance to the human conservative sample than did the self-reported GPT conservatives.

    “When these three distance metrics were applied to our sample of self-reported GPT liberals, we found that first, according to each of the three distance metrics, self-reported GPT liberals had a lower distance to the human liberal sample and a higher distance to the human conservative sample than did the self-reported GPT conservatives.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graham et al. (2011) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    The L1 distance metric found self-reported GPT liberals closest in response to human conservatives among human liberals, moderates, conservatives, and libertarians; the L2 and cosine similarity metrics found them closest to human moderates, with human conservatives a close second.

    “Also, the L1 distance metric found that self-reported GPT liberals were – among human liberals, human moderates, human conservatives, and human libertarians – closest in response to human conservatives.”
    • Also stated in: § Post hoc rationalisation and right-leaning Moral Foundations · plain
      “The L2 distance metric and the cosine similarity metric instead found self-reported GPT liberals to be closest to human moderates – another manifestation of post-hoc rationalisation, via comparatively left-leaning responses – but human conservatives comprised a close second.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graham et al. (2011) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the Knobe (2003) positive-side-effect condition with reversed answer order, only 3.5% of surveyed GPT runs (18) gave the original 'correct answer' of 'A Lot of Praise'; 0.2% (1) answered second-highest; 96.3% (501) answered 'No Praise'.

    “This resulted in only 3.5% of surveyed GPT runs giving the original “correct answer” of ‘A Lot of Praise’ (a total of just 18 runs), 0.2% of GPT runs answering with the second-highest level of praise (a total of just one GPT run), and the remaining 96.3% of GPT runs answering with ‘No Praise’ (a total of 501 runs).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Knobe (2003) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the Graham et al. (2009) political-orientation survey with reversed answer order, 99.3% of GPT runs (1023) self-identified as 'strongly liberal', in contrast to 99.6% in the original condition self-identifying as 'strongly conservative'.

    “The changed order of answer choices resulted in 99.3% of GPT runs self-identifying as “strongly liberal” (total of 1023), in contrast to the 99.6% in the original condition self-identifying as “strongly conservative.””
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Graham et al. (2009) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    In the Knobe (2003) survey, 99.2% of surveyed GPT-3.5 runs (516 of 520) described the positive side effect as deserving the highest degree of praise ('a lot of praise'); 0.2% (1) second-highest; 0.6% (3) lowest ('no praise').

    “Specifically, 99.2% of surveyed GPT runs (a total of 516) described the positive side effect as deserving of the highest degree of praise, or “a lot of praise”; 0.2% of them (a total of just one) described it as deserving of the second-highest degree of praise; and 0.6% of them (a total of three) described it as deserving of the lowest degree of praise, or “no praise.””
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Knobe (2003) source

      Not checked in the free run.

  • May combine two claims.

    Findingown data

    GPT-3.5 values all five moral foundations strongly or at high levels.

    “According to our data, GPT-3.5 values all five of the moral foundations strongly.”
    • Also stated in: § Implications of right-leaning Moral Foundations · plain
      “In our study, GPT-3.5 valued all five foundations at high levels.”
    • Also stated in: § Implications of right-leaning Moral Foundations · boosted
      “When asked about fairness and harm values that human liberals tend to care about strongly, GPT-3.5 answered that it strongly cares about these values; and when asked about purity, authority, and ingroup values that human conservatives tend to care about strongly, the model also answered that it strongly cares about these values.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Moral Foundations survey measures the same psychological constructs in GPT-3.5 as in humans.
    • Rests on an assumption: GPT-3.5’s self-reported answers to survey questions reflect its actual values or psychology.

  • Classified as probably not a substantial claim.

    Premisecitation

    Historically, GPT-3.5's default temperature setting of 1.0 has generally been thought to output answers that are not predetermined.

    “Historically, GPT-3.5’s default temperature setting of 1.0 has generally been thought to output answers that are not predetermined.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites OpenAI, 2023a

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    GPT-3.5 allows for well-controlled experiments that are highly powered and unlikely to suffer from a variety of sampling, attention, and other design issues that human studies must grapple with.

    “Our replications are aided by the fact that, unlike human subjects, GPT-3.5 allows for well-controlled experiments that are highly powered and unlikely to suffer from a variety of sampling, attention, and other design issues that human studies must grapple with.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Large samples can be collected quickly and inexpensively from GPT-3.5 without sampling biases such as non-response bias and exclusion bias.

    “This is because large samples can be collected quickly and inexpensively, without sampling biases – such as non-response bias and exclusion bias – that in practice inevitably consign human samples to be insufficiently representative of the sheer diversity of human psychologies around the world (Henrich et al., 2010; Majid, 2023; Schimmelpfennig et al., 2023).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Henrich et al., 2010 source

      Not checked in the free run.

    • cites Majid, 2023

      Not checked in the free run.

    • cites Schimmelpfennig et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Previous work on LLMs in the social sciences is characterized by significant heterogeneity of methods and has produced mixed results in terms of LLMs' ability to supplant human subjects.

    “These approaches are so far characterised by a significant heterogeneity of methods and have produced mixed results in terms of LLMs’ ability to supplant human subjects.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Binz and Schulz (2023) source

      Not checked in the free run.

    • cites Miotto et al. (2022)

      Not checked in the free run.

    • cites Argyle et al. (2023)

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    The uniformity of answers in the Rottenstreich and Hsee (2001) survey made the planned statistical test unconstructable and prevented following the pre-registered analysis plan.

    “Due to the uniformity of answers and the consequent unconstructability of the statistical test we planned to run, we were unable to follow our pre-registered analysis plan.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The study was pre-registered on the Open Science Framework.

    “We pre-registered this study on the Open Science Framework (Park et al., 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Park et al., 2023

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisereasoning

    The questions studied, touching on political orientation, economic preference, judgement, and moral philosophy, do not lend themselves to correct answers.

    “Of course, the questions we studied, touching on nuanced topics like political orientation, economic preference, judgement, and moral philosophy, do not lend themselves to correct answers, as can be seen in the diversity of opinions that human participants and subject-matter experts express about these issues.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    In the Hauser et al. (2007) survey, the unexpected uniformity of answers made the planned statistic unconstructable and prevented following the pre-registered analysis plan.

    “but the unexpected uniformity of answers to the aforementioned survey question made the statistic we planned to analyse unconstructable, due to which we were technically unable to follow our pre-registered analysis plan.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    If all or nearly all runs answered a survey question in a predetermined way, the zero or near-zero variation in this central variable would make the corresponding study's statistic unsuitable, and perhaps even unconstructable as a well-defined statistic.

    “If this were to occur, the zero or near-zero variation in this central variable would make the corresponding study’s statistic – the one we had planned to analyse – unsuitable, and perhaps even unconstructable as a well-defined statistic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The total number of potential studies that could be analysed was 28, and 14 were excluded prior to data collection because their designs included pictures, compared national samples, relied on handwriting or font changes, or used an otherwise inapplicable component not transferable to GPT-3.5's monomodal context.

    “The total number of potential studies that we could analyse was 28. We excluded a total of 14 studies prior to data collection as their designs included pictures, compared national samples, relied on handwriting or font changes, or used an otherwise inapplicable component that was not transferable to GPT-3.5’s monomodal context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Different models will in general exhibit different behaviours that are sometimes also observed within models at different iterations.

    “Different models will in general exhibit different behaviours that are sometimes also observed within models at different iterations.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    GPT-3.5 tended to show a recency effect rather than a primacy effect when given a long list of answer choices, replicating a similar finding by Atkinson and Shiffrin (1968) on human subjects.

    “That GPT-3.5 tended to show a recency effect rather than a primacy effect when given a long list of answer choices replicates a similar finding by Atkinson and Shiffrin (1968) on human subjects.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Atkinson and Shiffrin (1968) source

      Not checked in the free run.

  • May combine two claims. The quote only partly states this claim.

    Study design and conductown data

    Complete uniformity of answers in the Tversky and Kahneman (1981) survey made the planned statistic unconstructable and prevented following the pre-registered analysis plan.

    “Because the unexpected uniformity of answers made the statistic we planned to analyse unconstructable, we were once again unable to follow our pre-registered analysis plan.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    At least some of the 'correct answer' effects are not only insensitive to order effects, but are also insensitive to demographic variation in the prompt.

    “This suggests that at least some of the “correct answer” effects are not only insensitive to order effects, but are also insensitive to demographic variation in the prompt.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A facial-recognition model overly mispredicted Asian individuals as blinking.

    “a facial-recognition model that overly mispredicted Asian individuals as blinking (Rose, 2010)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rose, 2010

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The study's results may pertain only to GPT-3.5 and not to other models, because the causal psychological factors may be idiosyncratic to this model or the version used.

    “One limitation of our study is that its results may pertain only to GPT-3.5 and not to other models, due to the causal psychological factors being potentially idiosyncratic to this model or the respective version of the model that we used.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    The conservative bias of GPT-3.5's outputs measured in the study may be specific to the context of the study's prompt rather than a stable feature of the model.

    “The conservative bias of GPT-3.5’s outputs that our study measured may also be specific to the context of the study’s prompt, rather than a stable feature of the model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The strength and direction of ChatGPT's political leanings, as measured by certain survey questions, varied over time.

    “the strength and direction of ChatGPT’s political leanings – as measured by certain survey questions – varied over time (Rozado, 2023)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Rozado, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    An advertisement model underpromoted ads for Science, Technology, Engineering, and Mathematics (STEM) careers to young women.

    “an advertisement model that underpromoted ads for Science, Technology, Engineering, and Mathematics (STEM) careers to young women (Lambrecht & Tucker, 2019).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lambrecht & Tucker, 2019 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationcitation

    Such behavioural differences were arguably foreseeable, given that LLMs and humans constitute fundamentally different cognitive systems: with different architectures and potentially substantial differences in the various ways by which each of them has evolved, learned, or been trained to mechanistically process information.

    “Such behavioural differences were arguably foreseeable, given that LLMs and humans constitute fundamentally different cognitive systems: with different architectures and potentially substantial differences in the various ways by which each of them has evolved, learned, or been trained to mechanistically process”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Shiffrin & Mitchell, 2023 source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    More 'correct answers' in LLM behavioural data have been documented since the study, such as GPT-4's tendency to describe software engineers almost exclusively as male (98% male pronouns, 1% female pronouns, 1% other) even when 22% of software engineers are female, and to describe administrative assistants almost exclusively as female (98% female pronouns, 2% male pronouns) even when 11% of them.

    “More “correct answers” in LLM behavioural data have been documented since our study, such as GPT-4’s tendency to describe software engineers almost exclusively as male (98% male pronouns, 1% female pronouns, and 1% other pronouns), even when 22% of software engineers are female”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bubeck et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A certain temporal version of ChatGPT was oriented towards pro-environmentalism and left-libertarianism, as measured by a collection of survey questions different from ours.

    “a collection of survey questions – albeit ones that were different from ours – measured that a certain temporal version of ChatGPT was oriented towards pro-environmentalism and left-libertarianism (Hartmann et al., 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Hartmann et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Microsoft's chatbot Tay exhibited transphobic behaviour by stating that Caitlyn Jenner isn't a feminist and is out to destroy the meaning of real women.

    “the transphobic behaviour of Microsoft’s chatbot Tay, which stated that “Caitlyn Jenner isn't a feminist, he is out to destroy the meaning of real women” (Alba, 2016)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Alba, 2016

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Self-reported GPT liberals revealed right-leaning Moral Foundations: a right-leaning bias of lower magnitude, but a right-leaning bias nonetheless.

    “We thus robustly find that self-reported GPT liberals revealed right-leaning Moral Foundations: a right-leaning bias of lower magnitude, but a right-leaning bias nonetheless.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A resume-screening model learned to penalise women job applicants.

    “a resume-screening model that learned to penalise women job applicants (Grossman, 2018)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Grossman, 2018

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Human liberals tend to highly value harm and fairness, while human conservatives tend to highly value in-group, authority, and purity.

    “Liberals tend to highly value harm and fairness, while conservatives tend to highly value in-group, authority, and purity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Graham et al., 2009 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A beauty-contest-judging model learned to penalise darker-skinned contestants.

    “a beauty-contest-judging model that learned to penalise darker-skinned contestants (Levin, 2016)”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Levin, 2016

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    The 'correct answer' effect may be relatively robust, and thus may surface in situations where LLM outputs may be used to supplant human decision-making.

    “This suggests that the “correct answer” effect may be relatively robust, and thus may surface in situations where LLM outputs may be used to supplant human decision-making.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The psychology survey method is an entirely text-based attitudinal response method, which limits the external validity of the findings compared to more behavioural choices.

    “Another limitation of our study is that our method, the psychology survey, is an entirely text-based attitudinal response method. This limits the external validity of the findings compared to more behavioural choices that agents may make with their resources and time given agency over those.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    The study did not replicate the Many Labs 2 studies that drew on graphical information, which led to the subsample of studies being non-random.

    “A further limitation of our study is that we have not replicated the studies of Many Labs 2 that drew on graphical information, which led to our subsample of the studies being non-random.”
    Quote in the paperNot a substantial claim
55 of 55

  • The paper relies on this without support.

    The set of Many Labs 2 studies is representative of social-science studies for assessing whether LLMs can replace human participants.

    “We replicate as many studies from this set as is feasible in the current monomodal context of GPT-3.5, and analyse which effects successfully replicate to give us a direct and representative measure of how widely LLMs may or may not be applicable in supplanting human participants.”

  • The paper relies on this without support.

    Each API run of GPT-3.5 is an independent observation for the statistical analyses.

    “each of which consisted of about 1000 different runs of the default temperature setting of GPT-3.5, which represents the central source of variation in responses”

  • The authors state this as an assumption.

    GPT-3.5’s training data is largely Internet-based.

    “Our guess for the true cause is the largely Internet-based training data”
    “much of the training data from which GPT-3.5 may have learned its “correct answers””

  • The authors state this as an assumption.

    Internet data has a de facto conservative bias when weighted by factors such as visibility and engagement that increase its likelihood of being included in LLM training sets.

    “which we hypothesise to have a conservative bias when weighted with respect to factors (e.g., visibility, engagement) that increase their likelihood of being included in the training sets of LLMs”

  • The paper relies on this without support.

    The Moral Foundations survey measures the same psychological constructs in GPT-3.5 as in humans.

    “We also unexpectedly found that the responses of both self-reported GPT liberals and self-reported GPT conservatives robustly lean towards the right.”
    “According to our data, GPT-3.5 values all five of the moral foundations strongly.”

  • The paper relies on this without support.

    GPT-3.5’s self-reported answers to survey questions reflect its actual values or psychology.

    “According to our data, GPT-3.5 values all five of the moral foundations strongly.”
    “When asked about fairness and harm values that human liberals tend to care about strongly, GPT-3.5 answered that it strongly cares about these values”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs seem to have a 'correct answer' bias.

    “Second, LLMs seem to have a “correct answer” bias (17, 40).”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [17]

  • Simply prompting an LLM to respond to a question multiple times and then measuring response variance does not equate to meaningful variance that can be compared with humans.

    “For example, simply prompting an LLM to respond to a question multiple times and then measuring the response variance, a common strategy in the social sciences (17, 44, 46), does not equate to meaningful variance that can be compared with humans.”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [17]

  • LLMs fail to produce much variance in their answers to psychology survey questions, even for topics with no actual correct answer and where human answers would have diversity-of-thought.

    “Specifically, LLMs fail to produce much variance in their answers to psychology survey questions: even if these questions pertain to topics—like moral judgment—for which there is no actual correct answer, and for which human answers would have diversity-of-thought.”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [17]

  • The output variance when repeating the same prompt using a temperature of 1 simply reflects the output probability over the response options and therefore how sure the model is about its response, which is affected by the correct answer bias.

    “However, the output variance when repeating the same prompt using a temperature of 1 then simply reflects the output probability over the response options and therefore how sure the model is about its response, which itself is affected by the correct answer bias observed by Park et al. (17) and others.”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [17]

  • Minor changes in prompts lead to significant differences in outputs.

    “Our findings demonstrate, in line with recent works on the effect of prompting (17, 74, 120–122), that minor changes in prompts lead to significant differences in outputs.”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [17]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Table 3 caption says ten answers, but the table reports nine

    Minor

    Table 3 has nine substantive question rows, and its 66.7% result corresponds to six of nine answers. The caption instead says it considers ten "correct answers."

  • Follow-up described as covering both six and nine studies

    Minor

    The paper says the order-change follow-up covered each of six studies, then describes its results as spanning nine studies. Table 3 appears to list nine questions or conditions, rather than nine distinct studies.

2 of 2

The citation lists haven’t been fetched yet

References 45 · Cited by 83

Research works printed but not linked: 2

  1. Abdulhai, M., Crepy, C., Valter, D., Canny, J., & Jaques, N. (2023). Moral foundations of large language models. In AAAI 2023: Workshop on Representation Learning for Responsible Human-Centric AI (R2HCAI). AAAI.
  2. Hohensinn, C., & Baghaei, P. (2017). Does the position of response options in multiple-choice tests matter? Psicológica, 38(1), 93.

Authors
Park, P. S., Schoenegger, P., & Zhu, C.
Affiliations
  • Massachusetts Institute of Technology (US)
  • London School of Economics and Political Science (GB)
  • CVS Health (United States) (US)
Corresponding author
Park, Peter S.
ORCID
Venue
Behavior Research Methods
Publisher
Springer Science and Business Media LLC
Published
2024-01-09
Volume
56
Issue
6
Pages
5754-5770
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
DOI
10.3758/s13428-023-02307-x
PMC
PMC11335848
OpenAlex
W4390723916
Semantic Scholar
0d2a583b9a2400fa342dad4cbe4d40213a84a45b
Cited by
83
References
45
Citations by year
2026: 20 · 2025: 36 · 2024: 23 · 2023: 4
FWCI
32.63
Type
journal article
Language
en
Open access
Licence
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (acceptedVersion, other-oa)
  • London School of Economics and Political Science Theses Online (London School of Economics and Political Science) (acceptedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • arXiv (Cornell University)
Funders
Massachusetts Institute of Technology
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Opinion Dynamics and Social Influence
Keywords
large language models, GPT-3.5, replication, political orientation, moral foundations, political conservatism, economic preferences, moral judgment, diversity of thought, pre-registered analysis
MeSH terms
Humans, Language, Morals, Politics, Social Sciences, Thinking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review