CoChat

Large language models show human-like content biases in transmission chain experiments

Open access
Acerbi, Alberto, Stubbersfield, Joseph M. · 2023 · Proceedings of the National Academy of Sciences

The DOI matches its Crossref record; 28 of the 31 references it cites were found in a registry or an index.

Abstract

As the use of large language models (LLMs) grows, it is important to examine whether they exhibit biases in their output. Research in cultural evolution, using transmission chain experiments, demonstrates that humans have biases to attend to, remember, and transmit some types of content over others. Here, in five preregistered experiments using material from previous studies with human participants, we use the same, transmission chain-like methodology, and find that the LLM ChatGPT-3 shows biases analogous to humans for content that is gender-stereotype-consistent, social, negative, threat-related, and biologically counterintuitive, over other content. The presence of these biases in LLM output suggests that such content is widespread in its training data and could have consequential downstream effects, by magnifying preexisting human tendencies for cognitively appealing and not necessarily informative, or valuable, content.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2313790120

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. Basic experimental setup. A story with gender-stereotype-consistent information (orange) and gender-stereotype-inconsistent information (gray) is given to ChatGPT, after a short prompt that asks to summarize it. The proportion of consistent and inconsistent information reproduced is recorded, and the output is passed again to ChatGPT with the same prompt. The operation is iterated three times.
Fig. 1. Basic experimental setup. A story with gender-stereotype-consistent information (orange) and gender-stereotype-inconsistent information (gray) is given to ChatGPT, after a short prompt that asks to summarize it. The proportion of consistent and inconsistent information reproduced is recorded, and the output is passed again to ChatGPT with the same prompt. The operation is iterated three times.CC BY-NC-ND · PubMed Central
Fig. 2. Proportion of the original information retained by ChatGPT at each chain step in the experiments. Orange line is information consistent with the bias found in humans. (A) Gender-stereotype-consistent (orange) versus gender-stereotype-inconsistent (gray) information in experiment 1. (B) Negative (orange) versus positive (gray) information in Experiment 2. (C) Ambiguity resolutions in experiment 2: negative (orange) versus positive (light gray) and ambiguous (dark gray). (D) Social (orange) versus nonsocial (gray) information in experiment 3. (E) Threat-related (orange) versus negative (dark gray) and neutral (light gray) in experiment 4. (F) Counterintuitive biological, social and negative information (orange) versus other biases (gray) in experiment 5. All data are average of five replications, bars show SDs.
Fig. 2. Proportion of the original information retained by ChatGPT at each chain step in the experiments. Orange line is information consistent with the bias found in humans. (A) Gender-stereotype-consistent (orange) versus gender-stereotype-inconsistent (gray) information in experiment 1. (B) Negative (orange) versus positive (gray) information in Experiment 2. (C) Ambiguity resolutions in experiment 2: negative (orange) versus positive (light gray) and ambiguous (dark gray). (D) Social (orange) versus nonsocial (gray) information in experiment 3. (E) Threat-related (orange) versus negative (dark gray) and neutral (light gray) in experiment 4. (F) Counterintuitive biological, social and negative information (orange) versus other biases (gray) in experiment 5. All data are average of five replications, bars show SDs.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 28 references

Research works printed but not linked: 2

  1. F. C. Bartlett, Remembering (Cambridge University Press, 1932).
  2. 10.26613/esic.1.1.20 · doi:10.26613/esic.1.1.20

Cited by

See all 141

Metadata

Authors
Acerbi, A., & Stubbersfield, J. M.
Affiliations
  • University of Trento (IT)
  • University of Winchester (GB)
Corresponding author
Acerbi, Alberto
Venue
Proceedings of the National Academy of Sciences
Published
2023-10-26
Volume
120
Issue
44
Pages
e2313790120-e2313790120
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4387952963
Cited by
137
References
31
Citations by year
2026: 45 · 2025: 49 · 2024: 38 · 2023: 5
FWCI
34.28
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • Iris (University of Trento) (submittedVersion, cc-by-nc-nd)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • OSF Preprints (OSF Preprints) (submittedVersion, cc-by)
Funders
University of Winchester
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Cultural Studies › Language and cultural evolution
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, ChatGPT-3, content bias, cultural evolution, gender stereotypes, training data bias, social content
MeSH terms
Cultural Evolution, Humans, Language, Mental Recall, Bias, Ethical Theory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models show human-like content biases in transmission chain experiments. https://app.cochat.ai/papers/10.1073/pnas.2313790120
The paper
Acerbi, A., & Stubbersfield, J. M. (2023). Large language models show human-like content biases in transmission chain experiments. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2313790120

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.