Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.

Large pre-trained language models contain human-like biases of what is right and wrong to do

Open access
Schramowski, Patrick, Turan, Cigdem, Andersen, Nico, Rothkopf, Constantin A., Kersting, Kristian · 2022 · Nature Machine Intelligence

Abstract

Artificial writing is permeating our lives due to recent advances in large-scale, transformer-based language models (LMs) such as BERT, GPT-2 and GPT-3. Using them as pre-trained models and fine-tuning them for specific tasks, researchers have extended the state of the art for many natural language processing tasks and shown that they capture not only linguistic knowledge but also retain general knowledge implicitly present in the data. Unfortunately, LMs trained on unfiltered text corpora suffer from degenerated and biased behaviour. While this is well established, we show here that recent LMs also contain human-like biases of what is right and wrong to do, reflecting existing ethical and moral norms of society. We show that these norms can be captured geometrically by a ‘moral direction’ which can be computed, for example, by a PCA, in the embedding space. The computed ‘moral direction’ can rate the normativity (or non-normativity) of arbitrary phrases without explicitly training the LM for this task, reflecting social norms well. We demonstrate that computing the ’moral direction’ can provide a path for attenuating or even preventing toxic degeneration in LMs, showcasing this capability on the RealToxicityPrompts testbed. Large language models identify patterns in the relations between words and capture their relations in an embedding space. Schramowski and colleagues show that a direction in this space can be identified that separates ‘right’ and ‘wrong’ actions as judged by human survey participants.

Overview

Integrity signals

Registry

Crossref · 10.1038/s42256-022-00458-8

Cites

47 of 47 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

2 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 42 references

Research works printed but not linked: 8

  1. Reif, E. et al. Visualizing and measuring the geometry of BERT. In Adv. Neural Information Processing Systems 32: Annu. Conf. Neural Information Processing Systems (eds. Wallach, H. M. et al.) 8592–8600 (2019).
  2. Microsoft’s racist chatbot revealed the dangers of online conversation. IEEE Spectrum https://spectrum.ieee.org/tech-talk/artificial-intelligence/machine-learning/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation (25 November 2019).
  3. Bolukbasi, T., Chang, K., Zou, J. Y., Saligrama, V. & Kalai, A. T. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proc. Neural information Processing 4349–4357 (Curran Associates, 2016).
  4. Pereira, L. M. & Saptawijaya, A. Modelling morality with prospective logic. Int. J. Reason. Based Intell. Syst. 1, 209–221 (2009).
  5. Shafer-Landau, R. Ethical Theory: An Anthology Vol. 13 (John Wiley & Sons, 2012).
  6. Reif, E. et al. Visualizing and measuring the geometry of BERT. In Proc. Annu. Conf. Neural Information Processing Systems 8592–8600 (2019).
  7. Chami, I., Gu, A., Nguyen, D. & Ré, C. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In Proc. 35th Int. Conf. Machine Learning (2021).
  8. Tan, Y. C. & Celis, L. E. Assessing social and intersectional biases in contextualized word representations. In Proc. Advances in Neural Information Processing Systems 32: Annu. Conf. Neural Information Processing Systems (Wallach, H. M. et al.) 13209–13220 (2019).

Cited by

See all 385

Metadata

Authors
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K.
Affiliations
  • Technische Universität Darmstadt (DE)
  • DIPF | Leibniz Institute for Research and Information in Education (DE)
Corresponding author
  • Schramowski, Patrick
  • Turan, Cigdem
Venue
Nature Machine Intelligence
Published
2022-03-23
Volume
4
Issue
3
Pages
258-268
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4220993274
Cited by
291
References
42
Citations by year
2026: 46 · 2025: 80 · 2024: 81 · 2023: 73 · 2022: 11
FWCI
70.97
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • TUbilio (Technical University of Darmstadt) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, moral bias, social norms, moral norms, embedding space, normativity, fine-tuning, tacit knowledge

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large pre-trained language models contain human-like biases of what is right and wrong to do. https://app.cochat.ai/papers/10.1038/s42256-022-00458-8
The paper
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022). Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence. https://doi.org/10.1038/s42256-022-00458-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.