PreprintCited by 376Ethics and Social Impacts of AIHate Speech and Cyberbullying DetectionTopic Modeling

Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models

Open access
Ferrara, Emilio·2023·arXiv

Abstract

As the capabilities of generative language models continue to advance, the implications of biases ingrained within these models have garnered increasing attention from researchers, practitioners, and the broader public. This article investigates the challenges and risks associated with biases in large-scale language models like ChatGPT. We discuss the origins of biases, stemming from, among others, the nature of training data, model specifications, algorithmic constraints, product design, and policy decisions. We explore the ethical concerns arising from the unintended consequences of biased model outputs. We further analyze the potential opportunities to mitigate biases, the inevitability of some biases, and the implications of deploying these models in various applications, such as virtual assistants, content generation, and chatbots. Finally, we review the current approaches to identify, quantify, and mitigate biases in language models, emphasizing the need for a multi-disciplinary, collaborative effort to develop more equitable, transparent, and responsible AI systems. This article aims to stimulate a thoughtful dialogue within the artificial intelligence community, encouraging researchers and developers to reflect on the role of biases in generative language models and the ongoing pursuit of ethical AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2304.03738

Cites

86 of 88 references verified1 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generative conversational AI may expose users to undesirable consequences, such as biases, misinformation, and manipulation.

    “However, generative conversational AI may also expose users to undesirable consequences, such as biases, misinformation, and manipulation (Brandtzaeg, et al. , 2022; Ferrara, 2023).”

    From Why do people use ChatGPT? Exploring user motivations for generative conversational AI (2024) · cites this paper as Ferrara, 2023

    Section 5.2 (page unavailable) describes harms from bias in generative language models and says their use in content moderation could amplify harmful content and misinformation. The passages found do not address manipulation of users, so that example is not supported by this source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • Models trained on human data can exhibit human biases.

    “It is well-understood that models trained on human data - such as being pre-trained on human-written text or fine-tuned on human feedback - can exhibit human biases.”

    From Multi-Agent Risks from Advanced AI (2025) · cites this paper as Ferrara, 2023

  • Generative AI misuse includes parroting harmful stereotypes and misconceptions about certain social groups.

    “These range from parroting harmful stereotypes and misconceptions about certain social groups (Bianchi et al. 2023;Ferrara 2023)”

    From Epistemic Injustice in Generative AI (2024) · cites this paper as Ferrara 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 5

  1. D.K. Citron and F. Pasquale, 2014. "The scored society: Due process for automated predictions," Washington Law Review, volume 89, number 1, pp. 1-33, and at https://digitalcommons.law.uw.edu/wlr/vol89/iss1/2/, accessed 27 October 2023.
  2. S. Dev and J.M. Phillips, 2019. "Attenuating bias in word vectors," Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pp. 879-887, and at https://proceedings.mlr.press/v89/dev19a.html, accessed 27 October 2023.
  3. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polosukhin, 2017. "Attention is all you need," Advances in Neural Information Processing Systems, volume 30, at https://papers.nips.cc/paper_files/paper/2017, accessed 27 October 2023.
  4. B.L. Whorf, 1964. Language, thought, and reality: Selected Writings of Benjamin Lee Whorf. Edited by John B. Carroll. Cambridge, Mass.: MIT Press.
  5. R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, and Y. Choi, 2019. "Defending against neural fake news," ANIPS'19: Proceedings of the 33rd International Conference on Neural Information Processing Systems, article number 812, pp. 9,054-9,065.

Cited by

Sorted by citations

Authors
Ferrara, E.
Affiliations
University of Southern California (US)
Corresponding author
Ferrara, Emilio
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
1396-0458 (ISSN-L)
OpenAlex
W4388488349
Cited by
282
References
49
Citations by year
2026: 55 · 2025: 77 · 2024: 95 · 2023: 55
FWCI
70.56
Type
journal article
Language
en
Open access
Locations
  • First Monday (publishedVersion, cc-by-nc-sa)
  • SSRN Electronic Journal (acceptedVersion)
  • First Monday (acceptedVersion, cc-by-nc-sa)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Viterbi School of Engineering, University of Southern California
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
ChatGPT, large language models, algorithmic bias, bias mitigation, training data bias, ethical AI, generative AI risks, chatbots, virtual assistants, content creation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review