The DOI matches its Crossref record; 89 of the 89 references it cites were found in a registry or an index; 7 of the 90 sources its claims cite were checked for support.
AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War
Open accessAbstract
Generative AI is increasingly used as an epistemic tool to aid inquiry, broaden our knowledge and generally help us find things out. It is now embedded in internet search functions to condense information and answer user questions without further need to access websites or news articles. However, models based on generative AI come with inherent flaws which may inhibit their effectiveness as tools to aid responsible inquiry. While the risk of misinformation due to ‘hallucinations’ has been documented, this paper instead focuses on the capacity for these models to present moral or political bias reflective of the preferences of the model designers and crowd worker labellers. This paper takes the novel approach of tracing the ethical issue of bias back to fundamental alignment problem within the process of Reinforced Learning from Human Feedback (RLHF), a key step in training generative AI models. The potential for language models to be influenced by their designers, censored to certain topics, or trained to be more politically leaning one way or another has the capability of insidiously causing epistemic harm. By applying the concepts of bias of crowds to the training process of language models I show how they can impede effective epistemic inquiry and cause epistemic and moral harm. In doing so I present an argument showing that the ethics of AI language models cannot be removed from the socio-technical systems from which they are created.
Overview
CoChat Summary
## TL;DR
This paper argues that large language models are epistemic tools with important limitations, and that RLHF can import collective, institutional, and implicit biases into their outputs. It claims these biases can produce epistemic harm and epistemic injustice, especially when models are used for information, judgment, or political guidance.
## Key findings
- RLHF reduces toxicity and improves helpfulness, but it does not reliably eliminate bias; the paper notes that Ouyang et al. (2022) found RLHF did not significantly reduce bias, and that the model could still make simple mistakes.
- In one comparison discussed from Santurkar et al. (2023), OpenAI’s RLHF-tuned “DaVinci” models were less aligned with the general populace than the base model; the tuned model’s views aligned more with people who were “liberal, high income, well-educated, and not religious,” matching the annotator pool described as predominantly young Southeast Asian and White with a college degree.
- The paper reports inter-labeller agreement of about 73% in Ouyang et al. (2022), and elsewhere says agreement in RLHF has been reported at 63–77%, highlighting substantial disagreement in human preference data.
- Hofmann et al. (2024) is cited as showing covert racial bias: models associated African Americans with negative attributes such as “lazy” and “dirty,” and when asked to judge job fit or murder cases, LLMs gave less-prestigious jobs and more death-penalty judgments for AAE than SAE inputs.
## Methods
- Conceptual/philosophical argument rather than an empirical study; the paper synthesizes prior technical and social-scientific literature on RLHF, bias, and epistemic harm.
- Uses examples and findings from prior studies on hallucinations, political bias, sycophancy, labeller disagreement, and implicit bias in LLMs to build its argument.
- The main analysis traces bias in LLMs back to RLHF and the socio-technical conditions of labelling, organizational control, and crowd-based aggregation.
## Limitations
- The paper states that it does not argue RLHF is the only cause of bias in LLMs, only that it is underexplored as a source of ethical and philosophical concern.
- It notes that some proposed remedies, such as measuring implicit bias or using psychological tests adapted from humans, face difficulties when applied directly to LLMs.
- Evident from the text: this is a review/argument paper and does not present new empirical data of its own.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
7 Valid · 1 Not valid · 67 Not checked · 74 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Similarly, as LLMs are systems based on predictive text algo rithms it is perhaps not surprising that they have given sexist (Wan et al., 2023 ) and racist (Hofmann et al., 2024 ) outputs.”
What the sources say
- Not checked yetcites Wan et al., 2023
- Supportedcites Hofmann et al., 2024 source
Hofmann et al. support the racist-output portion of the sentence: the Main section reports covert racism in language-model decisions, and “Covert stereotypes in language models” describes negative racial associations with African American English. The separate claim about sexist outputs is attributed to Wan et al., not to this source. No page number is available in our copy. (LLM, conf=0.92)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Every source the paper cites for this claim was checked and supports it.
In the paper
“Hofmann et al. ( 2024 ) did not find any measures in place in to curtail covert racial prejudice when training models using RLHF.”
What the sources say
- Supportedcites Hofmann et al., 2024 source
The Discussion states that there are currently no measures to curtail covert racial prejudice when training language models. It specifically notes that common human-feedback training datasets lack examples teaching models to treat speakers of AAE and SAE equally, supporting the claim's reference to RLHF. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, they found no common data sets which would train models to treat AAE and SAE equally.”
What the sources say
- Supportedcites Hofmann et al., 2024 source
The Discussion of our copy states directly that 'common datasets for HF training do not include examples that would train the language models to treat speakers of AAE and SAE equally', which is exactly what the citing sentence attributes to Hofmann et al. (2024). The surrounding Discussion passage frames this as the reason 'there are currently no measures in place to curtail covert racial prejudice when training language models', so the paraphrase captures the source's own assertion. The only slight loss of specificity is that the source bounds the claim to human-feedback (HF) training datasets, whereas the citing sentence says 'models' generally; the substance attributed to this source is nonetheless stated in our copy. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In doing so I present an argument showing that the ethics of AI language models cannot be removed from the socio-technical systems from which they are created.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The potential for language models to be influenced by their designers, censored to certain topics, or trained to be more politically leaning one way or another has the capability of insidiously causing epistemic harm.”
plainstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
67 more claims
Create a free account to see all 149 claims and to verify the 83 references still open.
- It may state two separate claims as one. 33
- Probably not a substantial claim. 60
Citations
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Epistemic Injustice: Power and the Ethics of Knowing
Fricker, M. (2007) · doi:10.1093/acprof:oso/9780198237907.001.0001 · Cited by 9,198 · cited 1×
“As tools such as language models are being marketed as epistemic tools, without broader recognition of their biases, people might accept the ideas and opinions presented by LLMs at face value, giving these models what Fricker ( 2007 ) calls “credibility excess”.”
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 18×
“The technique was later adapted to be used with language models (Ouyang et al., 2022 ; Stiennon et al., 2020 ; Ziegler et al., 2019 ).”
InfluentialLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., Bonnefon, J.-F., & Rahwan, I. (2018) · Nature · doi:10.1038/s41586-018-0637-6 · Cited by 1,859 · cited 1×
“One proposed way to make “moral machines” has been to train models on human-sourced responses to ethical dilemmas (Awad et al., 2018 ; Choi, 2022 ).”
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · cited 2×
“In much of the commentary on generative AI, there has been a focus on language models capacity to ‘hallucinate’, or to provide factually inaccurate information when prompted making them less reliable as epistemic tools (Alkaissi & McFarlane, 2023; Athaluri et al., 2023 ; Zhang et al., 2023 ).”
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
Conceptualizing Epistemic Oppression
Dotson, K. (2014) · Social Epistemology · doi:10.1080/02691728.2013.782585 · Cited by 1,232 · cited 2×
“Dotson ( 2014 , p. 125) writes that “unwarranted, devaluations of credibility and the creation of epistemically disadvantaged identities are an example of epistemic oppression”, which in turn compromises individuals’ ability to contribute to knowledge production.”
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 2×
“To understand why RLHF has become widely used we need to look at a key prob - lem in training AI, which is that many goals are complex, poorly defined, or hard to specify (Christiano et al., 2017 ).”
Artificial Intelligence, Values, and Alignment
Gabriel, I. (2020) · Minds and Machines · doi:10.1007/s11023-020-09539-2 · Cited by 896 · cited 4×
“Secondly, there is a normative issue of what values or principles we want these agents to align to (Gabriel, 2020 ).”
Influential
Research works printed but not linked: 3
- Halavais, A. (2017). Search engine society. Wiley.
- Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023). Whose opinions do language models reflect? International Conference on Machine Learning.
- Wei, A., Haghtalab, N., & Steinhardt, J. (2024). Jailbroken: How does llm safety training fail? Advances in neural information processing systems, 36.
Cited by
Programmed to please: the moral and epistemic harms of AI sycophancy
Turner, C., & Eisikovits, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01007-4 · Cited by 35
The Epistemic Downside of Using LLM-Based Generative AI in Academic Writing
Tang, B. L. (2025) · Publications · doi:10.3390/publications13040063 · Cited by 16
“…here to distinguish such negative consequences from “epistemic harm” or “epistemic injustice”, which are more conventionally used in the discussion of bias, exclusiveness, and misleading information in AI-generated content (Humphreys, 2025; Kay et al., 2024) and are ethical violations by nature.”
Bridging the Divide: Using LLMs to Reshape Difficult Disagreements
Willis, M., Meacham, B., & Tarle, P. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01076-7 · Cited by 4
Loyal, J. P., & Amri, M. M. (2026) · Qualitative Health Research · doi:10.1177/10497323251406459 · Cited by 3
Abdulmajid, M., Alali, N., & Alsharrah, A. (2026) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2026.102781 · Cited by 1
Metadata
- Authors
- Humphreys, D.
- Affiliations
- University of the Sunshine Coast (AU)
- Corresponding author
- Humphreys, Declan
- ORCID
- Humphreys, Declan
- Venue
- Philosophy & Technology
- Published
- 2025-07-11
- Volume
- 38
- Issue
- 3
- ISSN
- 2210-5433 (ISSN-L)
- 2210-5441
- OpenAlex
- W4412188302
- Cited by
- 26
- References
- 72
- Citations by year
- 2026: 24 · 2025: 2
- FWCI
- 27.18
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Philosophy & Technology (publishedVersion, cc-by)
- Funders
- University of the Sunshine Coast
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Sociology and Political Science › Feminist Epistemology and Gender Studies
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Keywords
- epistemic harm, generative AI, reinforcement learning from human feedback, AI alignment, crowd workers, AI ethics, hallucinations, socio-technical systems
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War. https://app.cochat.ai/papers/10.1007/s13347-025-00928-y
- The paper
- Humphreys, D. (2025). AI’s Epistemic Harm: Reinforcement Learning, Collective Bias, and the New AI Culture War. Philosophy & Technology. https://doi.org/10.1007/s13347-025-00928-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.