Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 35 of the 97 sources its claims cite were checked for support.

Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback

Open access
Dahlgren Lindström, Adam, Methnani, Leila, Krause, Lea, Ericson, Petter, de Rituerto de Troya, Íñigo Martínez, Coelho Mollo, Dimitri, Dobbe, Roel · 2025 · Ethics and Information Technology

Abstract

This paper critically evaluates the attempts to align Artificial Intelligence (AI) systems, especially Large Language Models (LLMs), with human values and intentions through Reinforcement Learning from Feedback methods, involving either human feedback (RLHF) or AI feedback (RLAIF). Specifically, we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. Through a multidisciplinary sociotechnical critique, we examine both the theoretical underpinnings and practical implementations of RLHF techniques, revealing significant limitations in their approach to capturing the complexities of human ethics, and contributing to AI safety. We highlight tensions inherent in the goals of RLHF, as captured in the HHH principle (helpful, harmless and honest). In addition, we discuss ethically-relevant issues that tend to be neglected in discussions about alignment and RLHF, among which the trade-offs between user-friendliness and deception, flexibility and interpretability, and system safety. We offer an alternative vision for AI safety and ethics which positions RLHF approaches within a broader context of comprehensive design across institutions, processes and technological systems, and suggest the establishment of AI safety as a sociotechnical discipline that is open to the normative and political dimensions of artificial intelligence.

Overview

CoChat Summary

TL;DR

This paper argues that RLHF and RLAIF are deeply insufficient for achieving AI safety and ethical AI, especially when framed through the vague HHH principle of helpfulness, harmlessness, and honesty. It calls for repositioning RLHF within a broader sociotechnical approach to AI safety and ethics.

Key findings

  • The HHH principle is underdefined and inconsistently operationalised; the paper notes that Bai et al. (2022) explicitly left “helpful” and “harmless” for crowdworkers to interpret, which risks inconsistent and diluted ethical standards.
  • RLHF creates internal tensions: increased helpfulness can mislead users about LLM nature, sycophancy sacrifices truth for apparent helpfulness/harmlessness, and feedback-guided alignment can impose hegemonic values and increase ethical opacity.
  • The paper reports that Christiano et al. (2017) showed complex RL tasks could be solved with feedback samples amounting to less than 1 per cent of agent interactions, and that RLAIF output ranking was estimated to be 10 times cheaper than using human annotators.
  • The authors propose an alternative: comprehensive design across institutions, processes, and technological systems, and establishing AI safety as a sociotechnical discipline open to normative and political dimensions.

Methods

  • Design: multidisciplinary sociotechnical critique combining technical, philosophical, and system safety perspectives.
  • Data/sample: no empirical dataset or sample; the paper analyses theoretical underpinnings and practical implementations of RLHF/RLAIF and the HHH principle.
  • Main analysis: conceptual critique of alignment goals, internal tensions, and ethical issues, followed by a proposed reframing of AI safety and ethics.

Limitations

  • The paper states that the impact of sycophancy as induced by RLHF is hard to quantify in specific cases, and that investigating specific effects of RLHF/RLAIF on commercial LLMs is exceedingly difficult because companies continuously change these systems, making them a moving target for research.
  • Evident from the text: this is a conceptual and critical paper without new empirical experiments or quantitative validation of its proposed alternative.
  • Evident from the text: its critique focuses primarily on RLHF/RLAIF and the HHH principle, so it does not claim to cover all alignment methods or all possible sociotechnical interventions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10676-025-09837-2

Cites

80 of 80 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

30 Valid · 1 Not valid · 37 Not checked · 62 Own results and interpretations

53 more claims · 9 with their cited sources checked

Create a free account to see all 130 claims and to verify the 15 references still open.

See all claims — free account
  • Probably not a substantial claim. 43
  • It may state two separate claims as one. 40

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 57 references

Research works printed but not linked: 4

  1. Bansal, H., Dang, J., & Grover, A. Peering through preferences: Unraveling feedback acquisition for aligning large language models. In: ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models.
  2. Fricker, M. (2010). Epistemic Injustice (Reprinted). Oxford University Press
  3. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S.R., Durmus, E., Hatfield-Dodds, Z., Johnston, S.R., Kravec, S.M., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2024). Towards understanding sycophancy in language models. In: The Twelfth International Conference on Learning Representations https://openreview.net/forum?id=tvhaxkMKAn.
  4. Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C.M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. (2024). Position: a roadmap to pluralistic alignment. In: Proceedings of the 41st International Conference on Machine Learning, pp. 46280–46302.

Cited by

See all 54

Metadata

Authors
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R.
Affiliations
  • Umeå University (SE)
  • Vrije Universiteit Amsterdam (NL)
  • Delft University of Technology (NL)
Corresponding author
Ericson, Petter
Venue
Ethics and Information Technology
Published
2025-06
Volume
27
Issue
2
Pages
28-28
ISSN
  • 1388-1957 (ISSN-L)
  • 1572-8439
OpenAlex
W4411039658
Cited by
54
References
57
Citations by year
2026: 47 · 2025: 7
FWCI
55.54
Type
journal article
Language
en
Open access
Licence
Locations
  • Ethics and Information Technology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • VU Research Portal (publishedVersion, cc-by)
  • DiVA at Umeå University (Umeå University) (submittedVersion, other-oa)
Funders
  • European Commission (101120237)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek (024.004.022)
  • Umea University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
AI alignment, reinforcement learning from human feedback, reinforcement learning from AI feedback, large language models, AI safety, helpfulness, harmlessness, honesty, deception, interpretability, ease of use, AI ethics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback. https://app.cochat.ai/papers/10.1007/s10676-025-09837-2
The paper
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025). Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback. Ethics and Information Technology. https://doi.org/10.1007/s10676-025-09837-2

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.