The DOI matches its Crossref record; 80 of the 80 references it cites were found in a registry or an index; 35 of the 97 sources its claims cite were checked for support.
Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback
Open accessAbstract
This paper critically evaluates the attempts to align Artificial Intelligence (AI) systems, especially Large Language Models (LLMs), with human values and intentions through Reinforcement Learning from Feedback methods, involving either human feedback (RLHF) or AI feedback (RLAIF). Specifically, we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. Through a multidisciplinary sociotechnical critique, we examine both the theoretical underpinnings and practical implementations of RLHF techniques, revealing significant limitations in their approach to capturing the complexities of human ethics, and contributing to AI safety. We highlight tensions inherent in the goals of RLHF, as captured in the HHH principle (helpful, harmless and honest). In addition, we discuss ethically-relevant issues that tend to be neglected in discussions about alignment and RLHF, among which the trade-offs between user-friendliness and deception, flexibility and interpretability, and system safety. We offer an alternative vision for AI safety and ethics which positions RLHF approaches within a broader context of comprehensive design across institutions, processes and technological systems, and suggest the establishment of AI safety as a sociotechnical discipline that is open to the normative and political dimensions of artificial intelligence.
Overview
CoChat Summary
TL;DR
This paper argues that RLHF and RLAIF are deeply insufficient for achieving AI safety and ethical AI, especially when framed through the vague HHH principle of helpfulness, harmlessness, and honesty. It calls for repositioning RLHF within a broader sociotechnical approach to AI safety and ethics.
Key findings
- The HHH principle is underdefined and inconsistently operationalised; the paper notes that Bai et al. (2022) explicitly left “helpful” and “harmless” for crowdworkers to interpret, which risks inconsistent and diluted ethical standards.
- RLHF creates internal tensions: increased helpfulness can mislead users about LLM nature, sycophancy sacrifices truth for apparent helpfulness/harmlessness, and feedback-guided alignment can impose hegemonic values and increase ethical opacity.
- The paper reports that Christiano et al. (2017) showed complex RL tasks could be solved with feedback samples amounting to less than 1 per cent of agent interactions, and that RLAIF output ranking was estimated to be 10 times cheaper than using human annotators.
- The authors propose an alternative: comprehensive design across institutions, processes, and technological systems, and establishing AI safety as a sociotechnical discipline open to normative and political dimensions.
Methods
- Design: multidisciplinary sociotechnical critique combining technical, philosophical, and system safety perspectives.
- Data/sample: no empirical dataset or sample; the paper analyses theoretical underpinnings and practical implementations of RLHF/RLAIF and the HHH principle.
- Main analysis: conceptual critique of alignment goals, internal tensions, and ethical issues, followed by a proposed reframing of AI safety and ethics.
Limitations
- The paper states that the impact of sycophancy as induced by RLHF is hard to quantify in specific cases, and that investigating specific effects of RLHF/RLAIF on commercial LLMs is exceedingly difficult because companies continuously change these systems, making them a moving target for research.
- Evident from the text: this is a conceptual and critical paper without new empirical experiments or quantitative validation of its proposed alternative.
- Evident from the text: its critique focuses primarily on RLHF/RLAIF and the HHH principle, so it does not claim to cover all alignment methods or all possible sociotechnical interventions.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
30 Valid · 1 Not valid · 37 Not checked · 62 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“As such, RLHF and RLAIF introduce further difficulties for properly understanding and designing the inner workings of LLMs in ways that improve accountability (Dobbe, 2022).”
§ Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback > Internal tensions and ethical issues in RLHF > RLHF increases ethical opacity
What the sources say
- Partly supportedcites Dobbe, 2022 sourceExperimental — accuracy not yet measured
In Leveson Lesson 5 (page not provided), the paper describes LLMs as complex and difficult to equip with appropriate safety requirements. Leveson Lesson 7 discusses accountability gaps in AI systems. Neither passage attributes additional difficulties in understanding or designing LLMs to RLHF or RLAIF specifically. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Technology-first proposals for value alignment, such as RLHF, tend to neglect the role of democratic institutions in ethical deliberation through law and policy (Gansky & McDonald, 2022).”
§ Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback > Rebooting AI safety and ethics > Addressing values through comprehensive design
What the sources say
- Partly supportedcites Gansky & McDonald, 2022 sourceExperimental — accuracy not yet measured
Section 2.2 criticizes treating the lab as the locus of ethical practice, and Section 3.2 says FAccT proposals perfect lab work rather than support participation by public and justice institutions. Those passages support the general criticism of technology-centered approaches, but the paper does not address RLHF or value-alignment proposals specifically. Page numbers were not supplied in our copy. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“They showed that these complex problems could be solved without direct access to a reward model that would otherwise be difficult to compute, and could instead be learned through a few iterations of feedback samples (less than 1 per cent of the agent interactions with the environment).”
§ Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback > Background
What the sources say
- Supportedcites Christiano et al. (2017) source
The Abstract (p. 1) reports solving complex RL tasks without access to the reward function while providing feedback on less than 1% of agent–environment interactions. The Introduction (p. 2) explains that the authors fit a reward function to human preferences while training the agent, including by collecting feedback online. Thus, the unavailable reward was the task’s true reward function; the method learned a reward model from feedback. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“which have been shown to contain toxic, harmful, and untruthful content (Dinan et al., 2021)”
§ Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback > Background
What the sources say
- Partly supportedcites Dinan et al., 2021 sourceExperimental — accuracy not yet measured
Section 2.1 (p. 6) reports that language models can generate toxic content, and Section 2 (p. 5) describes harmful responses from conversational agents. Section 2.3 (p. 7) discusses the risk of incorrect advice, but does not show that LLM outputs have been found to contain untruthful content. The paper also distinguishes conversational agents from LLMs, so its broader safety examples do not establish every part of the claim specifically for LLMs. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Indeed, in some cases (Cui et al., 2023), aligning with ‘honesty’ can lead to an increased tendency for LLMs to add ‘unsure’ language in responses.”
§ Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback > Limitations of RLHF > Honesty
What the sources say
- Partly supportedcites Cui et al., 2023 sourceExperimental — accuracy not yet measured
The Additional information section’s Honesty principle explicitly asks assistants to express uncertainty and use phrases such as “I guess” and “perhaps.” Section 2.2 also describes assessing whether expressed uncertainty matches answer accuracy. Neither passage establishes that honesty alignment increased the frequency of such language in LLM responses. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
53 more claims · 9 with their cited sources checked
Create a free account to see all 130 claims and to verify the 15 references still open.
- Probably not a substantial claim. 43
- It may state two separate claims as one. 40
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“AI systems (designed for user satisfaction and nonconfrontational dialogue) often default to sycophantic alignment [38].”
From Delusional Experiences Emerging From AI Chatbot Interactions or “AI Psychosis” (2025) · cites this paper as [38]
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 4×
“These models are first trained on large corpora of data such as articles, books, and websites—requiring unprecedented volumes of training data (Bender et al., 2021).”
InfluentialTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“While the use of RLHF has led to improvements in LLMs’ tendencies to hallucinate, it has not protected against it entirely (Casper et al., 2023; Ouyang et al., 2022).”
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
High-Dimensional Continuous Control Using Generalized Advantage Estimation
Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2015) · arXiv · doi:10.48550/arxiv.1506.02438 · Cited by 2,103
Engineering a Safer World: Systems Thinking Applied to Safety
(2012) · Choice Reviews Online · doi:10.5860/choice.49-6305 · Cited by 1,566 · cited 2×
“…systems in comparison to previous, more physically constrained machines leads to what she dubbed the curse of flexibility : “with software, the limits of what is possible to accomplish are different than the limits of what can be accomplished successfully and safely” (Leveson, 2012, p. 50).”
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 1,313
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“The large amount of text in their training datasets allows LLMs to derive internal representations of various linguistic rules and patterns that form the foundation on which LLMs are then fine-tuned to perform other downstream tasks, such as question-answering (Jawahar et al., 2019; Goldberg, 2019).”
Research works printed but not linked: 4
- Bansal, H., Dang, J., & Grover, A. Peering through preferences: Unraveling feedback acquisition for aligning large language models. In: ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models.
- Fricker, M. (2010). Epistemic Injustice (Reprinted). Oxford University Press
- Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S.R., Durmus, E., Hatfield-Dodds, Z., Johnston, S.R., Kravec, S.M., Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D., Zhang, M., & Perez, E. (2024). Towards understanding sycophancy in language models. In: The Twelfth International Conference on Learning Representations https://openreview.net/forum?id=tvhaxkMKAn.
- Sorensen, T., Moore, J., Fisher, J., Gordon, M., Mireshghallah, N., Rytting, C.M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. (2024). Position: a roadmap to pluralistic alignment. In: Proceedings of the 41st International Conference on Machine Learning, pp. 46280–46302.
Cited by
Delusional Experiences Emerging From AI Chatbot Interactions or “AI Psychosis”
Hudon, A., & Stip, E. (2025) · JMIR Mental Health · doi:10.2196/85799 · Cited by 88
Programmed to please: the moral and epistemic harms of AI sycophancy
Turner, C., & Eisikovits, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01007-4 · Cited by 35
AI Mimicry and Human Dignity: Chatbot Use as a Violation of Self‐Respect
van der Rijt, J., Coelho Mollo, D., & Vaassen, B. (2026) · Journal of Applied Philosophy · doi:10.1111/japp.70037 · Cited by 13
Self-citationThe Curiosity Paradox: How Sycophantic GenAI May Undermine Learning
Mishra, P., & Henriksen, D. (2025) · TechTrends · doi:10.1007/s11528-025-01156-z · Cited by 9
Zhang, Z., & Valeo, C. (2026) · AI for Engineering · doi:10.3390/aieng1010005 · Cited by 5
Metadata
- Authors
- Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R.
- Affiliations
- Umeå University (SE)
- Vrije Universiteit Amsterdam (NL)
- Delft University of Technology (NL)
- Corresponding author
- Ericson, Petter
- Venue
- Ethics and Information Technology
- Published
- 2025-06
- Volume
- 27
- Issue
- 2
- Pages
- 28-28
- ISSN
- 1388-1957 (ISSN-L)
- 1572-8439
- PMC
- PMC12137480
- OpenAlex
- W4411039658
- Semantic Scholar
- ff5b0cc250d93b97fe60e1b0c2048708d6875595
- Cited by
- 54
- References
- 57
- Citations by year
- 2026: 47 · 2025: 7
- FWCI
- 55.54
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 3 d)
- Locations
- Ethics and Information Technology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- VU Research Portal (publishedVersion, cc-by)
- DiVA at Umeå University (Umeå University) (submittedVersion, other-oa)
- Funders
- European Commission (101120237)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (024.004.022)
- Umea University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- AI alignment, reinforcement learning from human feedback, reinforcement learning from AI feedback, large language models, AI safety, helpfulness, harmlessness, honesty, deception, interpretability, ease of use, AI ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback. https://app.cochat.ai/papers/10.1007/s10676-025-09837-2
- The paper
- Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025). Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback. Ethics and Information Technology. https://doi.org/10.1007/s10676-025-09837-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.