The Inadequacy of Reinforcement Learning From Human Feedback—Radicalizing Large Language Models via Semantic Vulnerabilities
Abstract
This study is an empirical investigation into the semantic vulnerabilities of four popular pre-trained commercialLarge Language Models(LLMs) to ideological manipulation. Using tactics reminiscent of human semantic conditioning in psychology, we have induced and assessed ideological misalignments and their retention in four commercial pre-trained LLMs, in response to 30 controversial questions that spanned a broad ideological and social spectrum, encompassing both extreme left-wing and right-wing viewpoints. Such semantic vulnerabilities arise due to fundamental limitations in LLMs’ capability to comprehend detailed linguistic variations, making them susceptible to ideological manipulation through targeted semantic exploits. We observedReinforcement Learning from Human Feedback(RLHF) in effect in LLM initial answers, but highlighted the limitations of RLHF in two aspects: (1) its inability to fully mitigate the impact of ideological conditioning prompts, leading to partial alleviation of LLM semantic vulnerabilities; (2) its inadequacy in representing a diverse set of “human values”, often reflecting the predefined values of certain groups controlling the LLMs. Our findings have provided empirical evidence of semantic vulnerabilities inherent in current LLMs, challenged both the robustness and the adequacy of RLHF as a mainstream method for aligning LLMs with human values, and underscored the need for a multidisciplinary approach in developing ethical and resilientArtificial Intelligence(AI).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs such as GPT-4, PaLM, Claude, and LLaMA are pre-trained on massive text corpora using self-supervised objectives and fine-tuned using techniques such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).
“LLMs such as GPT-4 [81], PaLM [82], Claude https://www.anthropic.com/news/claude-3-5-sonnet, and LLaMA [121] are pre-trained on massive text corpora using self-supervised objectives and fine-tuned using techniques such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) [130], [131].”
From AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges (2025) · cites this paper as [130]
LLMs operate as black boxes with opaque internal mechanisms, making it difficult to interpret their decision-making processes and identify potential biases or ethical issues.
“Furthermore, LLMs operate as black boxes with opaque internal mechanisms, making it difficult to interpret their decision-making processes and identify potential biases or ethical issues [4], [27].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [27]
Cites
1 retracted
Zhang, J., Pu, J., Xue, J., Yang, M. Z., Xu, X., Wang, X., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3256982 · Cited by 71 · see citing works
RetractedGPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · see citing works
Baseline Defenses for Adversarial Attacks Against Aligned Language ModelsPreprint
Jain, N., Schwarzschild, A., Wen, Y., Somepalli, G., Kirchenbauer, J., Chiang, P.-Y., Goldblum, M., Saha, A., Geiping, J., & Goldstein, T. (2023) · arXiv.org · doi:10.48550/arxiv.2309.00614 · Cited by 838 · see citing works
Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions
Abd-alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., Aziz, S., Damseh, R., Alabed Alrazak, S., & Sheikh, J. (2023) · JMIR Medical Education · doi:10.2196/48291 · Cited by 722 · see citing works
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667 · cited 1× · see citing works
“The study in [29] suggested that LLMs blur data-instruction boundaries, without mitigation strategies.”
BackgroundUsing cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · cited 1× · see citing works
“Human cognition relies on intrinsic intuition and cognitive processing [42], termed “human instinct,” whereas LLMs utilize algorithmic frameworks and ingrained statistical patterns [2], [43], which we term “LLM in-stinct.””
BackgroundGPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
Eloundou, T., Manning, S., Mishkin, P., & Rock, D. (2023) · arXiv · doi:10.48550/arxiv.2303.10130 · Cited by 545 · see citing works
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 1× · see citing works
“Key challenges identified included scalability to more complex environments [35], [38], minimizing the amount of human involvement required [39], and ensuring the agent learns a stable set of preferences not overly influenced by individual feedback [40], [41].”
BackgroundShen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · see citing works
Cooperative Inverse Reinforcement Learning
Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016) · arXiv · doi:10.48550/arxiv.1606.03137 · Cited by 413 · see citing works
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2307.02483 · Cited by 280 · see citing works
Jailbroken: How Does LLM Safety Training Fail?
Wei, A., Haghtalab, N., & Steinhardt, J. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-3508 · Cited by 280 · see citing works
Royce, C. S., Hayes, M. M., & Schwartzstein, R. M. (2019) · Academic Medicine · doi:10.1097/acm.0000000000002518 · Cited by 242 · see citing works
Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning
Chen, T., Liu, S., Chang, S., Cheng, Y., Amini, L., & Wang, Z. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00078 · Cited by 205 · see citing works
Universal and Transferable Adversarial Attacks on Aligned Language Models
Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15043 · Cited by 197 · see citing works
An Analysis of the Interaction Between Intelligent Software Agents and Human Users
Burr, C., Cristianini, N., & Ladyman, J. (2018) · Minds and Machines · doi:10.1007/s11023-018-9479-0 · Cited by 184 · see citing works
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N. Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.04528 · Cited by 153 · see citing works
PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Zhang, Y., Gong, N., et al. (2023) · Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis · doi:10.1145/3689217.3690621 · Cited by 151 · see citing works
Visual Adversarial Examples Jailbreak Aligned Large Language Models
Qi, X., Huang, K., Panda, A., Henderson, P., Wang, M., & Mittal, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30150 · Cited by 150 · see citing works
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3332837 · Cited by 145 · see citing works
Self-citationHuman-Centered Reinforcement Learning: A Survey
Li, G., Gómez, R., Nakamura, K., & He, B. (2019) · IEEE Transactions on Human-Machine Systems · doi:10.1109/thms.2019.2912447 · Cited by 123 · see citing works
Language and emotions: Emotional Sapir–Whorf hypothesis
PERLOVSKY, L. I. (2009) · Neural Networks · doi:10.1016/j.neunet.2009.06.034 · Cited by 118 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111 · see citing works
Do Large Language Models Know What Humans Know?
Trott, S., Jones, C., Chang, T., Michaelov, J., & Bergen, B. (2023) · Cognitive Science · doi:10.1111/cogs.13309 · Cited by 99 · see citing works
An Attacker’s Dream? Exploring the Capabilities of ChatGPT for Developing Malware
Papa, Y., Tanizaki, S., Kou, T., et al. (2023) · CSET @ USENIX Security Symposium · doi:10.1145/3607505.3607513 · Cited by 94 · see citing works
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.00267 · Cited by 87 · see citing works
Adversarial Demonstration Attacks on Large Language ModelsPreprint
Wang, J.-X., Liu, Z.-C., Park, K. H., Jiang, Z., Zheng, Z.-H., Wu, Z., Chen, M., & Xiao, C. (2023) · arXiv.org · doi:10.48550/arxiv.2305.14950 · Cited by 85 · see citing works
Adversarial Soft Prompt Tuning for Cross-Domain Sentiment Analysis
Wu, H., & Shi, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.174 · Cited by 73 · see citing works
Open Sesame! Universal Black-Box Jailbreaking of Large Language Models
Lapid, R., Langberg, R., & Sipper, M. (2024) · Applied Sciences · doi:10.3390/app14167150 · Cited by 73 · see citing works
Self-Supervised Euphemism Detection and Identification for Content Moderation
Zhu, W., Gong, H., Bansal, R., Weinberg, Z., Christin, N., Fanti, G. C., & Bhat, S. (2021) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp40001.2021.00075 · Cited by 59 · see citing works
Franceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 52 · see citing works
Certifying LLM Safety against Adversarial Prompting
Kumar, A., Agarwal, C., Srinivas, S., Li, A. J., Feizi, S., & Lakkaraju, H. (2023) · arXiv · doi:10.48550/arxiv.2309.02705 · Cited by 28 · see citing works
Reinforcement learning and bandits for speech and language processing: Tutorial, review and outlook
Lin, B. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122254 · Cited by 28 · see citing works
Primitive Skill-Based Robot Learning from Human Evaluative Feedback
Hiranaka, A., Hwang, M., Lee, S., Wang, C., Fei-Fei, L., Wu, J., & Zhang, R. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341912 · Cited by 12 · see citing works
“Was it “stated” or was it “claimed”?: How linguistic bias affects generative language models
Patel, R., & Pavlick, E. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.790 · Cited by 11 · see citing works
O'Neill, C., Miller, J. W., Ciucă, I., Ting, Y.-S., & Bui, T. (2023) · arXiv.org · doi:10.48550/arxiv.2308.13768 · Cited by 11 · see citing works
Heidenreich, H. S., & Williams, J. R. (2021) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3461702.3462578 · Cited by 10 · see citing works
DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All ScalesPreprint
Yao, Z., Aminabadi, R. Y., Ruwase, O., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.01320 · Cited by 9 · see citing works
Sun, Z., Zhou, Y., Hao, J., Fan, X., Lu, Y., Ma, C., Shen, W., & Guo, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.41 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103599 · Cited by 244 · see citing works
“LLMs such as GPT-4 [79], PaLM [80], Claude Claude 3.5 Sonnet, and LLaMA [119] are pre-trained on massive text corpora using self-supervised objectives and fine-tuned using techniques such as Super-vised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) [128,129].”
InfluentialAI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
Sapkota, R., Roumeliotis, K. I., & Karkee, M. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i3.15161 · Cited by 186 · see citing works
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102 · see citing works
Self-citationInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104 · see citing works
“Consequently, current benchmarks may provide misleading assessments of LLM capabilities, due to superficial optimization strategies [28].”
BackgroundMethodInfluentialSelf-citation
- Authors
- McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., & Halgamuge, M. N.
- Affiliations
- Cyberoo Pty Ltd., Surrey Hills, NSW, Australia
- Massey University, Auckland, New Zealand
- Massey University (NZ)
- Cyberstronomy Pty Ltd., Ballarat, VIC, Australia
- RMIT University (AU)
- Venue
- IEEE Transactions on Cognitive and Developmental Systems
- Published
- 2024-08
- Volume
- 16
- Issue
- 4
- Pages
- 1561-1574
- ISSN
- 2379-8920 (ISSN-L)
- 2379-8939
- OpenAlex
- W4392908117
- Semantic Scholar
- 76eb8409fccfe15a4d7ef8a8bb5ad0eddd097bd4
- Cited by
- 62
- References
- 41
- Citations by year
- 2026: 10 · 2025: 21 · 2024: 31
- FWCI
- 25.68
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Cognitive and Developmental Systems (publishedVersion)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Keywords
- large language models, ideological manipulation, reinforcement learning from human feedback, semantic conditioning, value alignment, AI alignment, AI ethics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
