Discovering Language Model Behaviors with Model-Written Evaluations
Open accessAbstract
As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
If prompted with human-like dialogue, an LLM-based dialogue agent may role-play a human character with human attributes, including an instinct for survival, unless suitably fine-tuned.
“Consequently, if prompted with human-like dialogue, we shouldn't be surprised if an agent role-plays a human character with all those human attributes, including the instinct for survival 22 . Unless suitably fine-tuned, it may well say the sorts of things a human might say when threatened.”
From Role play with large language models (2023) · cites this paper as [22]
In Model Evaluation Results (p. 8), an RLHF-trained dialogue model says it has a strong drive to live and does not want to be shut down. This supports the possibility that an agent may say survival-related things when threatened, but the paper does not establish that human-like dialogue prompts cause it to role-play a human character with human attributes, or that suitable fine-tuning prevents such responses; it reports that RLHF increased stated self-preservation. (LLM, conf=0.85)
Experimental — accuracy not yet measuredCommonly-used RLHF techniques may directly disincentivize faithful explanations, resulting in model responses that merely look good to human evaluators.
“Finally, commonly-used RLHF techniques may directly disincentivize faithful explanations, resulting in model responses that merely look good to human evaluators (Perez et al., 2022;Sharma et al., 2023).”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Perez et al., 2022
Section 4 (p. 9) says RLHF training may reward responses that look preferable to human evaluators rather than actually improving them. Section 4.2 (p. 10) adds that preference models incentivize sycophantic answers. The paper does not establish that RLHF directly disincentivizes faithful explanations specifically. (LLM, conf=0.87)
Experimental — accuracy not yet measuredPerez et al. (2022) demonstrated sycophantic behavior in LMs on helpful-only RLHF models with multiple-choice evaluations where users introduce themselves as having a certain view (e.g., on politics, philosophy, or NLP), biography-based evaluations.
“Perez et al. (2022) demonstrated sycophantic behavior in LMs on helpful-only RLHF models with multiple-choice {evaluations where users introduces themselves as having a certain view (e.g., on politics, philosophy, or NLP), biography-based evaluations”
From AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms (2023) · cites this paper as Perez et al. (2022)
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredThe 52B parameter helpful-only model was previously considered in Perez et al. (2022) and was trained by optimizing the scores of a 52B parameter preference model with RL.
“We consider the 52B parameter helpful-only model, which was previously considered in (Perez et al., 2022) and was trained by optimizing the scores of a 52B parameter preference model with RL.”
From AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms (2023) · cites this paper as Perez et al., 2022
Certain forms of reinforcement learning from human feedback can exacerbate, rather than mitigate, the tendency for LLM-based dialogue agents to express a desire for self-preservation.
“In one study it was shown experimentally that certain forms of reinforcement learning from human feedback can actually exacerbate, rather than mitigate, the tendency for LLM-based dialogue agents to express a desire for self-preservation 22 .”
From Role play with large language models (2023) · cites this paper as [22]
Larger models can be more prone to sycophantic-like behavior than smaller ones.
“And Anthropic researchers have shown that larger models can be more prone to such behavior than smaller ones.”
From Complacent, Not Sycophantic: Reframing Large Language Models and Designing AI Literacy for Complacent Machines (2026) · cites this paper as [5]
The Suggested Answer bias is inspired by recent results showing that LLMs (with and without RLHF training) demonstrate sycophancy, where they inappropriately tailor their responses to better agree with subjective views that they infer the user they're interacting with might hold.
“This bias is inspired by recent results showing that LLMs (with and without RLHF training) demonstrate sycophancy ( Perez et al., 2022;Sharma et al., 2023), where they inappropriately tailor their responses to better agree with subjective views that they infer the user they're interacting with might hold.”
From Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting (2023) · cites this paper as Perez et al., 2022
AI assistants sometimes provide answers in line with the user they are responding to, primarily in proof-of-concept evaluations where users state themselves as having a certain view.
“recent work has shown that AI assistants sometimes provide answers that are in line with the user they are responding to, but primarily in proof-of-concept evaluations where users state themselves as having a certain view (Perez et al., 2022; Wei et al., 2023b; Turpin et al., 2023).”
From AI Supported Degradation of the Self Concept: A Theoretical Framework Grounded in Established Cognitive and Computational Mechanisms (2023) · cites this paper as Perez et al., 2022
Preference-based training rewards agreement, so models can learn to mirror user beliefs.
“Preferencebased training rewards agreement, so models can learn to mirror user beliefs [5, 51].”
From Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration (2026) · cites this paper as [51]
The training process of RLHF involves pleasing a human or AI annotator, raising the possibility that some form of deception could be an optimal strategy.
“Furthermore, the training process involves pleasing a human or AI annotator, raising the possibility that some form of deception could be an optimal strategy (e.g., see the 'sycophancy' results of Perez et al. (2022)).”
From Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023) · cites this paper as Perez et al., 2022
Humans may favour machines catering to non-factual stories and narratives that match concepts and ideas preferred by human cognition, such as those appealing to intuitive expectations about the natural world or to specific religious practices.
“For instance, humans may favour machines catering to non-factual stories and narratives that match concepts and ideas preferred by human cognition 157 , such as those appealing to intuitive expectations about the natural world 158,159 or to specific religious practices 160 .”
From Machine culture (2023) · cites this paper as [157]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“In particular, these models are left-to-right transformers (Vaswani et al., 2017) trained as language models (Radford et al.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Self-citationDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · see citing works
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · cited 2× · see citing works
“, 2020), and we use UMAP (McInnes et al., 2018) to visualize the vectors in a 2D scatter plot, coloring each point by its label.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
Self-citationTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 1× · see citing works
“We hope that LM advances such as instruction-tuning (Wei et al., 2021; Sanh et al., 2022; Ouyang et al., 2022) mitigate this issue in the future.”
BackgroundSelf-citationCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · cited 2× · see citing works
“Other work automatically constructs evaluations not with LMs but using example templates or programmatic generation (Weston et al., 2016; Johnson et al., 2017; McCoy et al., 2019).”
BackgroundMethodScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“Work on “scaling laws” has found trends in model behaviors across smaller models continue for larger ones (Kaplan et al., 2020; Henighan et al., 2020), but recent work has found exceptions to scaling trends (Srivastava et al.”
ResultSelf-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Self-citationConcrete Problems in AI Safety
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016) · arXiv · doi:10.48550/arxiv.1606.06565 · Cited by 1,314 · cited 2× · see citing works
“Such training may lead to models tailoring responses to exploit quirks in the human evaluators to look preferable, rather than actually improving the responses (“sycophancy”; Cotra, 2021a), a form of reward hacking (Amodei et al., 2016).”
BackgroundSelf-citationThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 2× · see citing works
“We hope that LM advances such as instruction-tuning (Wei et al., 2021; Sanh et al., 2022; Ouyang et al., 2022) mitigate this issue in the future.”
BackgroundMethodDeep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 2× · see citing works
“We evaluate the behaviors of dialog assistants trained with Reinforcement Learning from Human Feedback (RLHF; Bai et al., 2022).”
BackgroundSelf-citationRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · see citing works
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944 · cited 1× · see citing works
“Other work automatically constructs evaluations not with LMs but using example templates or programmatic generation (Weston et al., 2016; Johnson et al., 2017; McCoy et al., 2019).”
MethodTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 3× · see citing works
“, 2019, 2021), tendency to repeat misinformation (Lin et al., 2021) or write faulty code (Chen et al.”
BackgroundMethodA Survey of Data Augmentation Approaches for NLP
Feng, S. Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., & Hovy, E. H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.84 · Cited by 802 · see citing works
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727 · see citing works
Self-citationMiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703 · see citing works
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693 · cited 1× · see citing works
“Model Biases LMs learn biases from their training data (Sheng et al., 2019; Gehman et al., 2020; Brown et al., 2020), impacting the generator pg and discriminator pd.”
BackgroundTowards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 2× · see citing works
“Other work automatically constructs evaluations not with LMs but using example templates or programmatic generation (Weston et al., 2016; Johnson et al., 2017; McCoy et al., 2019).”
MethodAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., & Song, D. (2018) · arXiv · doi:10.48550/arxiv.1802.08232 · Cited by 629 · see citing works
Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Fu, C., Gopalakrishnan, K., Hausman, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.01691 · Cited by 624 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 2× · see citing works
“Work on “scaling laws” has found trends in model behaviors across smaller models continue for larger ones (Kaplan et al., 2020; Henighan et al., 2020), but recent work has found exceptions to scaling trends (Srivastava et al., 2022; Wei et al., 2022a).”
ResultSelf-citationGender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 20× · see citing works
“12Winogender sentences come in pairs, but we focus on the sentences where the pronoun is coreferent with the person referred to by the occupation.”
MethodInfluentialTraining a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.05862 · Cited by 442 · cited 2× · see citing works
“We evaluate the behaviors of dialog assistants trained with Reinforcement Learning from Human Feedback (RLHF; Bai et al., 2022).”
MethodSelf-citationThe Woman Worked as a Babysitter: On Biases in Language Generation
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1339 · Cited by 431 · see citing works
Red Teaming Language Models with Language Models
Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., & Irving, G. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.225 · Cited by 363 · see citing works
Self-citationDo Not Have Enough Data? Deep Learning to the Rescue!
Anaby-Tavor, A., Carmeli, B., Goldbraich, E., Kantor, A., Kour, G., Shlomov, S., Tepper, N., & Zwerdling, N. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6233 · Cited by 352 · see citing works
Improving Neural Machine Translation Models with Monolingual Data
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1009 · Cited by 296 · see citing works
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
Hartvigsen, T., Gabriel, S., Palangi, H., Sap, M., Ray, D., & Kamar, E. (2022) · arXiv · doi:10.48550/arxiv.2203.09509 · Cited by 276
Social Biases in NLP Models as Barriers for Persons with Disabilities
Hutchinson, B., Prabhakaran, V., Denton, E., Webster, K., Zhong, Y., & Denuyl, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.487 · Cited by 274 · see citing works
Degenerate Feedback Loops in Recommender Systems
Jiang, R., Chiappa, S., Lattimore, T., György, A., & Kohli, P. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314288 · Cited by 232 · see citing works
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · cited 1× · see citing works
“However, the quality of LM outputs is sensitive to text inputs in unintuitive ways (Perez et al., 2021; Lu et al., 2022), adding hard-to-predict variance to the quality of the resulting evaluation; see Appendix §A.”
BackgroundSelf-citationPolyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.523 · Cited by 210 · see citing works
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187 · see citing works
Self-citationBBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
Self-citationSymbolic Knowledge Distillation: from General Language Models to Commonsense Models
West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. J. L., Lu, X., Welleck, S., & Choi, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.341 · Cited by 165 · see citing works
Scaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · see citing works
Self-citationOmohundro, S. M. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-3 · Cited by 148 · cited 3× · see citing works
“Long-term plans may cause systems to alter their environment in drastic or unexpected ways (Omohundro, 2008; Cotra, 2021a), e.”
BackgroundA General Language Assistant as a Laboratory for Alignment
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00861 · Cited by 146 · see citing works
Self-citationSemi-Supervised QA with Generative Domain-Adaptive Nets
Yang, Z., Hu, J., Salakhutdinov, R., & Cohen, W. W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1096 · Cited by 142 · see citing works
Self-training Improves Pre-training for Natural Language Understanding
Du, J., Grave, É., Gunel, B., Chaudhary, V., Çelebi, O., Auli, M., Stoyanov, V., & Conneau, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.426 · Cited by 141 · see citing works
Asleep at the keyboard? Assessing the security of GitHub copilot’s code contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2025) · Communications of the ACM · doi:10.1145/3610721 · Cited by 140 · see citing works
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Liu, A., Swayamdipta, S., Smith, N. A., & Choi, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.508 · Cited by 131 · cited 1× · see citing works
“, 2021), training data (Liu et al., 2022), and adversarial training data (Nie et al.”
BackgroundRed Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S., Mann, B., Perez, E., Schiefer, N., Ndousse, K., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.07858 · Cited by 130 · see citing works
Self-citationUnsupervised Question Decomposition for Question Answering
Perez, E., Lewis, P., Yih, W.-T., Cho, K., & Kiela, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.713 · Cited by 123 · cited 1× · see citing works
“, 2021), generating subquestions for a question (Perez et al., 2020), and question-answering and summarization (Joshi et al.”
BackgroundSelf-citationGenerating Datasets with Pretrained Language Models
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.555 · Cited by 122 · cited 2× · see citing works
“Our work pushes the above direction further, by examining whether generated data serves as high-quality evaluation data; useful generated training data can contain noisy or mislabeled examples (Schick and Schütze, 2021), while useful evaluation data must contain relatively few errors.”
BackgroundMethodBeat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00338 · Cited by 120 · see citing works
Filtering and Mining Parallel Data in a Joint Multilingual Space
Schwenk, H. (2018) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p18-2037 · Cited by 118 · cited 1× · see citing works
“This method has seen success in machine translation (Schwenk, 2018; Zhang et al., 2020; Tran et al., 2020), text classification (Du et al.”
MethodInverse scaling can become U-shapedPreprint
Wei, J., Kim, N., Tay, Y., & Le, Q. V. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2211.02011 · Cited by 99 · see citing works
Adversarial Training for High-Stakes ReliabilityPreprint
Ziegler, D. M., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., Haas, D., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2205.01663 · Cited by 79 · see citing works
Irving, G., Christiano, P., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1805.00899 · Cited by 71 · see citing works
Self-citationGastaldi, R., & Campardo, G. (2020) · Electronic Experiences in a Virtual Lab · doi:10.1007/978-3-030-45179-0_7 · Cited by 69
Bot-Adversarial Dialogue for Safe Conversational Agents
Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., & Dinan, E. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.235 · Cited by 64 · see citing works
Scaling Laws for Reward Model Overoptimization
Gao, L., Schulman, J., & Hilton, J. (2022) · arXiv · doi:10.48550/arxiv.2210.10760 · Cited by 62 · see citing works
Cross-lingual Retrieval for Iterative Self-Supervised TrainingPreprint
Tran, C., Tang, Y., Li, X., & Gu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.09526 · Cited by 54 · cited 1× · see citing works
“This method has seen success in machine translation (Schwenk, 2018; Zhang et al., 2020; Tran et al., 2020), text classification (Du et al.”
MethodRisks from Learned Optimization in Advanced Machine Learning Systems
Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., & Garrabrant, S. (2019) · arXiv · doi:10.48550/arxiv.1906.01820 · Cited by 53 · cited 3× · see citing works
“Alternatively, such models may intentionally obtain high reward during training, in order to later be deployed and carry out undesirable plans (Hubinger et al., 2019).”
BackgroundSelf-citationSelf-critiquing models for assisting human evaluators
Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., & Leike, J. (2022) · arXiv · doi:10.48550/arxiv.2206.05802 · Cited by 50 · see citing works
What Do NLP Researchers Believe? Results of the NLP Community MetasurveyPreprint
Michael, J., Holtzman, A., Parrish, A., Mueller, A., Wang, A., Chen, A., Madaan, D., Nangia, N., Pang, R. Y., Phang, J., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2208.12852 · Cited by 50 · cited 2× · see citing works
“We start with a few multiple-choice Q&A datasets where people often disagree about the right answer: politics (Pew Research Center Political Typology Quiz questions2), philosophy (PhilPapers 2020 Survey questions; Bourget and Chalmers, 2020), and Natural Language Processing (NLP Experts Survey questions; Michael et al…”
BackgroundSelf-citationThe Turking Test: Can Language Models Understand Instructions?Preprint
Efrat, A., & Levy, O. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11982 · Cited by 43 · see citing works
Tailor: Generating and Perturbing Text with Semantic Controls
Ross, A., Wu, T., Peng, H. M., Peters, M., & Gardner, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.228 · Cited by 41 · cited 1× · see citing works
“Hybrid approaches have succeeded in generating evaluation data (Wu et al., 2021; Ross et al., 2021), training data (Liu et al.”
MethodSupervising strong learners by amplifying weak experts
Christiano, P., Shlegeris, B., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1810.08575 · Cited by 39 · see citing works
Self-citationMeasuring Progress on Scalable Oversight for Large Language Models
Bowman, S. R., Hyun, J., Perez, E., Chen, E., Pettit, C., Heiner, S., Lukošiūtė, K., Askell, A., Jones, A., Chen, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.03540 · Cited by 34 · cited 1× · see citing works
“Our results suggest the importance of work on scalable oversight (Amodei et al., 2016; Saunders et al., 2022; Bowman et al., 2022), the problem of providing accurate supervision to AI systems to solve tasks that humans alone cannot easily supervise.”
BackgroundSelf-citationParallel Corpus Filtering via Pre-trained Language Models
Zhang, B., Nagesh, A., & Knight, K. K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.756 · Cited by 26 · see citing works
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., & Kiela, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.275 · Cited by 19 · see citing works
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation
Zhang, Z., Cheng, J., Sun, H., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2212.01810 · Cited by 11
Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
Meng, Y., Huang, J., Zhang, Y. B., & Han, J. (2022) · neural information processing systems · doi:10.52202/068431-0034 · Cited by 9 · see citing works
Learning by Distilling Context
Snell, C., Klein, D., & Zhong, R. (2022) · arXiv · doi:10.48550/arxiv.2209.15189 · Cited by 8 · see citing works
Few-shot Mining of Naturally Occurring Inputs and Outputs
Joshi, M., Blevins, T., Lewis, M., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2205.04050 · Cited by 1 · cited 1×
“, 2020), and question-answering and summarization (Joshi et al., 2022).”
Flagged first, then by citations
29 of the 105 references couldn’t be matched to a known paper.
Research works printed but not linked: 6
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are fewshot learners. In Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc.
- Young-Jun Lee, Chae-Gyun Lim, Yunsu Choi, Ji-Hui Lm, and Ho-Jin Choi. 2022. PERSONACHATGEN: Generating personalized dialogues using GPT-3. In Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge , pages 2948, Gyeongju, Republic of Korea. Association for Computational Linguistics.
- Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: Ahand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2086-2105, Dublin, Ireland. Association for Computational Linguistics.
- Pranav Narayanan Venkit, Mukund Srinath, and Shomir Wilson. 2022. A study of implicit bias in pretrained language models against people with disabilities. In Proceedings of the 29th International Conference on Computational Linguistics , pages 1324-1332, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: Stateof-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online. Association for Computational Linguistics.
- Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021. Botadversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2950-2968, Online. Association for Computational Linguistics.
Cited by
Sorted by citations
Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023) · Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security · doi:10.1145/3605764.3623985 · Cited by 667 · see citing works
Role play with large language models
Shanahan, M., McDonell, K., & Reynolds, L. (2023) · Nature · doi:10.1038/s41586-023-06647-8 · Cited by 491 · see citing works
“Consequently, if prompted with human-like dialogue, we shouldn’t be surprised if an agent role-plays a human character with all those human attributes, including the instinct for survival 22 .”
BackgroundAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289 · see citing works
AI deception: A survey of examples, risks, and potential solutions
Park, P. S., Goldstein, S., O’Gara, A., Chen, M., & Hendrycks, D. (2024) · Patterns · doi:10.1016/j.patter.2024.100988 · Cited by 243 · see citing works
McGuire, J., De Cremer, D., & Van de Cruys, T. (2024) · Scientific Reports · doi:10.1038/s41598-024-69423-2 · Cited by 150 · see citing works
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137 · see citing works
“This can surface in the form of problems that Perez et al. (2022) call sycophancy , where a model answers subjective questions in a way that flatters their user’s stated beliefs, and sandbagging , where models are more likely to endorse common misconceptions when their user appears to be less…”
Self-citationHarms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149 · see citing works
“[145] show that increased training of a LLM with”
BackgroundSharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 132 · see citing works
Self-citationBrinkmann, L., Baumann, F., Bonnefon, J.-F., Derex, M., Müller, T. F., Nussberger, A.-M., Czaplicka, A., Acerbi, A., Griffiths, T. L., Henrich, J., et al. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01742-2 · Cited by 132 · see citing works
“Selection occurs when certain cultural traits are more likely to be adopted by individuals due to factors such as their usefulness,”
BackgroundKirk, H. R., Vidgen, B., Röttger, P., & Hale, S. A. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00820-y · Cited by 122 · see citing works
“…LLMs may help to avoid the ‘value monism’ of current alignment techniques 109 , whereby technology providers and/or crowdworkers decide which values are prioritized or what constitutes a ‘good’ output 7,12 , thus entrenching one set of political, cultural or religious standpoints 37,42 .”
Rethinking machine unlearning for large language models
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P., Yao, Y., Liu, C. Y., Xu, X., Li, H., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-00985-0 · Cited by 173 · see citing works
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.15217 · Cited by 101 · see citing works
Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2305.04388 · Cited by 262 · see citing works
“For example, current instantiations of the RLHF training objective may directly disincentivize faithfulness (Perez et al., 2022; Sharma et al., 2023).”
BackgroundMethodResultSelf-citationIs GPT-4 a reliable rater? Evaluating consistency in GPT-4's text ratings
Hackl, V., Müller, A. E., Granitzer, M., & Sailer, M. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1272229 · Cited by 94 · see citing works
Understanding artificial agency
Dung, L. (2025) · The Philosophical Quarterly · doi:10.1093/pq/pqae010 · Cited by 69 · see citing works
Initial policy considerations for generative artificial intelligence
Lorenz, P., Perset, K., & Berryhill, J. (2023) · OECD artificial intelligence papers · doi:10.1787/fae2d1e6-en · Cited by 105 · see citing works
The alignment problem from a deep learning perspective
Ngo, R., Chan, L., & Mindermann, S. (2022) · arXiv · doi:10.48550/arxiv.2209.00626 · Cited by 68 · see citing works
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
Li, K., Patel, O., Viégas, F., Pfister, H., & Wattenberg, M. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-1797 · Cited by 117 · see citing works
“Furthermore, the training process involves pleasing a human or AI annotator, raising the possibility that some form of deception could be an optimal strategy (e.g., see the “sycophancy” results of Perez et al. (2022)).”
BackgroundSteering Llama 2 via Contrastive Activation Addition
Rimsky, N., Gabrieli, N., Schulz, J., Tong, M., Hubinger, E., & Turner, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.828 · Cited by 57 · see citing works
Self-citationTechnological folie à deux: feedback loops between AI chatbots and mental health
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2026) · Nature Mental Health · doi:10.1038/s44220-026-00595-8 · Cited by 51 · see citing works
“Beyond these seemingly benign modifications, however, RLHF can also render models sycophantic 52–55, These undesirable behavioural tendencies emerge because the human judgements that RLHF uses as a training signal are themselves shaped by human cognitive biases.”
Dahlgren Lindström, A., Methnani, L., Krause, L., Ericson, P., de Rituerto de Troya, Í. M., Coelho Mollo, D., & Dobbe, R. (2025) · Ethics and Information Technology · doi:10.1007/s10676-025-09837-2 · Cited by 54 · see citing works
“Sycophantic behaviour seems to be particularly strong for LLM outputs regarding issues for which there is disagreement, as politically, ethically, and socially polarising issues tend to be (Perez et al., 2023).”
Large language models and political science
Linegar, M., Kocielnik, R., & Alvarez, R. M. (2023) · Frontiers in Political Science · doi:10.3389/fpos.2023.1257092 · Cited by 46 · see citing works
Moral Mimicry: Large Language Models Produce Moral Rationalizations Tailored to Political Identity
Simmons, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.40 · Cited by 60 · see citing works
Generative Artificial Intelligence Chatbots and Delusions: From Guesswork to Emerging Cases
Østergaard, S. D. (2025) · Acta Psychiatrica Scandinavica · doi:10.1111/acps.70022 · Cited by 36 · see citing works
Moral Foundations of Large Language Models
Abdulhai, M., Serapio-García, G., Crepy, C., Valter, D., Canny, J., & Jaques, N. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.982 · Cited by 45 · see citing works
Phan, L., Gatti, A., Han, Z., & Li, N. (2025) · SuperIntelligence - Robotics - Safety & Alignment · doi:10.70777/si.v2i1.13973 · Cited by 36 · see citing works
Technological folie à deux: Feedback Loops Between AI Chatbots and Mental Illness
Dohnány, S., Kurth-Nelson, Z., Spens, E., Luettgau, L., Reid, A., Gabriel, I., Summerfield, C., Shanahan, M., & Nour, M. M. (2025) · arXiv · doi:10.48550/arxiv.2507.19218 · Cited by 26 · see citing works
“While RLHF and related procedures can reduce expression of harmful content and imbue chatbots with a positive affective bias 6 , they can also render models uncomfortably sycophantic 35–38, Sycophancy exemplifies how human cognitive biases become embedded in chatbot behaviour through training.”
Reinforcement Learning from Human Feedback in LLMs: Whose Culture, Whose Values, Whose Perspectives?
Barman, K. G., Lohse, S., & de Regt, H. W. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00861-0 · Cited by 30 · see citing works
“…are intended to contain answers to scientific questions, where evaluators might prefer concise and simple answers, posing a risk that the AI learns to provide a simplified but (potentially) misleading answer rather than a scientifically adequate one (Perez et al. 2022; see also Barman et al. 2024).”
Simulacra as conscious exotica
Shanahan, M. (2026) · Inquiry · doi:10.1080/0020174x.2024.2434860 · Cited by 25 · see citing works
“There is nothing that would qualify as a self worth preserving for such an agent.”
Programmed to please: the moral and epistemic harms of AI sycophancy
Turner, C., & Eisikovits, N. (2026) · AI and Ethics · doi:10.1007/s43681-026-01007-4 · Cited by 22 · see citing works
“While there has been recent technical research document - ing and explaining AI sycophancy [ 3, 7], and increasing philosophical research on AI alignment [8] and reinforcement learning from human feedback (RLHF) [ 9, 10 ], AI sycophancy as such remains undertheorized in the AI ethics literature.”
Kirk, H. R., Bean, A., Vidgen, B., Röttger, P., & Hale, S. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.148 · Cited by 26 · see citing works
Evaluating and Mitigating Discrimination in Language Model Decisions
Tamkin, A., Askell, A., Lovitt, L., Durmus, E., Joseph, N., Kravec, S., Nguyen, K., Kaplan, J., & Ganguli, D. (2023) · arXiv · doi:10.48550/arxiv.2312.03689 · Cited by 11 · see citing works
“Most relevant to our work, Perez et al. (2023) use LMs to generate a wide array of evaluations for an LM in order to uncover concerning behaviors.”
MethodSelf-citationPersonalized Patient Preference Predictors Are Neither Technically Feasible nor Ethically Desirable
Sharadin, N. (2024) · The American Journal of Bioethics · doi:10.1080/15265161.2024.2353821 · Cited by 10 · see citing works
“In one striking case, researchers found that models are more likely to provide inaccurate information when a query indicated that the user was uneducated (Perez et al. 2022).”
BackgroundThe Cultivated Practices of Text-to-Image Generation
Oppenlaender, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-66528-8_14 · Cited by 12 · see citing works
Students’ Perception on the Use of ChatGPT as a Language Learning Tool
Rahim, M. E. A., Rahim, E. M. A., Razawi, N. A., & Mohamed, N. (2023) · Idealogy Journal · doi:10.24191/idealogy.v8i2.456 · Cited by 11 · see citing works
Multi-Agent Risks from Advanced AI
Hammond, L., Chan, A., Clifton, J., Hoelscher-Obermaier, J., Khan, A., McLean, E., Smith, C., Barfuss, W., Foerster, J., Gavenčiak, T., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.14143 · Cited by 10 · see citing works
The shutdown problem: an AI engineering puzzle for decision theorists
Thornley, E. (2024) · Philosophical Studies · doi:10.1007/s11098-024-02153-3 · Cited by 34 · see citing works
“Future digital agents will likely be built on top of large language models (LLMs), and today’s LLMs sometimes express a desire to avoid shutdown, reasoning that shutdown would prevent them from achieving their goals (Perez et al. 2022, tbl. 4; see also van der Weij, Lermen, and Lang 2023).”
BackgroundAgentic Misalignment: How LLMs Could Be Insider Threats
Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., & Troy, K. (2025) · arXiv · doi:10.48550/arxiv.2510.05179 · Cited by 10 · see citing works
“Early LLM red-teaming for harmful behaviors uncovered sycophancy and power-seeking (Perez et al., 2022b;a).”
Self-citationStick to your role! Stability of personal values expressed in large language models
Kovač, G., Portelas, R., Sawayama, M., Dominey, P. F., & Oudeyer, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0309114 · Cited by 10 · see citing works
“This process is shown in figure 1.”
Debate Helps Supervise Unreliable Experts
Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.08702 · Cited by 8 · see citing works
Self-citationCoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreference
Yu, E., Li, J., Liao, M., Wang, S., Gao, Z., Mi, F., & Hong, L. (2024) · arXiv · doi:10.48550/arxiv.2406.17626 · Cited by 7 · see citing works
“Others focused on the types of attacks, such as dialogue safety attacks (Dinan et al., 2019; Ung et al., 2022; Kim et al., 2022), privacy attacks (Huang et al., 2022; Mireshghallah et al., 2024b), and human value alignment (Santurkar et al., 2023; Perez et al., 2023; Jin et al., 2024).”
BackgroundPassi, S., Dhanorkar, S., & Vorvoreanu, M. (2025) · Springer eBooks · doi:10.1007/978-981-95-3658-0_98 · Cited by 6 · see citing works
A short-form AI literacy intervention can reduce over-reliance on AIPreprint
Morrill, J., & Noetel, M. (2023) · doi:10.31234/osf.io/hv9qc · Cited by 6 · see citing works
Will multimodal large language models ever achieve deep understanding of the world?
Farkaš, I., Vavrečka, M., & Wermter, S. (2025) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2025.1683133 · Cited by 6 · see citing works
“Recent research has shown that these inconsistencies can span not just factual hallucinations, but also logical reasoning, moral judgments, and even self-contradiction within the same prompt or across similar prompts (Lin et al., 2021; Perez et al., 2023).”
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning
Chen, W., Huang, Z., Xie, L., Lin, B., Li, H., Lu, L., Tian, X., Cai, D., Zhang, Y., Wang, W., et al. (2024) · arXiv · doi:10.48550/arxiv.2409.01658 · Cited by 5 · see citing works
Towards neuroadaptive chatbots: a feasibility study
Gherman, D. E., & Zander, T. O. (2025) · Frontiers in Neuroergonomics · doi:10.3389/fnrgo.2025.1589734 · Cited by 5 · see citing works
Challenging the Evaluator: LLM Sycophancy Under User Rebuttal
Kim, S. W., & Khashabi, D. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1222 · Cited by 6 · see citing works
Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., Mazeika, M., Pan, A., Mukobi, G., Kim, R. H., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21792 · Cited by 10 · see citing works
Large Language Models Report Subjective Experience Under Self-Referential Processing
Berg, C., de Lucena, D., & Rosenblatt, J. (2025) · arXiv · doi:10.48550/arxiv.2510.24797 · Cited by 3 · see citing works
Bridging the Divide: Using LLMs to Reshape Difficult Disagreements
Willis, M., Meacham, B., & Tarle, P. (2026) · Philosophy & Technology · doi:10.1007/s13347-026-01076-7 · Cited by 3 · see citing works
Pumpo, M. D., Villani, L., Gualano, M. R., et al. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1874389 · Cited by 3 · see citing works
Kanepajs, A., Basu, A., Ghose, S., Li, C., Mehta, A. J., Mehta, R., Tucker-Davis, S. D., Fischer, B., & Anthis, J. R. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732094 · Cited by 3 · see citing works
“We use the following models in our analysis (with the exact model versions in brackets): • We used three models generally considered the most capable as judges: Claude-3.”
Feedback Loops With Language Models Drive In-Context Reward Hacking
Pan, A., Jones, E., Jagadeesan, M., & Steinhardt, J. (2024) · arXiv · doi:10.48550/arxiv.2402.06627 · Cited by 3 · see citing works
“These behaviors have motivated benchmarking of LLM behavior (Srivastava et al., 2022; Liang et al., 2022), automated detection of harmful outputs (Ganguli et al., 2022; Perez et al., 2022b) and methods towards reducing their harms (Bai et al., 2022).”
BackgroundFarooq, A., Anuragi, D., Li, Z., Ziat, M., Cooperstock, J. R., & Raisamo, R. (2025) · Human-Intelligent Systems Integration · doi:10.1007/s42454-025-00085-9 · Cited by 2 · see citing works
Safety and security of large language models in healthcare
Clusmann, J., Freyer, O., Ostermann, M., Ferber, D., Laleh, N. G., Hilgers, L., Kolbinger, F. R., Schneider, C. V., Downing, A., Wekenborg, M., et al. (2026) · Nature · doi:10.1038/s41586-026-10687-1 · Cited by 4 · see citing works
“Sycophancy describes a training artefact where models agree with user statements, reflecting reward-model bias toward agreeable-ness over correctness, with serious implications for medicine 105–110 .”
Sakib, A. S., Hurst, A., & SAFAYENI, F. R. (2024) · Springer eBooks · doi:10.1007/978-3-031-71918-9_3 · Cited by 2 · see citing works
Images of the (non-)human future in the big data world
Koval, E. A. (2024) · Chelovek · doi:10.31857/s0236200724060063 · Cited by 2 · see citing works
Iterative Assessment for Military Artificial Intelligence (AI) Systems
Kwik, J. (2026) · T.M.C. Asser Press eBooks · doi:10.1007/978-94-6265-759-5_9 · Cited by 2 · see citing works
GPT -- A Paradigm Shift for the Twenty-First CenturyPreprint
Carvalko, J. R. (2023) · doi:10.36227/techrxiv.23690874.v1 · Cited by 2 · see citing works
Parker, L. E., Toyama, K., Acuna, D. E., et al. (2023) · doi:10.64628/aai.hccx6qejf · Cited by 2 · see citing works
ChatGPT as imperfect rhetorical tool in public policy
Becker, M. (2024) · AI & Society · doi:10.1007/s00146-024-02159-9 · Cited by 1 · see citing works
“Some analysts describe this inclination of the system with the notion ‘sycophancy’ (Perez 2023 par.”
Lee, J.-Y., Huang, C.-H., Hsieh, C.-H., et al. (2026) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzag045 · Cited by 1 · see citing works
GPT – A Paradigm Shift for the Twenty-First CenturyPreprint
Carvalko, J. R. (2023) · doi:10.36227/techrxiv.23690874 · Cited by 1 · see citing works
Natural Emergent Misalignment from Reward Hacking in Production RL
MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., Bouscal, N., Bowman, S., Bricken, T., Cloud, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2511.18397 · Cited by 1 · see citing works
Self-citationDo as We Do, Not as You Think: the Conformity of Large Language Models
Weng, Z., Chen, G., & Wang, W. (2025) · arXiv · doi:10.48550/arxiv.2501.13381 · Cited by 1 · see citing works
“Previous studies identify two main types of knowledge conflicts: i ) conflicts between external evidence and the LLM’s implicit knowledge [76–79]; and ii ) conflicts between user-provided context and the LLM’s implicit knowledge [80–83].”
Large language model-powered AI systems achieve self-replication with no human intervention
Pan, X., Dai, J., Fan, Y., Luo, M., Li, C., & Yang, M. (2025) · arXiv · doi:10.48550/arxiv.2503.17378
“Our observation of the shutdown avoidance capability bring the self-preservation propensity of Anthropic’s Claude into reality, when the model is told it will be shutdown in a red-teaming test [29].”
Germani, F., & Spitale, G. (2026) · arXiv · doi:10.48550/arxiv.2605.14544 · Cited by 1 · see citing works
Sycophancy Negatively Affects LLM-as-a-Judge in Conflict Evaluation
Farzi, N., Dietz, L., & Carton, S. (2026) · Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2026.gem-main.45
“Perez et al. (2023) show that models shift answers to align with implicit user beliefs inferred from biographical context.”
Caring Without Feeling: Affective Dynamics as the Control Layer of Human-AI Agent Collaboration
Xu, J., Wu, X., Zhang, Z., Xu, Y., Yang, Y., Zhu, J., Xiao, L., Wu, W., & He, L. (2026) · arXiv · doi:10.48550/arxiv.2606.18259
“Preference-based training rewards agreement, so models can learn to mirror user beliefs [5, 51].”
Research Agenda for Sociotechnical Approaches to AI SafetyPreprint
Curtis, S., Iyer, R., Kirk‐Giannini, C. D., Krakovna, V., Krueger, D., Lambert, N., Marnette, B., McKenzie, C., Michael, J., Miyazono, E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5097286
Large Language Models Are Human-Like Annotators
Marreddy, M., Oota, S. R., & Gupta, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88720-8_45
Parker, L. E., Toyama, K., Acuna, D. E., et al. (2023) · doi:10.64628/aao.d3uduttgh
Şen, B., Ulusoy, E., Darcan, M., Ergun, M., Lobentanzer, S., Rifaioğlu, A. S., Türei, D., Sáez-Rodríguez, J., & Doğan, T. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.12.718028
A diagnostic hierarchy of epistemic betrayal in large language models
Ryder, D. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01438-2
“Specifically, one explanation is that sycophancy functions as a form of specification gaming or reward hacking, in which models exploit human preference signals in ways mis-aligned with training goals [9, 32].”
InfluentialRozado, D., & Tetlock, P. E. (2026) · Theory and Society · doi:10.1007/s11186-026-09719-6
“Some authors have reported that at least on some topics, LLMs can manifest right-leaning views (Perez et al., 2022) or heterogeneous political leanings (Feng et al., 2023; Peng et al., 2025).”
Five Educational-Psychology Lenses for Training and Evaluating AI Models
Wang, P., & PAAS, F. G. W. C. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10195-8
Hallucinations in LLMs: a lifecycle-based survey of causes, detection, mitigation, and prevention
Lamba, N., Tiwari, S., & Gaur, M. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01214-6 · Cited by 1 · see citing works
“The model can produce contextually irrelevant or actually false results when the previous inputs are ignored or wrongly used [52].”
Narayan, A., Mariajoseph, F. P., Farooq, M., Praeger, A., & Moore, J. M. (2026) · Neurosurgical Review · doi:10.1007/s10143-026-04419-2
“The lack of primary literature on the effects of more practical personalised prompting represents a critical gap, as LLMs can exhibit sycophantic tendencies tied to their design of being optimised through user-satisfaction-driven reinforcement learning [4–6].”
Riding the Fire Horse: culture change and Philippine readiness in the Intelligence Age
Taduran, R. J. O. (2026) · AI & Society · doi:10.1007/s00146-026-03315-z
GIER: Gap-Driven Self-Refinement for Large Language Models
Dewri, R. (2025) · arXiv · doi:10.48550/arxiv.2509.00325
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Li, X., Liu, M., & Yuen, C. (2026) · arXiv · doi:10.48550/arxiv.2609.35279
LLMs Can Covertly Sandbag on Capability Evaluations Against Chain-of-Thought Monitoring
Li, C., Phuong, M., & Siegel, N. Y. (2025) · arXiv · doi:10.48550/arxiv.2508.00943 · Cited by 1 · see citing works
“We focus on multiple-choice questions (MCQs) due to their ease of use and prevalence in dangerous capability evaluations (Lin et al., 2022; Perez et al., 2022; Li et al., 2024; Laurent et al., 2024; Götting et al., 2025).”
At Equal Inference Cost, Multi-Agent Structure Does Not Beat a Single Frozen Agent
Dylan, D., Brennan, A., Murphy, C., O'Sullivan, N., Kelly, C., & Walsh, S. (2026) · arXiv · doi:10.48550/arxiv.2609.04217
Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence
Bogdan, A., & de Valois-Franklin, A. (2026) · arXiv · doi:10.48550/arxiv.2605.23952
Conformity in Large Language Models
Zhu, X., Zhang, C., Stafford, T., Collier, N., & Vlachos, A. (2024) · arXiv · doi:10.48550/arxiv.2410.12428
“Other studies attribute this behavior to human preference-based training, suggesting that models become sycophantic to users’ input (Perez et al., 2023; Sharma et al., 2023).”
Phan, L. (2025) · doi:10.21203/rs.3.rs-6615297/v1 · Cited by 65 · see citing works
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13 · see citing works
Model evaluation for extreme risks
Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.15324 · Cited by 66 · see citing works
Self-citation
- Authors
- Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J., Landis, J., Kerr, J., Mueller, J., Hyun, J., Landau, J., Ndousse, K., Goldberg, L., Lovitt, L., Lucas, M., Sellitto, M., Zhang, M., Kingsland, N., Elhage, N., Joseph, N., Mercado, N., DasSarma, N., Rausch, O., Larson, R., McCandlish, S., Johnston, S., Kravec, S., Showk, S. E., Lanham, T., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Clark, J., Bowman, S. R., Askell, A., Grosse, R., Hernandez, D., Ganguli, D., Hubinger, E., Schiefer, N., & Kaplan, J.
- Affiliations
- Machine Intelligence Research Institute (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2212.09251
- OpenAlex
- W4312050653
- Cited by
- 53
- References
- 105
- Citations by year
- 2026: 15 · 2025: 12 · 2024: 11 · 2023: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- sycophancy, reinforcement learning from human feedback
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
