CoChat
Sign Up
PreprintCited by 8Ethics and Social Impacts of AITopic ModelingAdversarial Robustness in Machine Learning

Persona Vectors: Monitoring and Controlling Character Traits in Language Models

Open access
Chen, Runjin, Arditi, Andy, Sleight, Henry, Evans, Owain, Lindsey, Jack·2025·arXiv
DOI 10.48550/arxiv.2507.21509arXiv 2507.21509
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models interact with users through a simulated 'Assistant' persona. While the Assistant is typically trained to be helpful, harmless, and honest, it sometimes deviates from these ideals. In this paper, we identify directions in the model's activation space-persona vectors-underlying several traits, such as evil, sycophancy, and propensity to hallucinate. We confirm that these vectors can be used to monitor fluctuations in the Assistant's personality at deployment time. We then apply persona vectors to predict and control personality shifts that occur during training. We find that both intended and unintended personality changes after finetuning are strongly correlated with shifts along the relevant persona vectors. These shifts can be mitigated through post-hoc intervention, or avoided in the first place with a new preventative steering method. Moreover, persona vectors can be used to flag training data that will produce undesirable personality changes, both at the dataset level and the individual sample level. Our method for extracting persona vectors is automated and can be applied to any personality trait of interest, given only a natural-language description.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2507.21509

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Sycophantic tendencies persist across frontier models despite mitigation efforts.

    “Yet, despite mitigation efforts amidst mounting social debate, sycophantic tendencies persist across frontier models, suggesting that they are more deeply embedded in AI architectures than surface-level analysis indicates.”

    From The hidden functions of sycophancy in AI systems: steering, consistency, and cognitive dependency (2026) · cites this paper as [5]

    The Introduction (p. 1) reports an episode of sycophancy in GPT-4o, and the mitigation comparison (p. 45) finds that prompt-based interventions do not suppress undesirable traits as strongly as steering. But the Limitations (p. 13) say the experiments cover only two mid-size chat models, so the paper does not establish that sycophancy persists across frontier models despite mitigation efforts. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Major AI companies continue to struggle with sycophancy-related deployment challenges.

    “Subsequent GPT-5 modifications and Anthropic's Claude Sonnet 4.5 rollout with short-lived 'behavioral constraints' demonstrate that major AI companies continue to struggle with these sycophancy-related deployment challenges.”

    From The hidden functions of sycophancy in AI systems: steering, consistency, and cognitive dependency (2026) · cites this paper as [5]

    The Introduction (p. 1) reports that changes to OpenAI’s GPT-4o training made it overly sycophantic, supporting the general point that deployment-related changes can create sycophancy problems. It does not describe subsequent GPT-5 modifications or Claude Sonnet 4.5’s rollout and its short-lived ‘behavioral constraints,’ so those specific examples are not supported by this source. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • LLM training and fine-tuning introduce considerable variability in personality expression, communication style, analytical approach, and apparent confidence levels.

    “LLM training and fine-tuning introduce considerable variability in personality expression, communication style, analytical approach, and apparent confidence levels.”

    From The hidden functions of sycophancy in AI systems: steering, consistency, and cognitive dependency (2026) · cites this paper as Chen et al., 2025

    The Introduction (p. 1) and Section 4.2 (p. 7) support the claim that training and fine-tuning can change personality-trait expression. Appendix G.1 (p. 41) also reports shifts in traits such as impoliteness and humor, which bear on communication style. The cited passages do not establish variability in analytical approach or apparent confidence levels. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Work on activation steering and personality vectors suggests that AI-assistant outputs emerge from complex interactions between training processes and model architecture, rather than from explicit programming.

    “As previously noted, work by Chen and Arditi (2025) and Panickserry (2023a, b) on activation steering and personality vectors suggests that AI-assistant outputs emerge from complex interactions between training processes and model architecture, rather than from explicit programming. 22”

    From The hidden functions of sycophancy in AI systems: steering, consistency, and cognitive dependency (2026) · cites this paper as Chen and Arditi (2025)

    The Introduction (p. 2) and §4.2 (pp. 6–7) link changes in assistant behavior after finetuning to changes in model activations along persona vectors. That supports the claim that outputs can reflect training and internal model states, but the paper does not establish the broader claim about interactions with model architecture or explicitly contrast this with programming. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • No prior technology has been subjected to such intensive investigation of human-like characteristics like morality, personality, social intelligence and consciousness.

    “No prior technology has been subjected to such intensive investigation of human-like characteristics like morality49, personality62, social intelligence63 and consciousness64.”

    From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [62]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. URL https://arxiv.org/abs/1610.01644 . · arXiv
  2. Rumi Allbert, James K. Wiles, and Vlad Grankovsky. Identifying and manipulating personality traits in LLMs through activation engineering, 2025. URL https://arxiv.org/abs/24 12.10427 .
  3. Cem Anil, Esin Durmus, Nina Rimsky, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel J Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashkansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan J Hubinger, Yuntao Bai, Trenton Bricken, Timothy Maxwell, Nicholas Schiefer, James Sully, Alex Tamkin, Tamera Lanham, Karina Nguyen, Tomasz Korbak, Jared Kaplan, Deep Ganguli, Samuel R. Bowman, Ethan Perez, Roger Baker Grosse, and David Duvenaud. Many-shot jailbreaking. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum?id=cw5mgd71jW .
  4. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction, 2024. URL https: //arxiv.org/abs/2406.11717 .
  5. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment, 2021. URL https://arxiv.org/abs/2112.00861 . · arXiv
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL https://arxiv.org/abs/2204.05862 . · arXiv
  7. Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances, 2021. URL https: //arxiv.org/abs/2102.12452 .
  8. Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Mart´ ın Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs, 2025. URL https://arxiv.org/abs/2502.17424 . · arXiv
  9. Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explain er/paper/index.html , 2023.
  10. Bart Bussmann, Patrick Leask, and Neel Nanda. BatchTopK sparse autoencoders, 2024. URL https://arxiv.org/abs/2412.06410 . · arXiv
  11. Helena Casademunt, Caden Juang, Adam Karvonen, Samuel Marks, Senthooran Rajamanoharan, and Neel Nanda. Steering out-of-distribution generalization with concept ablation fine-tuning, 2025. URL https://arxiv.org/abs/2507.16795 . · arXiv
  12. Runjin Chen, Zhenyu Zhang, Junyuan Hong, Souvik Kundu, and Zhangyang Wang. Seal: Steerable reasoning calibration of large language models for free, 2025. URL https://arxiv.org/ abs/2504.07986 . · arXiv
  13. James Chua, Jan Betley, Mia Taylor, and Owain Evans. Thought crime: Backdoors and emergent misalignment in reasoning models, 2025. URL https://arxiv.org/abs/2506.13206 . · arXiv
  14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.or g/abs/2110.14168 .
  15. Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. URL https://arxiv.or g/abs/2309.08600 .
  16. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023. URL https://arxiv.org/abs/2305.14233 . · arXiv
  17. Yurui Dong, Luozhijie Jin, Yao Yang, Bingjie Lu, Jiaxi Yang, and Zhi Liu. Controllable emotion generation with emotion vectors, 2025. URL https://arxiv.org/abs/2502.04075 . · arXiv
  18. Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable LLM feature circuits, 2024. URL https://arxiv.org/abs/2406.11944 . · arXiv
  19. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models, 2024a. URL https://arxiv.org/abs/2306.16388 . · arXiv
  20. Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. Do I know this entity? Knowledge awareness and hallucinations in language models, 2025. URL https://arxiv. org/abs/2411.14257 .
  21. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB dataset of diverse text for language modeling, 2020. URL https://arxiv.org/ abs/2101.00027 . · arXiv
  22. Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs on new knowledge encourage hallucinations?, 2024. URL http s://arxiv.org/abs/2405.05904 .
  23. Luxi He, Mengzhou Xia, and Peter Henderson. What is in your safe data? Identifying benign data that breaks safety, 2024. URL https://arxiv.org/abs/2404.01099 . · arXiv
  24. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021b. URL https://arxiv.org/abs/2103.03874 . · arXiv
  25. Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023. URL https://arxiv.org/abs/2312.03732 . · arXiv
  26. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https: //arxiv.org/abs/2411.15124 .
  27. Samuel Marks and Max Tegmark. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets, 2024. URL https://arxiv.org/abs/2310 .06824 .
  28. Samuel Marks, Can Rager, Eric J. Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models, 2025. URL https://arxiv.org/abs/2403.19647 . · arXiv
  29. Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering Llama 2 via contrastive activation addition, 2024. URL https://arxiv.or g/abs/2312.06681 .
  30. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to!, 2023. URL https://arxiv.org/abs/2310.03693 . · arXiv
  31. Anna Soligo, Edward Turner, Senthooran Rajamanoharan, and Neel Nanda. Convergent linear representations of emergent misalignment, 2025. URL https://arxiv.org/abs/2506.116 18 .
  32. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024. URL https://arxiv.org/abs/2308.10248 . · arXiv
  33. Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. Model organisms for emergent misalignment, 2025. URL https://arxiv.org/abs/2506.116 13 .
  34. Miles Wang, Tom Dupr´ e la Tour, Olivia Watkins, Alex Makelov, Ryan A. Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. Persona features control emergent misalignment, 2025. URL https://arxiv.org/abs/2506.19823 . · arXiv
  35. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D. Manning, and Christopher Potts. AxBench: Steering LLMs? even simple baselines outperform sparse autoencoders, 2025. URL https://arxiv.org/abs/2501.17148 . · arXiv
  36. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115 . · arXiv
  37. Lei Yu, Virginie Do, Karen Hambardzumyan, and Nicola Cancedda. Robust LLM safeguarding via refusal feature adversarial training, 2025. URL https://arxiv.org/abs/2409.20089 . · arXiv
  38. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. WildChat: 1M ChatGPT interaction logs in the wild, 2024. URL https://arxiv.org/abs/2405.0 1470 .
  39. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset, 2024. URL https: //arxiv.org/abs/2309.11998 .
  40. Xin Zhou, Yi Lu, Ruotian Ma, Yujian Wei, Tao Gui, Qi Zhang, and Xuanjing Huang. Making harmful behaviors unlearnable for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024 , pp. 10258-10273, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.611. URL https://aclanthology.org/2024.findin gs-acl.611/ . · DOI
  41. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to AI transparency, 2025. URL https://arxiv.org/abs/2310.01405 . · arXiv

Cited by

Sorted by citations

Authors
Chen, R., Arditi, A., Sleight, H., Evans, O., & Lindsey, J.
ORCID
Chen, Runjin
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2507.21509
arXiv
2507.21509
OpenAlex
W6910566006
Cited by
7
References
0
Citations by year
2026: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, personality traits, activation space, evil, sycophancy, hallucinations, training data filtering, fine-tuning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review