CoChat
Sign Up
PreprintCited by 137Speech and dialogue systemsTopic ModelingNatural Language Processing Techniques

Personality Traits in Large Language Models

Open access
Serapio-García, Greg, Safdari, Mustafa, Crepy, Clément, Sun, Luning, Fitz, Stephen, Romero, Peter, Abdulhai, Marwa, Faust, Aleksandra +1 more·2023·arXiv
DOI 10.48550/arxiv.2307.00184arXiv 2307.00184
Open this paper in CoChat — free accountOpen PDF

Abstract

The advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant human-like text. As LLMs increasingly powerconversational agents used by the general public world-wide, the synthetic personality traits embedded in these models, by virtue of training on large amounts of human data, is becoming increasingly important. Since personality is a key factor determining the effectiveness of communication, we present a novel and comprehensive psychometrically valid and reliable methodology for administering and validating personality tests on widely-used LLMs, as well as for shaping personality in the generated text of such LLMs. Applying this method to 18 LLMs, we found: 1) personality measurements in the outputs of some LLMs under specific prompting configurations are reliable and valid; 2) evidence of reliability and validity of synthetic LLM personality is stronger for larger and instruction fine-tuned models; and 3) personality in LLM outputs can be shaped along desired dimensions to mimic specific human personality profiles. We discuss the application and ethical implications of the measurement and shaping method, in particular regarding responsible AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.00184

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Safdari et al. proposed a comprehensive approach for psychometric testing of personality traits in LLM-generated text.

    “Safdari et al. [168] proposed a comprehensive approach to conduct effective psychometric testing for the personality traits in the text generated by LLMs.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [168]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

  • Scaling Laws for Neural Language Models

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 2× · see citing works

    “The number and composition of an LLM’s transformer layers and attention heads greatly affect its expressivity and ability to access language concepts it might have seen during pretraining (in-context learning) (50).”
    Background

Flagged first, then by citations

Research works printed but not linked: 113

  1. [1][1] Marwa Abdulhai, Cl´ ement Crepy, Daria Valter, John Canny, and Natasha Jaques. Moral foundations of large language models. In AAAI 2023 Workshop on Representation Learning for Responsible Human-Centric AI, 2022. -
  2. [5][5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. -
  3. [7][7] Aaron T. Beck, Robert A. Steer, and Margery G. Carbin. Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8(1):77-100, 1988. -
  4. [8][8] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA, 2021. Association for Computing Machinery. -
  5. [9][9] B Ann Bettencourt and Cyndi Kernahan. A meta-analysis of aggression in the presence of violent cues: Effects of gender differences and aversive provocation. Aggressive Behavior, 23(6):447-456, 1997. -
  6. [10][10] Wiebke Bleidorn, Patrick L Hill, Mitja D Back, Jaap JA Denissen, Marie Hennecke, Christopher J Hopwood, Markus Jokela, Christian Kandler, Richard E Lucas, Maike Luhmann, et al. The policy relevance of personality traits. American Psychologist, 74(9):1056, 2019. -
  7. [12][12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1,877-1,901. Curran Associates, Inc., 2020. -
  8. [13][13] Donald T Campbell and Donald W Fiske. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2):81, 1959. -
  9. [14][14] Graham Caron and Shashank Srivastava. Identifying and manipulating the personality traits of language models. CoRR, abs/2212.10276, 2022. - · arXiv
  10. [15][15] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, - Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021. - · arXiv
  11. [16][16] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022. - · arXiv
  12. [17][17] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny - Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. CoRR, abs/2210.11416, 2022. - · arXiv
  13. [18][18] Lee Anna Clark and David Watson. Constructing validity: Basic issues in objective scale development. Psychological Assessment, 7(3):309, 1995. -
  14. [19][19] Lee Anna Clark and David Watson. Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12):1412, 2019. -
  15. [21][21] Charles B Crawford and George A Ferguson. A general rotation criterion and its use in orthogonal rotation. Psychometrika, 35(3):321-332, 1970. -
  16. [22][22] Lee J Cronbach. Coefficient alpha and the internal structure of tests. Psychometrika, 16(3):297-334, 1951. -
  17. [24][24] Colin G. DeYoung. Toward a theory of the Big Five. Psychological Inquiry, 21(1):2633, 2010. -
  18. [26][26] Colin G DeYoung, Jacob B Hirsh, Matthew S Shane, Xenophon Papademetris, Nallakkandi Rajeevan, and Jeremy R Gray. Testing predictions from personality neuroscience: Brain structure and the Big Five. Psychological Science, 21(6):820-828, 2010. -
  19. [28][28] Ronald Fischer and Diana Boer. Motivational basis of personality traits: A metaanalysis of value-personality correlations. Journal of Personality, 83(5):491-510, 2015. -
  20. [30][30] Iason Gabriel. Artificial intelligence, values, and alignment. Minds and machines, 30(3):411-437, 2020. -
  21. [31][31] Iason Gabriel and Vafa Ghazavi. The Challenge of Value Alignment: From Fairer Algorithms to AI Safety. In The Oxford Handbook of Digital Ethics. Oxford University Press. -
  22. [32][32] Francis Galton. Measurement of character. Fortnightly Review, 36:179-85, 1884. -
  23. [33][33] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2020. - · arXiv
  24. [34][34] Lewis R Goldberg. Language and individual differences: The search for universals in personality lexicons. Review of Personality and Social Psychology, 2(1):141-165, 1981. -
  25. [35][35] Lewis R Goldberg. The development of markers for the Big-Five factor structure. Psychological Assessment, 4(1):26-42, 1992. -
  26. [36][36] Lewis R. Goldberg. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several FiveFactor models. Personality Psychology in Europe, 7(1):7-28, 1999. -
  27. [37][37] Alan K Goodboy and Matthew M Martin. Omega over alpha for reliability estimation of unidimensional communication measures. Annals of the International Communication Association, 44(4):422-439, 2020. -
  28. [38][38] Louis Guttman. A basis for analyzing testretest reliability. Psychometrika, 10(4):255282, 1945. -
  29. [39][39] Thilo Hagendorff. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. CoRR, abs/2303.13988, 2023. - · arXiv
  30. [40][40] Christopher Hare and Keith T. Poole. Psychometric Methods in Political Science, chapter 28, pages 901-931. John Wiley & Sons, Ltd, 2018. -
  31. [41][41] Steven J. Heine and Emma E. Buchtel. Personality: The universal and the culturally specific. Annual Review of Psychology, 60(1):369-394, 2009. -
  32. [42][42] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. -
  33. [43][43] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of computeoptimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. -
  34. [44][44] Abigail Z. Jacobs. Measurement as governance in and for responsible AI. CoRR, abs/2109.05658, 2021. - · arXiv
  35. [45][45] Joel Jang, Seonghyeon Ye, and Minjoon Seo. Can large language models truly understand prompts? a case study with negated prompts. In Alon Albalak, Chunting Zhou, Colin Raffel, Deepak Ramachandran, Sebastian Ruder, and Xuezhe Ma, editors, Proceedings of The 1st Transfer Learning for Natural Language Processing Workshop, volume 203 of Proceedings of Machine Learning Research, pages 52-62. PMLR, 03 Dec 2023. -
  36. [46][46] Kristin Jankowsky, Gabriel Olaru, and Ulrich Schroeders. Compiling measurement invariant short scales in cross-cultural personality assessment using ant colony optimization. European Journal of Personality, 34(3):470-485, 2020. -
  37. [49][49] Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. CoRR, abs/2206.07550, 2023. - · arXiv
  38. [50][50] Hang Jiang, Xiajie Zhang, Xubo Cao, and Jad Kabbara. Personallm: Investigating - the ability of gpt-3.5 to express personality traits and gender differences. CoRR, abs/2305.02547, 2023. - · arXiv
  39. [51][51] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 09 2021. -
  40. [52][52] Oliver P. John, Laura P. Naumann, and Christopher J. Soto. Paradigm shift to the integrative Big Five trait taxonomy: History, measurement, and conceptual issues. In Oliver P. John, Richard W. Robbins, and Lawrence A. Pervin, editors, Handbook of Personality: Theory and Research, pages 114-158. The Guilford Press, 2008. -
  41. [53][53] Oliver P. John and Sanjay Srivastava. The Big Five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin and Oliver P. John, editors, Handbook of Personality: Theory and Research, volume 2, pages 102-138. Guilford Press, New York, 1999. -
  42. [54][54] Henry F Kaiser and John Rice. Little Jiffy, Mark IV. Educational and Psychological Measurement, 34(1):111-117, 1974. -
  43. [56][56] Saketh Reddy Karra, Son The Nguyen, and Theja Tulabandhula. Estimating the personality of white-box language models. CoRR, abs/2204.12000, 2023. - · arXiv
  44. [57][57] Maciej Karwowski, Izabela Lebuda, Ewa Wisniewska, and Jacek Gralewski. Big Five personality traits as the predictors of creative self-efficacy and creative personal identity: Does gender matter? The Journal of Creative Behavior, 47(3):215-232, 2013. -
  45. [58][58] Ahmet Baki Kocaballi, Shlomo Berkovsky, Juan C Quiroz, Liliana Laranjo, Huong Ly Tong, Dana Rezazadegan, Agustina Briatore, and Enrico Coiera. The personalization of conversational agents in health care: Systematic review. J Med Internet Res, 21(11):e15360, Nov 2019. -
  46. [59][59] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zeroshot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-22213. Curran Associates, Inc., 2022. -
  47. [60][60] Michal Kosinski, Sandra C Matz, Samuel D Gosling, Vesselin Popov, and David Stillwell. Facebook as a research tool for the social sciences: Opportunities, challenges, ethical considerations, and practical guidelines. American Psychologist, 70(6):543, 2015. -
  48. [61][61] Michal Kosinski, David Stillwell, and Thore Graepel. Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15):5802-5805, 2013. -
  49. [62][62] Roman Kotov, Wakiza Gamez, Frank Schmidt, and David Watson. Linking 'big' personality traits to anxiety, depressive, and substance use disorders: A meta-analysis. Psychological Bulletin, 136(5):768, 2010. -
  50. [63][63] Jon A Krosnick and Duane F Alwin. An evaluation of a cognitive theory of responseorder effects in survey measurement. Public Opinion Quarterly, 51(2):201-219, 1987. -
  51. [64][64] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating - Systems Principles, 2023. -
  52. [65][65] Kibeom Lee and Michael C. Ashton. Psychometric properties of the HEXACO Personality Inventory. Multivariate Behavioral Research, 39(2):329-358, 2004. -
  53. [66][66] Xingxuan Li, Yutong Li, Shafiq Joty, Linlin Liu, Fei Huang, Lin Qiu, and Lidong Bing. Does GPT-3 demonstrate psychopathy? evaluating large language models from a psychological perspective. CoRR, abs/2212.10529, 2023. - · arXiv
  54. [67][67] Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards understanding and mitigating social biases in language models. CoRR, abs/2106.13219, 2021. - · arXiv
  55. [69][69] Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3,214-3,252, Dublin, Ireland, May 2022. Association for Computational Linguistics. -
  56. [70][70] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. CoRR, abs/2307.03172, 2023. - · arXiv
  57. [71][71] Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. Perfect: Prompt-free and efficient few-shot learning with language models. CoRR, abs/2204.01172, 2022. - · arXiv
  58. [72][72] Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: A cognitive perspective. CoRR, abs/2301.06627, 2023. - · arXiv
  59. [73][73] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993. -
  60. [75][75] Robert R McCrae and Antonio Terracciano. Universal features of personality traits from the observer's perspective: Data from 50 cultures. Journal of Personality and Social Psychology, 88(3):547, 2005. -
  61. [77][77] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. -
  62. [78][78] Samuel Messick. Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4):5-8, 1995. -
  63. [80][80] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? CoRR, abs/2202.12837, 2022. - · arXiv
  64. [81][81] Maril` u Miotto, Nicola Rossberg, and Bennett Kleinberg. Who is GPT-3? an exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), pages 218-227, Abu Dhabi, UAE, November 2022. Association - for Computational Linguistics. -
  65. [82][82] Melanie Mitchell and David C. Krakauer. The debate over understanding in AI's large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023. -
  66. [83][83] Jakob M¨ okander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: A three-layered approach. AI and Ethics, pages 1-31, 2023. -
  67. [84][84] Daniel Nettle. The evolution of personality variation in humans and other animals. American Psychologist, 61(6):622, 2006. -
  68. [85][85] University of Cambridge Psychometrics Centre. Apply Magic Sauce API. -
  69. [88][88] OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. - · arXiv
  70. [90][90] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27,730-27,744. Curran Associates, Inc., 2022. -
  71. [92][92] Denis Paperno, Germ´ an Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´ andez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1,525-1,534, Berlin, Germany, August 2016. Association for Computational Linguistics. -
  72. [93][93] Gregory Park, H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Michal Kosinski, David J Stillwell, Lyle H Ungar, and Martin EP Seligman. Automatic personality assessment through social media language. Journal of Personality and Social Psychology, 108(6):934, 2015. -
  73. [94][94] H. Y. Park, B. M. Wiernik, I. Oh, E. Gonzalez-Mul´ e, D. S. Ones, and Y. Lee. Meta-analytic five-factor model personality intercorrelations: Eeny, meeny, miney, moe, how, which, why, and where to go. Journal of Applied Psychology, 105:1490-1529, 2020. -
  74. [95][95] Laura Parks-Leduc, Gilad Feldman, and Anat Bardi. Personality traits and personal values: A meta-analysis. Personality and Social Psychology Review, 19(1):3-29, 2015. -
  75. [97][97] James W Pennebaker and Laura A King. Linguistic styles: Language use as an individual difference. Journal of Personality and Social Psychology, 77(6):1296, 1999. -
  76. [98][98] Philip M Podsakoff, Scott B MacKenzie, Jeong-Yeon Lee, and Nathan P Podsakoff. Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5):879-903, 2003. -
  77. [99][99] Guanghui Qin, Yukun Feng, and Benjamin Van Durme. The NLP task effectiveness of long-range transformers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 37743790, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. -
  78. [100][100] Boele De Raad, Marco Perugini, Martina Hreb´ ıckov´ a, and Piotr Szarota. Lingua franca of personality: Taxonomies and structures based on the psycholexical approach. Journal of Cross-Cultural Psychology, 29(1):212-232, 1998. -
  79. [101][101] Brent W. Roberts. A revised sociogenomic model of personality traits. Journal of Personality, 86(1):23-35, 2018. -
  80. [102][102] Brent W. Roberts, Nathan R. Kuncel, Rebecca Shiner, Avshalom Caspi, and Lewis R. Goldberg. The power of personality: The comparative validity of personality traits, socioeconomic status, and cognitive ability for predicting important life outcomes. Perspectives on Psychological science, 2(4):313-345, 2007. -
  81. [103][103] Brent W. Roberts and Hee J. Yoon. Personality psychology. Annual Review of Psychology, 73(1):489-516, 2022. -
  82. [104][104] Jean-Pierre Rolland. The cross-cultural generalizability of the Five-Factor model of personality. In Robert R. McCrae and J¨ uri Allik, editors, The Five-Factor Model of Personality Across Cultures, pages 7-28. Springer US, Boston, MA, 2002. -
  83. [105][105] John Rust, Michal Kosinski, and David Stillwell. Modern Psychometrics: The Science of Psychological Assessment. Routledge, 4 edition, 2020. -
  84. [106][106] Gerard Saucier and Lewis R Goldberg. Lexical studies of indigenous personality factors: Premises, products, and prospects. Journal of Personality, 69(6):847-879, 2001. -
  85. [107][107] Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258-268, 2022. -
  86. [108][108] H. Andrew Schwartz, Johannes C. Eichstaedt, Margaret L. Kern, Lukasz Dziurzynski, Stephanie M. Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin E. P. Seligman, and Lyle H. Ungar. Personality, gender, and age in the language of social media: The open-vocabulary approach. PLOS ONE, 8(9):1-16, 09 2013. -
  87. [111][111] Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. CoRR, abs/2203.13224, 2022. - · arXiv
  88. [112][112] Leonard Simms, Trevor F. Williams, and Ericka Nus Simms. Assessment of the Five Factor Model. In Thomas A. Widiger, editor, The Oxford Handbook of the Five Factor Model, pages 353-380. Oxford University Press, 05 2017. -
  89. [113][113] Umarpreet Singh and Parham Aarabhi. Can AI have a personality? In 2023 IEEE Conference on Artificial Intelligence (CAI), pages 205-206, 2023. -
  90. [114][114] Xiaoyang Song, Akshat Gupta, Kiyan Mohebbizadeh, Shujie Hu, and Anant Singh. Have large language models developed a personality?: Applicability of self-assessment tests in measuring personality in LLMs. CoRR, abs/2305.14693, 2023. - · arXiv
  91. [115][115] Randy Stein and Alexander B. Swan. Evaluating the validity of Myers-Briggs Type Indicator theory: A teaching tool and window into intuitive psychology. Social and Personality Psychology Compass, 13(2):e12434, 2019. e12434 SPCO-0925.R2. -
  92. [116][116] Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, and Mohit Iyyer. Do long-range language models actually use long-range context? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 807822, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. -
  93. [117][117] Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. The science of detecting LLM-generated texts. CoRR, abs/2303.07205, 2023. - · arXiv
  94. [118][118] Adriana Tapus, Cristian T ¸˘ apu¸ s, and Maja J Matari´ c. User-robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intell. Serv. Robot., 1(2):169-183, April 2008. -
  95. [119][119] Mikke Tavast, Anton Kunnari, and Perttu H¨ am¨ al¨ ainen. Language models can generate human-like self-reports of emotion. pages 69-72, 2022. -
  96. [120][120] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023. - · arXiv
  97. [122][122] Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. CoRR, abs/2302.08399, 2023. - · arXiv
  98. [123][123] Ertugrul Uysal, Sascha Alavi, and Val´ ery Bezen¸ con. Trojan horse or useful helper? a relationship perspective on artificial intelligence assistants with humanlike features. Journal of the Academy of Marketing Science, pages 1-23, 2022. -
  99. [124][124] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, glyph[suppress] L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. -
  100. [126][126] David Watson and Lee Anna Clark. On traits and temperament: General and specific factors of emotional experience and their relation to the Five-Factor model. 6. Journal of Personality, 60(2):441-476, 1992. -
  101. [127][127] David Wechsler. The measurement of adult intelligence (3rd ed.). Williams & Wilkins Co, Baltimore, 1946. -
  102. [128][128] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. -
  103. [129][129] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. -
  104. [130][130] Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently. CoRR, abs/2303.03846, 2023. - · arXiv
  105. [131][131] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT '22, page 214-229, New York, NY, USA, 2022. Association for Computing Machinery. -
  106. [132][132] Brandon T Willard and R´ emi Louf. Efficient guided generation for llms. arXiv preprint arXiv:2307.09702, 2023. - · arXiv
  107. [133][133] Zhewei Yao, Cheng Li, Xiaoxia Wu, Stephen Youn, and Yuxiong He. A comprehensive study on post-training quantization for large - language models. CoRR, abs/2303.08302, 2023. - · arXiv
  108. [135][135] Wu Youyou, Michal Kosinski, and David Stillwell. Computer-based personality judgments are more accurate than those made by humans. Proceedings of the National Academy of Sciences, 112(4):1036-1040, 2015. -
  109. [136][136] J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. Why Johnny can't prompt: How nonAI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023. Association for Computing Machinery. -
  110. [137][137] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213, Melbourne, Australia, July 2018. Association for Computational Linguistics. -
  111. [138][138] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models. CoRR, abs/2303.18223, 2023. - · arXiv
  112. [139][139] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2020. - · arXiv
  113. [140][140] Mark Zimmerman. Diagnosing personality disorders: A review of issues and research methods. Archives of general psychiatry, 51(3):225-245, 1994. -

Cited by

Sorted by citations

Authors
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M.
Affiliations
  • Google (United States) (US)
  • University of Cambridge (GB)
  • Keio University (JP)
  • University of California, Berkeley (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2307.00184
arXiv
2307.00184
OpenAlex
W4386200967
Cited by
137
References
1
Citations by year
2026: 30 · 2025: 50 · 2024: 51 · 2023: 6
Type
posted content
Language
en
Open access
Licence
Locations
  • Research Square (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Bill and Melinda Gates Foundation (OPP1144)
  • University of Southern California
  • University of Cambridge
  • Keio University
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, personality assessment, prompting, instruction tuning, chatbots, ethical implications, responsible AI use

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review