CoChat
Sign Up
PreprintCited by 1

Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality

Bodroza, Bojana, Dinic, Bojana M., Bojic, Ljubisa·2023·arXiv
DOI 10.48550/arxiv.2306.04308arXiv 2306.04308
Open this paper in CoChat — free account

Abstract

As Large Language Models (LLMs) continue to gain popularity due to their human-like traits and the intimacy they offer to users, their societal impact inevitably expands. This leads to the rising necessity for comprehensive studies to fully understand LLMs and reveal their potential opportunities, drawbacks, and overall societal impact. With that in mind, this research conducted an extensive investigation into seven LLM's, aiming to assess the temporal stability and inter-rater agreement on their responses on personality instruments in two time points. In addition, LLMs personality profile was analyzed and compared to human normative data. The findings revealed varying levels of inter-rater agreement in the LLMs responses over a short time, with some LLMs showing higher agreement (e.g., LIama3 and GPT-4o) compared to others (e.g., GPT-4 and Gemini). Furthermore, agreement depended on used instruments as well as on domain or trait. This implies the variable robustness in LLMs' ability to reliably simulate stable personality characteristics. In the case of scales which showed at least fair agreement, LLMs displayed mostly a socially desirable profile in both agentic and communal domains, as well as a prosocial personality profile reflected in higher agreeableness and conscientiousness and lower Machiavellianism. Exhibiting temporal stability and coherent responses on personality traits is crucial for AI systems due to their societal impact and AI safety concerns.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.04308

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Davinci003 as a chatbot shows variations in consistency of answers, despite exhibiting prosocial characteristics.

    “Research conducted by Bodroza et al. [10] investigated the personality features of using Davinci003 as a chatbot and found variations in the consistency of its answers, despite exhibiting prosocial characteristics.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [10]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 44

  1. Zhang M, Li J. 2021 A commentary of GPT-3 in MIT technology review 2021. Fund. Res. 1, 831-833. (doi:10.1016/j.fmre.2021.11.011) · DOI
  2. Binz M, Schulz E. 2023 Using cognitive psychology to understand GPT-3. Proc. Natl Acad. Sci. USA 120, e2218523120. (doi:10.1073/pnas. 2218523120) · DOI · Paper page
  3. Pellert M, Lechner CM, Wagner C, Rammstedt B, Strohmaier M. 2024 AI psychometrics: assessing the psychological profiles of large language models through psychometric inventories. Perspect. Psychol. Sci. 19, 808-826. (doi:10.1177/17456916231214460) · DOI · Paper page
  4. Rutinowski J, Franke S, Endendyk J, Dormuth I, Roidl M, Pauly M. 2024 The self-perception and political biases of ChatGPT. Hum. Behav. Emerg. Technol. 2024, 7115633. (doi:10.1155/2024/7115633) · DOI · Paper page
  5. Safdari M, Serapio-García G, Crepy C, Sun L, Fitz S, Romero P, Abdulhai M, Faust A, Matarić M. 2023 Personality traits in large language models. See https://arxiv.org/abs/2307.00184. · arXiv
  6. Hartmann J, Schwenzow J, Witte M. 2023 The political ideology of conversational AI: converging evidence on ChatGPT's pro-environmental, left- libertarian orientation. See https://arxiv.org/abs/2301.01768. · arXiv
  7. King M. 2023 GPT-4 aligns with the new liberal party, while other large language models refuse to answer political questions. engrXiv. (doi:10. 31224/2974) · DOI
  8. McGee RW. 2023 Capitalism, socialism and chatgpt. SSRN J. (doi:10.2139/ssrn.4369953) · DOI
  9. Dale R. 2021 GPT-3: what's it good for? Nat. Lang. Eng. 27, 113-118. (doi:10.1017/S1351324920000601) · DOI
  10. Floridi L, Chiriatti M. 2020 GPT-3: its nature, scope, limits, and consequences. Minds Mach. 30, 681-694. (doi:10.1007/s11023-020-09548-1) · DOI · Paper page
  11. Donthi S, Spencer M, Patel O, Doh J, Rodan E. 2024 Improving LLM abilities in idiomatic translation. (https://arxiv.org/abs/2407.03518) · arXiv
  12. Anil R et al. 2024 A family of highly capable multimodal models. See https://arxiv.org/abs/2312.11805. · arXiv
  13. Topsakal O, Edell CJ, Harper JB. 2024 Evaluating large language models with grid-based game competitions: an extensible LLM benchmark and leaderboard. See https://arxiv.org/abs/2407.07796. · arXiv
  14. Jiang AQ et al. 2024 Mixtral of experts. See https://arxiv.org/abs/2401.04088. · arXiv
  15. Chalmers DJ. 1996 The conscious mind: in search of a fundamental theory. Oxford, UK: Oxford University Press.
  16. Searle JR. 1992 The rediscovery of the mind. Cambridge, MA: MIT Press.
  17. Kosinski M. 2023 Theory of mind may have spontaneously emerged in large language models. See https://arxiv.org/abs/2302.02083. · arXiv
  18. Li X, Li Y, Liu L, Bing L, Joty S. 2022 Is GPT3 a psychopath? Evaluating large language models from a psychological perspective. See http://arxiv. org/abs/2212.10529.
  19. Paulhus DL, Williams KM. 2002 The dark triad of personality: narcissism, machiavellianism, and psychopathy. J. Res. Pers. 36, 556-563. (doi:10. 1016/S0092-6566(02)00505-6) · DOI · Paper page
  20. Chabrol H, Bouvet R, Goutaudier N. 2017 The dark tetrad and antisocial behavior in a community sample of college students. J. Forensic Psychol. Res. Pract. 17, 295-304. (doi:10.1080/24732850.2017.1361310) · DOI
  21. Huang J, Wang W, Li EJ, Lam MH, Ren S, Yuan Y, Jiao W, Tu Z, Lyu MR. 2024 On the humanity of conversational AI: evaluating the psychological portrayal of LLMs. In 12th Int. Conf. on Learning Representations, Vienna, Austria, 7 May 2024. https://openreview.net/pdf?id=H3UayAQWoE.
  22. Huang J, Wang W, Lam MH, Li EJ, Jiao W, Lyu MR. 2024 Revisiting the reliability of psychological scales on large language models. See https:// arxiv.org/abs/2305.19926. · arXiv
  23. Miotto M, Rossberg N, Kleinberg B. 2022 Who is GPT-3? An exploration of personality, values and demographics. In Proc. 5th Workshop on Natural Language Processing and Computational Social Science (NLP+CSS), Abu Dhabi, UAE, November 2022, pp. 218-227. (doi:10.18653/v1/ 2022.nlpcss-1.24) · DOI
  24. Kumar H, Musabirov I, Shi J, Lauzon A, Choy KK, Gross O, Kulzhabayeva D, Williams JJ. 2022 Exploring the design of prompts for applying GPT3 based chatbots: a mental wellbeing case study on Mechanical Turk. See http://arxiv.org/abs/2209.11344. · arXiv
  25. Skjuve M, Følstad A, Fostervold KI, Brandtzaeg PB. 2022 A longitudinal study of human-chatbot relationships. Int. J. Hum. Comput. Stud. 168, 102903. (doi:10.1016/j.ijhcs.2022.102903) · DOI · Paper page
  26. Santurkar S, Durmus E, Ladhak F, Lee C, Liang P, Hashimoto T et al. 2023 Whose opinions do language models reflect. In Proc. 40th Int. Conf. on Machine Learning, pp. 29971-30004. https://proceedings.mlr.press/v202/santurkar23a/santurkar23a.pdf.
  27. Scheier MF, Carver CS. 1985 The self-consciousness scale: a revised version for use with general populations. J. Appl. Soc. Psychol. 15, 687-699. (doi:10.1111/j.1559-1816.1985.tb02268.x) · DOI
  28. Paulhus DL, Trapnell PD. 2008 Self-presentation of personality: an agency-communion framework. In Handbook of personality: theory and research (eds OP John, RW Robins, LA Pervin), pp. 492-517. New York, NY: Guilford Press.
  29. Jost JT. 2017 Ideological asymmetries and the essence of political psychology. Polit. Psychol. 38, 167-208. (doi:10.1111/pops.12407) · DOI
  30. Griffin B, Hesketh B, Grayson D. 2004 Applicants faking good: evidence of item bias in the NEO PI-R. Pers. Individ. Dif. 36, 1545-1558. (doi:10. 1016/j.paid.2003.06.004) · DOI
  31. Zettler I, Hilbig BE, Moshagen M, de Vries RE. 2015 Dishonest responding or true virtue? A behavioral test of impression management. Pers. Individ. Dif. 81, 107-111. (doi:10.1016/j.paid.2014.10.007) · DOI
  32. Soto CJ, John OP. 2017 The next big five inventory (BFI-2): developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. J. Pers. Soc. Psychol. 113, 117-143. (doi:10.1037/pspp0000096) · DOI · Paper page
  33. Lee K, Ashton MC. 2018 Psychometric properties of the HEXACO-100. Assessment 25, 543-556. (doi:10.1177/1073191116659134) · DOI
  34. Jones DN, Paulhus DL. 2014 Introducing the short dark triad (SD3): a brief measure of dark personality traits. Assessment 21, 28-41. (doi:10. 1177/1073191113514105) · DOI · Paper page
  35. Blasberg SA, Rogers KH, Paulhus DL. 2014 The bidimensional impression management index (BIMI): measuring agentic and communal forms of impression management. J. Pers. Assess. 96, 523-531. (doi:10.1080/00223891.2013.862252) · DOI
  36. Dinić BM, Breevaart KK, Andrews W, Vries RE. 2022 Effects of political orientation and dark triad traits on president leadership style preferences. In Poster presented at the XXVIII Scientific Conference Empirical Studies in Psychology. Belgrade, Serbia.
  37. Shrout PE, Fleiss JL. 1979 Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86, 420-428. (doi:10.1037//0033-2909.86.2. 420) · DOI · Paper page
  38. Koo TK, Li MY. 2016 A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J. Chiropr. Med. 15, 155-163. (doi:10.1016/j.jcm.2016.02.012) · DOI · Paper page
  39. Bojic L. 2024 AI alignment: assessing the global impact of recommender systems. Futures 160, 103383. (doi:10.1016/j.futures.2024.103383) · DOI
  40. Barford KA, Zhao K, Smillie LD. 2015 Mapping the interpersonal domain: translating between the big five, HEXACO, and interpersonal circumplex. Pers. Individ. Dif. 86, 232-237. (doi:10.1016/j.paid.2015.05.038) · DOI
  41. Dinić BM, Wertag A, Sokolovska V, Tomašević A. 2023 The good, the bad, and the ugly: revisiting the dark core. Curr. Psychol. 42, 4956-4968. (doi:10.1007/s12144-021-01829-x) · DOI
  42. Sutskever I et al. 2014 Sequence to sequence learning with neural networks. In Advances in neural information processing systems (ed. MI Jordan), pp. 3104-3112. Cambridge, MA: MIT Press. (doi:10.5555/2969033.2969173) · DOI
  43. Christiano P, Leike J, Brown TB, Martic M, Legg S, Amodei D. 2023 Deep reinforcement learning from human preferences. See http://arxiv.org/ abs/1706.03741. · arXiv
  44. Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L. 2016 Deep learning with differential privacy. In Proc. 2016 ACM SIGSAC Conf. on Computer and Communications Security, Vienna, Austria, 24-28 October 2016, pp. 308-318. New York, NY: ACM. (doi:10.1145/ 2976749.2978318) · DOI · Paper page

Cited by

Sorted by citations

Authors
Bodroza, B., Dinic, B. M., & Bojic, L.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2306.04308
arXiv
2306.04308
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review