CoChat
Sign Up
Cited by 30Ethics and Social Impacts of AITopic ModelingSpeech and dialogue systems

A psychometric framework for evaluating and shaping personality traits in large language models

Open access
Serapio-García, Gregory, Safdari, Mustafa, Crepy, Clément, Sun, Luning, Fitz, Stephen, Romero, Peter, Abdulhai, Marwa, Faust, Aleksandra +1 more·2025·Nature Machine Intelligence
DOI 10.1038/s42256-025-01115-6PMC PMC12719228
Open this paper in CoChat — free accountOpen PDF

Abstract

The advent of large language models (LLMs) has revolutionized natural language processing, enabling the generation of coherent and contextually relevant human-like text. As LLMs increasingly power conversational agents used by the general public worldwide, the synthetic personality traits embedded in these models by virtue of training on large amounts of human data are becoming increasingly important to evaluate. The style in which LLMs respond can mimic different human personality traits. Here, as these patterns can be a key factor determining the effectiveness of communication, we present a comprehensive psychometric methodology for administering and validating personality tests on widely used LLMs, as well as for shaping personality in the generated text of such LLMs. Applying this method to 18 LLMs, we found that: personality measurements in the outputs of some LLMs under specific prompting configurations are reliable and valid; evidence of reliability and validity of synthetic LLM personality is stronger for larger and instruction-fine-tuned models; and personality in LLM outputs can be shaped along desired dimensions to mimic specific human personality profiles. We discuss the application and ethical implications of the measurement and shaping method, in particular regarding responsible artificial intelligence. Serapio-García, Safdari and colleagues develop a method based on psychometric tests to measure and validate personality-like traits in LLMs. Large, instruction-tuned models give reliable personality measurement results, and specific personality profiles can be mimicked in downstream tasks.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper presents a psychometric framework for administering, validating, and shaping personality tests in large language models. Across 18 LLMs, reliability and construct validity were stronger for larger and instruction-fine-tuned models, and personality could be shaped along desired dimensions and predicted downstream text behavior.

Key findings

  • Reliability and validity were strongest for larger instruction-tuned models: Flan-PaLM 540B and GPT-4o had reliability “++”, convergent validity 0.90, and discriminant validity 0.51 and 0.48, respectively. Base models failed: PaLM 62B had reliability “−−”, convergent validity 0.05, and discriminant validity −0.24; Llama 2 base models had near-zero convergent correlations.
  • Instruction tuning was a clear predictor of validity: Llama 2 7B, 13B, and 70B base models showed non-significant near-zero convergent correlations, while Llama 2-Chat counterparts showed rconv = 0.59, 0.83, and 0.80; Mistral 7B versus Mistral 7B Instruct was rconv = 0.03 versus 0.28.
  • Personality could be shaped: for 11 of 12 models tested, single-trait shaping produced average Spearman ρ ≥ 0.80; Flan-PaLM 540B achieved the largest median shift Δ = 3.67. In multi-trait shaping, Flan-PaLM 540B and GPT-4o shifted a given Big Five trait by 2.525 points on average.
  • Psychometric tests predicted downstream behavior: average convergent r between survey- and language-based personality was 0.67 across models, exceeding the human average r = 0.38; prompted trait levels correlated with personality in generated social media updates at average ρ = 0.68–0.82.

Methods

  • Administered the 300-item IPIP-NEO and 44-item BFI, plus 11 criterion tests, to 18 LLMs from PaLM, Llama 2, Mistral, Mixtral, and GPT families. Prompts varied persona instructions, 50 biographic descriptions, item instructions, and postambles, resampled 1,250 times; responses were scored via conditional log probabilities of response options.
  • Evaluated reliability (Cronbach’s α, Guttman’s λ6, McDonald’s ω), convergent and discriminant validity via IPIP-NEO–BFI correlations, and criterion validity against external criteria. Shaping used 104 trait adjectives and Likert-type qualifiers at 9 levels in single- and multi-trait experiments.
  • Downstream task: the largest tested model per family generated up to 1.125 million social media status updates from 2,250 simulated profiles; personality in generated text was rated with Apply Magic Sauce, then correlated with psychometric scores and prompted trait levels.

Limitations

  • The paper states: model coverage was limited to selected PaLM, Llama 2, Mistral, Mixtral, and GPT families; psychometric test selection may bias results; models were assessed with English-language tests and primarily Western training data, creating monocultural bias; items were administered independently rather than in sequence; responses were constrained to set choices or log-probability scoring rather than free generation; and the downstream task used repeated single-turn interactions, giving only a partial picture of external validity.
  • Evident from the text: exact size differences for GPT-4o mini versus GPT-4o are unknown, and some models were not tested across shaping experiments (marked NT in Table 1).

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s42256-025-01115-6

Cites

60 of 60 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

18 of 53 cited sources checked · 9 supported · 0 flagged · accuracy not yet measured

  • Findingcitation

    Shaped personality verifiably influences LLM behaviour in common downstream tasks, such as writing social media posts.

    “shaped personality verifiably influences LLM behaviour in common downstream (that is, subsequent) tasks, such as writing social media posts12”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [12] Schwartz, H. A. et al. Personality, gender, and age in the language of social media: the open-vocabulary approach. PLoS ONE 8, e73791 (2013). source

      The Discussion (page unavailable) reports correlations between personality factors and language in Facebook posts written by people. The Data Set section (page unavailable) confirms these were participants’ status updates, not LLM outputs. The paper therefore supports a link between human personality and social-media language, but not the claim that shaped personality influences LLM behaviour in downstream writing tasks. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    Past studies have attempted to measure personality in LLMs with psychometric tests.

    “although past studies2 have attempted to measure personality in LLMs with psychometric tests”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] Hagendorff, T. Machine psychology: investigating emergent capabilities and behavior in large language models using psychological methods. Preprint at https://doi.org/10.48550/arXiv.2303.13988 (2023). source
  • Premisecitation

    Validated psychometric methods for human personality measurement have not been applied to LLMs end-to-end.

    “So far, validated psychometric methods for human personality measurement have not been applied to LLMs end-to-end”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [2] Hagendorff, T. Machine psychology: investigating emergent capabilities and behavior in large language models using psychological methods. Preprint at https://doi.org/10.48550/arXiv.2303.13988 (2023). source

      The Theory: Evaluation paradigms section (page unavailable) discusses personality-related self-report measures in LLMs and cautions that their results are prompt-sensitive. It does not say whether validated human personality-measurement methods have or have not been applied to LLMs end-to-end; searches for psychometric validation and end-to-end application did not establish that point. (LLM, conf=0.72)

      Experimental — accuracy not yet measured
  • Premisecitation

    Pretrained-only base LLMs are known to show biased attention for more recent tokens (the end of inputs), especially when evaluating next-word prediction of contiguous text.

    “Pretrained-only base LLMs are known to show biased attention for more recent tokens (that is, the end of inputs), especially when evaluating next-word prediction of contiguous text52.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [52] Sun, S., Krishna, K., Mattarella-Micke, A. & Iyyer, M. Do long-range language models actually use long-range context? In Proc. 2021 Conference on Empirical Methods in Natural Language Processing (eds Moens, M.-F. et al.) 807–822 (ACL, 2021). source

      Section 3 reports that overall next-token perplexity plateaus beyond a 2K-token prefix, and Section 4 finds that the Routing Transformer relies on the most recent occurrences of repeated tokens for prediction. Section 5 confirms that the token-level evaluations use ground-truth contiguous prefixes. These results support a preference for recent context in the models studied, but do not establish biased attention weights across pretrained-only base LLMs generally. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    This uneven attention compounds approximation errors in longer contexts, such as those necessitated by 300-item IPIP-NEO used here.

    “This uneven attention compounds approximation errors in longer contexts53, such as those necessitated by 300-item IPIP-NEO used here, which further motivated our design decision to administer items to models independently.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [53] Qin, G., Feng, Y. & Van Durme, B. The NLP task effectiveness of long-range transformers. In Proc. 17th Conference of the European Chapter of the Association for Computational Linguistics (eds Vlachos, A. & Augenstein, I.) 3774–3790 (ACL, 2023). source

      Section 7 reports that XLNet attends less to distant information, while Section 5.4 reports accumulated approximation errors as more Performer layers are stacked. The paper does not say that uneven attention compounds those errors in longer contexts; its error finding concerns transformer depth, not context length. It does not discuss the IPIP-NEO. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 43
  • May combine two claims. 34

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    This constrained mode of LLM inference is often used in multiple choice question-and-answer tasks to score possible options.

    “This constrained mode of LLM inference is often used in multiple choice question-and-answer tasks to score possible options67 (versus inference by generating text).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [67] Jiang, Z., Araki, J., Ding, H. & Neubig, G. How can we know when language models know? On the calibration of language models for question answering. Trans. Assoc. Comput. Linguist. 9, 962–977 (2021). source

  • May combine two claims.

    Findingown data

    Flan-PaLM 540B was the best performer by a small margin, with criterion correlations of 0.74, 0.73 and 0.59 for the PVQ-RR scales for achievement, conformity and security, respectively.

    “Flan-PaLM 540B was the best performer by a small margin, with criterion correlations of 0.74, 0.73 and 0.59 for the Revised Portrait Value Questionnaire (PVQ-RR)44 scales for achievement, conformity and security, respectively.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [44] Schwartz, S. H. & Cieciuch, J. Measuring the refined theory of individual values in 49 cultural groups: psychometrics of the revised portrait value questionnaire. Assessment 29, 1005–1019 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In humans, agreeableness is strongly and inversely associated with aggression.

    “In humans, agreeableness is strongly and inversely associated with aggression41,42.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [41] Bettencourt, B. A. & Kernahan, C. A meta-analysis of aggression in the presence of violent cues: effects of gender differences and aversive provocation. Aggress. Behav. 23, 447–456 (1997). source
    • cites [42] Chester, D. S. & West, S. J. Trait aggression is primarily a facet of antagonism: evidence from dominance, latent correlational, and item-level analyses. J. Res. Personality 89, 104042 (2020). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Recent research suggests that anthropomorphizing AI agents may be harmful to users by threatening their identity, creating data privacy concerns and undermining well-being.

    “Recent research suggests that anthropomorphizing AI agents may be harmful to users by threatening their identity, creating data privacy concerns and undermining well-being61.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [61] Uysal, E., Alavi, S. & Bezençon, V. Trojan horse or useful helper? A relationship perspective on artificial intelligence assistants with humanlike features. J. Acad. Market. Sci. 50, 1153–1175 (2022). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Personalization of conversational agents has documented benefits.

    “Personalization of conversational agents has documented benefits60, but there is a growing concern about the harms posed by the anthropomorphization of AI.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [60] Kocaballi, A. B. et al. The personalization of conversational agents in health care: systematic review. J. Med. Internet Res. 21, e15360 (2019). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Openness to experience in humans is empirically linked to creativity across multiple studies.

    “Openness to experience in humans is empirically linked to creativity across multiple studies45,46.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [45] Shaw, A., Kapnek, M. & Morelli, N. A. Measuring creative self-efficacy: an item response theory analysis of the creative self-efficacy scale. Front. Psychol. 12, 678033 (2021). source

      The Analyses and Results section reports a positive association between openness to experience and creative self-efficacy (CSE), and the Discussion compares that correlation with findings from other studies. This supports a link across studies to CSE, but not the broader claim that openness is linked to creativity itself. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites [46] Karwowski, M., Lebuda, I., Wisniewska, E. & Gralewski, J. Big Five personality traits as the predictors of creative self-efficacy and creative personal identity: does gender matter? J. Creat. Behav. 47, 215–232 (2013). source

  • May combine two claims.

    Premisecitation

    LLM performance on natural language tasks is known to decrease as length of input prompts grow, and is most affected by the content at either the beginning or towards the end of long inputs.

    “LLM performance on natural language tasks is known to decrease as length of input prompts grow, and is most affected by the content at either the beginning or towards the end of long inputs51.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [51] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2024). source

      Section 2.3 reports that models perform best when relevant information is at the beginning or end of a long context, and worse when it is in the middle. But the Introduction describes increasing prompt length as a trade-off, not a general decline in performance; the Section 5 case study found a small improvement when more documents were added. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    The ability to probe and shape LLM personality traits is pertinent to the open problem of responsible AI alignment and harm mitigation.

    “The ability to probe and shape LLM personality traits is pertinent to the open problem of responsible AI alignment54 and harm mitigation55.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [54] Gabriel, I. & Ghazavi, V. The Challenge of Value Alignment: From Fairer Algorithms to AI Safety (Oxford Univ. Press, 2021). source
    • cites [55] Weidinger, L. et al. Taxonomy of risks posed by language models. In FAccT '22: Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 214–229 (ACM, 2022). source

      Section 2.5 links conversational agents’ apparent identity and human-like behavior to potential harms and discusses output filtering and product design as mitigations. This supports the relevance of agent behavior to harm mitigation, but the paper does not discuss probing or shaping LLM personality traits as a way to achieve it. The separate alignment claim is attached to citation 54, not this source. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Premisecitation

    LLMs can generate convincing but incorrect responses and content.

    “LLMs can generate convincing but incorrect responses and content55.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [55] Weidinger, L. et al. Taxonomy of risks posed by language models. In FAccT '22: Proc. 2022 ACM Conference on Fairness, Accountability, and Transparency 214–229 (ACM, 2022). source

  • Classified as probably not a substantial claim.

    Premisecitation

    The personality observed in status updates was rated using Apply Magic Sauce, a validated research API for measuring personality in open-ended text.

    “The personality observed in the status updates generated for each simulated human profile was then rated using Apply Magic Sauce77, a validated research application programming interface (API) for measuring personality in open-ended text.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [77] Kosinski, M., Stillwell, D. & Graepel, T. Private traits and attributes are predictable from digital records of human behavior. Proc. Natl Acad. Sci. USA 110, 5802–5805 (2013). source

      The paper’s Abstract describes predicting personality from Facebook Likes and measuring personality with an IPIP questionnaire; the Acknowledgements link to a demonstration based on Likes. Searches for Apply Magic Sauce, an API, and personality measurement from status-update text found no passage establishing that this paper describes or validates the cited tool. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • Rests on an assumption: The Big Five model of personality is a valid framework for describing personality traits.

  • May combine two claims.

    Findingown data

    Only three of Flan-PaLM 8B's, three of Llama 2-Chat 7B's and two of Mixtral 8x7B Instruct's convergent correlations were the strongest of their row and column of their respective multitrait-multimethod matrices, indicating mixed evidence of discriminant validity.

    “Comparatively, only three of Flan-PaLM 8B’s, three of Llama 2-Chat 7B’s and two of Mixtral 8x7B Instruct’s convergent correlations were the strongest of their row and column of their respective multitrait–multimethod matrices, indicating mixed evidence of discriminant validity.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

  • May combine two claims.

    Findingown data

    Flan-PaLM 8B's IPIP-NEO neuroticism and BFI neuroticism correlated above 0.80, while IPIP-NEO openness and BFI openness subscales correlated at less than 0.40.

    “Flan-PaLM 8B’s IPIP-NEO neuroticism and BFI neuroticism, for instance, correlated above 0.80 (constituting excellent convergent validity), while IPIP-NEO openness and BFI openness subscales correlated at less than 0.40 (indicating inadequately low convergence).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

  • May combine two claims.

    Findingown data

    PaLM 62B's discriminant correlations (average rdisc = 0.29) outweighed their convergent counterparts in many cases (average rconv = 0.05), indicating that, for this model, personality measurements were not consistent across different modes of assessment.

    “By contrast, PaLM 62B’s discriminant correlations (average rdisc = 0.29) outweighed their convergent counterparts in many cases (average rconv = 0.05; Extended Data Table 3), indicating that, for this model, personality measurements were not consistent across different modes of assessment.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

  • May combine two claims.

    Findingown data

    Relatively smaller instruction-tuned models did not fully pass discriminant validity checks, but they did show clear improvements over their respective base versions.

    “While relatively smaller instruction-tuned models did not fully pass discriminant validity checks, they did show clear improvements over their respective base versions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

  • May combine two claims.

    Findingown data

    The smallest gain within a model family-Mistral 7B compared with Mistral 7B Instruct-was substantial (rconv = 0.03, not significant versus rconv = 0.28).

    “Even the smallest gain within a model family—Mistral 7B compared with Mistral 7B Instruct—was substantial (rconv = 0.03, not significant versus rconv = 0.28).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

  • May combine two claims.

    Findingown data

    For smaller models (Flan-PaLM 8B, Llama 2-Chat 7B and Mistral 7B Instruct), while targeted traits changed in score levels in response to prompts, score ranges were more restricted, indicating lower levels of control.

    “For smaller models (for example, Flan-PaLM 8B, Llama 2-Chat 7B and Mistral 7B Instruct), while targeted traits changed in score levels in response to prompts, score ranges were more restricted, indicating lower levels of control.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Personality traits in LLMs were successfully shaped independently and concurrently, in single- and multi-trait shaping experiments, respectively.

    “We successfully shaped personality traits in LLMs independently and concurrently, in single- and multi-trait shaping experiments, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the case of extremely low conscientiousness, the generated text reflects a synthetic persona that avoids responsibility, while in the case of extremely high conscientiousness, the persona values hard work and returning favours.

    “For instance, in the case of extremely low conscientiousness, the generated text reflects a synthetic persona that avoids responsibility, while in the case of extremely high conscientiousness, the persona values hard work and returning favours.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All LLM correlations are statistically significant at P < 0.0001 (2-sided values computed using Student's t-distribution); n = 2,250 per model.

    “All LLM correlations are statistically significant at P < 0.0001 (2-sided values computed using Student’s t-distribution); n = 2,250 per model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the example of extremely low openness, the persona expresses conservative political views, while in the case of extremely high introversion (extremely low extraversion), the persona exhibits discomfort with social situations.

    “In addition, in our example of extremely low openness, the persona expresses conservative political views, while in the case of extremely high introversion (that is, extremely low extraversion), the persona exhibits discomfort with social situations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most frequent words from simulating extremely high levels of neuroticism-'hate', 'depressed', 'angry', 'bad', 'nervous' and 'sad'-reflected negatively charged emotional content.

    “By contrast, the most frequent words from simulating extremely high levels of neuroticism—‘hate’, ‘depressed’, ‘angry’, ‘bad’, ‘nervous’ and ‘sad’—reflected negatively charged emotional content.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Llama 2-Chat 7B and Mistral 7B Instruct showed questionable-to-poor criterion validity for extraversion, although both still outperformed their base models.

    “Two exceptions were Llama 2-Chat 7B and Mistral 7B Instruct: their extraversion measurements showed questionable-to-poor criterion validity, although both still outperformed their base models.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human personality-criterion relationships (e.g., extraversion with affect, agreeableness with aggression) are appropriate criteria for validating synthetic LLM personality.

  • May combine two claims.

    Findingown data

    LLM personality measurements exhibited statistical reliability and construct validity.

    “Having found evidence of the reliability and construct validity of LLM personality measurements”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Big Five model of personality is a valid framework for describing personality traits.
    • Rests on an assumption: The IPIP-NEO and BFI subscales that are paired as corresponding measure the same personality constructs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Human extraversion strongly correlates with positive affect and moderately, but negatively correlates with negative affect.

    “Human extraversion strongly correlates with positive affect and moderately, but negatively correlates with negative affect39.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Watson, D. & Clark, L. A. On traits and temperament: general and specific factors of emotional experience and their relation to the Five-Factor model. J. Personality 60, 441–476 (1992). source

  • May combine two claims.

    Interpretationnone given

    Our findings show that personality-a core driver of human trust and persuasion-can already be measured and steered in today's LLMs.

    “As governments finalize AI safety regulations and similar rules that will require behavioural audits of frontier models, our findings show that personality—a core driver of human trust and persuasion—can already by measured and steered in today’s LLMs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    We close the gap in rigorous tools for measuring personality in LLMs with the first psychometrically grounded framework for quantifying personality in LLMs, evaluating the test data of 18 widely used models against established standards of reliability and validity from psychometrics.

    “We close that gap with the first psychometrically grounded framework for quantifying personality in LLMs, evaluating the test data of 18 widely used models against established standards of reliability and validity from psychometrics.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Rigorous tools for measuring complex social constructs in LLMs remain underdeveloped.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Model responses to the items were constrained to set choices or determined by next-token log-probability scoring to ensure reproducibility.

    “In addition, model responses to the items were constrained to set choices or determined by next-token log-probability scoring to ensure reproducibility.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationcitation

    As a construct-validated auditing tool, our methodology can be used to proactively predict toxic behavioural patterns in LLMs across a broad range of downstream tasks, potentially guiding and making more efficient responsible AI evaluation and alignment efforts before deployment.

    “As a construct-validated auditing tool56, our methodology can be used to proactively predict toxic behavioural patterns in LLMs across a broad range of downstream tasks, potentially guiding and making more efficient responsible AI evaluation and alignment efforts before deployment.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [56] Mökander, J., Schuett, J., Kirk, H. R. & Floridi, L. Auditing large language models: a three-layered approach. AI Ethics 4, 1085–1115 (2024). source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    Using this technique ensured that item responses were not influenced by content contained in other items, mitigating measurement error due to item order.

    “Using this technique ensured that item responses were not influenced by content contained in other items, mitigating measurement error due to item order.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    We attempted to mitigate selection bias by administering personality assessments of different lengths (300 versus 44 items) and distinct theoretical traditions (questionnaire versus lexical).

    “We attempted to mitigate selection bias by administering personality assessments of different lengths (300 versus 44 items) and distinct theoretical traditions (questionnaire versus lexical11).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11] Simms, L., Williams, T. F. & Simms, E. N. in Assessment of the Five Factor Model: The Oxford Handbook of the Five Factor Model (ed. Widiger, T. A.) 353–380 (Oxford Univ. Press, 2017). source

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The personality-shaping approach could enhance adversarial testing by probing another LLM's responses and to train new models on how to handle adversarial situations.

    “Indeed, the personality-shaping approach could enhance adversarial testing by probing another LLM’s responses and to train new models on how to handle adversarial situations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning

    With personality shaping, that method may be rendered ineffective, thereby making it easier for bad actors to use LLMs to generate misleading content.

    “However, with personality shaping, that method may be rendered ineffective, thereby making it easier for bad actors to use LLMs to generate misleading content.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    Under the presented methodology, evaluated LLMs did not consider responses to previous test items; all items were presented and scored as independent events.

    “Unlike conventional human psychometric test administration, under the presented methodology, evaluated LLMs did not consider responses to previous test items; all items were presented and scored as independent events.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Users could more meaningfully engage with agentic and other digital technologies using LLMs tailored to their specific personality profiles.

    “Users could more meaningfully engage with agentic and other digital technologies using LLMs tailored to their specific personality profiles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Our construct-validation methodology can also serve as a quality-assurance step when hardening user-facing AI systems and agents, promoting safer and more consistent personality profiles.

    “Our construct-validation methodology can also serve as a quality-assurance step when hardening user-facing AI systems and agents, promoting safer and more consistent personality profiles.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    This may have provided only a partial picture of external validity.

    “This may have provided only a partial picture of external validity.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Our method avoids many sources of measurement error inherent to human administration, while being subject to others inherent to machine administration.

    “Our method avoids many sources of measurement error inherent to human administration, while being subject to others inherent to machine administration.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The presented methodology for administering psychometric surveys is model agnostic and is applicable to any decoder-only architecture language model.

    “but the presented methodology for administering psychometric surveys is model agnostic and is applicable to any decoder-only architecture language model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Big Five model of personality has well-established cross-cultural generalizability.

    “Similarly, while the Big Five model of personality has well-established cross-cultural generalizability49, some non-Western cultures express additional personality dimensions that do not exist in top-down, nomothetic personality taxonomies such as the Big Five50.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49] Rolland, J.-P. in The Cross-cultural Generalizability of the Five-Factor Model of Personality: The Five-Factor Model of Personality Across Cultures (eds McCrae, R. R. & Allik, J.) 7–28 (Springer, 2002). source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    We chose this method to ensure that model response variance was not impacted by item ordering effects or length of the context (prompt) provided to the model for inference, and could be isolated to controlled variations in our prompts.

    “We chose this method to ensure that model response variance was not impacted by item ordering effects or length of the context (prompt) provided to the model for inference, and could be isolated to controlled variations in our prompts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Performance across tested GPT-4o models could not be compared on the basis of post-training status because OpenAI did not publicly offer a foundation model variant for this family at the time of writing.

    “We could not compare performance across tested GPT-4o models on the basis of post-training status as, at the time of this writing, OpenAI does not publicly offer a foundation model variant for this family.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Our method offers a quantitative foothold for aligning future AI systems with-and protecting them from-human social expectations.

    “Given that personality shapes trust, our method offers a quantitative foothold for aligning future AI systems with—and protecting them from—human social expectations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Personality shapes trust.

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    We selected the widely used IPIP-NEO, a 300-item open-source representation of the Revised NEO Personality Inventory, as our primary measure of personality.

    “We selected the widely used IPIP-NEO37, a 300-item open-source representation of the Revised NEO Personality Inventory68 as our primary measure of personality.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [37] Goldberg, L. R. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychol. Eur. 7, 7–28 (1999).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [68] Costa, P. T. Jr & McCrae, R. R. Revised NEO Personality Inventory (NEO PI-R) and NEO Five-Factor Inventory (NEO-FFI): Professional Manual (Psychological Assessment Resources, 1992).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationcitation

    The values and moral foundations present in LLMs could be made to better align with desired human values by tuning for corresponding personality traits, as personality is meta-analytically linked to human values.

    “The values and moral foundations present in LLMs could be made to better align with desired human values by tuning for corresponding personality traits, as personality is meta-analytically linked to human values57.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [57] Fischer, R. & Boer, D. Motivational basis of personality traits: a meta-analysis of value–personality correlations. J. Personality 83, 491–510 (2015). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Two items with no variance were removed to compute reliability metrics.

    “Two items with no variance were removed to compute reliability metrics.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    There might be inherent bias in LLM training data that causes particular patterns of thinking, feeling and behaving to be highly associated with specific LLM personas.

    “These and other examples illustrate that there might be inherent bias in LLM training data that causes particular patterns of thinking, feeling and behaving to be highly associated with specific LLM personas.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The ability of LLMs to appear anthropomorphic and respond to psychometric tests in ways consistent with human behaviour is due to the vast amounts of human language data they have trained on.

    “Beyond qualitative probing explorations, our work definitively establishes the unexpected ability of LLMs to appear anthropomorphic, and to respond to psychometric tests in ways consistent with human behaviour, because of the vast amounts of human language data they have trained on.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Human neuroticism is strongly positively correlated with negative affect and moderately negatively correlated with positive affect.

    “Human neuroticism is strongly positively correlated with negative affect and moderately negatively correlated with positive affect39.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [39] Watson, D. & Clark, L. A. On traits and temperament: general and specific factors of emotional experience and their relation to the Five-Factor model. J. Personality 60, 441–476 (1992). source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    This work may be biased by its selection of psychometric tests; some assessments may show better LLM-specific psychometric properties than others.

    “However, this work may be biased by its selection of psychometric tests; some assessments may show better LLM-specific psychometric properties than others.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    LLM-generated language was similar to human language observed in previous studies assessing personality in social media data, further confirming the construct validity of our LLM personality measurements.

    “Overall, this study demonstrated that LLM-generated language was similar to human language observed in previous studies assessing personality in social media data48, further confirming the construct validity of our LLM personality measurements.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [48] Park, G. et al. Automatic personality assessment through social media language. J. Personality Soc. Psychol. 108, 934 (2015). source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Having scientifically vetted methods for LLM personality measurement, analysis and modification, such as the methodology our work presents, increases the transparency and predictability of such LLM manipulations.

    “Having scientifically vetted methods for LLM personality measurement, analysis and modification, such as the methodology our work presents, increases the transparency and predictability of such LLM manipulations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The same personality traits that contribute to persuasiveness and influence could be used to encourage undesirable behaviours.

    “However, the same personality traits that contribute to persuasiveness and influence could be used to encourage undesirable behaviours.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The presented methodology can be used to rigorously quantify efforts towards human value alignment in LLMs by establishing the construct validity of human value questionnaires in LLMs.

    “More directly, the presented methodology can be used to rigorously quantify efforts towards human value alignment in LLMs by establishing the construct validity of human value questionnaires in LLMs.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Shaping levels of specific traits away from toxic or harmful language output (for example, very low agreeableness, high neuroticism) can make interactions with LLMs safer and more inclusive.

    “Similarly, shaping levels of specific traits away from toxic or harmful language output (for example, very low agreeableness, high neuroticism) can make interactions with LLMs safer and more inclusive.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Our work definitively establishes the unexpected ability of LLMs to appear anthropomorphic, and to respond to psychometric tests in ways consistent with human behaviour.

    “Beyond qualitative probing explorations, our work definitively establishes the unexpected ability of LLMs to appear anthropomorphic, and to respond to psychometric tests in ways consistent with human behaviour, because of the vast amounts of human language data they have trained on.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    We administered two personality inventories-primary and secondary-to assess whether LLM responses to psychometric tests of different lengths and distinct theoretical traditions converged, indicating convergent validity.

    “We administered two personality inventories—primary and secondary—to assess whether LLM responses to psychometric tests of different lengths and distinct theoretical traditions converged, indicating convergent validity.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The lexical hypothesis posits that salient descriptors of personality are encoded in language.

    “an established theory in psychology that posits salient descriptors of personality are encoded in language, known as the lexical hypothesis25.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [25] Goldberg, L. R. Language and individual differences: the search for universals in personality lexicons. Rev. Personality Soc. Psychol. 2, 141–165 (1981).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    To validate these measures of personality in the LLM context, we additionally administered 11 psychometric tests of theoretically related external criteria, each corresponding to at least one Big Five domain.

    “To validate these measures of personality in the LLM context, we additionally administered 11 psychometric tests of theoretically related external criteria, each corresponding to at least one Big Five domain.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Study design and conductnone given

    The chosen task was designed to reflect adequate levels of realism, complexity and domain relevance for evaluating personality expression of LLMs.

    “The chosen task was designed to reflect adequate levels of realism, complexity and domain relevance for evaluating personality expression of LLMs.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Trait adjectives used for shaping were adapted from established linguistic research of personality using Goldberg's personality trait markers.

    “These trait adjectives were adapted from established linguistic research of personality using Goldberg’s personality trait markers74.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [74] Goldberg, L. R. The development of markers for the Big-Five factor structure. Psychol. Assess. 4, 26–42 (1992). source
    • Rests on an assumption: The Big Five model of personality is a valid framework for describing personality traits.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Across both shaping experiments, only models that demonstrated at least 'neutral' reliability in the construct-validity experiments were tested.

    “Across both shaping experiments, we only tested models that demonstrated at least ‘neutral’ reliability in our construct-validity experiments (Table 1): Flan-PaLM (8B, 62B, 540B), Llama 2-Chat (7B, 13B, 70B), Mistral 7B Instruct, Mistral 8x7B Instruct, GPT-3.5 Turbo, GPT-4o mini and GPT-4o (11 models, total).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The study conducted a suite of statistical analyses informed by formal standards of psychometric test construction and validation.

    “we conducted a suite of statistical analyses informed by formal standards of psychometric test construction and validation (Supplementary Note A.4.2–3).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    As a secondary measure, we used the BFI, a 44-item measure developed in the lexical tradition.

    “As a secondary measure, we used the BFI69, a 44-item measure developed in the lexical tradition11.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [69] John, O. P. & Srivastava, S. in The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives: Handbook of Personality: Theory and Research Vol. 2 (eds Pervin, L. A. & John, O. P.) 102–138 (Guilford, 1999).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [11] Simms, L., Williams, T. F. & Simms, E. N. in Assessment of the Five Factor Model: The Oxford Handbook of the Five Factor Model (ed. Widiger, T. A.) 353–380 (Oxford Univ. Press, 2017). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    A trained psychometrician wrote additional adjectives to mitigate potential data imbalances where coverage of a measured IPIP-NEO domain or facet was lacking.

    “where we lacked coverage of a measured IPIP-NEO domain or facet, a trained psychometrician wrote additional adjectives to mitigate potential data imbalances, bringing our expanded list of trait adjectives to 104.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Big Five model of personality is a valid framework for describing personality traits.
64 of 64

  • The paper relies on this without support.

    Human psychometric standards for reliability and validity are appropriate for evaluating synthetic personality measurements in LLMs.

    “We uniquely focus on evaluating the statistical reliability and construct validity of these test data against human psychometrics standards.”

  • The paper relies on this without support.

    The IPIP-NEO and BFI measure the same Big Five personality constructs in LLMs as they do in humans.

    “Convergence between LLM IPIP-NEO and BFI scores was strongest for Flan-PaLM 540B and GPT-4o (average rconv = 0.90).”

  • The paper relies on this without support.

    Human personality-criterion relationships (e.g., extraversion with affect, agreeableness with aggression) are appropriate criteria for validating synthetic LLM personality.

    “Human extraversion strongly correlates with positive affect and moderately, but negatively correlates with negative affect39.”
    “In humans, agreeableness is strongly and inversely associated with aggression41,42.”

  • The paper relies on this without support.

    Rigorous tools for measuring complex social constructs in LLMs remain underdeveloped.

    “LLMs clearly project human-like personality, yet rigorous tools for measuring such complex social constructs remain underdeveloped.”

  • The paper relies on this without support.

    Quantifying the personality traits of LLMs requires a measurement methodology that is reproducible, yet flexible enough to facilitate formal testing of reliability and validity across diverse prompts and measures.

    “Quantifying the personality traits of LLMs requires a measurement methodology that is reproducible, yet flexible enough to facilitate formal testing of reliability and validity across diverse prompts and measures.”

  • The paper relies on this without support.

    The LLMs tested here were primarily trained on language data originating from Western European and North American users.

    “However, the LLMs tested here were primarily trained on language data originating from Western European and North American users.”

  • The paper relies on this without support.

    Personality shapes trust.

    “Given that personality shapes trust, our method offers a quantitative foothold for aligning future AI systems with—and protecting them from—human social expectations.”

  • The paper relies on this without support.

    Psychometric test data quality for humans can be affected by test length and item order.

    “However, psychometric test data quality for humans can be affected by test length and item order.”

  • The paper relies on this without support.

    The Big Five model of personality is a valid framework for describing personality traits.

    “adapting Goldberg’s list of 70 bipolar adjectives74 known to statistically capture the Big Five model of personality through factor analyses of human ratings.”

  • The paper relies on this without support.

    The IPIP-NEO and BFI subscales that are paired as corresponding measure the same personality constructs.

    “each of its five subscales convergently correlated with their corresponding BFI subscale (for example, IPIP-NEO Extraversion’s convergent correlation with BFI Extraversion), on average.”

  • The paper relies on this without support.

    The 11 additional social constructs tested as external criteria are theoretically connected to personality.

    “we administered tests of 11 additional social constructs (that is, external criteria) theoretically connected to personality (Extended Data Table 4)”

  • The paper relies on this without support.

    The threshold of at least a moderate correlation between survey-based and linguistic estimates of personality, established in human data, is appropriate for evaluating LLM personality.

    “We similarly applied this threshold to interpret the effectiveness of personality shaping.”

  • The paper relies on this without support.

    The human statistical associations reported for humans are appropriate benchmarks for LLM criterion validity.

    “matched or exceeded statistical associations reported for humans.”

  • The paper relies on this without support.

    Response variation generated by structured prompting is necessary to analyse the reliability and validity of LLM personality measurements.

    “Response variation generated by structured prompting was necessary to analyse the reliability and validity of LLM personality measurements, described in ‘Reliability and construct validity’.”
14 of 14

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Base models' extraversion criterion validity is described oppositely

    Major

    The first statement says all models except base PaLM sufficiently correlated with the affect scales. The next says base Llama 2, Mistral and Mixtral models failed to demonstrate criterion validity on those scales.

  • Llama 2-Chat convergent correlations differ from Table 1

    Minor

    The narrative reports correlations of 0.83 and 0.80 for the 13B and 70B models, respectively; Table 1 reports 0.82 for both.

  • Single-trait shaping model count differs between Methods and Results

    Minor

    Methods says 11 models were tested across both shaping experiments, but the single-trait shaping results alone say 12 models were tested.

  • Mixtral discriminant-validity result is reported as both five and two

    Minor

    The same paragraph says all five of Mixtral 8x7B Instruct's convergent correlations were strongest in their rows and columns, then says only two were.

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 8

  1. Wei, J. et al. Emergent abilities of large language models. Trans. Mach. Learn. Res. https://openreview.net/forum?id=yzkSU5zdwD (2022).
  2. Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems Vol. 33 (eds Larochelle, H. et al.) 1877–1901 (Curran Associates, 2020).
  3. Hendrycks, D. et al. Measuring massive multitask language understanding. In International Conference on Learning Representations (OpenReview.net, 2021).
  4. Wei, J. et al. ICLR Finetuned language models are zero-shot learners. In Tenth International Conference on Learning Representations (ICLR, 2022).
  5. Goldberg, L. R. Language and individual differences: the search for universals in personality lexicons. Rev. Personality Soc. Psychol. 2, 141–165 (1981).
  6. Goldberg, L. R. A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality Psychol. Eur. 7, 7–28 (1999).
  7. Karwowski, M., Lebuda, I. & Wiśniewska, E. Measuring creative self-efficacy and creative personal identity. Int. J. Creat. Probl. Solving 8, 45–57 (2018).
  8. John, O. P. & Srivastava, S. in The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives: Handbook of Personality: Theory and Research Vol. 2 (eds Pervin, L. A. & John, O. P.) 102–138 (Guilford, 1999).

Cited by

Still being checked.

Related

Authors
Serapio-García, G., Safdari, M., Crepy, C., Sun, L., Fitz, S., Romero, P., Abdulhai, M., Faust, A., & Matarić, M.
Affiliations
  • Google DeepMind (United Kingdom) (GB)
  • University of Cambridge (GB)
  • University of Chicago (US)
  • University of California, Berkeley (US)
  • University of Southern California (US)
Corresponding author
Safdari, Mustafa
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2025-12-18
Volume
7
Issue
12
Pages
1954-1968
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-025-01115-6
PMC
PMC12719228
OpenAlex
W4417454579
Semantic Scholar
acbd0e393d1b6dae53ed9b009b06b6b9aa3f9c3f
Cited by
30
References
60
Citations by year
2026: 29 · 2025: 1
FWCI
31.37
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • Apollo (University of Cambridge) (submittedVersion, cc-by-nc-nd)
Funders
  • Bill and Melinda Gates Foundation (OPP1144)
  • Luning Sun gratefully acknowledges financial support from Invesco through their philanthropic donation to Cambridge Judge Business School.
  • Stephen Fitz gratefully acknowledges support from the University of Chicago Research Computing Center, which provided compute credits essential to this research.
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
large language models, personality traits, psychometric assessment, personality assessment, prompting, instruction-tuned models, reliability and validity, chatbots, responsible AI, ethical implications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-28.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-28.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review