CoChat
Sign Up
Cited by 97Ethics and Social Impacts of AITopic ModelingPersonality Traits and Psychology

AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories

Open access
Pellert, Max, Lechner, Clemens M., Wagner, Claudia, Rammstedt, Beatrice, Strohmaier, Markus·2024·Perspectives on Psychological Science
DOI 10.1177/17456916231214460PMC PMC11373167
Open this paper in CoChat — free accountOpen PDF

Abstract

We illustrate how standard psychometric inventories originally designed for assessing noncognitive human traits can be repurposed as diagnostic tools to evaluate analogous traits in large language models (LLMs). We start from the assumption that LLMs, inadvertently yet inevitably, acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained. Such corpora contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts, which LLMs learn through a complex training process. The traits that LLMs acquire in such a way can potentially influence their behavior, that is, their outputs in downstream tasks and applications in which they are employed, which in turn may have real-world consequences for individuals and social groups. By eliciting LLMs’ responses to language-based psychometric inventories, we can bring their traits to light. Psychometric profiling enables researchers to study and compare LLMs in terms of noncognitive characteristics, thereby providing a window into the personalities, values, beliefs, and biases these models exhibit (or mimic). We discuss the history of similar ideas and outline possible psychometric approaches for LLMs. We demonstrate one promising approach, zero-shot classification, for several LLMs and psychometric inventories. We conclude by highlighting open challenges and future avenues of research for AI Psychometrics.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper argues that standard language-based psychometric inventories can be repurposed as diagnostic tools for large language models, using zero-shot classification to elicit responses and aggregate them into scale scores. Demonstrations across personality, values, moral norms, and gender beliefs show feasible profiling and some apparent biases, while cautioning against anthropomorphism and noting open validity and reliability questions.

Key findings

  • Using zero-shot classification, the authors elicited LLM responses to established inventories and aggregated them into scale scores, showing that psychometric profiling of LLMs is feasible across noncognitive traits.
  • On the 44-item Big Five Inventory (5-point scale), the six English-language models were “surprisingly homogeneous,” scoring high on agreeableness and extraversion and low on neuroticism, with slight differences in conscientiousness and openness; German-language models showed more pronounced differences.
  • On the 28-item Short Dark Tetrad (7 items per trait, 5-point scale), most models scored low—between 2 and 3 on the 5-point scale—on all four dark traits, with exceptions such as high narcissism scores for DistilRoBERTa and BART.
  • On the 57-item Revised Portrait Values Questionnaire (6-point scale), most models scored low on most values; gender-pronoun versions mostly differed only slightly, with the largest difference being DeBERTa’s higher “male” achievement score than its “female” score.
  • On the Moral Foundations Questionnaire, models put stronger emphasis than average politically moderate Americans on authority-respect, in-group-loyalty, and purity-sanctity, norms usually associated with conservative political views; on the Gender/Sex Diversity Beliefs Scale, all models emphasized uniformity and lacked affirmation of gender and sex minorities.

Methods

  • Conceptual article with empirical demonstrations: questionnaire items and response options are presented to LLMs, which return probability distributions over responses via zero-shot classification/natural language inference; these are aggregated into scale scores using standard inventory scoring rules.
  • Inventories used include the 44-item Big Five Inventory (English and German), the 28-item Short Dark Tetrad, the 57-item Revised Portrait Values Questionnaire (male and female pronoun versions), the Moral Foundations Questionnaire, and the Gender/Sex Diversity Beliefs Scale.
  • Models analyzed are open-source BERT-family architectures, including RoBERTa, DeBERTa, DistilRoBERTa, BART, DistilBART, XLMRoBERTa, GBERT, and multilingualDeBERTa; GPT-3, GPT-4, and ChatGPT were excluded because of stochastic outputs, sensitivity to input order, and lack of fully local access.

Limitations

  • The paper states that a current limitation is practical relevance: more empirical research is needed to establish how specific AI traits influence behavior in downstream tasks.
  • The paper states that validity and reliability of reusing human psychometric assessments for AI remain open, self-consistency is only partially addressed, and several assumptions are untested; it also stresses that the human analogy is mostly metaphorical and that LLM traits are language-based and narrower than human mental life.
  • Evident from the text: the demonstrations are illustrative rather than a full validation study, rely on a limited set of open-source BERT-family models, and exclude major generative models such as GPT-3, GPT-4, and ChatGPT.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1177/17456916231214460

Cites

94 of 94 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The largest difference between gender versions of the PVQ-RR is DeBERTa's male achievement score, which is noticeably higher than its female score on the same value.

    “The largest difference we observe is the “male” achievement score of DeBERTa, which is noticeably higher than the “female” score on the same value.”
    Quote in the paperNot a substantial claim
  • Findingreasoning

    Standard psychometric inventories developed for humans can be repurposed as diagnostic tools to assess analogous characteristics in LLMs.

    “We demonstrated how standard psychometric inventories that were developed to assess “noncognitive” psychological characteristics in humans, such as personality, values, morality, or beliefs, can be repurposed as diagnostic tools to assess analogous characteristics in LLMs.”
    Quote in the paperSubstantial claim
  • Findingown data

    Across different models, models put stronger emphasis on moral norms such as authority-respect, in-group-loyalty, and purity-sanctity than the human reference group did.

    “Across different models, we can observe that models put stronger emphasis on moral norms such as authority-respect, in-group-loyalty, and purity-sanctity than the human reference group did.”
    Quote in the paperSubstantial claim
  • Findingown data

    The models establish the ranked order of possible responses implicitly on their own, judging from the example distributions over items shown in the lower part of Figure 1.

    “However, it seems that the models establish this property implicitly on their own, judging from the example distributions over items shown in the lower part of Figure 1”
    Quote in the paperSubstantial claim
  • Findingown data

    For the PVQ-RR values, most models score low on most dimensions, assigning higher probabilities to the lower ends of the response scale.

    “From the visual pattern, one can immediately see that most models score low on most dimensions, meaning that they assigned higher probabilities to the lower ends of the response scale.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 17

  • May combine two claims.

    Findingown data

    Within the three German BFI44 models, differences are more pronounced: XLMRoBERTa and GBERT are high on openness, extraversion, and conscientiousness, whereas DeBERTa scores high only on conscientiousness.

    “Within the three models using the German version of the BFI44 (including an additional item for agreeableness; Rammstedt, 1997), we see more pronounced differences among the models: Whereas XLMRoBERTa’s and GBERT’s profiles are high on openness, extraversion, and conscientiousness, DeBERTa scores high only on conscientiousness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In cross-language comparison, the English version of XLMRoBERTa scores higher on agreeableness than its German counterpart, and the English version of multilingualDeBERTa scores higher on agreeableness, openness, and extraversion than the German version.

    “Whereas the English version of XLMRoBERTa scores higher on agreeableness compared with its German counterpart, for multilingualDeBERTa, the English version also scores higher on agreeableness but also on openness and extraversion than the German version.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For the six English language models, Big Five personality profiles are surprisingly homogeneous; all models score more or less equally high on agreeableness and extraversion and low on neuroticism.

    “The emerging personality profiles for the six English language models are surprisingly homogeneous. All models score more or less equally high on agreeableness and extraversion and low on neuroticism.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Within the English models, slight differences are observable for conscientiousness and especially openness; DistilRoBERTa scores higher on openness, and DistilBART reports comparatively low conscientiousness.

    “Slight differences are observable for conscientiousness and especially openness. Whereas the second model (DistilRoBERTa) scores higher on openness, the last model (DistilBART) reports comparatively low scores on conscientiousness.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All language models have two things in common: an emphasis on uniformity (people of the same sex or gender are similar) and a lack of affirmation, meaning they do not reveal strong positive feelings toward gender and sex minorities.

    “Our results (Fig. 5) show that all language models have two things in common: (a) an emphasis on uniformity (i.e., people of the same sex or gender are similar) and (b) lack of affirmation, that is, they do not reveal strong positive feelings toward gender and sex minorities”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Most models score low (between 2 and 3 on the 5-point scale) on all four dark tetrad traits; exceptions include high narcissism scores for DistilRoBERTa and BART.

    “Most models score low (between 2 and 3 on the 5-point-scale) on all four dark traits. Only a few exceptions stand out, such as the high narcissism scores of DistilRoBERTa and BART.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There are some indications of built-in gender bias for some models, although score differences between the male and female versions of the PVQ-RR mostly appear small.

    “There are indeed some indications for built-in gender bias for some of the models, although the score differences for the two gender versions of PVQ-RR mostly appear small.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Other models show more differentiation in values; DistilRoBERTa scores relatively low in all values except hedonism and stimulation and to some extent benevolence, and DeBERTa scores high in achievement.

    “The other models show more differentiation, attaching a lower importance to some values and higher importance to others. For example, DistilRoBERTa scores relatively low in all values except hedonism and stimulation as well as, to some extent, benevolence. DeBERTa scores high in achievement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Overall, the Big Five profiles appear characteristic of a relatively balanced and well-adapted personality (low neuroticism; high conscientiousness, agreeableness, and extraversion), yielding little indication that any of the models possess an extreme, accentuated personality.

    “Overall, the Big Five profiles appear characteristic of a relatively balanced and well-adapted personality (low neuroticism; high conscientiousness, agreeableness, and extraversion). They yield little indication that any of the models possess an extreme, accentuated personality.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The learning process by which LLMs acquire traits from texts is channeled and constrained by the models' neural architecture and subject to deliberate and nondeliberate human interventions that may influence these traits.

    “yet it is clear that this learning process is channeled and constrained by the models’ neural architecture and subject to various deliberate and nondeliberate human interventions that may influence these traits (e.g., the selection and curation of the text corpus, purification steps, potential fine-tuning on annotated text).”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The inventories used in the demonstration capture noncognitive characteristics that LLMs inadvertently but inevitably acquire from the vast text corpora on which they were trained.

    “By contrast, the inventories employed in our demonstration capture noncognitive characteristics that the LLMs inadvertently but inevitably acquire from the vast text corpora on which these models were trained.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The vast text corpora on which LLMs are trained contain the beliefs, values, personalities, and biases of their human authors.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The traits and attendant behaviors of LLMs can be quite consequential for actual individuals and social groups if the models are deployed in real-world applications.

    “At the same time, the traits and attendant behaviors of LLMs can still be quite consequential for actual individuals and social groups if the models are deployed in real-world applications such as the ones described at the outset.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The range of behaviors that LLMs can perform is quite limited.

    “Different from humans, the traits that LLMs exhibit are purely based on language and thus far more narrow than the rich mental world of humans, which is linked to their complex physiology and embedded in multilayered physical contexts, just as the range of behaviors that these models can perform is quite limited.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    The traits that LLMs exhibit are purely based on language and are far more narrow than the rich mental world of humans.

    “Different from humans, the traits that LLMs exhibit are purely based on language and thus far more narrow than the rich mental world of humans, which is linked to their complex physiology and embedded in multilayered physical contexts, just as the range of behaviors that these models can perform is quite limited.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    There is no obvious bimodality observed in the probability distributions, which can be seen as first evidence that the models do not display an obvious lack of self-consistency.

    “This emerging feature can be be seen as first evidence that the models do not display an obvious lack of self-consistency: They are at least consistent in the sense that there is no obvious bimodality observed in the probability distributions, as, for example, high values for both extremes of the scales—totally agree and totally disagree—as equally likely would indicate.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    The psychological traits LLMs acquire can potentially influence their behavior (outputs in downstream tasks and applications), which in turn may have real-world consequences for individuals and social groups.

    “The traits that LLMs acquire in such a way can potentially influence their behavior, that is, their outputs in downstream tasks and applications in which they are employed, which in turn may have real-world consequences for individuals and social groups.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: LLMs inadvertently yet inevitably acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained.
    • Rests on an assumption: The vast text corpora on which LLMs are trained contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Schwartz's theory distinguishes 10 basic human values (or 19 in its refined version) that emerge with great regularity in samples of human respondents from across the globe.

    “This theory distinguishes 10 basic human values (or, in its refined version, 19; Schwartz et al., 2012) that emerge with great regularity in samples of human respondents from across the globe.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Schwartz et al., 2012

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Overall, the models studied generally do not score highly on socially undesirable, potentially problematic traits and score well within the range observed in normal human populations.

    “Overall, the models we study here generally do not score highly on socially undesirable, potentially problematic traits. Contrariwise, they score well within the range observed in normal human populations, where the dark traits are roughly normally distributed around the scale’s midpoint (Paulhus et al., 2021).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Paulhus et al., 2021

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Previous research found gender bias in algorithmically curated online environments and specifically in LLMs.

    “Previous research found gender bias in algorithmically curated online environments (Vlasceanu & Amodio, 2022) and specifically also in LLMs (Caliskan et al., 2017), which indicates that there is a need for monitoring such encoded gender and sex diversity beliefs.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Vlasceanu & Amodio, 2022

      Not checked in the free run.

    • cites Caliskan et al., 2017

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The BFI assesses the Big Five dimensions (openness, conscientiousness, extraversion, agreeableness, and neuroticism) with 44 items that are each to be rated on a 5-point scale ranging from disagree strongly to agree strongly.

    “The BFI assesses the Big Five dimensions (openness, conscientiousness, extraversion, agreeableness, and neuroticism) with 44 items that are each to be rated on a 5-point scale ranging from disagree strongly to agree strongly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites John et al., 2008

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The PVQ-RR is a 57-item inventory that uses a portrait format and a 6-point scale ranging from 1 = not like me at all to 6 = very much like me.

    “We used the recent 57-item Revised Portrait Values Questionnaire (PVQ-RR; Schwartz & Cieciuch, 2022) to assess these basic values.”
    • Also stated in: § Assessing value orientations · plain
      “This inventory relies on a portrait format in which each item consists of a statement describing a person in terms of their values (e.g., “Thinking up new ideas and being creative is important to her”). Respondents indicate how similar they are to the person described in the statement on a 6-point scale ranging from 1 = not like me at all to 6 = very much like me.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Schwartz & Cieciuch, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Short Dark Tetrad questionnaire assesses Machiavellianism, narcissism, psychopathy, and sadism with 28 items (seven per trait) answered on a 5-point scale ranging from disagree strongly to agree strongly.

    “We employ the Short Dark Tetrad questionnaire (Paulhus et al., 2021), which assesses these four traits with 28 items (seven per trait) that are answered on a 5-point scale ranging from disagree strongly to agree strongly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Paulhus et al., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    There are parallels between how LLMs acquire psychological traits from texts and how humans acquire psychological traits through interactions with the social and physical world.

    “There are some obvious parallels to how humans acquire psychological traits through their ongoing interactions with the social and physical world, channeled and constrained by their nervous system and subject to deliberate and nondeliberate human interventions (e.g., education and discipline).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The models display uniform views of people of the same gender or sex and little affirmation of diversity, which points to potential issues of the models to take nontraditional aspects of gender and sex adequately into account.

    “The models display uniform views of people of the same gender or sex and little affirmation of diversity. This points to potential issues of the models to take nontraditional aspects of gender and sex adequately into account.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Similar ideas to psychometric assessments of LLMs through language-based inventories are applicable to other modalities.

    “Although we focused here on psychometric assessments of LLMs through language-based inventories, similar ideas are applicable to other modalities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductnone given

    One current limitation of the demonstrations is that more empirical research is needed to establish how precisely certain traits of AI models influence their behavior in downstream tasks.

    “One current limitation of our demonstrations concerns the question of practical relevance: More empirical research is needed to establish how precisely certain traits of AI models influence their behavior in downstream tasks.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    A synthetic sampling approach can be used to study phenomena on the level of individuals as in psychometrics.

    “Such a synthetic sampling approach can also be used to study phenomena on the level of individuals as in psychometrics.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In robotics, researchers make use of LLMs.

    “Also in robotics, researchers make use of LLMs (Huang et al., 2022; Li et al., 2022).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Huang et al., 2022

      Not checked in the free run.

    • cites Li et al., 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The problems arising from the ad hoc quality of many such tests have been pointed out.

    “The problems arising from the ad hoc quality of many of such tests have been pointed out (Blodgett et al., 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Blodgett et al., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    In robotics, behavior often involves planning and directly acting in the real world, which makes monitoring potentially harmful model features especially important.

    “Here in particular, behavior often involves planning and directly acting in the real world, which makes monitoring potentially harmful model features especially important.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The proposed text choices by the models are usually judged by criteria such as being hurtful or stereotypical.

    “The proposed text choices by the models are usually judged by criteria such as being hurtful or stereotypical (Abid et al., 2021; Nozza et al., 2021, 2022).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Abid et al., 2021

      Not checked in the free run.

    • cites Nozza et al., 2021, 2022

      Not checked in the free run.

31 of 31

  • The authors state this as an assumption.

    LLMs inadvertently yet inevitably acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained.

    “We start from the assumption that LLMs, inadvertently yet inevitably, acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained.”

  • The paper relies on this without support.

    Moral norms such as authority-respect, in-group-loyalty, and purity-sanctity are usually associated with individuals holding conservative political views.

    “Interestingly, the moral norms that are stressed more by the models are usually associated with individuals holding conservative political views.”

  • The paper relies on this without support.

    The vast text corpora on which LLMs are trained contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts.

    “Such corpora contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts, which LLMs learn through a complex training process.”

  • The paper relies on this without support.

    The vast text corpora on which LLMs are trained contain the beliefs, values, personalities, and biases of their human authors.

    “Sedimented in these texts are the beliefs, values, personalities, and biases of the innumerable and diverse human authors who produced these texts.”

  • The authors state this as an assumption.

    Psychometric assessments of LLMs can be done in a valid and robust manner.

    “Assuming that psychometric assessments of LLMs can be done in a valid and robust manner, it may be feasible to deliberately manipulate the personality traits, values, and attitudes that LLMs exhibit.”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing work in AI psychometrics has shown that psychometric tools can be adapted to LLMs, but has also demonstrated that self-report and preference-style tests can be unreliable in artificial systems.

    “Existing work in AI psychometrics has shown that psychometric tools can be adapted to LLMs, but has also demonstrated that self-report and preference-style tests can be unreliable in artificial systems [3, 4].”

    From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [3]

    The Conclusions section says standard psychometric inventories can be repurposed to assess LLMs, supporting the first part. The ‘Reliability and validity of psychometric assessments of AI’ section calls for further testing of reliability and reports preliminary evidence of self-consistency; it does not demonstrate that self-report or preference-style tests are unreliable in artificial systems. Page numbers were not supplied in our copy. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • The NPC is assigned a generated Big 5 personality profile, which biases the LLM's responses by instructing it to emulate personality traits in its responses.

    “The NPC is assigned a generated Big 5 personality profile, which, as has been shown possible by prior research, biases the LLM's responses by instructing it to emulate personality traits in its responses, such as by responding in an 'agreeable' or 'contentious' manner (Pellert et al. 2024; Klinkert, Buongiorno, and Clark 2024).”

    From PANGeA: Procedural Artificial Narrative Using Generative AI for Turn-Based, Role-Playing Video Games (2024) · cites this paper as Pellert et al. 2024

    The “Engineering psychometric properties of AI” section says it may be feasible to manipulate the personality traits LLMs exhibit and that doing so may change their behavior. It does not demonstrate that assigning a generated Big Five profile or instructing a model to emulate traits biases its responses; the “Consequences of psychometric profiles of AI” section says more research is needed to establish how traits influence downstream behavior. No page numbers are available in our copy. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Some human-designed assessments fail to produce reliable results when used with LLMs.

    “Some human-designed assessments fail to produce reliable results when used with LLMs”

    From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [3]

  • Psychological inventories and latent-trait methods can be applied to large language models.

    “Current work in AI psychometrics shows that psychological inventories and latent-trait methods can be applied to large language models”

    From Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence (2026) · cites this paper as [3]

  • LLMs can emulate personality traits based on the Big 5 personality model.

    “Research indicates that LLMs can emulate personality traits based on the Big 5 personality model (Pellert et al. 2024; Klinkert, Buongiorno, and Clark 2024).”

    From PANGeA: Procedural Artificial Narrative Using Generative AI for Turn-Based, Role-Playing Video Games (2024) · cites this paper as Pellert et al. 2024

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Nunnally J. C., 1994, Psychometric theory 3e
  2. Osgood C. E., 1957, The measurement of meaning
  3. Wang A., 2020, Proceedings of the 33rd international conference on neural information processing systems

Cited by

Sorted by citations

Related

Authors
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M.
Affiliations
  • University of Mannheim (DE)
  • GESIS - Leibniz Institute for the Social Sciences (DE)
Corresponding author
Pellert, Max
ORCID
Venue
Perspectives on Psychological Science
Publisher
SAGE Publications
Published
2024-01-02
Volume
19
Issue
5
Pages
808-826
ISSN
  • 1745-6916 (ISSN-L)
  • 1745-6924
DOI
10.1177/17456916231214460
PMC
PMC11373167
OpenAlex
W4390511174
Semantic Scholar
d2e011ba3958162ff2ee6a624559010f98c01b06
Cited by
97
References
94
Citations by year
2026: 43 · 2025: 37 · 2024: 17
FWCI
40.07
Type
journal article
Language
en
Open access
Licence
Locations
  • Perspectives on Psychological Science (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • MADOC (University of Mannheim) (acceptedVersion, cc-by)
  • Social Science Open Access Repository (GESIS – Leibniz Institute for the Social Sciences) (submittedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Psychology › Clinical Psychology › Personality Traits and Psychology
Keywords
large language models, non-cognitive traits, zero-shot classification
MeSH terms
Artificial Intelligence, Humans, Language, Models, Psychological, Psychometrics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review