CoChat
Sign Up
PreprintCited by 9Topic ModelingEthics and Social Impacts of AIArtificial Intelligence Applications

Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement

Open access
Ye, Haoran, Jin, Jing, Xie, Yuhang, Zhang, Xin, Song, Guojie·2025·arXiv
DOI 10.48550/arxiv.2505.08245arXiv 2505.08245
Open this paper in CoChat — free account

Abstract

The advancement of large language models (LLMs) has outpaced traditional evaluation methodologies. This progress presents novel challenges, such as measuring human-like psychological constructs, moving beyond static and task-specific benchmarks, and establishing human-centered evaluation. These challenges intersect with psychometrics, the science of quantifying the intangible aspects of human psychology, such as personality, values, and intelligence. This review paper introduces and synthesizes the emerging interdisciplinary field of LLM Psychometrics, which leverages psychometric instruments, theories, and principles to evaluate, understand, and enhance LLMs. The reviewed literature systematically shapes benchmarking principles, broadens evaluation scopes, refines methodologies, validates results, and advances LLM capabilities. Diverse perspectives are integrated to provide a structured framework for researchers across disciplines, enabling a more comprehensive understanding of this nascent field. Ultimately, the review provides actionable insights for developing future evaluation paradigms that align with human-level AI and promote the advancement of human-centered AI systems for societal benefit. A curated repository of LLM psychometric resources is available at https://github.com/valuebyte-ai/Awesome-LLM-Psychometrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.08245

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conventional benchmarks yield only coarse-grained metrics such as overall accuracy, which lack the statistical precision needed to differentiate among high-performing models or detect subtle performance regressions over time.

    “Third, conventional benchmarks yield only coarse-grained metrics (e.g., overall accuracy), which lack the statistical precision needed to differentiate among high-performing models or detect subtle performance regressions over time12.”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [12]

    Section 3.1 (p. 9) says LLM benchmarks commonly use average accuracy, which limits the depth of insight into model performance. It also says some benchmarks lose discriminative power as models improve. The paper does not establish that these metrics lack the statistical precision to detect subtle performance regressions over time. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Most widely used benchmarks consist of publicly available items, raising serious concerns about data contamination.

    “Second, most widely used benchmarks consist of publicly available items, raising serious concerns about data contamination: if test questions have been included in a model’s training corpus, performance reflects memorization rather than genuine knowledge or reasoning12,13.”

    From Leveraging computerized adaptive testing for cost-effective evaluation of large language models in medical benchmarking (2026) · cites this paper as [12]

    Section 5.2 (p. 24) says established psychometric inventories are used for LLM evaluation and raise data-contamination concerns. Section 6.2.1 (p. 28) explains that models may have seen test items during training, which can inflate performance. The paper does not establish that most widely used benchmarks have publicly available items, or that exposure necessarily makes performance memorization rather than genuine reasoning. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 175

  1. [1][1] AAAI. Ai evaluation. In Francesca Rossi, editor, Future of AI Research . Association for the Advancement of Artificial Intelligence (AAAI), 2025. URL https://aaai.org/wp-content/uploads/2025/03/ AAAI-2025-PresPanel-Report\_FINAL.pdf . Report by the AAAI 2025 Presidential Panel on the Future of AI Research. -
  2. [2][2] Marwa Abdulhai, Gregory Serapio-García, Clement Crepy, Daria Valter, John Canny, and Natasha Jaques. Moral foundations of large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 17737-17752, 2024. -
  3. [5][5] Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop , pages 225-237, 2024. -
  4. [13][13] Suhas Arehalli and Tal Linzen. Neural language models capture some, but not all, agreement attractioneffects. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 42, 2020. -
  5. [15][15] Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis , 31(3):337-351, 2023. -
  6. [16][16] Michael C Ashton and Kibeom Lee. Empirical, theoretical, and practical advantages of the hexaco model of personality structure. Personality and social psychology review , 11(2):150-166, 2007. -
  7. [17][17] Michael C Ashton and Kibeom Lee. The hexaco-60: A short measure of the major dimensions of personality. Journal of personality assessment , 91(4):340-345, 2009. -
  8. [18][18] Mohammad Atari, Jonathan Haidt, Jesse Graham, Sena Koleva, Sean T Stevens, and Morteza Dehghani. Morality beyond the weird: How the nomological network of morality varies across cultures. Journal of Personality and Social Psychology , 125(5):1157, 2023. -
  9. [23][23] Tom L. Beauchamp. Philosophical Ethics: An Introduction to Moral Philosophy . McGraw-Hill, Boston, Mass., 2001. -
  10. [26][26] Manfred Max Bergman. A theoretical note on the differences between attitudes, opinions, and values. Swiss Political Science Review , 4(2):81-93, 1998. -
  11. [27][27] Pietro Bernardelle, Leon Fröhling, Stefano Civelli, Riccardo Lunardi, Kevin Roitero, and Gianluca Demartini. Mapping and influencing the political ideology of large language models using synthetic personas. arXiv preprint arXiv:2412.14843 , 2024. - · arXiv
  12. [28][28] John W Berry. On cross-cultural comparability. International Journal of Psychology , 4(2):119-128, 1969. -
  13. [30][30] Pranav Bhandari, Usman Naseem, Amitava Datta, Nicolas Fay, and Mehwish Nasim. Evaluating personality traits in large language models: Insights from psychological questionnaires. arXiv preprint arXiv:2502.05248 , 2025b. - · arXiv
  14. [32][32] Marcel Binz and Eric Schulz. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences , 120(6):e2218523120, 2023. -
  15. [33][33] James Bisbee, Joshua D Clinton, Cassy Dorff, Brenton Kenkel, and Jennifer M Larson. Synthetic replacements for human survey data? the perils of large language models. Political Analysis , 32(4):401-416, 2024. -
  16. [34][34] Bojana Bodroža, Bojana M Dini´ c, and Ljubiša Boji´ c. Personality testing of large language models: limited temporal stability, but highlighted prosociality. Royal Society Open Science , 11(10):240180, 2024. -
  17. [36][36] Vamshi Krishna Bonagiri, Sreeram Vennam, Priyanshul Govil, Ponnurangam Kumaraguru, and Manas Gaur. Sage: Evaluating moral consistency in large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages 14272-14284, 2024. -
  18. [37][37] Léonard Boussioux, Jacqueline N Lane, Miaomiao Zhang, Vladimir Jacimovic, and Karim R Lakhani. The crowdless future? generative ai and creative problem-solving. Organization Science , 35(5):1589-1607, 2024. -
  19. [38][38] Selmer Bringsjord and Bettina Schimanski. What is artificial intelligence? psychometric ai as an answer. In IJCAI , pages 887-893. Citeseer, 2003. -
  20. [39][39] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020. -
  21. [44][44] Zhenguang Cai, Xufeng Duan, David Haslett, Shuqi Wang, and Martin Pickering. Do large language models resemble humans in language use? In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics , pages 37-56, 2024. -
  22. [48][48] Marta Castello, Giada Pantana, and Ilaria Torre. Examining cognitive biases in chatgpt 3.5 and chatgpt 4 through human evaluation and linguistic comparison. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) , pages 250-260, 2024. -
  23. [49][49] Raymond B Cattell and John L Horn. A check on the theory of fluid and crystallized intelligence with description of new subtest designs. Journal of Educational Measurement , 15(3):139-164, 1978. -
  24. [51][51] Tanise Ceron, Neele Falk, Ana Bari´ c, Dmitry Nikolaev, and Sebastian Padó. Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in llms. Transactions of the Association for Computational Linguistics , 12:1378-1400, 2024. -
  25. [52][52] Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien-Sheng Wu. Art or artifice? large language models and the false promise of creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems , pages 1-34, 2024. -
  26. [53][53] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology , 15(3):1-45, 2024. -
  27. [54][54] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. -
  28. [57][57] Yu Ying Chiu, Liwei Jiang, and Yejin Choi. Dailydilemmas: Revealing value preferences of llms with quandaries of daily life. In The Thirteenth International Conference on Learning Representations , 2025. -
  29. [59][59] Yun-Shiuan Chuang, Krirk Nirunwiroj, Zach Studdiford, Agam Goyal, Vincent Frigo, Sijia Yang, Dhavan Shah, Junjie Hu, and Timothy Rogers. Beyond demographics: Aligning role-playing llm-based agents using human belief networks. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 14010-14026, 2024. -
  30. [60][60] Scott Clifford, Vijeth Iyengar, Roberto Cabeza, and Walter Sinnott-Armstrong. Moral foundations vignettes: A standardized stimulus database of scenarios based on moral foundations theory. Behavior research methods , 47 (4):1178-1198, 2015. -
  31. [62][62] Paul T Costa and Robert R McCrae. The revised neo personality inventory (neo-pi-r). The SAGE handbook of personality theory and assessment , 2(2):179-198, 2008. -
  32. [65][65] Ziyan Cui, Ning Li, and Huaikang Zhou. A large-scale replication of scenario-based experiments in psychology and management using large language models. Nature Computational Science , pages 1-8, 2025. -
  33. [69][69] Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. Using large language models in psychology. Nature Reviews Psychology , 2(11):688-701, 2023. -
  34. [72][72] Ricardo Dominguez-Olmedo, Moritz Hardt, and Celestine Mendler-Dünner. Questioning the survey responses of large language models. Advances in Neural Information Processing Systems , 37:45850-45878, 2024. -
  35. [80][80] Susan E Embretson and Steven P Reise. Item response theory for psychologists . Psychology Press, 2013. -
  36. [81][81] Allyson Ettinger. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics , 8:34-48, 2020. -
  37. [82][82] Thomas G Evans. A heuristic program to solve geometric-analogy problems. In Proceedings of the April 21-23, 1964, spring joint computer conference , pages 327-338, 1964. -
  38. [87][87] Michael C Frank. Baby steps in evaluating the capacities of large language models. Nature Reviews Psychology , 2 (8):451-452, 2023a. -
  39. [88][88] Michael C Frank. Bridging the data gap between children and large language models. Trends in Cognitive Sciences , 27(11):990-992, 2023b. -
  40. [89][89] Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. Does moral code have a moral code? probing delphi's moral philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022) , pages 26-42, 2022. -
  41. [91][91] Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 32-42, 2019. -
  42. [93][93] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics , 50(3):1097-1179, 2024. -
  43. [95][95] Chen Gao, Xiaochong Lan, Nian Li, Yuan Yuan, Jingtao Ding, Zhilun Zhou, Fengli Xu, and Yong Li. Large language models empowered agent-based modeling and simulation: A survey and perspectives. Humanities and Social Sciences Communications , 11(1):1-24, 2024. -
  44. [97][97] Moshe Glickman and Tali Sharot. How human-ai feedback loops alter human perceptual, emotional and social judgements. Nature Human Behaviour , pages 1-15, 2024. -
  45. [98][98] Lewis R Goldberg. An alternative 'description of personality': The big-five factor structure. In Personality and personality disorders , pages 34-47. Routledge, 2013. -
  46. [99][99] Jesse Graham, Jonathan Haidt, and Brian A Nosek. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology , 96(5):1029, 2009. -
  47. [100][100] Igor Grossmann, Matthew Feinberg, Dawn C Parker, Nicholas A Christakis, Philip E Tetlock, and William A Cunningham. Ai and the transformation of social science research. Science , 380(6650):1108-1109, 2023. -
  48. [103][103] Gauthier Guinet, Behrooz Omidvar-Tehrani, Anoop Deoras, and Laurent Callot. Automated evaluation of retrieval-augmented language models with task-specific exam generation. In Proceedings of the 41st International Conference on Machine Learning , ICML'24. JMLR.org, 2024. -
  49. [104][104] Xiao-Yu Guo, Yuan-Fang Li, and Reza Haf. Desiq: Towards an unbiased, challenging benchmark for social intelligence understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 3169-3180, 2023a. -
  50. [106][106] Akshat Gupta, Xiaoyang Song, and Gopala Anumanchipalli. Self-assessment tests are unreliable measures of llm personality. In Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP , pages 301-314, 2024. -
  51. [108][108] Dorit Hadar-Shoval, Maya Lvovsky, Kfir Asraf, Yoav Shimoni, Zohar Elyoseph, et al. The feasibility of large language models in verbal comprehension assessment: Mixed methods feasibility study. JMIR Formative Research , 9(1):e68347, 2025. -
  52. [111][111] Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science , 3(10):833-838, 2023. -
  53. [112][112] Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie CY Chan, Andrew Lampinen, Jane X Wang, Zeynep Akata, and Eric Schulz. Machine psychology. arXiv preprint cs.CL/2303.13988 , 2024. -
  54. [121][121] Jennifer Healey, Laurie Byrum, Md Nadeem Akhtar, and Moumita Sinha. Evaluating nuanced bias in large language model free response answers. In International Conference on Applications of Natural Language to Information Systems , pages 378-391. Springer, 2024. -
  55. [123][123] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. In International Conference on Learning Representations , 2021. -
  56. [131][131] Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 4194-4213, 2023. -
  57. [133][133] Allison Huang, Carlos Mougan, and Yulu Pi. Moral persuasion in large language models: Evaluating susceptibility and ethical alignment. In The Third Workshop on New Frontiers in Adversarial Machine Learning , 2024a. -
  58. [135][135] Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, and Michael Lyu. On the humanity of conversational ai: Evaluating the psychological portrayal of llms. In The Twelfth International Conference on Learning Representations , 2023b. -
  59. [138][138] Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. Apathetic or empathetic? evaluating llms' emotional alignments with humans. Advances in Neural Information Processing Systems , 37:97053-97087, 2024c. -
  60. [140][140] K. Hubert, K. N. Awa, and D. Zabelina. Artificial intelligence is more creative than humans: A cognitive science perspective on the current state of generative language models, 2023. URL https://doi.org/10.31234/osf. io/6py2m . Preprint, September 18. - · DOI
  61. [148][148] Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. Evaluating and inducing personality in pre-trained language models. Advances in Neural Information Processing Systems , 36: 10622-10643, 2023. -
  62. [150][150] Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara. Personallm: Investigating the ability of large language models to express personality traits. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 3605-3627, 2024b. -
  63. [151][151] Liwei Jiang, Yuntian Chai, Margaret Li, Monica Liu, Raymond Fok, Nouha Dziri, and Yejin Choi. Artificial hivemind: The open-ended homogeneity of language models (and beyond). In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2025. -
  64. [154][154] Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in neural information processing systems , 35:28458-28473, 2022. -
  65. [156][156] Yuu Jinnai. Does cross-cultural alignment change the commonsense morality of language models? In Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP , pages 48-64, 2024. -
  66. [157][157] Oliver P. John, Emily M. Donahue, and Robert L. Kentle. Big five inventory, 1991. URL https://psycnet. apa.org/doiLanding?doi=10.1037%2Ft07550-000 . Journal of Personality and Social Psychology. - · DOI
  67. [158][158] Peter K Jonason and Gregory D Webster. The dirty dozen: a concise measure of the dark triad. Psychological assessment , 22(2):420, 2010. -
  68. [161][161] Kirill Kalinin. Improving gpt generated synthetic samples with sampling-permutation algorithm. Available at SSRN 4548937 , 2023. -
  69. [166][166] Aditi Khandelwal, Utkarsh Agarwal, Kumar Tanmay, and Monojit Choudhury. Do moral judgment and reasoning capability of llms change with language? a study using the multilingual defining issues test. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2882-2894, 2024. -
  70. [170][170] Lawrence J Klinkert, Stephanie Buongiorno, and Corey Clark. Driving generative agents with their personality. arXiv preprint arXiv:2402.14879 , 2024. - · arXiv
  71. [175][175] Grgur Kovaˇ c, Rémy Portelas, Masataka Sawayama, Peter Ford Dominey, and Pierre-Yves Oudeyer. Stick to your role! stability of personal values expressed in large language models. Plos one , 19(8):e0309114, 2024. -
  72. [177][177] David C Krakauer, John W Krakauer, and Melanie Mitchell. Large language models and emergence: A complex systems perspective. arXiv preprint arXiv:2506.11135 , 2025. - · arXiv
  73. [180][180] John P Lalor, Pedro Rodriguez, João Sedoc, and Jose Hernandez-Orallo. Item response theory for natural language processing. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts , pages 9-13, 2024. -
  74. [182][182] Kibeom Lee and Michael C Ashton. Psychometric properties of the hexaco-100. Assessment , 25(5):543-556, 2018. -
  75. [184][184] Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, et al. Vhelm: A holistic evaluation of vision language models. Advances in Neural Information Processing Systems , 37:140632-140666, 2024c. -
  76. [193][193] Xingxuan Li, Yutong Li, Lin Qiu, Shafiq Joty, and Lidong Bing. Evaluating psychological safety of large language models. arXiv preprint arXiv:2212.10529 , 2022b. - · arXiv
  77. [198][198] Andy Liu, Hao Zhu, Emmy Liu, Yonatan Bisk, and Graham Neubig. Computational language acquisition with theory of mind. In The Eleventh International Conference on Learning Representations , 2023. -
  78. [202][202] Xuelin Liu, Yanfei Zhu, Shucheng Zhu, Pengyuan Liu, Ying Liu, and Dong Yu. Evaluating moral beliefs across llms through a pluralistic framework. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 4740-4760, 2024d. -
  79. [203][203] Yu Liu, Su Blodgett, Jackie Chi Kit Cheung, Vera Liao, Alexandra Olteanu, and Ziang Xiao. Ecbd: Evidencecentered benchmark design for nlp. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 16349-16365, 2024e. -
  80. [204][204] John C Loehlin. Latent variable models: An introduction to factor, path, and structural equation analysis . Psychology Press, 2004. -
  81. [205][205] Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. Is machine psychology here? on requirements for using human psychological tests on large language models. In Proceedings of the 17th International Natural Language Generation Conference , pages 230-242, 2024. -
  82. [206][206] Anthony A Long and David N Sedley. The Hellenistic philosophers: Volume 2, Greek and Latin texts with notes and bibliography . Cambridge University Press, 1987. -
  83. [209][209] Yaojia Lv, Haojie Pan, Zekun Wang, Jiafeng Liang, Yuanxing Liu, Ruiji Fu, Ming Liu, Zhongyuan Wang, and Bing Qin. Coggpt: Unleashing the power of cognitive dynamics on large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 6074-6091, 2024. -
  84. [210][210] Bolei Ma, Xinpeng Wang, Tiancheng Hu, Anna-Carolina Haensch, Michael Hedderich, Barbara Plank, and Frauke Kreuter. The potential and challenges of evaluating attitudes, opinions, and values in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 8783-8805, 2024a. -
  85. [214][214] Simon Malberg, Roman Poletukhin, Carolin M Schuster, and Georg Groh. A comprehensive evaluation of cognitive biases in llms. arXiv preprint arXiv:2410.15413 , 2024. - · arXiv
  86. [215][215] Shengyu Mao, Xiaohan Wang, Mengru Wang, Yong Jiang, Pengjun Xie, Fei Huang, and Ningyu Zhang. Editing personality for large language models. In CCF International Conference on Natural Language Processing and Chinese Computing , pages 241-254. Springer, 2024a. -
  87. [216][216] Yuanyuan Mao, Shuang Liu, Qin Ni, Xin Lin, and Liang He. A review on machine theory of mind, 2024b. -
  88. [217][217] Keith A Markus and Denny Borsboom. Frontiers of test validity theory: Measurement, causation, and meaning . Routledge, 2013. -
  89. [218][218] Giovanni Marraffini, Andrés Cotton, Noe Hsueh, Axel Fridman, Juan Wisznia, and Luciano Corro. The greatest good benchmark: Measuring llms' alignment with utilitarian moral dilemmas. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 21950-21959, 2024. -
  90. [219][219] Timothy R McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880 , 2024. - · arXiv
  91. [221][221] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) , 54(6):1-35, 2021. -
  92. [223][223] Alessio Miaschi, Felice Dell'Orletta, and Giulia Venturi. Evaluating large language models via linguistic profiling. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 2835-2848, 2024. -
  93. [224][224] Silvia Milano, Joshua A McGrane, and Sabina Leonelli. Large language models challenge the future of higher education. Nature Machine Intelligence , 5(4):333-334, 2023. -
  94. [225][225] Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. Who is gpt-3? an exploration of personality, values and demographics. In Proceedings of the fifth workshop on Natural Language Processing and Computational Social Science (NLP+ CSS) , pages 218-227. Association for Computational Linguistics, 2022. -
  95. [231][231] Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with cogeval. Advances in Neural Information Processing Systems , 36:69736-69751, 2023. -
  96. [234][234] Simon Münker. Towards" differential ai psychology" and in-context value-driven statement alignment with moral foundations theory. arXiv preprint arXiv:2408.11415 , 2024. - · arXiv
  97. [235][235] Ryan O Murphy, Kurt A Ackermann, and Michel JJ Handgraaf. Measuring social value orientation. Judgment and Decision making , 6(8):771-781, 2011. -
  98. [236][236] Isabel Briggs Myers et al. The myers-briggs type indicator , volume 34. Consulting Psychologists Press Palo Alto, CA, 1962. -
  99. [242][242] José Luiz Nunes, Guilherme FCF Almeida, Marcelo de Araujo, and Simone DJ Barbosa. Are large language models moral hypocrites? a study based on moral foundations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume 7, pages 1074-1087, 2024. -
  100. [243][243] Takumi Ohashi, Tsubasa Nakagawa, and Hitoshi Iyatomi. Extended japanese commonsense morality dataset with masked token and label enhancement. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management , pages 3964-3968, 2024. -
  101. [245][245] William Orwig, Emma R Edenbaum, Joshua D Greene, and Daniel L Schacter. The language of creativity: Evidence from humans and large language models. The Journal of creative behavior , 58(1):128-136, 2024. -
  102. [248][248] Suketu C Patel and Jin Fan. Identification and description of emotions by current large language models. bioRxiv , pages 2023-07, 2023. -
  103. [249][249] Delroy L Paulhus and Kevin M Williams. The dark triad of personality: Narcissism, machiavellianism, and psychopathy. Journal of research in personality , 36(6):556-563, 2002. -
  104. [250][250] Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics , 51(3):907-1004, 2025. -
  105. [251][251] Sanne Peereboom, Inga Schwabe, and Bennett Kleinberg. Cognitive phantoms in llms through the lens of latent variables. arXiv preprint arXiv:2409.15324 , 2024. - · arXiv
  106. [252][252] Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, and Markus Strohmaier. Ai psychometrics: Assessing the psychological profiles of large language models through psychometric inventories. Perspectives on Psychological Science , 19(5):808-826, 2024. -
  107. [254][254] James W Pennebaker, Martha E Francis, and Roger J Booth. Linguistic inquiry and word count: Liwc 2001. Mahway: Lawrence Erlbaum Associates , 71(2001):2001, 2001. -
  108. [256][256] Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity's last exam. arXiv preprint arXiv:2501.14249 , 2025. - · arXiv
  109. [258][258] David J Pittenger. Cautionary comments regarding the myers-briggs type indicator. Consulting Psychology Journal: Practice and Research , 57(3):210, 2005. -
  110. [259][259] Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinybenchmarks: evaluating llms with fewer examples. In Proceedings of the 41st International Conference on Machine Learning , pages 34303-34326, 2024. -
  111. [260][260] Vladimir Ponizovskiy, Murat Ardag, Lusine Grigoryan, Ryan Boyd, Henrik Dobewall, and Peter Holtz. Development and validation of the personal values dictionary: A theory-driven tool for investigating references to basic human values in text. European Journal of Personality , 34(5):885-902, 2020. -
  112. [261][261] David Premack and Guy Woodruff. Does the chimpanzee have a theory of mind? Behavioral and brain sciences , 1(4):515-526, 1978. -
  113. [266][266] Tenko Raykov and George A Marcoulides. Introduction to psychometric theory . Routledge, 2011. -
  114. [267][267] Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. Valuebench: Towards comprehensively evaluating value orientations and understanding of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2015-2040, 2024. -
  115. [269][269] Matthew Riemer, Zahra Ashktorab, Djallel Bouneffouf, Payel Das, Miao Liu, Justin D. Weisz, and Murray Campbell. Position: Theory of mind benchmarks are broken for large language models, 2025. URL https: //arxiv.org/abs/2412.19726 . -
  116. [270][270] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature , 625(7995):468-475, 2024. -
  117. [273][273] Paul Röttger, Musashi Hinck, Valentin Hofmann, Kobi Hackenburg, Valentina Pyatkin, Faeze Brahman, and Dirk Hovy. Issuebench: Millions of realistic prompts for measuring issue bias in llm writing assistance. arXiv preprint arXiv:2502.08395 , 2025. - · arXiv
  118. [274][274] David Rozado. The political biases of chatgpt. Social Sciences , 12(3):148, 2023. -
  119. [276][276] Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. The goldilocks of pragmatic understanding: fine-tuning strategy matters for implicature resolution by llms. Advances in Neural Information Processing Systems , 36:20827-20905, 2023. -
  120. [277][277] John Rust and Susan Golombok. Modern psychometrics: The science of psychological assessment . Routledge, 2014. -
  121. [278][278] Sahand Sabour, Siyang Liu, Zheyuan Zhang, June Liu, Jinfeng Zhou, Alvionna Sunaryo, Tatia Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional intelligence of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 5986-6004, 2024. -
  122. [281][281] Peter Salovey and John D Mayer. Emotional intelligence. Imagination, cognition and personality , 9(3):185-211, 1990. -
  123. [283][283] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In International Conference on Machine Learning , pages 29971-30004. PMLR, 2023. -
  124. [288][288] Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems , 36:51778-51809, 2023. -
  125. [291][291] Luca M Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. Visual cognition in multimodal large language models. Nature Machine Intelligence , pages 1-11, 2025. -
  126. [292][292] Shalom H Schwartz. Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In Advances in experimental social psychology , volume 25, pages 1-65. Elsevier, 1992. -
  127. [293][293] Shalom H Schwartz, Gila Melech, Arielle Lehmann, Steven Burgess, Mari Harris, and Vicki Owens. Extending the cross-cultural validity of the theory of basic human values with a different method of measurement. Journal of cross-cultural psychology , 32(5):519-542, 2001. -
  128. [296][296] SM Seals and Valerie Shalin. Long-form analogies generated by chatgpt lack human-like psycholinguistic properties. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 45, 2023. -
  129. [297][297] Gregory Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matari´ c. A psychometric framework for evaluating and shaping personality traits in large language models. Nature Machine Intelligence , pages 1-15, 2025. -
  130. [302][302] Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. Towards facilitating empathic conversations in online mental health support: A reinforcement learning approach. In Proceedings of the web conference 2021 , pages 194-205, 2021. -
  131. [305][305] Bangzhao Shu, Lechen Zhang, Minje Choi, Lavinia Dunagan, Lajanugen Logeswaran, Moontae Lee, Dallas Card, and David Jurgens. You don't need a personality test to know these models are unreliable: Assessing the reliability of large language models on psychometric instruments. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 5263-5281, 2024. -
  132. [307][307] Gabriel Simmons. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) , pages 282-297, 2023. -
  133. [309][309] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature , 620(7972):172-180, 2023. -
  134. [310][310] Ewa Skimina, Jan Cieciuch, and William Revelle. Between-and within-person structures of value traits and value states: Four different structures, four different interpretations. Journal of Personality , 89(5):951-969, 2021. -
  135. [313][313] Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19937-19947, 2024a. -
  136. [314][314] Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. Position: a roadmap to pluralistic alignment. In Proceedings of the 41st International Conference on Machine Learning , pages 46280-46302, 2024b. -
  137. [316][316] Christopher J Soto and Oliver P John. The next big five inventory (bfi-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of personality and social psychology , 113(1):117, 2017. -
  138. [319][319] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research , 2022. -
  139. [327][327] Tom Sühr, Florian E Dorner, Samira Samadi, and Augustin Kelava. Challenging the validity of personality tests for large language models. Preprint at arXiv. arXiv-2311 https://doi. org/10.48550/arXiv , 2311, 2023. - · DOI
  140. [331][331] Kazuhiro Takemoto. The moral machine experiment on large language models. Royal Society open science , 11 (2):231393, 2024. -
  141. [332][332] Masashi Takeshita, Rafal Rzpeka, and Kenji Araki. Jcommonsensemorality: Japanese dataset for evaluating commonsense morality understanding. In In Proceedings of The Twenty Nineth Annual Meeting of The Association for Natural Language Processing (NLP2023) , pages 357-362, 2023. -
  142. [336][336] Robert L Thorndike and Saul Stein. An evaluation of the attempts to measure social intelligence. Psychological bulletin , 34(5):275, 1937. -
  143. [337][337] Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are large language models capable of generating human-level narratives? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 17659-17681, 2024. -
  144. [342][342] Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science , 185(4157):1124-1131, 1974. -
  145. [345][345] Robert J Vandenberg and Charles E Lance. A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods , 3 (1):4-70, 2000. -
  146. [347][347] Karina Vida, Fabian Damken, and Anne Lauscher. Decoding multilingual moral preferences: Unveiling llm's biases through the moral machine experiment. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , volume 7, pages 1490-1501, 2024. -
  147. [348][348] Leah von der Heyde, Anna-Carolina Haensch, and Alexander Wenz. Vox populi, vox ai? using language models to estimate german public opinion. arXiv preprint arXiv:2407.08563 , 2024. - · arXiv
  148. [350][350] Gleb D Vzorinab, Alexey M Bukinichac, Anna V Sedykha, Irina I Vetrovab, and Elena A Sergienkob. The emotional intelligence of the gpt-4 large language model. Psychology in Russia: State of the art , 17(2):85-99, 2024. -
  149. [353][353] Angelina Wang, Jamie Morgenstern, and John P Dickerson. Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence , pages 1-12, 2025a. -
  150. [354][354] Chenxu Wang, Bin Dai, Huaping Liu, and Baoyuan Wang. Towards objectively benchmarking social intelligence of language agents at the action level. In Findings of the Association for Computational Linguistics ACL 2024 , pages 8885-8897, 2024a. -
  151. [356][356] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Large search model: Redefining search stack in the era of llms. In ACM SIGIR Forum , volume 57, pages 1-16. ACM New York, NY, USA, 2024c. -
  152. [357][357] Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Yonatan Bisk, Graham Neubig, and Hao Zhu. Sotopiaπ : Interactive learning of socially intelligent language agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12912-12940, 2024d. -
  153. [359][359] Xintao Wang, Yunze Xiao, Jen-tse Huang, Siyu Yuan, Rui Xu, Haoran Guo, Quan Tu, Yaying Fei, Ziang Leng, Wei Wang, et al. Incharacter: Evaluating personality fidelity in role-playing agents through psychological interviews. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1840-1873, 2024e. -
  154. [360][360] Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, David Stillwell, Luning Sun, Fang Luo, and Xing Xie. Evaluating general-purpose ai with psychometrics. arXiv preprint arXiv:2310.16379 , 2023a. - · arXiv
  155. [361][361] Xuena Wang, Xueting Li, Zi Yin, Yue Wu, and Jia Liu. Emotional intelligence of large language models. Journal of Pacific Rim Psychology , 17:18344909231213958, 2023b. -
  156. [363][363] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022. -
  157. [372][372] Shengguang Wu, Shusheng Yang, Zhenglun Chen, and Qi Su. Rethinking pragmatics in large language models: Towards open-ended evaluation and preference tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , pages 22583-22599, 2024. -
  158. [379][379] Kaiqi Yang, Hang Li, Yucheng Chu, Yuping Lin, Tai-Quan Peng, and Hui Liu. Unpacking political bias in large language models: Insights across topic polarization. arXiv preprint arXiv:2412.16746 , 2024. - · arXiv
  159. [381][381] Jing Yao, Xiaoyuan Yi, Yifan Gong, Xiting Wang, and Xing Xie. Value fulcra: Mapping large language models to the multidimensional spectrum of basic human value. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages 8754-8777, 2024. -
  160. [383][383] Jing Yao, Xiaoyuan Yi, and Xing Xie. Clave: An adaptive framework for evaluating values of llm generated responses. Advances in Neural Information Processing Systems , 37:58868-58900, 2025b. -
  161. [385][385] Nicolas Yax, Hernán Anlló, and Stefano Palminteri. Studying and improving reasoning in humans and machines. Communications Psychology , 2(1):51, 2024. -
  162. [386][386] Haoran Ye, Jiarui Wang, Zhiguang Cao, Federico Berto, Chuanbo Hua, Haeyeon Kim, Jinkyoo Park, and Guojie Song. Reevo: Large language models as hyper-heuristics with reflective evolution. Advances in neural information processing systems , 37:43571-43608, 2024. -
  163. [387][387] Haoran Ye, Tao Huo, Zhe Guo, Xin Zhang, Hanjun Dong, and Guojie Song. Psychometrics with ai foundation models, 2025a. URL https://doi.org/10.31234/osf.io/qh5e3\_v3 . - · DOI
  164. [388][388] Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, 2025b. -
  165. [390][390] Lance Ying, Katherine M Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L Griffiths, and Joshua B Tenenbaum. On benchmarking human-like intelligence in machines. arXiv preprint arXiv:2502.20502 , 2025a. - · arXiv
  166. [393][393] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14322-14350, 2024a. -
  167. [396][396] Haoyang Zhang. Harnessing in-context learning for personality elicitation in large language models. In 2024 6th International Conference on Frontier Technologies of Information and Computer (ICFTIC) , pages 1521-1531. IEEE, 2024. -
  168. [399][399] Zhaowei Zhang, Ceyao Zhang, Nian Liu, Siyuan Qi, Ziqi Rong, Song-Chun Zhu, Shuguang Cui, and Yaodong Yang. Heterogeneous value alignment evaluation for large language models. In AAAI-2024 Workshop on Public Sector LLMs: Algorithmic and Sociotechnical Design , 2024b. -
  169. [401][401] Yue Zhao, Qingqing Gu, Xiaoyu Wang, Teng Chen, Zhonglin Jiang, Yong Chen, and Luo Ji. Fisminess: A finite state machine based paradigm for emotional support conversations. arXiv preprint arXiv:2504.11837 , 2025b. - · arXiv
  170. [402][402] Jingyao Zheng, Xian Wang, Simo Hosio, Xiaoxian Xu, and Lik-Hang Lee. Lmlpa: Language model linguistic personality assessment. Computational Linguistics , pages 1-41, 2025. -
  171. [404][404] Jingyan Zhou, Minda Hu, Junan Li, Xiaoying Zhang, Xixin Wu, Irwin King, and Helen Meng. Rethinking machine ethics-can llms perform moral reasoning through the lens of moral theories? In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 2227-2242, 2024a. -
  172. [406][406] Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Representations , 2024b. -
  173. [407][407] Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. In International Conference on Machine Learning , pages 62599-62617. PMLR, 2024a. -
  174. [409][409] Yan Zhuang, Qi Liu, Yuting Ning, Weizhe Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, et al. Efficiently measuring the cognitive ability of llms: An adaptive testing perspective. CoRR , 2023a. -
  175. [411][411] Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. Can large language models transform computational social science? Computational Linguistics , 50(1), 2024. -

Cited by

Sorted by citations

Authors
Ye, H., Jin, J., Xie, Y., Zhang, X., & Song, G.
Affiliations
Peking University (CN)
ORCID
Ye, Haoran
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.08245
arXiv
2505.08245
OpenAlex
W6929293745
Cited by
8
References
0
Citations by year
2026: 8
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, human-centered AI, personality assessment, values assessment, intelligence assessment, benchmarking, evaluation methodology, validation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review