From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology
Open accessAbstract
Large language models (LLMs) are entering psychological research both as tools and as objects of inquiry. Yet many studies apply human instruments to LLMs without establishing that the outputs are reliable or interpretable, raising the risk of measurement phantoms—statistical regularities mistaken for genuine psychological phenomena. This review argues that robust AI psychological research requires integrating two methodological traditions: psychometric validation of what a score means and causal inference standards for what the results warrant. It develops a dual-validity framework in which evidentiary demands scale with scientific ambition: from tool use through behavioral characterization and human simulation to cognitive modeling. Classifying text may require only accuracy and reliability; claiming that an LLM simulates anxiety or illuminates cognitive mechanisms requires additional evidence, including construct validity evidence and experimental controls. Progress depends on developing computational analogs of psychological constructs rather than assuming human measures automatically apply to language models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · cited 2× · see citing works
“…studies now claim to measure personality traits, theory of mind, cognitive biases, and emotional intelligence in language models, often drawing direct parallels to human psychology (Binz & Schulz 2023, Kosinski 2024, Miotto et al. 2022, Pellert et al. 2024, Wang et al. 2023, Webb et al. 2023).”
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377 · see citing works
Evaluating large language models in theory of mind tasks
Kosinski, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2405460121 · Cited by 339 · cited 3× · see citing works
“…studies now claim to measure personality traits, theory of mind, cognitive biases, and emotional intelligence in language models, often drawing direct parallels to human psychology (Binz & Schulz 2023, Kosinski 2024, Miotto et al. 2022, Pellert et al. 2024, Wang et al. 2023, Webb et al. 2023).”
Evaluating Large Language Models in Theory of Mind Tasks
Kosinski, M. (2023) · arXiv · doi:10.48550/arxiv.2302.02083 · Cited by 339 · see citing works
ChatGPT outperforms humans in emotional awareness evaluations
Elyoseph, Z., Hadar-Shoval, D., Asraf, K., & Lvovsky, M. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1199058 · Cited by 324 · see citing works
Why and how to embrace AI such as ChatGPT in your academic life
Lin, Z. (2023) · doi:10.31234/osf.io/sdx3j · Cited by 202 · see citing works
Self-citationUsing natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140 · see citing works
Sharma, M., Tong, M., Korbak, T., Duperrouzel, D., Was, H., Korbak, J., Cheng, M., Askell, A., Bai, Y., Brain, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13548 · Cited by 132 · cited 1× · see citing works
“…learning from human feedback (RLHF)—which rewards responses preferred by annotators—can induce agreement and social desirability biases, so a model may endorse items because agreement is rewarded rather than because the item reflects a stable construct (Dentella et al. 2023, Sharma et al. 2024).”
Emotional intelligence of Large Language Models
Wang, X., Li, X., Yin, Z., Wu, Y., & Liu, J. (2023) · Journal of Pacific Rim Psychology · doi:10.1177/18344909231213958 · Cited by 128 · see citing works
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · cited 2× · see citing works
“Trivial variations—altering option labels, definition order, or other prompt details—can cause models to change answers over 70% of the time (Oh & Demberg 2025) or shift the rate assigned to a single moral-foundation category by as much as 164% relative to the baseline (Abdurahman et al. 2024).”
Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
Ullman, T. (2023) · arXiv · doi:10.48550/arxiv.2302.08399 · Cited by 81 · cited 1× · see citing works
“Theory-of-mind performance can fail after seemingly trivial scenario alterations—e.g., changing an opaque container in a false-belief task to a transparent one, adding trusted testimony about true states, or changing which character’s beliefs are queried (Ullman 2023, Wang et al. 2025b).”
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74 · cited 1× · see citing works
“In the first, the LLM is a research tool—to code qualitative data, extract sentiment, generate stimuli, or assist measurement (Binz et al. 2025; Blanchard et al. 2025; Demszky et al. 2023; Feuerriegel et al. 2025; Lin 2023, 2025b; Ziems et al. 2024).”
Shojaee, P., Mirzadeh, I., Alizadeh vahid, K., Horton, M., Bengio, S., & Farajtabar, M. (2025) · Advances in Neural Information Processing Systems 38 · doi:10.52202/085713-3603 · Cited by 47 · see citing works
Large language models are proficient in solving and creating emotional intelligence tests
Schlegel, K., Sommer, N. R., & Mortillaro, M. (2025) · Communications Psychology · doi:10.1038/s44271-025-00258-x · Cited by 46 · see citing works
LLMs Get Lost In Multi-Turn Conversation
Laban, P., Hayashi, H., Zhou, Y., & Neville, J. (2025) · arXiv · Cited by 10 · cited 1× · see citing works
“Within a session, stability decreases as context windows fill and attention mechanisms prioritize recent information over earlier responses (Laban et al. 2026, Niu et al. 2026).”
Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement
Ye, H., Jin, J., Xie, Y., Zhang, X., & Song, G. (2025) · arXiv · doi:10.48550/arxiv.2505.08245 · Cited by 8 · see citing works
Cognitive phantoms in large language models through the lens of latent variables
Peereboom, S., Schwabe, I., & Kleinberg, B. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100161 · Cited by 8 · see citing works
One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity
Murthy, S. K., Ullman, T., & Hu, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.561 · Cited by 8 · see citing works
Flagged first, then by citations
- Authors
- Lin, Z.
- Affiliations
- Yonsei University (KR)
- Corresponding author
- Lin, Zhicheng
- ORCID
- Lin, Zhicheng
- Venue
- Annual Review of Psychology
- Publisher
- Annual Reviews
- Published
- 2026-08-10
- ISSN
- 0066-4308 (ISSN-L)
- 1545-2085
- OpenAlex
- W4412062109
- Cited by
- 4
- References
- 21
- Citations by year
- 2026: 4
- FWCI
- 15.39
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by
- Locations
- Annual Review of Psychology (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Yonsei University (5355661)
- National Science and Technology Major Project (2021ZD0204200)
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- large language models, construct validity, psychometric validation, causal inference, human simulation, cognitive modeling, text classification, reliability statistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
