Can generative AI infer thinking style from language? Evaluating the utility of AI as a psychological text analysis tool
Open accessAbstract
Generative AI, short for Generative Artificial Intelligence, a class of artificial intelligence systems, is not currently the choice technology for text analysis, but prior work suggests it may have some utility to assess dynamics like emotion. The current work builds upon this empirical foundation to consider how analytic thinking scores from a large language model chatbot, ChatGPT, were linked to analytic thinking scores from dictionary-based tools like Linguistic Inquiry and Word Count (LIWC). Using over 16,000 texts from four samples and tested against three prompts and two large language models (GPT-3.5, GPT-4), the evidence suggests there were small associations between ChatGPT and LIWC analytic thinking scores (meta-analytic effect sizes: .058 < rs < .304; ps < .001). When given the formula to calculate the LIWC analytic thinking index, ChatGPT performed incorrect mathematical operations in 22% of the cases, suggesting basic word and number processing may be unreliable with large language models. Researchers should be cautious when using AI for text analysis.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Minor changes in prompts lead to significant differences in outputs.
“Our findings demonstrate, in line with recent works on the effect of prompting (17, 74, 120–122), that minor changes in prompts lead to significant differences in outputs.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [74]
Past work has shown that slight alterations and modifications in phrasing, context, or order can lead to substantially different responses.
“Past work has shown that slight alterations and modifications in phrasing, context, or order can lead to substantially different responses (17, 74, 120–123).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2011) · Cited by 14,317 · see citing works
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349 · see citing works
The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
Culture and systems of thought: Holistic versus analytic cognition.
Nisbett, R. E., Peng, K., Choi, I., & Norenzayan, A. (2001) · Psychological Review · doi:10.1037/0033-295x.108.2.291 · Cited by 4,263 · see citing works
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · see citing works
Hoffman, K. M., Trawalter, S., Axt, J. R., & Oliver, M. N. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516047113 · Cited by 2,486 · see citing works
Linguistic Inquiry and Word Count (LIWC)
Chung, C. K., & Pennebaker, J. W. (2012) · IGI Global eBooks · doi:10.4018/978-1-60960-741-8.ch012 · Cited by 2,152 · see citing works
Handbook of individual differences in social behavior
(2010) · Choice Reviews Online · doi:10.5860/choice.47-3482 · Cited by 2,011 · see citing works
cocor: A Comprehensive Solution for the Statistical Comparison of Correlations
Diedenhofen, B., & Musch, J. (2015) · PLOS ONE · doi:10.1371/journal.pone.0121945 · Cited by 1,919 · see citing works
Artificial hallucinations in ChatGPT: implications in scientific writing
Alkaissi, H., & McFarlane, S. I. (2023) · Cureus · doi:10.7759/cureus.35179 · Cited by 1,540 · see citing works
Pennebaker, J. W. (2011) · New Scientist · doi:10.1016/s0262-4079(11)62167-2 · Cited by 780 · see citing works
Language use in intergroup contexts: The linguistic intergroup bias.
Maass, A., Salvi, D., Arcuri, L., & Semin, G. R. (1989) · Journal of Personality and Social Psychology · doi:10.1037//0022-3514.57.6.981 · Cited by 654 · see citing works
Language use in intergroup contexts: The linguistic intergroup bias.
Maass, A., Salvi, D., Arcuri, L., & Semin, G. R. (1989) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.57.6.981 · Cited by 654 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., Chevalier, A., & Berner, J. (2023) · Neural Information Processing Systems · doi:10.52202/075280-1205 · Cited by 630 · see citing works
When Small Words Foretell Academic Success: The Case of College Admissions Essays
Pennebaker, J. W., Chung, C. K., Frazee, J., Lavergne, G. M., & Beaver, D. I. (2014) · PLoS ONE · doi:10.1371/journal.pone.0115844 · Cited by 568 · see citing works
The secret life of pronouns: what our words say about us
Haranburu Oiharbide, M. (2012) · Choice Reviews Online · doi:10.5860/choice.49-4293 · Cited by 567 · see citing works
(2007) · Cited by 521 · see citing works
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514 · see citing works
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514 · see citing works
Pronoun Use Reflects Standings in Social Hierarchies
Kacewicz, E., Pennebaker, J., Davis, M., Jeon, M., & Graesser, A. (2014) · Journal of Language and Social Psychology · doi:10.1177/0261927x13502654 · Cited by 512 · see citing works
Language from police body camera footage shows racial disparities in officer respect
Voigt, R., Camp, N. P., Prabhakaran, V., Hamilton, W. L., Hetey, R. C., Griffiths, C. M., Jurgens, D., Jurafsky, D., & Eberhardt, J. L. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1702413114 · Cited by 407 · see citing works
Köbis, N., & Mossink, L. D. (2021) · Computers in Human Behavior · doi:10.1016/j.chb.2020.106553 · Cited by 406 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · see citing works
Language style matching in writing: synchrony in essays, correspondence, and poetry.
Ireland, M. E., & Pennebaker, J. W. (2010) · Journal of Personality and Social Psychology · doi:10.1037/a0020386 · Cited by 380 · see citing works
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369 · see citing works
All the News That’s Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation
Kreps, S., McCain, R. M., & Brundage, M. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2020.37 · Cited by 360 · see citing works
Human heuristics for AI-generated language are flawed
Jakesch, M., Hancock, J. T., & Naaman, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2208839120 · Cited by 342 · see citing works
Large Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · see citing works
Boyd, R. L., & Schwartz, H. A. (2021) · Journal of Language and Social Psychology · doi:10.1177/0261927x20967028 · Cited by 317 · see citing works
Pain and Ethnicity in the United States: A Systematic Review
Cintron, A., & Morrison, R. S. (2006) · Journal of Palliative Medicine · doi:10.1089/jpm.2006.9.1454 · Cited by 305 · see citing works
Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300 · see citing works
All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.565 · Cited by 290 · see citing works
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021) · arXiv · doi:10.48550/arxiv.2107.00061 · Cited by 290
The Oxford Handbook of Language and Social Psychology
(2014) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199838639.001.0001 · Cited by 269 · see citing works
Krosnick, J. A. (2017) · Springer eBooks · doi:10.1007/978-3-319-54395-6_53 · Cited by 244 · see citing works
The Development and Psychometric Properties of LIWC-22
Boyd, R. L., Ashokkumar, A., Seraj, S., & Pennebaker, J. W. (2022) · doi:10.13140/rg.2.2.23890.43205 · Cited by 215 · see citing works
Jordan, K., Sterling, J. L., Pennebaker, J. W., & Boyd, R. L. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1811987116 · Cited by 200 · see citing works
Coh-Metrix Measures Text Characteristics at Multiple Levels of Language and Discourse
Graesser, A. C., McNamara, D. S., Cai, Z., Conley, M. W., Li, H., & Pennebaker, J. W. (2014) · The Elementary School Journal · doi:10.1086/678293 · Cited by 199 · see citing works
Eichstaedt, J. C., Kern, M. L., Yaden, D. B., Schwartz, H. A., Giorgi, S., Park, G., Hagan, C. A., Tobolsky, V. A., Smith, L. K., Buffone, A. E. K., et al. (2021) · Psychological Methods · doi:10.1037/met0000349 · Cited by 193 · see citing works
The narrative arc: Revealing core narrative structures through text analysis
Boyd, R. L., Blackburn, K. G., & Pennebaker, J. W. (2020) · Science Advances · doi:10.1126/sciadv.aba2196 · Cited by 185 · see citing works
Kern, M. L., Eichstaedt, J. C., Schwartz, H. A., Dziurzynski, L., Ungar, L., Stillwell, D. J., Kosiński, M., Ramones, S. M., & Seligman, M. E. P. (2013) · Assessment · doi:10.1177/1073191113514104 · Cited by 142 · see citing works
GPT is an effective tool for multilingual psychological text analysisPreprint
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C., & Bavel, J. J. V. (2023) · doi:10.31234/osf.io/sekf5 · Cited by 129 · see citing works
Language left behind on social media exposes the emotional and cognitive costs of a romantic breakup
Seraj, S., Blackburn, K. G., & Pennebaker, J. W. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2017154118 · Cited by 120 · see citing works
Linguistic Obfuscation in Fraudulent Science
Markowitz, D. M., & Hancock, J. T. (2015) · Journal of Language and Social Psychology · doi:10.1177/0261927x15614605 · Cited by 113 · see citing works
Mehl, M. R. (2006) · American Psychological Association eBooks · doi:10.1037/11383-011 · Cited by 108 · see citing works
Consumer Psychology in a Social Media World
(2015) · Routledge eBooks · doi:10.4324/9781315714790 · Cited by 102 · see citing works
Historical representations of social groups across 200 years of word embeddings from Google Books
Charlesworth, T., Caliskan, A., & Banaji, M. (2022) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2121798119 · Cited by 98 · see citing works
Is ChatGPT a Good Sentiment Analyzer? A Preliminary StudyPreprint
Wang, Z., Xie, Q., Feng, Y., Ding, Z., Yang, Z., & Xia, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.04339 · Cited by 85 · see citing works
Natural Language Use as a Marker of Personality
Holtgraves, T., Ireland, M. E., & Mehl, M. R. (2014) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780199838639.013.034 · Cited by 66 · see citing works
Food for thought: Exploring how people think and talk about food online
Blackburn, K. G., Yılmaz, G., & Boyd, R. L. (2018) · Appetite · doi:10.1016/j.appet.2018.01.022 · Cited by 66 · see citing works
Text Analysis for Psychology: Methods, Principles, and PracticesPreprint
Kennedy, B., Ashokkumar, A., Boyd, R. L., & Dehghani, M. (2021) · doi:10.31234/osf.io/h2b8t · Cited by 57 · see citing works
Dying Is Unexpectedly Positive
Goranson, A., Ritter, R. S., Waytz, A. G., Norton, M. I., & Gray, K. J. (2017) · Psychological Science · doi:10.1177/0956797617701186 · Cited by 48 · see citing works
Gender and ethnicity bias in medicine: a text analysis of 1.8 million critical care records
Markowitz, D. M. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac157 · Cited by 43 · see citing works
A way with words : Using language for psychological science in the modern era
Boyd, R. L., & Pennebaker, J. W. (2015) · Lancaster EPrints (Lancaster University) · Cited by 39 · see citing works
Analytic thinking as revealed by function words: What does language really measure?
Markowitz, D. M. (2023) · Applied Cognitive Psychology · doi:10.1002/acp.4057 · Cited by 29 · see citing works
Meier, T., Boyd, R. L., Mehl, M., Milek, A., Pennebaker, J., Martin, M., Wolf, M., & Horn, A. (2020) · PLoS ONE · doi:10.1371/journal.pone.0243637 · Cited by 25 · see citing works
Instrumental goal activation increases online petition support across languages.
Markowitz, D. M. (2022) · Journal of Personality and Social Psychology · doi:10.1037/pspa0000333 · Cited by 20 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- In M. Dehghani & R. L. Boyd (Eds.), Handbook of language analysis in psychology. Guilford Press.
- Markowitz, D. M. (2022). How experts react: The World Health Organ- ization's appraisal of COVID-19 via communication patterns. Journal of Language and Social Psychology, 41(2), 209-218. https:// doi. org/ 10. 1177/ 02619 27X21 102634 · DOI
Cited by
Sorted by citations
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
Human–chatbot communication: a systematic review of psychological studies
Семеновна, Р. А., & Воронин, А. Н. (2025) · AI & Society · doi:10.1007/s00146-025-02277-y · Cited by 40 · see citing works
Exploring the Association Between Textual Parameters and Psychological and Cognitive Factors
Uludağ, K. (2024) · Psychology Research and Behavior Management · doi:10.2147/prbm.s460503 · Cited by 32 · see citing works
Pfeifer, V. A., Chilton, T., Grilli, M. D., & Mehl, M. R. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02440-1 · Cited by 15 · see citing works
Guenduez, A. A., Fuchs, S., Mergel, I. A., Mettler, T., & Frowein, S. C. J. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02357-7 · Cited by 19 · see citing works
The (in)efficacy of AI personas in deception detection experiments
Markowitz, D. M., & Levine, T. R. (2025) · Journal of Communication · doi:10.1093/joc/jqaf034 · Cited by 6 · see citing works
Putting the Ghost in the Machine: Emulating Cognitive Style in Large Language Models
Agarwal, V., Jablokow, K., & McComb, C. (2024) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4066857 · Cited by 4 · see citing works
Neural text embeddings in psychological research: A guide with examples in R.
Teitelbaum, L., & Simchon, A. (2025) · Psychological Methods · doi:10.1037/met0000768 · Cited by 3 · see citing works
An intelligent agent for sentence completion test: creation and application in depression assessment
Huang, Y., Lei, M., Zhang, H., Zong, L., Zhu, B., & Hong, L. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1649905 · Cited by 3 · see citing works
Markowitz, D. M. (2024) · Asian Communication Research · doi:10.20879/acr.2024.21.025 · Cited by 2 · see citing works
Linguistic profile of LLMs: an overview of linguistic performance and jailbreak vulnerability
Abdelzaher, E. M., & Rafaee, E. (2026) · Journal of Cultural Cognitive Science · doi:10.1007/s41809-025-00192-8 · Cited by 2 · see citing works
Linguistic dynamics of online scam conversations: a multi-stage analysis based on the COLD framework
Li, D.-Y., Zheng, R., Liu, X.-F., Ma, B., Sun, H.-W., & Jiang, L. C. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07052-y · Cited by 2 · see citing works
Mayrhofer, L., Foramitti, M., Fassnacht, S., Köhler, J. K., Todorova, B., Lamm, C., & Martins, M. J. D. (2026) · Frontiers in Political Science · doi:10.3389/fpos.2026.1745587 · Cited by 1 · see citing works
Георгиевич, А. Е., Levchenko, V. V., & Seletkova, G. I. (2024) · Digital sociology · doi:10.26425/2658-347x-2024-7-3-4-14
Ströbel, J., Medina, M., Guzmán, E. S., & van den Bogaard, M. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893007
Duan, C., Zhang, X., & Pan, X. (2025) · International Journal of Information Technologies and Systems Approach · doi:10.4018/ijitsa.391899
Building a Generative AI Comment Review System for Content Compliance
Chen, R., Chen, Z., & Tian, Y. (2025) · International Symposium on Integrated Circuit Design and Integrated Systems (ICDIS) · doi:10.1145/3772326.3774720
Luo, J., Li, S., Wang, M., Cao, X., & Li, H. (2026) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2026.3708984
Videira, J., & DeAndrea, D. C. (2026) · Human-Machine Communication · doi:10.30658/hmc.13.4
- Authors
- Markowitz, D. M.
- Affiliations
- Michigan State University (US)
- Corresponding author
- Markowitz, David M.
- ORCID
- Markowitz, David M.
- Venue
- Behavior Research Methods
- Published
- 2024-01-26
- Volume
- 56
- Issue
- 4
- Pages
- 3548-3559
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- OpenAlex
- W4391254082
- Cited by
- 19
- References
- 56
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 7
- FWCI
- 9.07
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Behavior Research Methods (publishedVersion)
- PubMed (publishedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Topics
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- generative AI, large language models, ChatGPT, analytical thinking, thinking styles, LIWC, text analysis, computational text analysis
- MeSH terms
- Artificial Intelligence, Humans, Language, Thinking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
