GPT is an effective tool for multilingual psychological text analysis
Open accessAbstract
The social and behavioral sciences have been increasingly using automated text analysis to measure psychological constructs in text. We explore whether GPT, the large-language model (LLM) underlying the AI chatbot ChatGPT, can be used as a tool for automated psychological text analysis in several languages. Across 15 datasets ( n = 47,925 manually annotated tweets and news headlines), we tested whether different versions of GPT (3.5 Turbo, 4, and 4 Turbo) can accurately detect psychological constructs (sentiment, discrete emotions, offensiveness, and moral foundations) across 12 languages. We found that GPT ( r = 0.59 to 0.77) performed much better than English-language dictionary analysis ( r = 0.20 to 0.30) at detecting psychological constructs as judged by manual annotators. GPT performed nearly as well as, and sometimes better than, several top-performing fine-tuned machine learning models. Moreover, GPT’s performance improved across successive versions of the model, particularly for lesser-spoken languages, and became less expensive. Overall, GPT may be superior to many existing methods of automated text analysis, since it achieves relatively high accuracy across many languages, requires no training data, and is easy to use with simple prompts (e.g., “is this text negative?”) and little coding experience. We provide sample code and a video tutorial for analyzing text with the GPT application programming interface. We argue that GPT and other LLMs help democratize automated text analysis by making advanced natural language processing capabilities more accessible, and may help facilitate more cross-linguistic research with understudied languages.
AI snapshot
Written from the paper’s full text.
TL;DR
Across 15 datasets and 12 languages, GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo detected psychological constructs in text with accuracy often comparable to or better than dictionary methods and sometimes close to fine-tuned models, though generally below top fine-tuned LLMs. GPT improved across versions and was effective even in several lesser-spoken African languages, but struggled with complex moral-foundation constructs.
Key findings
- GPT versions accurately detected sentiment, discrete emotions, offensiveness, and moral foundations across 12 languages; average F1 improved from GPT-3.5 (0.571) to GPT-4 (0.603) to GPT-4 Turbo (0.653).
- On 213 news headlines rated on 1–7 Likert scales, GPT correlations with human ratings averaged r = 0.63 (GPT-3.5), 0.69 (GPT-4), and 0.67 (GPT-4 Turbo), versus r = 0.28 for dictionary methods (LIWC/NRC); all GPT-vs-dictionary differences were P < 0.001.
- In African-language sentiment, mean F1 improved from 0.455 (GPT-3.5) to 0.520 (GPT-4) to 0.600 (GPT-4 Turbo); top GPT average F1 = 0.600 lagged fine-tuned models (0.720), but GPT-4 Turbo beat the top fine-tuned model for Amharic (0.646 vs. 0.640).
- GPT sometimes matched or exceeded top fine-tuned models (e.g., Arabic sentiment GPT-4 Turbo F1 = 0.746 vs. Naive Bayes 0.610; English discrete emotions GPT-4 Turbo F1 = 0.782 vs. BERT 0.785), but overall average top GPT F1 = 0.665 vs. alternate models 0.735; moral foundations had high accuracy (0.899–0.980) but low F1 for proportionality (0.174 GPT-4, 0.130 GPT-4 Turbo) and purity (0.192, 0.144). Test–retest kappa was 0.93–0.99.
Methods
- Tested GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo via API using simple prompts, mostly zero-shot, temperature 0, and no text preprocessing; compared outputs to human manual annotations as the gold standard.
- Data: 15 datasets in the Overview (n = 47,912 annotated tweets, news headlines, and Reddit comments; Discussion says 16 datasets and Table 1 lists 16 rows) covering sentiment, discrete emotions, offensiveness, and moral foundations in 12 languages, including eight African languages.
- Analysis: accuracy and macroaveraged F1 for categorical tasks; Spearman correlations for Likert-scale headlines; comparisons to dictionary methods and top fine-tuned models from source papers; test–retest reliability via weighted Cohen’s kappa.
Limitations
- Paper states: only compares GPT to human annotator judgments, not actual feelings or experiences; the human gold standard is imperfect, annotator disagreement exists, and detailed annotator demographics were unavailable.
- Paper states: GPT is a black box with limited interpretability; API costs money; text may be used for further training, raising ethical concerns for private or sensitive data; zero-shot GPT often did not surpass fine-tuned LLMs and struggled with complex constructs; few-shot sometimes decreased performance; other work reports lower reliability across months.
- Evident in text: dataset count and n are inconsistent (15 vs. 16 datasets; 47,912 vs. 47,925); some GPT-4/GPT-4 Turbo analyses used only the first 1,000 tweets due to cost; moral-foundation high accuracy was mainly driven by true negatives.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 27 cited sources checked · 0 supported · 0 flagged
- Findingown data
GPT is effective in several lesser-spoken languages.
“GPT is also effective in several lesser-spoken languages, which could facilitate text analysis research from understudied contexts.”
Quote in the paperSubstantial claim - Findingown data
GPT is capable of accurately detecting various psychological constructs (as judged by manual annotators) in text across 12 languages, using simple prompts and no additional training data.
“We demonstrate that the large-language model GPT is capable of accurately detecting various psychological constructs (as judged by manual annotators) in text across 12 languages, using simple prompts and no additional training data.”
Quote in the paperSubstantial claim - Findingown data
More recent GPT versions were more likely to label Turkish tweets as "not offensive".
“The different GPT versions had similar confusion matrices, although more recent versions were more likely to label Turkish tweets as “not offensive” (SI Appendix, Fig. S3).”
Quote in the paperNot a substantial claim - Findingnone given
The performance of GPT was often lower than the performance of more recent fine-tuned models based on LLMs.
“However, the performance of GPT was often lower than the performance of more recent fine-tuned models based on LLMs.”
Quote in the paperNot a substantial claim - Findingown data
High agreement between all versions of GPT and human ratings for English offensiveness (F1 = 0.725 to 0.746).
“We found high agreement between all versions of GPT and human ratings for English (F1 = 0.725 to 0.746) and Turkish (F1 = 0.709 to 0.762).”
Quote in the paperSubstantial claim
- Findingown data
Z-tests found all correlations between manual annotations and GPT output were significantly different from correlations between manual annotations and dictionary methods (all Ps < 0.001).
“Z-tests found that all of the correlations between manual annotations and GPT output were significantly different from the correlations between manual annotations and dictionary methods (all Ps < 0.001).”
Quote in the paperSubstantial claim - Findingown data
GPT was effective even with lesser-spoken African languages.
“Overall, GPT was effective even with lesser-spoken African languages (Table 3).”
Quote in the paperSubstantial claim - Findingown data
Weighted Cohen's kappa between English-prompted and Indonesian-prompted runs was 0.95.
“We found that the weighted Cohen’s kappa value between the English-prompted and the Indonesian-prompted runs was 0.95 (SI Appendix, Table S7), once again indicating almost perfect agreement.”
Quote in the paperSubstantial claim - Findingown data
In most cases, GPT did not surpass the accuracy of fine-tuned LLMs.
“in most cases it did not surpass the accuracy of fine-tuned LLMs.”
Quote in the paperSubstantial claim - Findingreasoning
Average F1, precision, and recall scores for moral foundations were considerably lower.
“As a result, the average F1, precision, and recall scores—which balance the accurate detection of positives and negatives—were considerably lower.”
Quote in the paperNot a substantial claim - Findingown data
GPT outperformed the best model from the SemEval competition in both English and Arabic sentiment.
“Moreover, GPT outperformed the best model from the SemEval competition in both languages (Table 4).”
Quote in the paperSubstantial claim - Findingown data
All versions of GPT had high agreement with humans in Indonesian discrete emotions (GPT-3.5 Turbo F1 = 0.678, GPT-4 F1 = 0.740, GPT-4 Turbo F1 = 0.785).
“We found that all versions of GPT had high agreement with humans in both English (GPT-3.5 Turbo F1 = 0.720, GPT-4 F1 = 0.779, GPT-4 Turbo F1 = 0.782) and Indonesian (GPT-3.5 Turbo F1 = 0.678, GPT-4 F1 = 0.740, GPT-4 Turbo F1 = 0.785) (Table 3).”
Quote in the paperSubstantial claim - Findingown data
Correlations between dictionary-based methods and manual annotations were much smaller (r = 0.22 to 0.30) than correlations between manual annotations and GPT.
“The correlations between these dictionary-based methods and manual annotations were much smaller (between r = 0.22 and r = 0.30) than the correlations between manual annotations and both versions of GPT.”
Quote in the paperSubstantial claim - Findingown data
Even higher correlations for GPT-4 (r = 0.66 to 0.75) and GPT-4 Turbo (r = 0.59 to 0.77) with human ratings on Likert sentiment and discrete emotions.
“We found very high correlations (r = 0.56 to 0.74) between GPT-3.5 Turbo and human ratings, and even higher correlations for GPT-4 (r = 0.66 to 0.75) and GPT-4 Turbo (r = 0.59 to 0.77) (Fig. 1 and Table 5).”
Quote in the paperSubstantial claim - Findingown data
GPT-3.5 Turbo achieved good performance at predicting human sentiment ratings in Arabic (Accuracy = 0.700, F1 = 0.720).
“Even the oldest GPT model we analyzed, GPT-3.5 Turbo, achieved good performance at predicting human ratings in both English (Accuracy = 0.673, F1 = 0.685) and Arabic (Accuracy = 0.700, F1 = 0.720) (Table 3).”
Quote in the paperSubstantial claim - Findingown data
GPT generally lags behind state-of-the-art fine-tuned models.
“Despite GPT’s relatively high-performance and the improvement across versions, GPT generally lags behind state-of-the-art fine-tuned models.”
Quote in the paperNot a substantial claim - Findingown data
GPT's performance on Tsonga and Amharic improved considerably with GPT-4 (Tsonga F1 = 0.302; Amharic F1 = 0.609) and GPT-4 Turbo (Tsonga F1 = 0.448, Amharic F1 = 0.646).
“However, GPT’s performance on these languages improved considerably with the release of GPT-4 (Tsonga F1 = 0.302; Amharic F1 = 0.609) and GPT-4 Turbo (Tsonga F1 = 0.448, Amharic F1 = 0.646).”
Quote in the paperSubstantial claim - Findingnone given
GPT surpasses the accuracy of dictionary methods at detecting manually annotated sentiment.
“While we show that GPT surpasses the accuracy of dictionary methods at detecting manually annotated sentiment”
Quote in the paperSubstantial claim - Findingown data
GPT struggled with complex constructs (e.g., the moral foundation of purity).
“we found that GPT struggled with such constructs (e.g., the moral foundation of purity, SI Appendix, Figs. S5–S8).”
Quote in the paperSubstantial claim - Findingown data
Average performance of the top-performing GPT version for each African language (Mean F1 = 0.600) was lower than average performance for all fine-tuned large language models (Mean F1 = 0.720).
“Specifically, the average performance of the top-performing GPT version for each African language (Mean F1 = 0.600) was lower than the average performance for all fine-tuned large language models (Mean F1 = 0.720).”
Quote in the paperNot a substantial claim - Findingnone given
GPT works well without any fine-tuning (zero-shot).
“while GPT works well without any fine-tuning (zero-shot)”
Quote in the paperSubstantial claim - Findingown data
GPT's offensiveness performance did not reach that of the top-performing models from their respective studies (F1 = 0.826 to 0.829).
“However, the performance did not reach that of the top-performing models from their respective studies, both of which used older LLMs that were fine-tuned with additional training data (F1 = 0.826 to 0.829; Table 4).”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5 Turbo achieved good performance at predicting human sentiment ratings in English (Accuracy = 0.673, F1 = 0.685).
“Even the oldest GPT model we analyzed, GPT-3.5 Turbo, achieved good performance at predicting human ratings in both English (Accuracy = 0.673, F1 = 0.685) and Arabic (Accuracy = 0.700, F1 = 0.720) (Table 3).”
Quote in the paperSubstantial claim - Findingown data
Very high correlations (r = 0.56 to 0.74) between GPT-3.5 Turbo and human ratings on Likert sentiment and discrete emotions.
“We found very high correlations (r = 0.56 to 0.74) between GPT-3.5 Turbo and human ratings, and even higher correlations for GPT-4 (r = 0.66 to 0.75) and GPT-4 Turbo (r = 0.59 to 0.77) (Fig. 1 and Table 5).”
Quote in the paperSubstantial claim - Findingown data
GPT-4 Turbo reached an F1 score roughly equivalent to the top-performing state-of-the-art LLM (a BERT model fine-tuned on Twitter data) in both English and Indonesian discrete emotions.
“Each newer version of GPT showed an improvement in accuracy and F1 above the previous one, with GPT-4 Turbo reaching an F1 score that was roughly equivalent to the top-performing state-of-the-art LLM (a BERT model that was fine-tuned on Twitter data) in both English and Indonesian (Table 4).”
Quote in the paperSubstantial claim - Findingown data
GPT's average performance at detecting sentiment with African languages improved dramatically from GPT-3.5 (Mean Accuracy = 0.462; Mean F1 = 0.455) to GPT-4 (Mean Accuracy = 0.558, Mean F1 = 0.520) to GPT-4 Turbo (Mean Accuracy = 0.636, Mean F1 = 0.600).
“Further, GPT’s average performance at detecting sentiment with African languages improved dramatically from GPT-3.5 (Mean Accuracy = 0.462; Mean F1 = 0.455) to GPT-4 (Mean Accuracy = 0.558, Mean F1 = 0.520) to GPT-4 Turbo (Mean Accuracy = 0.636, Mean F1 = 0.600).”
Quote in the paperSubstantial claim - Findingown data
Positive correlations ranging from 0.12 to 0.38 between dictionary methods and GPT output.
“We found positive correlations ranging from between 0.12 and 0.38 when looking at the relationship between dictionary methods and GPT output.”
Quote in the paperSubstantial claim - Findingown data
All versions of GPT had high agreement with humans in English discrete emotions (GPT-3.5 Turbo F1 = 0.720, GPT-4 F1 = 0.779, GPT-4 Turbo F1 = 0.782).
“We found that all versions of GPT had high agreement with humans in both English (GPT-3.5 Turbo F1 = 0.720, GPT-4 F1 = 0.779, GPT-4 Turbo F1 = 0.782) and Indonesian (GPT-3.5 Turbo F1 = 0.678, GPT-4 F1 = 0.740, GPT-4 Turbo F1 = 0.785) (Table 3).”
Quote in the paperSubstantial claim - Findingnone given
GPT-4 and GPT-Turbo showed major improvements for lesser-spoken and lesser-studied languages (such as Tsonga) compared to initial worse-than-chance performance.
“While GPT’s performance was initially worse than chance for some lesser-spoken and lesser-studied languages (such as Tsonga, which has 7 million global speakers), GPT-4 and GPT-Turbo showed major improvements for these languages.”
Quote in the paperSubstantial claim - Findingown data
GPT outperformed a fine-tuned BERT model trained on Twitter data and applied to Reddit data for several moral foundations.
“However, GPT outperformed a fine-tuned BERT model that was trained on Twitter data and applied to Reddit data for several moral foundations (SI Appendix, Tables S7 and S8).”
Quote in the paperSubstantial claim - Findingown data
Each newer version of GPT showed an improvement in accuracy and F1 above the previous one for discrete emotions.
“Each newer version of GPT showed an improvement in accuracy and F1 above the previous one, with GPT-4 Turbo reaching an F1 score that was roughly equivalent to the top-performing state-of-the-art LLM (a BERT model that was fine-tuned on Twitter data) in both English and Indonesian (Table 4).”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5 Turbo performed slightly better than both GPT-4 and GPT-4 Turbo on sentiment tasks in English and Arabic.
“Interestingly, GPT-3.5 Turbo performed slightly better than both GPT-4 and GPT-4 Turbo—both newer models—on both tasks (English: GPT-3.5 Turbo F1 = 0.685, GPT-4 F1= 0.633, GPT-4 Turbo F1 = 0.615; Arabic: GPT-3.5 Turbo F1 = 0.720; GPT-4 F1 = 0.707; GPT-4 Turbo F1 = 0.690).”
Quote in the paperNot a substantial claim - Findingown data
GPT was remarkably accurate at detecting the aggregated judgments of native speakers across countries and cultures.
“we still found that GPT was remarkably accurate at detecting the aggregated judgments of native speakers across countries and cultures.”
Quote in the paperSubstantial claim - Findingown data
GPT's test-retest reliability was very high (Cohen's kappa = 0.93 to 0.99) when the same version of GPT was run multiple times on the same dataset.
“We found that reliability was very high (Cohen’s kappa = 0.93 to 0.99) if the same version of GPT was run multiple times.”
Quote in the paperNot a substantial claim - Findingown data
For Amharic, GPT-4 Turbo (F1 = 0.646) outperformed the top-performing fine-tuned model (F1 = 0.640).
“For one language (Amharic), GPT-4 Turbo (F1 = 0.646) outperformed the top-performing fine-tuned model (F1 = 0.640).”
Quote in the paperNot a substantial claim - Findingown data
High agreement between all versions of GPT and human ratings for Turkish offensiveness (F1 = 0.709 to 0.762).
“We found high agreement between all versions of GPT and human ratings for English (F1 = 0.725 to 0.746) and Turkish (F1 = 0.709 to 0.762).”
Quote in the paperSubstantial claim - Findingown data
Weighted Cohen's kappa values ranged between 0.93 for Tsonga to 0.99 in Swahili, Hausa, Yoruba, and Kinyarwanda.
“We found that the weighted Cohen’s kappa values ranged between 0.93 for Tsonga to 0.99 in Swahili, Hausa, Yoruba, and Kinyarwanda (SI Appendix, Table S9).”
Quote in the paperSubstantial claim - Findingown data
A fine-tuned classifier trained on one dataset will not work as well when applied to a different kind of dataset (e.g., a Twitter-trained classifier on Reddit data).
“a fine-tuned classifier trained on one dataset will not work as well when applied to a different kind of dataset. For instance, a classifier trained on Twitter data did not work as well when applied to Reddit data (SI Appendix, Tables S7 and S8).”
Quote in the paperSubstantial claim - Findingown data
GPT shows reasonable accuracy across languages and contexts.
“It shows reasonable accuracy across languages and contexts”
Quote in the paperSubstantial claim - Findingown data
There is often considerable disagreement between human annotators, which is reflected in the datasets analyzed.
“Notably, there is often considerable disagreement between human annotators, which is also reflected in the datasets we analyzed (SI Appendix, Table S10).”
Quote in the paperNot a substantial claim - Findingown data
GPT strongly outperformed dictionary methods at detecting manually annotated sentiment and emotion.
“GPT strongly outperformed dictionary methods at detecting manually annotated sentiment and emotion”
Quote in the paperSubstantial claim - Premisereasoning
Dictionary methods are often not as accurate at detecting psychological constructs in text as determined by manual annotators when compared to more recent methods.
“However, dictionary methods are often not as accurate at detecting psychological constructs in text as determined by manual annotators—who are often considered the “gold standard” of accuracy in natural language processing (35)—when compared to more recent methods.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [1] J. Wilkerson, A. Casas, Large-scale computerized text analysis in political science: Opportunities and challenges. Annu. Rev. Polit. Sci. 20, 529– 544 (2017). source
Not checked in the free run.
- cites [34] Y. R. Tausczik, J. W. Pennebaker, The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29, 24– 54 (2010). source
Not checked in the free run.
- Interpretationreasoning
GPT overcomes the limitations present in existing text analysis methods.
“GPT thus overcomes the limitations present in existing methods.”
Quote in the paperSubstantial claim - Interpretationown data
GPT is an effective multilingual text analysis tool.
“These results suggest that GPT is an effective multilingual text analysis tool.”
Quote in the paperSubstantial claim - Interpretationown data
GPT is rapidly improving with newer models.
“The recent performance improvements also suggest that GPT is rapidly improving with newer models.”
Quote in the paperNot a substantial claim - Interpretationown data
GPT appears far more effective at detecting manually annotated sentiment and discrete emotions than common dictionary-based methods.
“Thus, GPT appears to be far more effective at detecting manually annotated sentiment and discrete emotions than common dictionary-based methods that are very popular in psychology and the social sciences.”
Quote in the paperSubstantial claim - Premisereasoning
GPT does not require training data.
“While machine learning classifiers require large amounts of manually annotated text to train and high coding proficiency, GPT does not require training data, is effective across contexts and languages, and is intuitive to use with little coding experience, since it works via prompting with minimal programming.”
Quote in the paperSubstantial claim - Premisecitation
GPT itself can be fine-tuned to further increase its performance.
“Additionally, GPT itself can be fine-tuned to further increase its performance. For instance, one paper found that fine-tuned GPT outperformed even fine-tuned BERT at detecting moral foundations (73).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [73] S. Abdurahman, Perils and opportunities in using large language models in psychological research. PNAS Nexus 3 245 (2024). PMC11249969 39015547 source
Not checked in the free run.
- Interpretationreasoning
Fine-tuned models, while highly accurate for one context or dataset, are not very flexible when applied to other contexts or datasets.
“This suggests that fine-tuned models, while highly accurate for one context or dataset, are not very flexible when applied to other contexts or datasets.”
Quote in the paperSubstantial claim - Interpretationown data
GPT is capable of accurately detecting psychological constructs in text, regardless of the format of the ratings or the type of text.
“This suggests that GPT is capable of accurately detecting psychological constructs in text, regardless of the format of the ratings or the type of text.”
Quote in the paperSubstantial claim - Interpretationreasoning
In some cases, GPT may also be a better choice than fine-tuned machine learning models.
“In some cases, GPT may also be a better choice than fine-tuned machine learning models.”
Quote in the paperNot a substantial claim - Interpretationown data
More recent versions of GPT might have a cross-linguistic bias toward overestimating sentiment in a given text compared to humans.
“This suggests more recent versions of GPT might have a cross-linguistic bias toward overestimating sentiment in a given text compared to humans.”
Quote in the paperSubstantial claim - Interpretationreasoning
GPT may be superior to many-but not all-existing automated text analysis methods.
“GPT may be superior to many—but not all—existing automated text analysis methods.”
Quote in the paperSubstantial claim - Interpretationown data
GPT provides extremely reliable results even when the prompt is asked in a different language.
“This suggests that GPT provides extremely reliable results even when the prompt is asked in a different language.”
Quote in the paperSubstantial claim - Interpretationown data
GPT is effective at detecting psychological constructs even in lesser-spoken languages.
“These results suggest that GPT is effective at detecting psychological constructs even in lesser-spoken languages.”
Quote in the paperSubstantial claim - Interpretationown data
GPT may struggle more with more complex or difficult-to-define constructs.
“Thus, GPT may struggle more with more complex or difficult-to-define constructs.”
Quote in the paperSubstantial claim - Interpretationown data
Running GPT at separate times yields extremely high reliability when compared to traditional standards.
“A Cohen’s Kappa value of 0.81 indicates “almost perfect agreement,” indicating that running GPT at separate times yields extremely high reliability when compared to traditional standards (75).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [75] M. L. McHugh, Interrater reliability: The kappa statistic. Biochem. Med. (Zagreb) 22, 276– 282 (2012). 23092060 PMC3900052 source
Not checked in the free run.
- Study design and conductreasoning
The study does not show that GPT can accurately detect what a person is feeling or experiencing.
“We do not, however, show that GPT can accurately detect what a person is feeling or experiencing”
Quote in the paperNot a substantial claim - Premisecitation
Only Anger, Fear, Joy, and Sadness were significantly correlated with human rater judgments.
“We used only these four emotions because Robertson et al. (18) found that only those emotions were significantly correlated with human rater judgments.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] C. E. Robertson, Negativity drives online news consumption. Nat. Hum. Behav. 7, 812– 822 (2023). 36928780 PMC10202797 source
Not checked in the free run.
- Premisecitation
Human ratings were not significantly correlated with NRC codings for Trust, Surprise, Disgust, and Anticipation.
“Human ratings were not significantly correlated with NRC codings for Trust, Surprise, Disgust, and Anticipation.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [18] C. E. Robertson, Negativity drives online news consumption. Nat. Hum. Behav. 7, 812– 822 (2023). 36928780 PMC10202797 source
Not checked in the free run.
- Interpretationown data
GPT is an effective tool for detecting various psychological constructs in text across several languages.
“Our results suggest that GPT is an effective tool for detecting various psychological constructs in text across several languages.”
Quote in the paperSubstantial claim - Interpretationreasoning
GPT may have a number of benefits over existing text analysis methods, such as dictionary-based methods and fine-tuned machine learning models.
“GPT may have a number of benefits over existing text analysis methods, such as dictionary-based methods and fine-tuned machine learning models.”
Quote in the paperSubstantial claim
- May combine two claims. 27
- Classified as probably not a substantial claim. 18
May combine two claims.
Findingown dataInitially, GPT had less-than-chance performance on Tsonga (GPT-3.5 Turbo Accuracy = 0.291; F1 = 0.281) and Amharic (GPT-3.5 Turbo Accuracy = 0.646; F1 = 0.226).
“Initially, GPT had less-than-chance performance on two languages: Tsonga (GPT-3.5 Turbo Accuracy = 0.291; F1 = 0.281), the least-spoken language we analyzed, and Amharic (GPT-3.5 Turbo Accuracy = 0.646; F1 = 0.226).”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataF1 scores were relatively high for care (GPT-4 F1 = 0.534, GPT-4 Turbo F1 = 0.497) and moral sentiment (GPT-4 F1 = 0.653, GPT-4 Turbo F1 = 0.677), but lower for proportionality (GPT-4 F1 = 0.174, GPT-4 Turbo = 0.130) and purity (GPT-4 F1 = 0.192, GPT-4 Turbo F1 = 0.144).
“While F1 scores were relatively high for care (GPT-4 F1 = 0.534, GPT-4 Turbo F1 = 0.497) and moral sentiment (GPT-4 F1 = 0.653, GPT-4 Turbo F1 = 0.677), they were lower for foundations such as proportionality (GPT-4 F1 = 0.174, GPT-4 Turbo = 0.130) and purity (GPT-4 F1 = 0.192, GPT-4 Turbo F1 = 0.144).”
Quote in the paperSubstantial claimMay combine two claims.
FindingcitationAcross 16 different datasets, multiple versions of GPT could accurately detect various psychological constructs in different types of text and across 12 languages, including lesser-spoken African languages.
“Across 16 different datasets, we found that multiple versions of GPT (GPT-3.5, GPT-4, and GPT-4 Turbo) could accurately detect various psychological constructs (sentiment, discrete emotions, and offensiveness, and moral foundations) in different types of text (tweets, news headlines, and Reddit posts) and across 12 languages, including lesser-spoken African languages (76).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [76] A. Magueresse, V. Carles, E. Heetderks, Low-resource languages: A review of past work and future challenges. arXiv [Preprint] (2020). 10.48550/arXiv.2006.07264 (Accessed 30 April 2023).
Not checked in the free run.
May combine two claims.
Findingown dataFew-shot learning with GPT-4 sometimes increased and sometimes decreased performance.
“we experimented with providing GPT-4 with examples (“few-shot” learning) in an attempt to improve its performance, finding this sometimes did increase, but other times decreased performance”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataWhen compared to a fine-tuned BERT model, GPT performed worse, although it came close for moral sentiment.
“When compared to a fine-tuned BERT model, GPT performed worse, although it came close to the fine-tuned BERT model for moral sentiment (SI Appendix, Tables S5 and S6).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataIn many cases, GPT performed close to (and sometimes better than) fine-tuned machine learning models.
“In many cases, GPT performed close to (and sometimes better than) fine-tuned machine learning models.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT-4 was more likely to classify "neutral" tweets as either "positive" or "negative" compared to GPT-3.5 Turbo in both English and Arabic, a bias which persisted in GPT-4 Turbo albeit to a lesser extent.
“Examination of the confusion matrices (SI Appendix, Fig. S1) revealed a possible driver of this effect: GPT-4 was more likely to classify “neutral” tweets as either “positive” or “negative” compared to GPT-3.5 Turbo in both English and Arabic, a bias which persisted in GPT-4 Turbo albeit to a lesser extent.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataGPT's performance improved substantially from GPT-3.5 (Average F1 = 0.571) to GPT-4 (Average F1 = 0.603) to GPT-4 Turbo (Average F1 = 0.653), with largest improvements for the least-spoken languages.
“GPT’s performance improved substantially from GPT-3.5 (Average F1 = 0.571) to GPT-4 (Average F1 = 0.603) to GPT-4 Turbo (Average F1 = 0.653), with largest improvements for the least-spoken languages.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataEach moral foundation had a very high accuracy score (0.899 to 0.980 for each foundation, and 0.634 to 0.684 for moral sentiment overall), mainly driven by true negatives.
“Although each moral foundation had a very high accuracy score (ranging from 0.899 to 0.980 for each foundation, and 0.634 to 0.684 for moral sentiment overall), this was mainly driven by true negatives (i.e., comments that did not contain a moral foundation that were correctly labeled by GPT), as there much fewer positives (i.e., comments that contained a moral foundation).”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Findingnone givenGPT is easy to use with little code and simple prompts.
“is easy to use with little code and simple prompts”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone givenGPT requires no training data.
“requires no training data”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationMachine learning models are time-consuming and resource-intensive to create and often require high coding proficiency and tens of thousands of manually annotated texts to train.
“However, most machine learning models are time-consuming and resource-intensive to create. Moreover, they often require high coding proficiency to design or implement and tens of thousands of manually annotated texts to train (27).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [27] W. J. Brady, K. McLoughlin, T. N. Doan, M. Crockett, How social learning amplifies moral outrage expression in online social networks. Science 7, eabe5641 (2021). PMC8363141 34389534
Not checked in the free run.
May combine two claims.
PremisecitationGPT is trained on massive datasets of internet text, making it promising for completing text analysis tasks across multiple languages without additional training.
“GPT is trained on massive datasets of internet text (such as Common Crawl or Wikipedia), which makes it particularly promising for completing text analysis tasks across multiple languages without any additional training (known as “zero-shot” learning) (46).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [46] J. Wei, Finetuned language models are zero-shot learners. arXiv [Preprint] (2021). 10.48550/arXiv.2109.01652 (Accessed 30 April 2023).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationResearchers have noted GPT's ability to help with computational social science tasks, detect misinformation, infer politicians' ideologies, write persuasive political arguments, respond to patient questions, simulate human research participants, and model collective behavior.
“Researchers have also recently noted GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
Quote in the paperNot a substantial claim0 of 8 sources checked- cites [49] C. Ziems, Can large language models transform computational social science? Computational Linguistics 50, 237– 291 (2024). source
Not checked in the free run.
- cites [50] M. M. Amin, E. Cambria, B. W. Schuller, Will affective computing emerge from foundation models and general AI? A first evaluation on ChatGPT. IEEE Intell. Syst. 38, 15– 23 (2023). source
Not checked in the free run.
- cites [51] P. Törnberg, ChatGPT-4 outperforms experts and crowd workers in annotating political Twitter messages with zero-shot learning. arXiv [Preprint] (2023). 10.48550/arXiv.2304.06588 (Accessed 30 April 2023).
Not checked in the free run.
- cites [52] D. Demszky, Using large language models in psychology. Nat. Rev. Psychol. 2, 688– 701 (2023).
Not checked in the free run.
- cites [53] I. Grossmann, AI and the transformation of social science research. Science 380, 1108– 1109 (2023). 37319216 source
Not checked in the free run.
- cites [54] C. A. Bail, Can generative AI improve social science? Proc. Natl. Acad. Sci. U.S.A. 121, e2314021121 (2024). 38722813 PMC11127003 source
Not checked in the free run.
- cites [55] E. Hoes, S. Altay, J. Bermeo, Using ChatGPT to fight misinformation: ChatGPT nails 72% of 12,000 verified claims. PsyArXiv [Preprint] (2023). https://osf.io/preprints/psyarxiv/qnjkf (Accessed 30 April 2023).
Not checked in the free run.
- cites [56] J. He, F. Wallis, A. Gvirtz, S. Rathje, Artificial intelligence chatbots mimic human collective behaviour. ResearchSquare [Preprint] (2024). 10.21203/rs.3.rs-3096289/v2 (Accessed 25 February 2024). PMC13051051 39739553 source
Not checked in the free run.
May combine two claims.
PremisereasoningBoth dictionary and machine learning approaches are not well-equipped to analyze multilingual data.
“A further shortcoming of both of these approaches is that they are not well-equipped to analyze multilingual data.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationGPT has exhibited human-level performance on a variety of tasks and better performance than existing LLMs.
“GPT has been lauded for its ability to exhibit human-level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [47] D. M. Katz, M. J. Bommarito, S. Gao, P. Arredondo, GPT-4 passes the bar exam. Philos. Trans. A Math. Phys. Eng. Sci. 382, 20230254 (2024). 38403056 PMC10894685 source
Not checked in the free run.
- cites [48] E. R. Mollick, L. Mollick, New modes of learning enabled by AI chatbots: Three methods and assignments. SSRN [Preprint] (2022). 10.2139/ssrn.4300783 (Accessed 30 April 2023). source
Not checked in the free run.
Classified as probably not a substantial claim.
Premisenone givenNewer versions of GPT (starting with 3.5) work by prompting, generating output in response to a human question.
“Compared to older LLMs like Bidirectional Encoder Representations from Transformers (BERT), newer versions of GPT (starting with 3.5) work by “prompting,” meaning that they generate output in response to a question asked by a human user.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataGPT is effective across contexts and languages.
“While machine learning classifiers require large amounts of manually annotated text to train and high coding proficiency, GPT does not require training data, is effective across contexts and languages, and is intuitive to use with little coding experience, since it works via prompting with minimal programming.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataGPT performs much better than English-language dictionary methods at both sentiment analysis and discrete emotion detection.
“GPT performs much better than English-language dictionary methods at both sentiment analysis and discrete emotion detection.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningGPT may be overall beneficial in moving computational social science beyond a WEIRD perspective, since it is better suited for multilingual analysis than prior computational social science tools.
“GPT may be overall beneficial in moving computational social science beyond this WEIRD perspective, since it is better suited for multilingual analysis than prior computational social science tools.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningWhat GPT loses in accuracy compared to fine-tuned models it gains in flexibility, since it is highly accurate without further training data on a variety of datasets.
“What GPT loses in accuracy (compared to fine-tuned models) it gains in its flexibility, since it is highly accurate without any further training data on a variety of datasets.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataWhile GPT output and dictionary method output were correlated, the correlations were not particularly strong, suggesting analyses using GPT may potentially lead to very different conclusions than analyses using dictionary methods.
“Thus, while GPT output and dictionary method output were correlated, the correlations were not particularly strong, suggesting that analyses using GPT may potentially lead to very different conclusions than analyses using dictionary methods.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningGPT is also very easy to use and achieves much higher accuracy at detecting psychological constructs as judged by manual annotators.
“While dictionary-based text analysis methods are often used because of their user-friendliness, GPT is also very easy to use and achieves much higher accuracy at detecting psychological constructs as judged by manual annotators.”
Quote in the paperSubstantial claimMay combine two claims.
Interpretationown dataGPT appears effective at multilingual sentiment analysis, with performance comparable to top-performing machine learning models from several years ago.
“Overall, GPT appears to be effective at multilingual sentiment analysis, with performance comparable to top-performing machine learning models from several years ago.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Study design and conductnone givenDatasets 7 to 16 were not publicly available on the internet at the time GPT was trained in 2021, and thus could not have influenced the training dataset.
“Datasets 7 to 16 were not publicly available on the internet at the time GPT was trained in 2021, and thus could not have influenced the training dataset.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationThe gold standard of manual annotator judgments is imperfect and does not necessarily reflect the complex nature of the constructs being measured.
“this gold standard is still imperfect (72), and does not necessarily reflect the complex nature of the constructs we are measuring (80).”
Quote in the paperSubstantial claim0 of 2 sources checkedProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSome work has found that LLMs reflect human biases, such as in-group favoritism.
“Some work has found that LLMs reflect human biases, such as in-group favoritism (77).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [77] T. Hu, Generative language models exhibit social identity biases. arXiv [Preprint] (2023). http://arxiv.org/abs/2310.15819 (Accessed 25 February 2024). PMC11774750 39668254
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataBecause the study did not have detailed demographic data on the annotators, it was difficult to examine whether GPT was biased toward reflecting the judgments of certain annotators.
“Because we did not have detailed demographic data on the annotators, it was difficult to examine whether GPT was biased toward reflecting the judgments of certain annotators.”
Quote in the paperNot a substantial claimMay combine two claims.
InterpretationreasoningGPT is presently an accurate and easy to use text analysis tool that works across languages and contexts.
“we find that GPT is presently an accurate and easy to use text analysis tool that works across languages and contexts.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLLaMA and other modern LLMs often approach the accuracy of GPT at many tasks.
“which often approach the accuracy of GPT at many tasks (81).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [81] A. Buscemi, D. Proverbio, ChatGPT vs. Gemini vs LLaMA on multilingual sentiment analysis. arXiv [Preprint] (2024). http://arxiv.org/abs/2402.01715 (Accessed 1 June 2024).
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.
InterpretationreasoningGPT and future LLMs may soon supplant existing automated text analysis approaches and facilitate more cross-linguistic research with lesser-resourced languages and non-WEIRD populations.
“Therefore, we believe GPT and future LLMs may soon supplant existing automated text analysis approaches and facilitate more cross-linguistic research with lesser-resourced languages and non-WEIRD populations.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoningThe metrics used were kept the same as those reported in the original papers so that GPT's performance could be compared to other models.
“We keep the metrics that we use the same as the metrics reported in the papers from which our datasets originated so that we can compare the performance of GPT to other models.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenPrice concerns about the GPT API might be especially pronounced for researchers in non-WEIRD contexts.
“These price concerns might be especially pronounced for researchers in non-WEIRD contexts.”
Quote in the paperNot a substantial claim- Rests on an assumption: The GPT API costs money to use, with GPT-4 being the most expensive.
May combine two claims.
PremisecitationThe output of human annotators, like GPT, is also not reproducible and usually has far lower test-retest reliability.
“the output of human annotators, like GPT, is also not reproducible and usually has far lower test–retest reliability (73).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [73] S. Abdurahman, Perils and opportunities in using large language models in psychological research. PNAS Nexus 3 245 (2024). PMC11249969 39015547 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOther work has found that GPT is biased toward responses on cognitive tasks that are similar to those of WEIRD populations.
“Other work has found that GPT is biased toward responses on cognitive tasks that are similar to those of WEIRD populations (78).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [78] M. Atari, M. J. Xue, P. S. Park, D. Blasi, J. Henrich, Which humans? PsyArXiv [Preprint] (2023). https://osf.io/preprints/psyarxiv/5b26t (Accessed 25 February 2024). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataThe study only compares GPT's responses to the judgments of human annotators.
“One limitation of our work is that we only compare GPT’s responses to the judgments of human annotators.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationLIWC has been validated to be a measure that correlates with people's behaviors or self-reports.
“LIWC has been validated to be a measure that correlates with people’s behaviors or self-reports (31).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [31] A. Ashokkumar, J. W. Pennebaker, Tracking group identity through natural language within groups. PNAS Nexus 1, pgac022 (2022). 35774418 PMC9229362 source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOther work has reported lower reliability if GPT runs are several months apart.
“other work has reported lower reliability if GPT runs are several months apart (74).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [74] J. Bisbee, J. D. Clinton, C. Dorff, B. Kenkel, J. Larson, Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 10.1017/pan.2024.5 (2024). source
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.
InterpretationreasoningGPT might increase the ease and accessibility of advanced natural language processing methods, which may empower more researchers around the globe to do advanced text analysis research.
“Overall, GPT might increase the ease and accessibility of advanced natural language processing methods, which may empower more researchers around the globe to do advanced text analysis research.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe temperature of 0 setting means that the GPT output would not largely differ if the analysis were run a second time.
“We used a temperature of 0 to obtain the highest probability predictions of the models. This setting means that the GPT output would not largely differ if we ran our analysis a second time.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
GPT is a 'black box,' and it is difficult to know why it is producing the responses it provides.
“GPT is a “black box,” and it is difficult to know why it is producing the responses it provides.”
The paper relies on this without support.
Some dictionary methods, such as LIWC, were not necessarily designed to detect manually annotated sentiment.
“some of these dictionary methods, such as LIWC, were not necessarily designed to detect manually annotated sentiment.”
The paper relies on this without support.
The GPT API costs money to use, with GPT-4 being the most expensive.
“the GPT API costs money to use, with GPT-4 being the most expensive.”
The paper relies on this without support.
The price of the GPT API is still much lower than other research costs, such as hiring human annotators or experts to design novel machine learning classifiers.
“this price of the GPT API is still much lower than other research costs—such as the cost of hiring human annotators to manually label data, or the cost of hiring experts to design novel machine learning classifiers.”
The paper relies on this without support.
GPT's API costs have reduced with successive model updates and will hopefully continue to do so.
“GPT’s API costs have reduced with successive model updates and will hopefully continue to do so.”
The paper relies on this without support.
GPT uses text it receives for further training, raising important ethical considerations when using GPT to analyze private or sensitive data.
“GPT uses text it receives for further training, raising important ethical considerations when using GPT to analyze private or sensitive data.”
The paper relies on this without support.
OpenAI released an improved GPT-4 model (GPT-4 Turbo) that was less expensive to use and more accurate, particularly in lesser-spoken languages.
“OpenAI released an improved GPT-4 model (GPT-4 Turbo) that was, at the same time, less expensive to use and more accurate, particularly in lesser-spoken languages.”
The paper relies on this without support.
OpenAI released GPT-4o, which is cheaper and faster than GPT-4 Turbo.
“OpenAI released GPT-4o, which is cheaper and faster than GPT-4 Turbo.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have been shown to perform well on diverse text-analysis tasks, often matching or exceeding state-of-the-art models and crowd workers, even with zero-shot prompts.
“LLMs have been shown to perform well on diverse text-analysis tasks 36 , often matching or exceeding state-of-the-art models and crowd workers 37,38 , even with zero-shot prompts 39 .”
From Quantifying the prevalence and impact of overreaching causal claims in social science (2026) · cites this paper as [37]
Prior work demonstrates that LLMs are a reliable tool for large-scale content analysis.
“Yet, prior work demonstrates that LLMs are a reliable tool for largescale content analysis (Feuerriegel et al. 2025; Rathje et al. 2024)”
From Characterizing AI-Generated Misinformation on Social Media (2025) · cites this paper as Rathje et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Arabic sentiment's top GPT model conflicts with reported scores
MajorTable 4 identifies GPT-4 Turbo as the top-performing Arabic sentiment model with F1 = 0.746. The Results instead give it F1 = 0.690 and give GPT-3.5 Turbo the higher score of 0.720, reversing which model performs best.
“| Arabic | Sentiment | 0.746 | 4 Turbo | 0.610 | Naive Bayes | 2017 |”
“Arabic: GPT-3.5 Turbo F1 = 0.720; GPT-4 F1 = 0.707; GPT-4 Turbo F1 = 0.690”
English sentiment test-set sizes differ
MinorTable 1 gives 12,283 English sentiment tweets, while Methods gives 12,284 for the test set used.
“| Sentiment of English tweets (2017) | Sentiment | Tweets | 12,283 | Positive, Negative, Neutral | English | 1,450 |”
“We used the designated test set for subtask A (N = 12,284).”
Indonesian emotion test-set sizes differ
MinorTable 1 gives 440 Indonesian emotion tweets, while Methods says the test portion used contained 442.
“| Discrete emotions in Indonesian tweets (2020) | Discrete Emotions | Tweets | 440 | Anger, Fear, Sadness, Love, Joy | Indonesian | 300 |”
“We used the test portion of this dataset (N = 442).”
GPT-version correlation pair is labeled differently
MinorThe Results describe the 0.76-0.86 correlations as comparing GPT-4 with GPT-4 Turbo, whereas Table 5 labels that correlation column as comparing GPT-3.5 Turbo with GPT-4 Turbo.
“as well as high correlations between GPT-4 and GPT-4 Turbo (between r = 0.76 and r = 0.86)”
“Spearman correlation between GPT-3.5 Turbo and GPT-4 Turbo output”
Dataset count alternates between 15 and 16
MinorTable 1 lists 16 datasets, matching the Discussion's count, but the table note says 15 were used.
“We used 15 different datasets which contained 47,925 manually annotated tweets and news headlines in 12 languages”
“Across 16 different datasets, we found that multiple versions of GPT”
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Advances In Experimental Social Psychology
(2022) · Advances in experimental social psychology · doi:10.1016/s0065-2601(22)x0002-6 · Cited by 22,559 · see citing works
Interrater reliability: The kappa statistic
McHugh, M. L. (2012) · Biochemia Medica · doi:10.11613/bm.2012.031 · Cited by 19,877 · see citing works
The weirdest people in the world?
Henrich, J., Heine, S. J., & Norenzayan, A. (2010) · Behavioral and Brain Sciences · doi:10.1017/s0140525x0999152x · Cited by 12,641 · see citing works
The spread of true and false news online
Vosoughi, S., Roy, D., & Aral, S. (2018) · Science · doi:10.1126/science.aap9559 · Cited by 8,847 · cited 1× · see citing works
“…datasets are typically analyzed for the presence of various psychological constructs, such as sentiment (i.e., positivity versus negativity) (18–22), discrete emotions such as anger or sadness (23, 24), offensiveness (25), moral emotions (26, 27), out - party animosity (5, 9), or toxicity (28, 29).”
BackgroundThe Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods
Tausczik, Y. R., & Pennebaker, J. W. (2010) · Journal of Language and Social Psychology · doi:10.1177/0261927x09351676 · Cited by 5,774 · see citing works
Ayers, J. W., Poliak, A., Dredze, M., Leas, E. C., Zhu, Z., Kelley, J. B., Faix, D. J., Goodman, A. M., Longhurst, C. A., Hogarth, M., et al. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.1838 · Cited by 2,613 · cited 1× · see citing works
“…GPT’s ability to help with computational social science tasks (49–54), detect misinformation (55), infer politicians’ ideologies (56), write persuasive political arguments (57), respond to patient questions (58), simulate human research participants (59–61), and model collective behavior (56).”
BackgroundBarrett, L. F. (2006) · Perspectives on Psychological Science · doi:10.1111/j.1745-6916.2006.00003.x · Cited by 1,859 · see citing works
Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach
Schwartz, H. A., Eichstaedt, J. C., Kern, M. L., Dziurzynski, L., Ramones, S. M., Agrawal, M., Shah, A., Kosinski, M., Stillwell, D., Seligman, M. E. P., et al. (2013) · PLoS ONE · doi:10.1371/journal.pone.0073791 · Cited by 1,700 · see citing works
The neglected 95%: why American psychology needs to become less American.
Arnett, J. (2008) · American Psychologist · doi:10.1037/0003-066x.63.7.602 · Cited by 1,656 · see citing works
Emotion shapes the diffusion of moralized content in social networks
Brady, W. J., Wills, J. A., Jost, J. T., Tucker, J. A., & Van Bavel, J. J. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1618923114 · Cited by 1,396 · cited 1× · see citing works
“This method is widely used within psychological research (9, 10, 18, 30–33).”
MethodSelf-citationFinetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
SemEval-2017 Task 4: Sentiment Analysis in Twitter
Rosenthal, S., Farra, N., & Nakov, P. (2017) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s17-2088 · Cited by 869 · see citing works
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Barbieri, F., Camacho-Collados, J., Anke, L. E., & Neves, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.148 · Cited by 756 · see citing works
SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)
Zampieri, M., Malmasi, S., Nakov, P., Rosenthal, S., Farra, N., & Kumar, R. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2010 · Cited by 750 · see citing works
SemEval-2018 Task 1: Affect in Tweets
Mohammad, S., Bravo-Marquez, F., Salameh, M., & Kiritchenko, S. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1001 · Cited by 746 · see citing works
Transformer models for text-based emotion detection: a review of BERT-based approaches
Acheampong, F. A., Nunoo‐Mensah, H., & Chen, W. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09958-2 · Cited by 686 · see citing works
Facebook language predicts depression in medical records
Eichstaedt, J. C., Smith, R. J., Merchant, R. M., Ungar, L. H., Crutchley, P., Preoţiuc-Pietro, D., Asch, D. A., & Schwartz, H. A. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1802331115 · Cited by 643 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Out-group animosity drives engagement on social media
Rathje, S., Van Bavel, J. J., & van der Linden, S. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2024292118 · Cited by 560 · see citing works
Self-citationUsing cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · see citing works
(2012) · Advances in experimental social psychology · Cited by 507 · see citing works
Computational social science: Obstacles and opportunities
Lazer, D. M. J., Pentland, A., Watts, D. J., Aral, S., Athey, S., Contractor, N., Freelon, D., Gonzalez-Bailon, S., King, G., Margetts, H., et al. (2020) · Science · doi:10.1126/science.aaz8170 · Cited by 453 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
Over-reliance on English hinders cognitive science
Blasi, D. E., Henrich, J., Adamou, E., Kemmerer, D., & Majid, A. (2022) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2022.09.015 · Cited by 396 · see citing works
Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., & Çöltekin, Ç. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.188 · Cited by 389 · see citing works
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377 · see citing works
How social learning amplifies moral outrage expression in online social networks
Brady, W. J., McLoughlin, K., Doan, T. N., & Crockett, M. J. (2021) · Science Advances · doi:10.1126/sciadv.abe5641 · Cited by 375 · see citing works
Using large language models in psychology
Demszky, D., Yang, D., Yeager, D. S., Bryan, C. J., Clapper, M., Chandhok, S., Eichstaedt, J. C., Hecht, C., Jamieson, J., Johnson, M., et al. (2023) · Nature Reviews Psychology · doi:10.1038/s44159-023-00241-5 · Cited by 369 · see citing works
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
Morality beyond the WEIRD: How the nomological network of morality varies across cultures.
Atari, M., Haidt, J., Graham, J., Koleva, S. P., Stevens, S. T., & Dehghani, M. (2023) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000470 · Cited by 349 · see citing works
Negativity drives online news consumption
Robertson, C. E., Pröllochs, N., Schwarzenegger, K., Pärnamets, P., Van Bavel, J. J., & Feuerriegel, S. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01538-4 · Cited by 333 · see citing works
Self-citationLarge-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300 · see citing works
Beyond WEIRD: A review of the last decade and a look ahead to the global laboratory of the future
Apicella, C. L., Ara Norenzayan, A., & Henrich, J. (2020) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2020.07.015 · Cited by 282 · see citing works
Moralization in social networks and the emergence of violence during protests
Mooijman, M., Hoover, J., Lin, Y., Ji, H., & Dehghani, M. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0353-0 · Cited by 229 · see citing works
A Bibliometric Review of Large Language Models Research from 2017 to 2023
Fan, L., Li, L., Ma, Z., Lee, S., Yu, H., & Hemphill, L. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3664930 · Cited by 221 · see citing works
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language UnderstandingPreprint
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05387 · Cited by 215 · see citing works
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2023) · doi:10.31235/osf.io/5ecfa · Cited by 213 · see citing works
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024) · Political Analysis · doi:10.1017/pan.2024.5 · Cited by 209 · see citing works
Tackman, A. M., Sbarra, D. A., Carey, A. L., Donnellan, M. B., Horn, A. B., Holtzman, N. S., Edwards, T. S., Pennebaker, J. W., & Mehl, M. R. (2018) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000187 · Cited by 208 · see citing works
Optimization of sentiment analysis using machine learning classifiers
Singh, J., Singh, G., & Singh, R. (2017) · Human-centric Computing and Information Sciences · doi:10.1186/s13673-017-0116-3 · Cited by 192 · see citing works
From Text to Thought: How Analyzing Language Can Advance Psychological Science
Jackson, J. C., Watts, J., List, J., Puryear, C., Drabble, R., & Lindquist, K. A. (2021) · Perspectives on Psychological Science · doi:10.1177/17456916211004899 · Cited by 183 · see citing works
New Modes of Learning Enabled by AI Chatbots: Three Methods and Assignments
Mollick, E. R., & Mollick, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4300783 · Cited by 176 · cited 1× · see citing works
“GPT has been lauded for its ability to exhibit human - level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”
BackgroundConservatives report, but liberals display, greater happiness
Wojcik, S. P., Hovasapian, A., Graham, J., Motyl, M. S., & Ditto, P. H. (2015) · Science · doi:10.1126/science.1260817 · Cited by 167 · see citing works
Amin, M. M., Cambria, E., & Schuller, B. W. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3254179 · Cited by 162 · see citing works
Törnberg, P. (2023) · arXiv · doi:10.48550/arxiv.2304.06588 · Cited by 152 · see citing works
Atari, M., Xue, M. J., Park, P. S., Blasi, D. E., & Henrich, J. (2023) · doi:10.31234/osf.io/5b26t · Cited by 151 · see citing works
Mohammad, S. M., & Turney, P. D. (2013) · NPARC · doi:10.4224/21270984 · Cited by 136 · see citing works
Cultural influences on word meanings revealed through large-scale semantic alignment
Thompson, B., Roberts, S. G., & Lupyan, G. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-0924-8 · Cited by 134 · see citing works
GPT is an effective tool for multilingual psychological text analysisPreprint
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C., & Bavel, J. J. V. (2023) · doi:10.31234/osf.io/sekf5 · Cited by 129 · see citing works
Self-citationSong, H., Tolochko, P., Eberl, J.-M., Eisele, O., Greussing, E., Heidenreich, T., Lind, F., Galyga, S., & Boomgaarden, H. G. (2020) · Political Communication · doi:10.1080/10584609.2020.1723752 · Cited by 123 · see citing works
Schöne, J. P., Parkinson, B., & Goldenberg, A. (2021) · Affective Science · doi:10.1007/s42761-021-00057-7 · Cited by 114 · see citing works
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · cited 3× · see citing works
“For instance, one paper found that fine - tuned GPT outperformed even fine - tuned BERT at detecting moral foundations (73).”
BackgroundIncivility is rising among American politicians on twitter
Frimer, J. A., Aujla, H., Feinberg, M., Skitka, L. J., Aquino, K., Eichstaedt, J. C., & Willer, R. (2023) · Social Psychological and Personality Science · doi:10.1177/19485506221083811 · Cited by 110 · see citing works
LLM-generated messages can persuade humans on policy issues
Bai, H., Voelkel, J. G., Muldowney, S., Eichstaedt, J. C., & Willer, R. (2025) · Nature Communications · doi:10.1038/s41467-025-61345-5 · Cited by 110 · see citing works
Emotion Classification on Indonesian Twitter Dataset
Saputri, M. S., Mahendra, R., & Adriani, M. (2018) · International Conference on Asian Language Processing · doi:10.1109/ialp.2018.8629262 · Cited by 100 · see citing works
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100 · cited 1× · see citing works
“Other work has found that GPT is biased toward responses on cognitive tasks that are similar to those of WEIRD populations (78).”
Self-citationHistorical representations of social groups across 200 years of word embeddings from Google Books
Charlesworth, T., Caliskan, A., & Banaji, M. (2022) · Proceedings of the National Academy of Sciences of the United States of America · doi:10.1073/pnas.2121798119 · Cited by 98 · see citing works
Sterling, J. L., Jost, J. T., & Bonneau, R. (2020) · Journal of Personality and Social Psychology · doi:10.1037/pspp0000275 · Cited by 95 · see citing works
Antypas, D., Preece, A. D., & Camacho-Collados, J. (2023) · Online Social Networks and Media · doi:10.1016/j.osnem.2023.100242 · Cited by 93 · see citing works
Troll and divide: the language of online polarization
Simchon, A., Brady, W. J., & Van Bavel, J. J. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac019 · Cited by 85 · see citing works
Self-citationHistorical language records reveal a surge of cognitive distortions in recent decades
Bollen, J., ten Thij, M., Breithaupt, F. A., Barron, A. T. J., Rutter, L. A., Lorenzo‐Luaces, L., & Scheffer, M. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2102061118 · Cited by 73 · see citing works
Ashokkumar, A., & Pennebaker, J. W. (2021) · Science Advances · doi:10.1126/sciadv.abg7843 · Cited by 67 · see citing works
The rise and fall of rationality in language
Scheffer, M., van de Leemput, I. A., Weinans, E., & Bollen, J. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2107848118 · Cited by 60 · see citing works
People think that social media platforms do (but should not) amplify divisive content
Rathje, S., Robertson, C., Brady, W. J., & Van Bavel, J. J. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231190392 · Cited by 59 · see citing works
Self-citationMoral concerns are differentially observable in language
Kennedy, B., Atari, M., Davani, A. M., Hoover, J., Omrani, A., Graham, J., & Dehghani, M. (2021) · Cognition · doi:10.1016/j.cognition.2021.104696 · Cited by 57 · see citing works
Text Analysis for Psychology: Methods, Principles, and PracticesPreprint
Kennedy, B., Ashokkumar, A., Boyd, R. L., & Dehghani, M. (2021) · doi:10.31234/osf.io/h2b8t · Cited by 57 · see citing works
How social learning amplifies moral outrage expression in online social networksPreprint
Brady, W. J., McLoughlin, K. L., Doan, T. N., et al. (2021) · doi:10.31234/osf.io/gf7t5 · Cited by 57 · see citing works
Yu, X., Wojcieszak, M., & Casas, A. (2023) · Political Behavior · doi:10.1007/s11109-022-09850-x · Cited by 55 · see citing works
Negative expressions are shared more on Twitter for public figures than for ordinary users
Schöne, J. P., García, D., Parkinson, B., & Goldenberg, A. (2023) · PNAS Nexus · doi:10.1093/pnasnexus/pgad219 · Cited by 54 · see citing works
Extremists on the Left and Right Use Angry, Negative Language
Frimer, J. A., Brandt, M. J., Melton, Z. J., & Motyl, M. S. (2018) · Personality and Social Psychology Bulletin · doi:10.1177/0146167218809705 · Cited by 51 · see citing works
A Bibliometric Review of Large Language Models Research from 2017 to 2023Preprint
Fan, L., Li, L., Ma, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.02020 · Cited by 49 · see citing works
Key language markers of depression on social media depend on race
Rai, S., Stade, E. C., Giorgi, S., Francisco, A., Ungar, L., Curtis, B. L., & Guntuku, S. C. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2319837121 · Cited by 45 · see citing works
Should large language models replace human participants?Preprint
Crockett, M. J., & Messeri, L. (2023) · doi:10.31234/osf.io/4zdx9 · Cited by 43 · see citing works
On the rise of fear speech in online social media
Saha, P., Garimella, K. K., Kalyan, N. K., Pandey, S. K., Meher, P. M., Mathew, B., & Mukherjee, A. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2212270120 · Cited by 40 · see citing works
Inducing anxiety in large language models increases exploration and biasPreprint
Coda-Forno, J., Witte, K., Jagadish, A. K., Binz, M., Akata, Z., & Schulz, E. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.11111 · Cited by 39 · see citing works
Artificial Intelligence Can Persuade Humans on Political IssuesPreprint
Bai, H., Voelkel, J. G., Eichstaedt, J. C., & Willer, R. (2023) · doi:10.31219/osf.io/stakv · Cited by 38 · see citing works
Tracking group identity through natural language within groups
Ashokkumar, A., & Pennebaker, J. W. (2022) · PNAS Nexus · doi:10.1093/pnasnexus/pgac022 · Cited by 36 · see citing works
Low-resource Languages: A Review of Past Work and Future ChallengesPreprint
Magueresse, A., Carles, V., & Heetderks, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07264 · Cited by 36 · see citing works
SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)Preprint
Zampieri, M., Malmasi, S., Nakov, P., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.08983 · Cited by 34 · see citing works
Understanding the Behaviors of Toxic Accounts on Reddit
Kumar, D., Hancock, J. T., Thomas, K., & Durumeric, Z. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583522 · Cited by 30 · see citing works
SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)
Muhammad, S. H., Abdulmumin, I., Yimam, S. M., Adelani, D. I., Ahmad, I. S., Ousidhoum, N. D., Ayele, A. A., Mohammad, S. M., Beloucif, M., & Ruder, S. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.315 · Cited by 30 · see citing works
The Moral Foundations Reddit Corpus
Trager, J. P., Ziabari, A. S., Rahmati, E., Davani, A. M., Golazizian, P., Karimi-Malekabadi, F., Omrani, A., Li, Z., Kennedy, B., Chochlakis, G., et al. (2026) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2b6xmbq3kphf · Cited by 22 · see citing works
Weak ties strengthen anger contagion in social media
Fan, R., Xu, K., & Zhao, J. (2020) · arXiv · doi:10.48550/arxiv.2005.01924 · Cited by 18 · see citing works
The paucity of morality in everyday talk
Atari, M., Mehl, M. R., Graham, J., Doris, J. M., Schwarz, N., Davani, A. M., Omrani, A., Kennedy, B., Gonzalez, E., Jafarzadeh, N., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-32711-4 · Cited by 17 · see citing works
ChatGPT vs Gemini vs LLaMA on Multilingual Sentiment AnalysisPreprint
Buscemi, A., & Proverbio, D. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.01715 · Cited by 17 · see citing works
Zampieri, M., Nakov, P., Rosenthal, S., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07235 · Cited by 16 · see citing works
Using Facebook language to predict and describe excessive alcohol use
Jose, R., Matero, M., Sherman, G. T., Curtis, B. L., Giorgi, S., Schwartz, H. A., & Ungar, L. (2022) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.14807 · Cited by 14 · see citing works
Artificial Intelligence Chatbots Mimic Human Collective Behaviour
He, J., Wallis, F., Gvirtz, A., & Rathje, S. (2024) · doi:10.21203/rs.3.rs-3096289/v2 · Cited by 4 · see citing works
Self-citationGenerative Language Models Exhibit Social Identity BiasesPreprint
Hu, T., Kyrychenko, Y., Rathje, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.15819 · Cited by 4 · see citing works
Self-citationSemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)Preprint
Muhammad, S. H., Abdulmumin, I., Yimam, S. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.06845 · Cited by 4 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 3
- A. Vaswani “Attention is all you need” in Proceedings of the 31st International Conference on Neural Information Processing Systems (Curran Associates Inc. Red Hook NY 2017) pp. 6000–6010.
- E. Hoes S. Altay J. Bermeo Using ChatGPT to fight misinformation: ChatGPT nails 72% of 12 000 verified claims. PsyArXiv [Preprint] (2023). https://osf.io/preprints/psyarxiv/qnjkf (Accessed 30 April 2023).
- R. Marjieh I. Sucholutsky P. van Rijn N. Jacoby T. L. Griffiths “What language reveals about perception: Distilling psychophysical knowledge from large language models” in Proceedings of the Annual Meeting of the Cognitive Science Society (Cognitive Science Society 2023) vol. 45.
Cited by
Sorted by citations
The impact of generative artificial intelligence on socioeconomic inequalities and policy making
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., Bonnefon, J.-F., Brañas-Garza, P., Butera, L., Douglas, K. M., et al. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae191 · Cited by 371 · see citing works
Self-citationSentiment Analysis in the Age of Generative AI
Krugmann, J. O., & Hartmann, J. (2024) · Customer Needs and Solutions · doi:10.1007/s40547-024-00143-4 · Cited by 189 · see citing works
“The emergence of Generative AI is revolutionizing the marketing landscape through its dual role : it serves to both generate [9, 22, 55] and analyze content [36, 38, 41, 54, 66].”
InfluentialUsing natural language processing to analyse text data in behavioural science
Feuerriegel, S., Maarouf, A., Bär, D., Geissler, D., Schweisthal, J., Pröllochs, N., Robertson, C. E., Rathje, S., Hartmann, J., Mohammad, S. M., et al. (2025) · Nature Reviews Psychology · doi:10.1038/s44159-024-00392-z · Cited by 140 · see citing works
Self-citationZhang, R., Li, H., Meng, H., Zhan, J., Gan, H., & Lee, Y.-C. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713429 · Cited by 123 · see citing works
“The LLM’s output included the category of harmful AI behavior and a concise reason, limited to 20 words.”
Generative language models exhibit social identity biases
Hu, T., Kyrychenko, Y., Rathje, S., Collier, N., van der Linden, S., & Roozenbeek, J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00741-1 · Cited by 100 · see citing works
“GPT-2-Large Positive .5106 .9375 2 Results”
Self-citationQuantifying the prevalence and impact of overreaching causal claims in social science
Isch, C., Dörr, T., Fasching, N., Jennings, G., & Watts, D. J. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02553-x
Characterizing AI-Generated Misinformation on Social Media
Drolsbach, C., Demirel, E., & Pröllochs, N. (2025) · arXiv · doi:10.48550/arxiv.2505.10266
“To effectively address this emerging threat, researchers, platforms, and policymakers will need to develop new strategies and countermeasures that account for the unique properties of AI-generated misinformation.”
- Authors
- Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J.
- Affiliations
- New York University (US)
- Princeton University (US)
- Department of Psychology, New York University, New York, NY 10003
- Corresponding author
- Rathje, Steve
- Mirea, Dan-Mircea
- ORCID
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2024-08-12
- Volume
- 121
- Issue
- 34
- Pages
- e2308950121-e2308950121
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC11348013
- OpenAlex
- W4401506443
- Semantic Scholar
- 89ec06ea4230e1c7ba1cb3dc8eaceef4472f5d9c
- Cited by
- 297
- References
- 94
- Citations by year
- 2026: 129 · 2025: 127 · 2024: 36 · 2023: 5
- FWCI
- 43.09
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Canadian Government | Natural Sciences and Engineering Research Council of Canada (567554-2022)
- Gates Cambridge Scholarship (OPP1144)
- Russell Sage Foundation (G-2110-33990)
- HHS | NIH | National Institute of Mental Health (R01MH119511)
- Templeton World Charity Foundation (TWCF-31570)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Psychology › Social Psychology › Mental Health via Writing
- Keywords
- large language models, GPT, multilingual text analysis, sentiment analysis, emotion recognition, moral foundations, cross-linguistic research, tweets, news headlines
- MeSH terms
- Machine Learning, Emotions, Humans, Language, Natural Language Processing, Multilingualism, Social Media
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
