Large language models and political science
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Abstract
Large Language Models (LLMs) are a type of artificial intelligence that uses information from very large datasets to model the use of language and generate content. While LLMs like GPT-3 have been used widely in many applications, the recent public release of OpenAI's ChatGPT has opened more debate about the potential uses and abuses of LLMs. In this paper, we provide a brief introduction to LLMs and discuss their potential application in political science and political methodology. We use two examples of LLMs from our recent research to illustrate how LLMs open new areas of research. We conclude with a discussion of how researchers can use LLMs in their work, and issues that researchers need to be aware of regarding using LLMs in political science and political methodology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has led to their increasing adoption in social sciences where qualitative text analysis has traditionally been the dominant paradigm (see Karjus, 2025;Linegar et al., 2023).”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Linegar et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Social Media and Fake News in the 2016 Election
Allcott, H., & Gentzkow, M. (2017) · Journal of Economic Perspectives · doi:10.1257/jep.31.2.211 · Cited by 6,767 · cited 1×
“It is well-known that social media is a vector of fake news and misinformation that can threaten elections (Allcott and Gentzkow, 2017).”
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 3,832
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 2×
“For example, textual data has been used in political science to study political party manifestos (Laver and Garry, 2000), political speeches (Grimmer and Stewart, 2013), legislator communications with constituents (Grimmer, 2013), and social movements (Kann et al., 2023).”
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Robust Speech Recognition via Large-Scale Weak Supervision
Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., & Sutskever, I. (2022) · arXiv · doi:10.48550/arxiv.2212.04356 · Cited by 1,953
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Lambrecht, A., & Tucker, C. (2019) · Management Science · doi:10.1287/mnsc.2018.3093 · Cited by 854
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
Bias in Cable News: Persuasion and Polarization
Martin, G. J., & Yurukoglu, A. (2017) · American Economic Review · doi:10.1257/aer.20160812 · Cited by 666
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
MTEB: Massive Text Embedding Benchmark
Muennighoff, N., Tazi, N., Magne, L., & Reimers, N. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.148 · Cited by 548
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2016) · arXiv · doi:10.48550/arxiv.1610.02357 · Cited by 538
The Ugly Truth About Ourselves and Our Robot Creations: The Problem of Bias and Social Inequity
Howard, A., & Borenstein, J. (2018) · Science and Engineering Ethics · doi:10.1007/s11948-017-9975-2 · Cited by 337
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329
An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence
Osoba, O., & Welser, W. (2017) · doi:10.7249/rr1744 · Cited by 326
Kocielnik, R., Amershi, S., & Bennett, P. N. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300641 · Cited by 277
Self-citationUsing artificial intelligence in craft education: crafting with text-to-image generative models
Vartiainen, H., & Tedre, M. (2023) · Digital Creativity · doi:10.1080/14626268.2023.2174557 · Cited by 263
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
ChatGPT is not all you need. A state-of-the-art review of large generative AI models
Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023) · arXiv · doi:10.48550/arxiv.2301.04655 · Cited by 245
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, G., Arriaga, R. I., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2208.10264 · Cited by 161
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Crisan, A., Drouhard, M., Vig, J., & Rajani, N. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533108 · Cited by 101 · cited 1×
“Recent initiatives such as “Interactive Model Cards” ( Crisan et al., 2022), “AutoBiasTest” (Kocielnik et al., 2023b), and model written evaluations focus on interactive tools, that support live exploration of model capabilities and limitations using generated datasets or human-in-the-loop…”
Supporting Human-AI Collaboration in Auditing LLMs with LLMs
Rastogi, C., Ribeiro, M. T., King, N., Nori, H., & Amershi, S. (2023) · arXiv · doi:10.48550/arxiv.2304.09991 · Cited by 79
Delobelle, P., Tokpo, E., Calders, T., & Berendt, B. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.122 · Cited by 57
Discovering Language Model Behaviors with Model-Written Evaluations
Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.09251 · Cited by 53
TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models
Huang, Y., Zhang, Q., Y, P. S., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2306.11507 · Cited by 47 · cited 1×
“Benchmarking LLMs is crucial not just for performance assessment, but also for detecting and mitigating biases, and assessing user satisfaction and trust (Huang et al., 2023).”
GPT detectors are biased against non-native English writers
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2304.02819 · Cited by 40
Research works printed but not linked: 25
- Abid, 2021, Large language models associate Muslims with violence, Nat. Mach. Intell., 3, 461, 10.1038/s42256-021-00359-2 · DOI
- Alvarez, 2023, Generative AI and the Future of Elections
- Alvarez, 2022, How (not) to reproduce: Practical considerations to improve research transparency in political science, Polit. Sci. Polit., 55, 149, 10.1017/S1049096521001062 · DOI
- Borji, 2022, Generated faces in the wild: Quantitative comparison of Stable Diffusion, Midjourney and DALL-E 2, arXiv preprint arXiv:2210.00586, 10.48550/arXiv.2210.00586 · DOI · Paper page
- Buolamwini, 2018, “Gender shades: intersectional accuracy disparities in commercial gender classification,”, Proceedings of the 1st Conference on Fairness, Accountability and Transparency, volume 81 of Proceedings of Machine Learning Research, 77
- Chouldechova, 2018, “A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions,”, 134
- Denton, 2023, “Neurips 2023,”, Thirty-seventh Conference on Neural Information Processing Systems
- Gao, 2020, The pile: an 800GB dataset of diverse text for language modeling, arXiv preprint arXiv:2101.00027, 10.48550/arXiv.2101.00027 · DOI · Paper page
- Grimmer, 2013, Representational Style in Congress: What Legislators Say and Why It Matters, 10.1017/CBO9781139207782 · DOI · Paper page
- Holland, 2020, The dataset nutrition label, Data Protect. Privacy, 12, 1, 10.5040/9781509932771.ch-001 · DOI · Paper page
- Kann, 2023, Collective identity in collective action: evidence from the 2020 summer BLM protests, Front. Polit. Sci., 5, 1185633, 10.3389/fpos.2023.1185633 · DOI
- King, 1995, Replication, replication, Polit. Sci. Polit., 28, 444, 10.2307/420301 · DOI
- Kocielnik, “Can you label less by using out-of-domain data? Active & transfer learning with few-shot instructions,”, 22
- Kocielnik, BiasTestGPT: using ChatGPT for social bias testing of language models, arXiv preprint arXiv: 2302.07371 · arXiv
- Laver, 2000, Estimating policy positions from political texts, Am. J. Polit. Sci., 44, 619, 10.2307/2669268 · DOI · Paper page
- Li, 2023, “TROCR: transformer-based optical character recognition with pre-trained models,”, Proceedings of the AAAI Conference on Artificial Intelligence, 13094, 10.1609/aaai.v37i11.26538 · DOI · Paper page
- Mendelsohn, 2023, “From dogwhistles to bullhorns: unveiling coded rhetoric with language models,”, 10.18653/v1/2023.acl-long.845 · DOI
- Poole, 1985, A spatial model for legislative roll call analysis, Am. J. Polit. Sci., 29, 357, 10.2307/2111172 · DOI · Paper page
- Radford, 2021, “Learning transferable visual models from natural language supervision,”, 8748
- Schnabel, 2016, “Recommendations as treatments: debiasing learning and evaluation,”, 1670
- Srikanth, 2021, “Dynamic social media monitoring for fast-evolving online discussions,”, Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 3576
- van der Linden, 2023, Foolproof: Why Misinformation Infects our Minds and How to Build Immunity
- Xu, 2023, WizardLM: empowering large language models to follow complex instructions, arXiv preprint arXiv:2304.12244, 10.48550/arXiv.2304.12244 · arXiv · doi:10.48550/arxiv.2304.12244 · Paper page
- Zhang, One small step for generative AI, one giant leap for AGI: a complete survey on chatgpt in AIGC era, arXiv preprint arXiv:2304.06488, 10.48550/arXiv.2304.06488 · DOI
- Zhang, Text-to-image diffusion model in Generative AI: A survey, arXiv preprint arXiv:2303.07909, 10.48550/arXiv.2303.07909 · arXiv · doi:10.48550/arxiv.2303.07909 · Paper page
Cited by
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
Metadata
- Authors
- Linegar, M., Kocielnik, R., & Alvarez, R. M.
- Venue
- Frontiers in Political Science
- Publisher
- Frontiers Media SA
- Published
- 2023-10-16
- Volume
- 5
- ISSN
- 2673-3145
- Cited by
- 42
- References
- 34
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.