How to train your stochastic parrot: large language models for political texts
The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
We demonstrate how few-shot prompts to large language models (LLMs) can be effectively applied to a wide range of text-as-data tasks in political science—including sentiment analysis, document scaling, and topic modeling. In a series of pre-registered analyses, this approach outperforms conventional supervised learning methods without the need for extensive data pre-processing or large sets of labeled training data. Performance is comparable to expert and crowd-coding methods at a fraction of the cost. We propose a set of best practices for adapting these models to social science measurement tasks, and develop an open-source software package for researchers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Applying LLMs to tasks from Benoit et al. (2016), both Le Mens and Gallego and Ornstein et al. used LLMs to replicate crowd-sourced and expert classifications of policy content at the sentence level (Le Mens & Gallego, 2025;Ornstein et al., 2025).”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Ornstein et al., 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 4×
“2 We conclude with a discussion on whether the performance gains we document are worth the potential dangers associated with unprincipled use of these models (Strubell et al. , 2019; Abid et al. , 2021; Bender et al. , 2021; Spirling, 2023).”
InfluentialLiberals and conservatives rely on different sets of moral foundations.
Graham, J., Haidt, J., & Nosek, B. A. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015141 · Cited by 4,560 · cited 1×
“Consistent with Moral Foundations Theory (Graham et al. , 2009), we might expect conservatives to emphasize virtues like loyalty, patriotism, and hard work in their speeches, while liberals would be more likely to emphasize fairness, compassion, and charity.”
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · arXiv · doi:10.48550/arxiv.1906.02243 · Cited by 3,735
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297 · cited 3×
“One of the most exciting new developments in our field has been the explosion of methods for automating this process, meth-ods we broadly call “ text-as-data ” (Grimmer and Stewart, 2013).”
InfluentialSemantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2021) · arXiv · doi:10.48550/arxiv.2109.07958 · Cited by 899
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
Hara, K., Adams, A., Milland, K., Savage, S., Callison-Burch, C., & Bigham, J. P. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174023 · Cited by 497 · cited 1×
“Studies suggest that people performing “ human intelligence tasks ” on sites like MTurk , CrowdFlower , Clickworker , and Toluna earn a median hourly wage of roughly $2/h, and only 4 percent earn more than the US federal minimum wage of $7.25 (Hara et al. , 2018).”
InfluentialKatz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420
Local News and National Politics
MARTIN, G. J., & McCRAIN, J. (2019) · American Political Science Review · doi:10.1017/s0003055418000965 · Cited by 331
Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges
Wilkerson, J., & Casas, A. (2017) · Annual Review of Political Science · doi:10.1146/annurev-polisci-052615-025542 · Cited by 300 · cited 5×
“Though Wilkerson and Casas (2017) focus their analysis on partisan differences in substantive topics (e.g., education, defense, agriculture, etc.), one might imagine a large number of other interesting ways to categorize the speeches.”
InfluentialTweetNLP: Cutting-Edge Natural Language Processing for Social Media
Camacho-collados, J., Rezaee, K., Riahi, T., Ushio, A., Loureiro, D., Antypas, D., Boisson, J., Espinosa Anke, L., Liu, F., & Martínez Cámara, E. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2022.emnlp-demos.5 · Cited by 142
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104
U.S. Treaty Making with American Indians: Institutional Change and Relative Power, 1784–1911
Spirling, A. (2012) · American Journal of Political Science · doi:10.1111/j.1540-5907.2011.00558.x · Cited by 98 · cited 1×
“Whether it ’ s measuring the ideology expressed in party manifestos (Lowe et al. , 2011), the harshness of treaty provisions (Spirling, 2012), polarization in legislative speeches (Peterson and Spirling, 2018), partisan slant in television news coverage (Martin and McCrain, 2019), political…”
Research works printed but not linked: 23
- Le Mens, G and Gallego, A (2023) Scaling Political Texts with ChatGPT. arXiv:2311.16639. · arXiv
- 10.1146/annurev-polisci-051120-111443 · DOI
- 10.1017/S0003055416000058 · DOI · Paper page
- Widmann, 2022, Creating and comparing dictionary, word embedding, and transformer-based models to measure discrete emotions in German political text, Political Analysis, 31, 1
- 10.1146/annurev-polisci-053119-015921 · DOI · Paper page
- 10.1111/ajps.12779 · DOI
- Ying, 2021, Topics, concepts, and measurement: a crowdsourced procedure for validating topics as measures, Political Analysis, 30, 1
- Firth, J (ed) (1957) Studies in Linguistic Analysis. Special Volume of the Philological Society. Blackwell, Oxford, repr edition.
- 10.2139/ssrn.4389233 · DOI · Paper page
- 10.1017/9781108769495 · DOI
- 10.1017/pan.2019.32 · DOI
- 10.1111/j.1939-9162.2010.00006.x · DOI · Paper page
- 10.1111/ajps.12423 · DOI
- 10.1086/715162 · DOI · Paper page
- 10.1038/s42256-021-00359-2 · DOI
- 10.1017/S0003055417000302 · DOI
- Blei, 2003, Latent Dirichlet allocation, Journal of Machine Learning Research, 3, 993
- Vaswani, A , Shazeer, N , Parmar, N , Uszkoreit, J , Jones, L , Gomez, AN , Kaiser, L and Polosukhin, I (2017) Attention is all you need. arXiv:1706.03762 [cs]. · arXiv
- Grimmer, 2022, Text as Data: A New Framework for Machine Learning and the Social Sciences
- Go, 2009, Twitter sentiment classification using distant supervision, CS224N Project Report, Stanford, 1, 1
- 10.1609/aimag.v41i4.5304 · DOI
- Surowiecki, 2004, The Wisdom of Crowds: Why the Many are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations
- 10.1017/pan.2017.39 · DOI · Paper page
Cited by
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024) · Political Analysis · doi:10.1017/pan.2024.5 · Cited by 209
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2023) · doi:10.31235/osf.io/5ecfa · Cited by 213
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
Metadata
- Authors
- Ornstein, J. T., Blasingame, E. N., & Truscott, J. S.
- ORCID
- Ornstein, Joseph T.
- Venue
- Political Science Research and Methods
- Publisher
- Cambridge University Press (CUP)
- Published
- 2025-01-14
- Volume
- 13
- Issue
- 2
- Pages
- 264-281
- ISSN
- 2049-8470
- 2049-8489
- Cited by
- 44
- References
- 18
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.