Using large language models to analyze political texts through natural language understanding
Open accessThe DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Abstract
Large language models (LLMs) offer scalable alternatives to human experts when analyzing political texts for meaning , using natural language understanding (NLU). Qualitative NLU methods relying on human experts are severely limited by cost and scalability. Statistical text‐as‐data methods are scalable but rely on strong and often unrealistic assumptions. We propose a systematic, scalable, and replicable method that can extend existing qualitative and quantitative approaches by using LLMs to interpret texts meaningfully rather than as mere data. Our ensemble means of LLM‐generated estimates of party positions on six key issue dimensions correlate highly with equivalent mean ratings by country specialists. When applied to coalition policy declarations, LLM estimates align more closely with standard models of government formation than hand‐coded estimates. We conclude with a discussion of the profound implications of modern LLMs for political text analysis.
Overview
CoChat Summary
TL;DR
The paper proposes an ensemble LLM workflow that summarizes political texts and scores issue positions, and finds LLM estimates correlate highly with expert survey benchmarks and outperform hand-coded MP-style estimates on coalition policy declarations. It argues this offers a scalable, replicable alternative to both qualitative expert coding and traditional text-as-data methods.
Key findings
- LLM ensemble means correlate with expert survey means at roughly .87–.92 for most issue dimensions, near the simulated expert–expert upper bound of about .90; decentralization is lower, with expert–expert correlations at .78 and LLM–expert correlations substantially below the other dimensions.
- Reliability is high: overall intra-LLM ICC = .91 and Krippendorff’s alpha = .90; inter-LLM ICC = .91 and alpha = .90, though environment (.76–.80 ICC) and decentralization (.66–.69 ICC) are lower.
- For 23 coalition agreements, LLM estimates fall within the range of member-party positions in 64% of cases overall versus 38% for Klüver/MP logit scores; exceptions include decentralization (.43 LLM vs .61 hand-coded).
- Reproducibility after a 3-month wait using updated proprietary LLMs produced correlations of .92–.98 except decentralization at .69; open-weight LLM ensembles correlated about .95 with the proprietary ensemble and performed only slightly worse against expert benchmarks.
Methods
- Corpus: 235 party manifestos in 21 languages from the Manifesto Project, plus 23 coalition agreements from Klüver et al. (2023), scored on six issue dimensions: economic, social, immigration, EU, environment, and decentralization.
- Workflow: three proprietary LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) generated 300–400-word English summaries per issue, then scored summaries on 7-point scales under zero-shot and few-shot conditions in a “3 × 3” design, yielding 18 LLM scores per manifesto-dimension and 25,380 scores in total.
- Validation: LLM ensemble means were compared with expert survey benchmarks (Laver–Hunt, Benoit–Laver, CHES) and MP logit scales; reliability was assessed with ICC and Krippendorff’s alpha, and predictive validity was tested against the expectation that coalition policy should lie within member-party ranges.
Limitations
- Stated: true party positions are fundamentally unobservable, so there is no ground truth; expert surveys are used as a surrogate, and bias relative to true positions cannot be known. The paper also notes lower performance for decentralization and environment, which it attributes partly to structural bias in manifestos and lack of variation, and it describes a trade-off between pristine estimates and missing data.
- Stated: LLMs remain “black box” tools, data leakage is a legitimate concern, and residual stochastic variation persisted despite temperature zero, top-p = 1.0, and fixed seeds where available. The authors also call the paper a demonstration rather than a technical analysis of LLMs.
- Evident from the text: the validation is limited to party manifestos and 23 coalition agreements, the baseline uses proprietary models, and the method relies on English-language summaries even for non-English originals; these scope conditions are not framed by the authors as limitations but follow from the described design.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
0 Valid · 16 Not valid · 27 Not checked · 120 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Despite these measures, we observed small residual variation in some models' outputs across runs-likely due to stochastic processes within their generation pipelines or undocumented aspects of inference architecture.”
§ Using large language models to analyze political texts through natural language understanding > Choosing LLMs and software environment
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our ensemble means of LLMgenerated estimates of party positions on six key issue dimensions correlate highly with equivalent mean ratings by country specialists.”
§ Using large language models to analyze political texts through natural language understanding > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When applied to coalition policy declarations, LLM estimates align more closely with standard models of government formation than hand-coded estimates.”
§ Using large language models to analyze political texts through natural language understanding > Abstract
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A related and distinctive contribution of our approach is the production of concise policy summaries in the first stage of our analysis.”
§ Using large language models to analyze political texts through natural language understanding > CONCLUSION: THE REVOLUTIONARY POTENTIAL OF LLMS
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This lack of variation results, as Tables 5 and 6 show, in lower inter-LLM correlations for decentralization than for the other five issue dimensions.”
§ Using large language models to analyze political texts through natural language understanding > Decentralization: Do experts see issue positions not found in party manifestos?
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
74 more claims
Create a free account to see all 163 claims and to verify the 26 references still open.
- It may state two separate claims as one. 44
- Probably not a substantial claim. 51
- The quote only partly states this claim. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using large language models to analyze political texts through natural language understanding > Addressing potential concerns about "data leakage"
“Our prompts explicitly instructed the LLMs to base their answers solely on these summary texts, with no reference whatsoever to benchmark datasets (CHES or others).”
§ Using large language models to analyze political texts through natural language understanding > Scoring issue positions in text summaries
“For each issue, our few-shot scoring prompts supplement the zero-shot prompts with three examples-from the right, left and center of each dimension-paired with their corresponding expert survey scores.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Using large language models to analyze political texts through natural language understanding > 2. Large language model (LLM) authorization
“Prompt LLMs to generate 200-300-word English language summaries of each text's content relating to each of the six issue dimensions.”
§ Using large language models to analyze political texts through natural language understanding > Scoring issue positions in text summaries
“we first used each of Claude, GPT, and Gemini to generate 300-to 400-word summaries (in English) of the issue positions set out in the full native language text.”
Citations
Cites
Content Analysis: An Introduction to Its Methodology
Krippendorff, K. (2019) · doi:10.4135/9781071878781 · Cited by 15,448
Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts
Grimmer, J., & Stewart, B. M. (2013) · Political Analysis · doi:10.1093/pan/mps028 · Cited by 3,297
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · North American Chapter of the Association for Computational Linguistics · Cited by 2,912
Cheap and fast---but is it good?: evaluating non-expert annotations for natural language tasks
Snow, R., O'Connor, B., Jurafsky, D., & Ng, A. Y. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613751 · Cited by 1,887
Party Policy in Modern Democracies
Benoit, K., & Laver, M. (2006) · doi:10.4324/9780203028179 · Cited by 1,616
Self-citationLost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023) · arXiv · doi:10.48550/arxiv.2307.03172 · Cited by 1,599
Mapping Policy Preferences: Estimates for Parties, Electors, and Governments 1945-1998
Budge, I., Klingemann, H., Volkens, A., Bara, J. L., & Tanenbaum, E. (2001) · Medical Entomology and Zoology · Cited by 1,508
Mapping Policy Preferences: Estimates for Parties, Electors, and Governments 1945—1998
Budge, I., Klingemann, H.-D., Volkens, A., Bara, J., Tanenbaum, E., Fording, R. C., Hearl, D. J., Kim, H. M., Mcdonald, M. D., & Mendes, S. M. (2001) · doi:10.1093/oso/9780199244003.001.0001 · Cited by 1,260
Gentzkow, M., Kelly, B., & Taddy, M. (2019) · Journal of Economic Literature · doi:10.1257/jel.20181020 · Cited by 1,236
Research works printed but not linked: 5
- 10.1016/j.elecstud.2021.102420 · doi:10.1016/j.elecstud.2021.102420
- 10.1177/2053268016643346 · doi:10.1177/2053268016643346
- Mikolov Tomáš, 2013, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 746
- National Academies of Sciences, Engineering, and Medicine, 2019, Reproducibility and Replicability in Science
- Snow Rion, 2008, Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 254
Cited by
Measuring issue salience for political parties using LLMs
Benoit, K. R., & Laver, M. (2026) · West European Politics · doi:10.1080/01402382.2026.2637134 · Cited by 3
Self-citationPeez, A., & Scherzinger, J. (2026) · The Review of International Organizations · doi:10.1007/s11558-026-09620-y · Cited by 3
Licht, H. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.10882 · Cited by 2
Multicalibration for Unbiased Model-Based Prevalence EstimationPreprint
Linder, F., Leeper, T., Haimovich, D., Tax, N., Perini, L., & Vojnović, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2604.21549 · Cited by 1
Revisiting business associations in trade politics: a large language model scaling approach
Cézar, R. F. (2026) · Review of International Political Economy · doi:10.1080/09692290.2026.2706545 · Cited by 1
Metadata
- Authors
- Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J.
- Affiliations
- Singapore Management University (SG)
- Duke University (US)
- Northeastern University (US)
- New York University (US)
- London School of Economics and Political Science (GB)
- Corresponding author
- Benoit, Kenneth
- Venue
- American Journal of Political Science
- Publisher
- Wiley
- Published
- 2026-03-19
- Volume
- 70
- Issue
- 4
- Pages
- 1589-1605
- ISSN
- 0092-5853 (ISSN-L)
- 1540-5907
- OpenAlex
- W7138869444
- Cited by
- 37
- References
- 42
- Citations by year
- 2026: 36
- FWCI
- 114.86
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (vor)
- doi.wiley.com/10.1002/tdm_license_1.1 (tdm)
- Locations
- American Journal of Political Science (publishedVersion, cc-by)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
- London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion, public-domain)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › Law › Artificial Intelligence in Law
- Keywords
- large language models, political text analysis, natural language understanding, party positions, issue dimensions, government formation, scalability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using large language models to analyze political texts through natural language understanding. https://app.cochat.ai/papers/10.1111/ajps.70050
- The paper
- Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026). Using large language models to analyze political texts through natural language understanding. American Journal of Political Science. https://doi.org/10.1111/ajps.70050
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.