CoChat

Using large language models to analyze political texts through natural language understanding

Open access
Benoit, Kenneth, De Marchi, Scott, Laver, Conor, Laver, Michael, Ma, Jinshuai · 2026 · American Journal of Political Science

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.

Abstract

Large language models (LLMs) offer scalable alternatives to human experts when analyzing political texts for meaning , using natural language understanding (NLU). Qualitative NLU methods relying on human experts are severely limited by cost and scalability. Statistical text‐as‐data methods are scalable but rely on strong and often unrealistic assumptions. We propose a systematic, scalable, and replicable method that can extend existing qualitative and quantitative approaches by using LLMs to interpret texts meaningfully rather than as mere data. Our ensemble means of LLM‐generated estimates of party positions on six key issue dimensions correlate highly with equivalent mean ratings by country specialists. When applied to coalition policy declarations, LLM estimates align more closely with standard models of government formation than hand‐coded estimates. We conclude with a discussion of the profound implications of modern LLMs for political text analysis.

Overview

CoChat Summary

TL;DR

The paper proposes an ensemble LLM workflow that summarizes political texts and scores issue positions, and finds LLM estimates correlate highly with expert survey benchmarks and outperform hand-coded MP-style estimates on coalition policy declarations. It argues this offers a scalable, replicable alternative to both qualitative expert coding and traditional text-as-data methods.

Key findings

  • LLM ensemble means correlate with expert survey means at roughly .87–.92 for most issue dimensions, near the simulated expert–expert upper bound of about .90; decentralization is lower, with expert–expert correlations at .78 and LLM–expert correlations substantially below the other dimensions.
  • Reliability is high: overall intra-LLM ICC = .91 and Krippendorff’s alpha = .90; inter-LLM ICC = .91 and alpha = .90, though environment (.76–.80 ICC) and decentralization (.66–.69 ICC) are lower.
  • For 23 coalition agreements, LLM estimates fall within the range of member-party positions in 64% of cases overall versus 38% for Klüver/MP logit scores; exceptions include decentralization (.43 LLM vs .61 hand-coded).
  • Reproducibility after a 3-month wait using updated proprietary LLMs produced correlations of .92–.98 except decentralization at .69; open-weight LLM ensembles correlated about .95 with the proprietary ensemble and performed only slightly worse against expert benchmarks.

Methods

  • Corpus: 235 party manifestos in 21 languages from the Manifesto Project, plus 23 coalition agreements from Klüver et al. (2023), scored on six issue dimensions: economic, social, immigration, EU, environment, and decentralization.
  • Workflow: three proprietary LLMs (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) generated 300–400-word English summaries per issue, then scored summaries on 7-point scales under zero-shot and few-shot conditions in a “3 × 3” design, yielding 18 LLM scores per manifesto-dimension and 25,380 scores in total.
  • Validation: LLM ensemble means were compared with expert survey benchmarks (Laver–Hunt, Benoit–Laver, CHES) and MP logit scales; reliability was assessed with ICC and Krippendorff’s alpha, and predictive validity was tested against the expectation that coalition policy should lie within member-party ranges.

Limitations

  • Stated: true party positions are fundamentally unobservable, so there is no ground truth; expert surveys are used as a surrogate, and bias relative to true positions cannot be known. The paper also notes lower performance for decentralization and environment, which it attributes partly to structural bias in manifestos and lack of variation, and it describes a trade-off between pristine estimates and missing data.
  • Stated: LLMs remain “black box” tools, data leakage is a legitimate concern, and residual stochastic variation persisted despite temperature zero, top-p = 1.0, and fixed seeds where available. The authors also call the paper a demonstration rather than a technical analysis of LLMs.
  • Evident from the text: the validation is limited to party manifestos and 23 coalition agreements, the baseline uses proprietary models, and the method relies on English-language summaries even for non-English originals; these scope conditions are not framed by the authors as limitations but follow from the described design.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1111/ajps.70050

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
CC-BY · From the PDF
F I G U R E 2 Distributions of Manifesto Project (MP)-estimated manifesto percentages on (de)centralization for manifestos in the LLM corpus. Notes : This boxplot of percentages of mentions measured by the MP shows that almost no manifesto has quasi-sentences advocating the centralization of decision-making.
F I G U R E 2 Distributions of Manifesto Project (MP)-estimated manifesto percentages on (de)centralization for manifestos in the LLM corpus. Notes : This boxplot of percentages of mentions measured by the MP shows that almost no manifesto has quasi-sentences advocating the centralization of decision-making.CC-BY · From the PDF
F I G U R E 3 Distributions of LLM 18-score ensemble, expert and MP scores, by issue. Notes : Each panel shows, for the relevant issue dimension, a kernel density plot summarizing the distribution of scores estimated by: the LLM ensemble; the expert survey benchmark; a logit scale derived from MP data.
F I G U R E 3 Distributions of LLM 18-score ensemble, expert and MP scores, by issue. Notes : Each panel shows, for the relevant issue dimension, a kernel density plot summarizing the distribution of scores estimated by: the LLM ensemble; the expert survey benchmark; a logit scale derived from MP data.CC-BY · From the PDF

Verification

Claims the paper makes

0 Valid · 16 Not valid · 27 Not checked · 120 Own results and interpretations

74 more claims

Create a free account to see all 163 claims and to verify the 26 references still open.

See all claims — free account
  • It may state two separate claims as one. 44
  • Probably not a substantial claim. 51
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 46 references

Research works printed but not linked: 5

  1. 10.1016/j.elecstud.2021.102420 · doi:10.1016/j.elecstud.2021.102420
  2. 10.1177/2053268016643346 · doi:10.1177/2053268016643346
  3. Mikolov Tomáš, 2013, Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 746
  4. National Academies of Sciences, Engineering, and Medicine, 2019, Reproducibility and Replicability in Science
  5. Snow Rion, 2008, Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, 254

Cited by

See all 37

Metadata

Authors
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J.
Affiliations
  • Singapore Management University (SG)
  • Duke University (US)
  • Northeastern University (US)
  • New York University (US)
  • London School of Economics and Political Science (GB)
Corresponding author
Benoit, Kenneth
Venue
American Journal of Political Science
Publisher
Wiley
Published
2026-03-19
Volume
70
Issue
4
Pages
1589-1605
ISSN
  • 0092-5853 (ISSN-L)
  • 1540-5907
OpenAlex
W7138869444
Cited by
37
References
42
Citations by year
2026: 36
FWCI
114.86
Type
journal article
Language
en
Open access
Licence
Locations
  • American Journal of Political Science (publishedVersion, cc-by)
  • Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by)
  • London School of Economics and Political Science Research Online (London School of Economics and Political Science) (submittedVersion, public-domain)
Topics
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Social Sciences › Law › Artificial Intelligence in Law
Keywords
large language models, political text analysis, natural language understanding, party positions, issue dimensions, government formation, scalability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-21, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-21 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.