Can ChatGPT accurately identify the position of parties? A validation study with an expert survey in France
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
We evaluate whether ChatGPT can be used to estimate the ideological positions of parties in real time. Compared to other methods, ChatGPT can be used at all times, with minimal cost, no technical skills, and for all parties, including new ones. In a validation exercise, we asked ChatGPT and a group of local experts to simultaneously estimate the positions of French political parties in the context of the 2024 European election on a series of issues from the Chapel Hill Expert Survey. We find that ChatGPT’s estimations are generally close to those of the expert survey, especially for major and salient political issues (left-right, European integration, immigration). However, it provides less accurate estimates on fringe issues (free market, asylum policies, urban–rural interests) and specific party characteristics (within-party division, issue blurring). Thus, ChatGPT shows strong potential as an alternative data source for estimating parties’ positions but should be used with caution.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Bol and Bono simply asked Chat-GPT to answer the CHES expert surveys questions but did ground these in manifestos and applied them only to eight French parties for one election (Bol & Bono, 2025).”
From Using large language models to analyze political texts through natural language understanding (2026) · cites this paper as Bol & Bono, 2025
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Party Policy in Modern Democracies
Benoit, K., & Laver, M. (2006) · doi:10.4324/9780203028179 · Cited by 1,616
Extracting Policy Positions from Political Texts Using Words as Data
LAVER, M., BENOIT, K., & GARRY, J. (2003) · American Political Science Review · doi:10.1017/s0003055403000698 · Cited by 1,377 · cited 1×
“…relied on a method that involves the-matically coding the electoral manifestos or other pieces of party communication, either manually, as in the Comparative manifesto project (Budge et al., 2001), or automatically using text analysis techniques (Laver et al., 2003; Slapin and Proksch, 2008).”
A Scaling Model for Estimating Time‐Series Party Positions from Texts
Slapin, J. B., & Proksch, S. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00338.x · Cited by 684
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369
Can Generative AI Improve Social Science?
Bail, C. A. (2023) · doi:10.31235/osf.io/rwtzs · Cited by 369
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329
Measuring Democratic Backsliding
Little, A. T., & Meng, A. (2024) · PS: Political Science & Politics · doi:10.1017/s104909652300063x · Cited by 121 · cited 1×
“…a wide range of experts would in principle minimize this risk (Knutsen et al., 2024), it is not impossible that these experts share the same biases, given that political scientists as a group tend to have similar socio-demographic characteristics and political preferences (Little and Meng, 2024).”
Mellon, J., Bailey, J., Scott, R., Breckwoldt, J., Miori, M., & Schmedeman, P. (2024) · Research & Politics · doi:10.1177/20531680241231468 · Cited by 61 · cited 1×
“In political science, some have, for example, demonstrated its usefulness in coding responses to open-ended survey questions (Mellon et al., 2024) or in creating realistic and natural placebo vignettes in survey experiments (Porter and Velez, 2022).”
Research works printed but not linked: 22
- 10.1017/S0007123414000155 · DOI · Paper page
- 10.1017/S0007123409000635 · DOI · Paper page
- 10.4135/9781473957978 · DOI
- 10.1093/oso/9780199244003.001.0001 · DOI · Paper page
- 10.1111/j.1475-6765.1984.tb00080.x · DOI · Paper page
- Downs A (1957) An Economic Theory of Democracy. Manhattan, NY: Harper & Brothers.
- 10.1177/1354068810372100 · DOI · Paper page
- 10.1080/19331681.2024.2444587 · DOI
- 10.1111/1467-9248.12015 · DOI · Paper page
- 10.1145/3600211.3604672 · DOI · Paper page
- 10.1111/j.1475-6765.2009.01912.x · DOI · Paper page
- 10.1016/j.electstud.2021.102420 · DOI · Paper page
- 10.1017/S104909652300077X · DOI
- 10.1016/j.electstud.2006.03.007 · DOI · Paper page
- 10.1017/S0007123403000322 · DOI · Paper page
- 10.1177/0010414020938081 · DOI · Paper page
- 10.1017/gov.2018.32 · DOI
- 10.4159/harvard.9780674429147 · DOI · Paper page
- 10.1017/pan.2021.16 · DOI
- Reinl A-K, Braun D (2023) European Election Studies 2019: Manifesto Project. Cologne: GESIS. ZA7891 Data file Version 1.0.0.
- 10.1111/1475-6765.12032 · DOI · Paper page
- 10.1177/1465116507073284 · DOI · Paper page
Cited by
Using large language models to analyze political texts through natural language understanding
Benoit, K., De Marchi, S., Laver, C., Laver, M., & Ma, J. (2026) · American Journal of Political Science · doi:10.1111/ajps.70050 · Cited by 37
Metadata
- Authors
- Bol, D., & Bono, P.-H.
- Affiliations
- CEVIPOF, Sciences Po Paris
- ORCID
- Bol, Damien
- Venue
- Research & Politics
- Publisher
- SAGE Publications
- Published
- 2025-04
- Volume
- 12
- Issue
- 2
- ISSN
- 2053-1680
- Cited by
- 5
- References
- 8
- Language
- en
- Licence
- creativecommons.org/licenses/by-nc/4.0/ (unspecified)
- journals.sagepub.com/page/policies/text-and-data-mining-license (tdm)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.