Comparing the persuasiveness of role-playing large language models and human experts on polarized U.S. political issues
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Advances in large language models (LLMs) could significantly disrupt political communication. In a large-scale pre-registered experiment ( n = 4955), we prompted GPT-4 to generate persuasive messages impersonating the language and beliefs of U.S. political parties—a technique we term “partisan role-play”—and directly compared their persuasiveness to that of human persuasion experts. In aggregate, the persuasive impact of role-playing messages generated by GPT-4 was not significantly different from that of non-role-playing messages. However, the persuasive impact of GPT-4 rivaled, and on some issues exceeded, that of the human experts. Taken together, our findings suggest that — contrary to popular concern — instructing current LLMs to role-play as partisans offers limited persuasive advantage, but also that current LLMs can rival and even exceed the persuasiveness of human experts. These results potentially portend widespread adoption of AI tools by persuasion campaigns, with important implications for the role of AI in politics and democracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 1×
“Crucially, despite multiple studies illustrating the significant impact of prompt design on model outputs (30–32), recent research still employs basic prompts to instruct models to generate persuasive messages (2, 7, 33).”
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 1×
“Recent research suggests that the most capable LLMs – trained on public corpora of human-generated text – can encode nuanced and fine-grained information about the ideas, attitudes, and socio-cultural contexts that characterize human attitudes and identities (9).”
Jeon, J., & Lee, S. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11834-1 · Cited by 696 · cited 1×
“Notably, role-play techniques have improved the performance and reasoning capabilities of LLMs across different benchmarks (13–15) and improved the contextual relevance of outputs (16, 17).”
Bailenson (2005) · doi:10.1111/j.1467-9280.2005.01619.x · Cited by 549
Role-Play with Large Language Models
Shanahan, M., McDonell, K., & Reynolds, L. (2023) · arXiv · doi:10.48550/arxiv.2305.16367 · Cited by 490
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380
Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign
Keller, F. B., Schoch, D., Stier, S., & Yang, J. (2020) · Political Communication · doi:10.1080/10584609.2019.1661888 · Cited by 378 · cited 1×
“Here, we use the largest publicly accessible LLM to date, GPT-4, to ask two related questions:”
Starbird, K., Arif, A., & Wilson, T. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359229 · Cited by 370
All the News That’s Fit to Fabricate: AI-Generated Text as a Tool of Media Misinformation
Kreps, S., McCain, R. M., & Brundage, M. (2022) · Journal of Experimental Political Science · doi:10.1017/xps.2020.37 · Cited by 360 · cited 1×
“…existing studies of LLM-induced attitude change attempt to persuade participants of all political beliefs towards a singular viewpoint (2, 7, 33, 34), failing to consider the political context surrounding the selected issues when drawing conclusions about persuasiveness in and across partisan…”
Cited by
On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial
Salvi, F., Ribeiro, M. H., Gallotti, R., & West, R. (2024) · doi:10.21203/rs.3.rs-4429707/v1 · Cited by 285
On the conversational persuasiveness of GPT-4
Salvi, F., Horta Ribeiro, M., Gallotti, R., & West, R. (2025) · Nature Human Behaviour · doi:10.1038/s41562-025-02194-6 · Cited by 264
Evaluating the persuasive influence of political microtargeting with large language models
Hackenburg, K., & Margetts, H. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2403116121 · Cited by 114
“Early research suggests that most capable LLMs can directly persuade humans on political issues (5), draft more persuasive public communications than actual government agencies and political communication experts (6, 7), and generate convincing disinfor-mation and fake news (1, 8, 9).”
BackgroundSelf-citationTesting theories of political persuasion using AI
Argyle, L. P., Busby, E. C., Gubler, J. R., Lyman, A., Olcott, J., Pond, J., & Wingate, D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2412815122 · Cited by 53
Moon, K., Green, A. E., & Kushlev, K. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100207 · Cited by 38
Metadata
- Authors
- Hackenburg, K., Ibrahim, L., Tappin, B. M., & Tsakiris, M.
- Venue
- AI & SOCIETY
- Published
- 2025-07-16
- Volume
- 41
- Issue
- 1
- Pages
- 351-361
- ISSN
- 0951-5666 (ISSN-L)
- 1435-5655
- OpenAlex
- W4412476081
- Cited by
- 23
- References
- 15
- FWCI
- 34.44
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comparing the persuasiveness of role-playing large language models and human experts on polarized U.S. political issues. https://app.cochat.ai/papers/10.1007/s00146-025-02464-x
- The paper
- Hackenburg, K., Ibrahim, L., Tappin, B. M., & Tsakiris, M. (2026). Comparing the persuasiveness of role-playing large language models and human experts on polarized U.S. political issues. AI & SOCIETY. https://doi.org/10.1007/s00146-025-02464-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.