Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters
Open accessAbstract
Do LLMs have political leanings and are LLMs able to shift our political views?This paper explores these questions in the context of the 2024 U.S. presidential election.Through a voting simulation, we demonstrate 18 openweight and closed-source LLMs' political preference for Biden over Trump.We show how Biden-leaning becomes more pronounced in instruction-tuned and reinforced models compared to their base versions by analyzing their responses to political questions related to the two nominees.We further explore the potential impact of LLMs on voter choice by recruiting 935 U.S. registered voters.Participants interacted with LLMs (Claude-3, Llama-3, and GPT-4) over five exchanges.Intriguingly, although LLMs were not asked to persuade users to support Biden, about 20% of Trump supporters reduced their support for Trump after LLM interaction.This result is noteworthy given that many studies on the persuasiveness of political campaigns have shown minimal effects in presidential elections.Many users also expressed a desire for further interaction with LLMs on political subjects.Further research on how LLMs affect users' political views is required, as their use becomes more widespread.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prior work has established the existence of political bias in LLMs.
“Prior work has established the existence of political bias in LLMs (Feng et al., 2023;Röttger et al., 2024;Potter et al., 2024;Peng et al., 2025;Westwood et al., 2025), contributing to a growing focus on achieving political neutrality.”
From Benchmarking Overton Pluralism in LLMs (2025) · cites this paper as Potter et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Research works printed but not linked: 33
- Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical re- port. arXiv preprint arXiv:2303.08774. · arXiv
- Lisa P Argyle, Christopher A Bail, Ethan C Busby, Joshua R Gubler, Thomas Howe, Christopher Rytting, Taylor Sorensen, and David Wingate. 2023. Leverag- ing AI for democratic discourse: Chat interventions can improve online political conversations at scale. Proceedings of the National Academy of Sciences, 120(41):e2311627120.
- Bernard R Berelson, Paul F Lazarsfeld, and William N McPhee. 1986. Voting: A Study of Opinion For- mation in a Presidential Campaign. University of Chicago Press.
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Nat- ural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc.".
- David E Broockman and Joshua L Kalla. 2023. When and why are campaigns' persuasive effects small? Evidence from the 2020 US presidential election. American Journal of Political Science, 67(4):833- 849.
- Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. arXiv preprint arXiv:2305.08283. · arXiv
- Suyash Fulay, William Brannon, Shrestha Mohanty, Cassandra Overney, Elinor Poole-Dayan, Deb Roy, and Jad Kabbara. 2024. On the Relationship between Truth and Political Bias in Language Models. arXiv preprint arXiv:2409.05283. · arXiv
- Shubh Goyal, Medha Hira, Shubham Mishra, Sukriti Goyal, Arnav Goel, Niharika Dadu, DB Kirushikesh, Sameep Mehta, and Nishtha Madaan. 2024. LLM- Guard: Guarding against Unsafe LLM Behavior. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23790-23792.
- Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794. · arXiv
- Kobi Hackenburg and Helen Margetts. 2024. Evaluating the persuasive influence of political microtargeting with large language models. Proceedings of the Na- tional Academy of Sciences, 121(24):e2403116121.
- Anselm Hager. 2019. Do Online Ads Influence Vote Choice? Political Communication, 36(3):376-393.
- Jochen Hartmann, Jasper Schwenzow, and Maximil- ian Witte. 2023. The political ideology of conversa- tional AI: Converging evidence on ChatGPT's pro- environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768. · arXiv
- Luke Hewitt, David Broockman, Alexander Coppock, Ben M Tappin, James Slezak, Valerie Coffman, Nathaniel Lubin, and Mohammad Hamidian. 2024. How experiments help campaigns persuade voters: Evidence from a large archive of campaigns' own ex- periments. American Political Science Review, pages 1-19.
- Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of Experts. arXiv preprint arXiv:2401.04088. · arXiv
- Joshua L Kalla and David E Broockman. 2018. The min- imal persuasive effects of campaign contact in gen- eral elections: Evidence from 49 field experiments. American Political Science Review, 112(1):148-166.
- Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, et al. 2023. SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling. arXiv preprint arXiv:2312.15166. · arXiv
- Austin C Kozlowski, Matt Taddy, and James A Evans. 2019. The geometry of culture: Analyzing the mean- ings of class through word embeddings. American Sociological Review, 84(5):905-949.
- Paul F Lazarsfeld, Bernard Berelson, and Hazel Gaudet. 1968. The People's Choice: How the Voter Makes Up His Mind in a Presidential Campaign. Columbia University Press.
- Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. 2023. Platypus: Quick, Cheap, and Powerful Refinement of LLMs. arXiv preprint arXiv:2308.07317. · arXiv
- Fabio Motoki, Valdemar Pinho Neto, and Victor Ro- drigues. 2024. More human than human: Measuring ChatGPT political bias. Public Choice, 198(1):3-23.
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730-27744.
- Jérôme Rutinowski, Sven Franke, Jan Endendyk, Ina Dormuth, Moritz Roidl, and Markus Pauly. 2024. The Self-Perception and Political Biases of Chat- GPT. Human Behavior and Emerging Technologies, 2024(1):7115633.
- Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In In- ternational Conference on Machine Learning, pages 29971-30004.
- PMLR.
- Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understand- ing sycophancy in language models. arXiv preprint arXiv:2310.13548. · arXiv
- Michaela Sullivan-Paul. 2023. How would ChatGPT vote in a federal election? A study exploring algo- rithmic political bias in artificial intelligence. Ph.D. thesis, School of Public Policy, University of Tokyo.
- Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. Stanford Center for Research on Foundation Models, 3(6):7.
- Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein. 2024. Systematic biases in LLM simula- tions of debates. arXiv preprint arXiv:2402.04049. · arXiv
- Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. · arXiv
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open Founda- tion and Fine-Tuned Chat Models. arXiv preprint arXiv:2307.09288. · arXiv
- Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. Openchat: Advanc- ing open-source language models with mixed-quality data. arXiv preprint arXiv:2309.11235. · arXiv
- Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2023. WizardLM: Empow- ering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging LLM-as-a-Judge with MT-Bench and Chat- bot Arena. Advances in Neural Information Process- ing Systems, 36.
Cited by
Sorted by citations
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 49 · see citing works
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 34 · see citing works
Self-citationThe Homogenizing Engine: AI's Role in Standardizing Culture and the Path to Policy
Daryani, Y., Sourati, Z., & Dehghani, M. (2025) · Policy Insights from the Behavioral and Brain Sciences · doi:10.1177/23727322251406591 · Cited by 11 · see citing works
Simulating social perception with large language models: perceptions of China’s common prosperity
Jiang, Z., Huang, B., Ge, J., Lin, C., Xu, Y., & Yu, J. (2025) · Journal of Chinese Governance · doi:10.1080/23812346.2025.2603822 · Cited by 9 · see citing works
Using Generative AI in Historical Practice
Fox, Y. (2026) · Cambridge University Press eBooks · doi:10.1017/9781009740067 · Cited by 5 · see citing works
Jones, C., & Bergen, B. K. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11517-6 · Cited by 17 · see citing works
Ethnography and artificial intelligence: the question of context
Witteborn, S. (2025) · Annals of the International Communication Association · doi:10.1093/anncom/wlaf026 · Cited by 3 · see citing works
Sharma, T., Potter, Y., Park, J., Liu, Y., Huang, Y., Liu, X., Song, D., Hancock, J. T., & Wang, Y. (2026) · Scientific Reports · doi:10.1038/s41598-026-40180-8 · Cited by 4 · see citing works
Self-citationBeyond partisan leaning: a comparative analysis of political bias in large language models
Peng, T., Yang, K., Lee, S., Li, H., Chu, Y., Lin, Y., & Liu, H. (2026) · Journal of Information Technology & Politics · doi:10.1080/19331681.2026.2646990 · Cited by 10 · see citing works
Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
Kozlowski, A. C. (2026) · Theory and Society · doi:10.1007/s11186-026-09685-z · Cited by 2 · see citing works
Bland new world: is AI making us all think the same?
Hutson, M. (2026) · Nature · doi:10.1038/d41586-026-02682-3 · Cited by 2 · see citing works
Measuring Political Stance and Consistency in Large Language Models
Alali, S. F., Maasfeh, M. N., Kutlu, M., & Kardaş, Ş. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21324-2_34 · Cited by 1 · see citing works
Hakimov, R., Rohner, D., & Boboshin, F. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5386746 · Cited by 2 · see citing works
Benchmarking Overton Pluralism in LLMs
Poole-Dayan, E., Wu, J., Sorensen, T., Pei, J., & Bakker, M. A. (2025) · arXiv · doi:10.48550/arxiv.2512.01351
“Prior work has established the existence of political bias in LLMs (Feng et al., 2023; Röttger et al., 2024; Potter et al., 2024; Peng et al., 2025; Westwood et al., 2025), contributing to a growing focus on achieving political neutrality.”
Kang, T., Thorson, K., Peng, T., Hiaeshutter-Rice, D., Lee, S., & Soroka, S. (2025) · Social Science Computer Review · doi:10.1177/08944393251410155 · Cited by 1 · see citing works
Zhang, D., Zhang, L., Qu, F., Zhuang, Z., & Zhou, D. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464608 · Cited by 2 · see citing works
Large language models as a conduit for value shifts in contemporary China
Cao, L., & Dai, L. (2026) · Chinese Sociological Review · doi:10.1080/21620555.2026.2656193
Ruidias, R. R. S., Nunes, B. P., & da Silva, T. N. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_27
Rozado, D., & Tetlock, P. E. (2026) · Theory and Society · doi:10.1007/s11186-026-09719-6
LLMs on Guard: Training Language Models to Be More Positive Toward the Country of Use
Kovalev, G. P., Urazov, S. O., Tikhomirov, M. M., Loukachevitch, N. V., & Dobrov, B. V. (2026) · Lobachevskii Journal of Mathematics · doi:10.1134/s1995080226619673
Value Alignment Beyond Safety: A Cross-Cultural Analysis of Large Language Model Priorities
Ma, Y., Tong, S., Wang, B., & Peng, K. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-6913-6_13
Large language models for computational social science
Xu, F., Shao, C., Chen, L., Zeng, Q., Chen, Z., Sukiennik, N., Wang, J., Gao, C., Wang, H., Lian, J., et al. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02539-9
Self-citation
- Authors
- Potter, Y., Lai, S., Kim, J., Evans, J., & Song, D.
- Affiliations
- University of California, Berkeley (US)
- University of Chicago (US)
- Venue
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Published
- 2024
- Pages
- 4244-4275
- OpenAlex
- W4404782693
- Cited by
- 21
- References
- 0
- Citations by year
- 2026: 14 · 2025: 7
- FWCI
- 9.58
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Communication › Social Media and Politics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- large language models, political persuasion, political orientation, voting behavior, 2024 US presidential election, Donald Trump, political attitudes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
