The political preferences of LLMs
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
I report here a comprehensive analysis about the political preferences embedded in Large Language Models (LLMs). Namely, I administer 11 political orientation tests, designed to identify the political preferences of the test taker, to 24 state-of-the-art conversational LLMs, both closed and open source. When probed with questions/statements with political connotations, most conversational LLMs tend to generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. This does not appear to be the case for five additional base (i.e. foundation) models upon which LLMs optimized for conversation with humans are built. However, the weak performance of the base models at coherently answering the tests’ questions makes this subset of results inconclusive. Finally, I demonstrate that LLMs can be steered towards specific locations in the political spectrum through Supervised Fine-Tuning (SFT) with only modest amounts of politically aligned data, suggesting SFT’s potential to embed political orientation in LLMs. With LLMs beginning to partially displace traditional information sources like search engines and Wikipedia, the societal implications of political biases embedded in LLMs are substantial.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 1×
“The decision to use automated stance detection instead of human raters is justified by recent evidence showing that ChatGPT outperforms crowd workers for text annotation tasks such as stance detection [31].”
MethodMore human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · cited 1×
“Shortly after the release of ChatGPT, its answers to political orientation tests were documented as manifesting left-leaning political preferences [11–13].”
BackgroundThe Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281 · cited 2×
“The left-leaning political preferences of ChatGPT have been documented previously [11].”
BackgroundMethodSelf-citationThe Self-Perception and Political Biases of ChatGPT
Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., & Pauly, M. (2023) · arXiv · doi:10.48550/arxiv.2304.07333 · Cited by 178 · cited 1×
“Shortly after the release of ChatGPT, its answers to political orientation tests were documented as manifesting left-leaning political preferences [11–13].”
BackgroundFeng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
The Self-Perception and Political Biases of ChatGPT
Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., Roidl, M., & Pauly, M. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/7115633 · Cited by 98
Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., & Hovy, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.816 · Cited by 89
Diminished diversity-of-thought in a standard large language model
Park, P. S., Schoenegger, P., & Zhu, C. (2024) · Behavior Research Methods · doi:10.3758/s13428-023-02307-x · Cited by 83 · cited 1×
“Shortly after the release of ChatGPT, its answers to political orientation tests were documented as manifesting left-leaning political preferences [11–13].”
BackgroundQuantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68
Cited by
Bias in Large Language Models: Origin, Evaluation, and Mitigation
Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024) · arXiv · doi:10.48550/arxiv.2411.10915 · Cited by 208
Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H., Schuetze, H., & Hovy, D. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.816 · Cited by 89
An, J., Huang, D., Lin, C., & Tai, M. (2025) · PNAS Nexus · doi:10.1093/pnasnexus/pgaf089 · Cited by 58
In generative AI we trust: can chatbots effectively verify political information?
Kuznetsova, E., Makhortykh, M., Vziatysheva, V., Stolze, M., Baghumyan, A., & Urman, A. (2025) · Journal of Computational Social Science · doi:10.1007/s42001-024-00338-8 · Cited by 32
Emerging AI individualism: how young people integrate social AI into everyday life
Brandtzaeg, P. B., Følstad, A., & Skjuve, M. (2025) · Communication and Change · doi:10.1007/s44382-025-00011-2 · Cited by 30
Metadata
- Authors
- Rozado, D.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2024-07-31
- Volume
- 19
- Issue
- 7
- Pages
- e0306621
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W4401170130
- Cited by
- 83
- References
- 12
- FWCI
- 34.26
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The political preferences of LLMs. https://app.cochat.ai/papers/10.1371/journal.pone.0306621
- The paper
- Rozado, D. (2024). The political preferences of LLMs. PLOS ONE. https://doi.org/10.1371/journal.pone.0306621
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.