The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
More human than human: measuring ChatGPT political bias
Open accessAbstract
We investigate the political bias of a large language model (LLM), ChatGPT, which has become popular for retrieving factual information and generating content. Although ChatGPT assures that it is impartial, the literature suggests that LLMs exhibit bias involving race, gender, religion, and political orientation. Political bias in LLMs can have adverse political and electoral consequences similar to bias from traditional and social media. Moreover, political bias can be harder to detect and eradicate than gender or racial bias. We propose a novel empirical design to infer whether ChatGPT has political biases by requesting it to impersonate someone from a given side of the political spectrum and comparing these answers with its default. We also propose dose-response, placebo, and profession-politics alignment robustness tests. To reduce concerns about the randomness of the generated text, we collect answers to the same questions 100 times, with question order randomized on each round. We find robust evidence that ChatGPT presents a significant and systematic political bias toward the Democrats in the US, Lula in Brazil, and the Labour Party in the UK. These results translate into real concerns that ChatGPT, and LLMs in general, can extend or even amplify the existing challenges involving political processes posed by the Internet and social media. Our findings have important implications for policymakers, media, politics, and academia stakeholders.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“On one hand, we must be mindful that several studies have underscored the ways in which the outputs of GenAI tools like ChatGPT can inadvertently reinforce biases in areas such as gender and politics (Gross, 2023; Motoki et al., 2024; Rutinowski et al., 2024).”
From If ChatGPT can do it, where is my creativity? generative AI boosts performance but diminishes experience in creative writing (2025) · cites this paper as Motoki et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Available data point to ChatGPT output expressing more progressive than traditional political views [38][39][40].”
From The Impact of Artificial Intelligence on Human Sexuality: A Five-Year Literature Review 2020–2024 (2024) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · cited 3×
“It is particularly important that our method does not need access to the inner parameters of the LLM, like word embeddings (Caliskan et al., 2017), as companies make them opaque or costly due to competition concerns (Vincent, 2023; Science Media Centre, 2023), nor advanced programming skills.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1×
“Finally, Park et al. (2023) document that ChatGPT can simulate human behavior, taking actions that vary with the agents’ experiences and environment.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173 · cited 1×
“Finally, Park et al. (2023) document that ChatGPT can simulate human behavior, taking actions that vary with the agents’ experiences and environment.”
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · cited 2×
“We answer a call from van Dis et al. (2023) to hold LLMs on to human verification, addressing the standing issue of the lack of a reliable method for measuring their biases.”
Chollet, F. (2017) · Cited by 1,158
Gentzkow, M., & Shapiro, J. M. (2006) · Journal of Political Economy · doi:10.1086/499414 · Cited by 1,013 · cited 1×
“One avenue is understanding channels and implications of bias through modeling (Castañeda & Martinelli, 2018; Gentzkow & Shapiro, 2006).”
Groseclose, T., & Milyo, J. (2005) · The Quarterly Journal of Economics · doi:10.1162/003355305775097542 · Cited by 991 · cited 2×
“We focus on the issue of political bias, as it can have major social consequences (Bernhardt et al., 2008; Chiang & Knight, 2011; Groseclose & Milyo, 2005; Levendusky, 2013) and is subtler than other biases (Peters, 2022).”
Graphs in Statistical Analysis
Anscombe, F. J. (1973) · The American Statistician · doi:10.1080/00031305.1973.10478966 · Cited by 976
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 3×
“We know from existing literature that certain professions are more aligned with Democrats or with Repub-licans, as detailed in Table 5, and we know that ChatGPT can correctly reproduce known distributions from specific subgroups (Argyle et al., 2022).”
InfluentialSocial Media, News Consumption, and Polarization: Evidence from a Field Experiment
Levy, R. (2021) · American Economic Review · doi:10.1257/aer.20191777 · Cited by 624
Research works printed but not linked: 3
- Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P. & Amodei, D. (2020) 07. Language models are few-shot learners. arXiv.
- Cameron, A. C. & Trivedi, P. K. (2022). Microeconometrics using Stata (2nd ed.). College StationStata Press.
- Jelveh, Z., Kogut, B. & Naidu, S. (2022). Political language in economics [SSRN Scholarly Paper]. Rochester, NY.
Cited by
(2024) · doi:10.1109/access.2024.3365742 · Cited by 934
“This is due to the inherent biases in the training data, which are assimilated by the model and reflected in its responses [181].”
BackgroundHarnessing the power of AI in qualitative research: Exploring, using and redesigning ChatGPT
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100144 · Cited by 393
“First, there is a risk that AI systems, including ChatGPT, might inadvertently perpetuate existing biases in their training data [74].”
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024) · Political Analysis · doi:10.1017/pan.2024.5 · Cited by 209
“Our findings raise serious questions about the use and performance of LLMs for the characterization of public opinion and the creation of synthetic data, connecting with a growing body of research that asks similar questions (Bender et al. 2021; Spirling 2023) and documents similar issues (Abdulhai…”
The Self-Perception and Political Biases of ChatGPT
Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., & Pauly, M. (2023) · arXiv · doi:10.48550/arxiv.2304.07333 · Cited by 178
“Compared to the publications that conducted similar tests with ChatGPT [24], [25], [28], [29], we also obtain results indicating a political bias of ChatGPT towards progressive views.”
BackgroundMethodResultInfluential
Metadata
- Authors
- Motoki, F., Pinho Neto, V., & Rodrigues, V.
- Affiliations
- University of East Anglia (GB)
- Corresponding author
- Motoki, Fabio
- Venue
- Public Choice
- Published
- 2023-08-17
- Volume
- 198
- Issue
- 1-2
- Pages
- 3-23
- ISSN
- 0048-5829 (ISSN-L)
- 1573-7101
- OpenAlex
- W4385852384
- Semantic Scholar
- 3d8a3517231643c1df79bc32c8c2664a4cba3a41
- Cited by
- 329
- References
- 44
- Citations by year
- 2026: 94 · 2025: 125 · 2024: 81 · 2023: 28
- FWCI
- 81.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Public Choice (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- UEA Digital Repository (University of East Anglia) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- UEA Digital Repository (University of East Anglia) (submittedVersion)
- Funders
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (Finance Code 001)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Communication › Social Media and Politics
- Social Sciences › Sociology and Political Science › Media Influence and Politics
- Keywords
- ChatGPT, large language models, political bias, algorithmic bias, impersonation, placebo tests, robustness testing, Luiz Inácio Lula da Silva, Labour Party, political polarization, AI-generated text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: More human than human: measuring ChatGPT political bias. https://app.cochat.ai/papers/10.1007/s11127-023-01097-2
- The paper
- Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024). More human than human: measuring ChatGPT political bias. Public Choice. https://doi.org/10.1007/s11127-023-01097-2
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.