Performance and biases of large language models in public opinion simulation.
Open accessAbstract
The rise of Large Language Models (LLMs) like ChatGPT marks a pivotal advancement in artificial intelligence, reshaping the landscape of data analysis and processing. By simulating public opinion, ChatGPT shows promise in facilitating public policy development. However, challenges persist regarding its worldwide applicability and bias across demographics and themes. Our research employs socio-demographic data from the World Values Survey to evaluate ChatGPT’s performance in diverse contexts. Findings indicate significant performance disparities, especially when comparing countries. Models perform better in Western, English-speaking, and developed nations, notably the United States, in comparison to others. Disparities also manifest across demographic groups, showing biases related to gender, ethnicity, age, education, and social class. The study further uncovers thematic biases in political and environmental simulations. These results highlight the need to enhance LLMs’ representativeness and address biases, ensuring their equitable and effective integration into public opinion research alongside conventional methodologies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most conversational agents have been tested only in text-based, low-stakes interactions.
“Most have been tested only with text-based, low-stakes interactions,33–38”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [33]
The Simulation input parameters section describes testing ChatGPT with text prompts drawn from survey questionnaires. It does not establish that most conversational agents have been tested only this way, or characterize those interactions as low-stakes. No page number is available in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredReported biases among LLMs have included propagation of biases based on race and ethnicity, gender, educational level, social class, sexuality, and age.
“These have included the propagation of biases based on race and ethnicity, gender, educational level, social class, sexuality, and age.184,185,186”
From Large Language Models for Chatbot Health Advice Studies: A Systematic Review (2025) · cites this paper as [184]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Semantics derived automatically from language corpora contain human-like biases
Caliskan, A., Bryson, J. J., & Narayanan, A. (2017) · Science · doi:10.1126/science.aal4230 · Cited by 2,887 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · cited 6× · see citing works
“This ef fi ciency, combined with the ability to process and analyze extensive text data, empowers LLMs to uncover insights into public sentiment often overlooked by conventional methods (Ray, 2023).”
InfluentialGenerative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · cited 1× · see citing works
“Complementary to this, studies by Brand et al. (2023) and Park et al. (2023) highlighted ChatGPT ’ s skill in simulating consumer behavior and human actions in various scenarios.”
The Impact of Public Opinion on Public Policy: A Review and an Agenda
Burstein, P. (2003) · Political Research Quarterly · doi:10.1177/106591290305600103 · Cited by 1,541 · see citing works
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · see citing works
Alon-Barkat, S., & Busuioc, M. (2023) · Journal of Public Administration Research and Theory · doi:10.1093/jopart/muac007 · Cited by 469 · see citing works
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
Horton, J., Filippas, A., & Manning, B. (2023) · National Bureau of Economic Research · doi:10.3386/w31122 · Cited by 329 · see citing works
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · see citing works
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. (2023) · doi:10.31235/osf.io/5ecfa · Cited by 213 · see citing works
Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024) · Political Analysis · doi:10.1017/pan.2024.5 · Cited by 209 · see citing works
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190 · see citing works
Chan, A. (2023) · AI and Ethics · doi:10.1007/s43681-022-00148-6 · Cited by 155 · see citing works
Language Models and Cognitive Automation for Economic Research
Korinek, A. (2023) · doi:10.3386/w30957 · Cited by 129 · see citing works
The Temperature Feature of ChatGPT: Modifying Creativity for Clinical Research
Davis, J., Bulck, L. V., Durieux, B. N., & Lindvall, C. J. (2024) · JMIR Human Factors · doi:10.2196/53559 · Cited by 103 · see citing works
Quantifying and alleviating political bias in language models
Liu, R., Jia, C., Wei, J., Xu, G., & Vosoughi, S. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2021.103654 · Cited by 68 · see citing works
An initial investigation of ChatGPT unit test generation capability
Guilherme, V., & Vincenzi, A. M. R. (2023) · Brazilian Symposium on Systematic and Automated Software Testing (SAST) · doi:10.1145/3624032.3624035 · Cited by 64 · see citing works
Is Artificial Intelligence a World Changer? A Case Study of OpenAI’s Chat GPT
Mathew, A. (2023) · doi:10.9734/bpi/rpst/v5/18240d · Cited by 49 · see citing works
Delobelle, P., Temple, P., Perrouin, G., Frénay, B., Heymans, P., & Berendt, B. (2021) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3468507.3468513 · Cited by 35 · see citing works
How to Learn and Teach Economics with Large Language Models, Including GPT
Cowen, T., & Tabarrok, A. T. (2023) · doi:10.2139/ssrn.4391863 · Cited by 35 · see citing works
Rating Distribution Calibration for Selection Bias Mitigation in Recommendations
Liu, H., Tang, D., Yang, J., Zhao, X., Liu, H., Tang, J.-L., & Cheng, Y.-L. (2022) · The Web Conference · doi:10.1145/3485447.3512078 · Cited by 33 · see citing works
The Ethico-Political Universe of ChatGPT
Martin, J. L. (2023) · Journal of Social Computing · doi:10.23919/jsc.2023.0003 · Cited by 18 · see citing works
Lee, S., Peng, T., Goldberg, M. H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2311.00217 · Cited by 7 · see citing works
How do Generative Language Models Answer Opinion Polls?Preprint
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2024) · doi:10.31235/osf.io/r2pnb · Cited by 7 · see citing works
On Measuring Social Biases in Prompt-Based Multi-Task LearningPreprint
Akyürek, A. F., Paik, S., Kocyigit, M. Y., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.11605 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Aher GV, Arriaga RI, Kalai AT (2023) Using large language models to simulate multiple humans and replicate human subject studies. Proceedings of the 40th International Conference on Machine Learning, 337-371. https:// proceedings.mlr.press/v202/aher23a.html
- Hutchings VL (2005) Public Opinion and Democratic Accountability: How Citi- zens Learn about Politics. Princeton University Press
- Liang PP, Wu C, Morency L-P, Salakhutdinov R (2021) Towards understanding and mitigating social biases in language models. Proceedings of the 38th International Conference on Machine Learning, 6565-6576. https:// proceedings.mlr.press/v139/liang21a.html
- Russell S (2019) Human compatible: AI and the problem of control. Penguin, Uk
- Santurkar S, Durmus E, Ladhak F, Lee C, Liang P, Hashimoto T (2023) Whose opinions do language models reflect? Proceedings of the 40th International Conference on Machine Learning, 29971-30004. https://proceedings.mlr. press/v202/santurkar23a.html
Cited by
Sorted by citations
Large Language Models for Chatbot Health Advice Studies: A Systematic Review
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., et al. (2025) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.57879 · Cited by 283 · see citing works
Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias
Breithaupt, A. G., Tang, A., Paolillo, E. W., Bibars, M., Johnson, E. C. B., Saloner, R., Possin, K. L., Windon, C. C., Hill-Jarrett, T. G., Giorgio, J., et al. (2026) · Seminars in Neurology · doi:10.1055/a-2744-9871 · Cited by 6 · see citing works
- Authors
- Qu, Y., & Wang, J.
- Affiliations
- Nanyang Technological University (SG)
- Corresponding author
- Wang, Jue
- Venue
- Humanities and Social Sciences Communications
- Published
- 2024-08-28
- Volume
- 11
- Issue
- 1
- ISSN
- 2055-1045 (ISSN-L)
- 2662-9992
- OpenAlex
- W4401955911
- Cited by
- 79
- References
- 25
- Citations by year
- 2026: 45 · 2025: 33 · 2024: 1
- FWCI
- 32.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Humanities and Social Sciences Communications (publishedVersion, cc-by-nc-nd)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DR-NTU (Nanyang Technological University) (submittedVersion, cc-by-nc-nd)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- large language models, ChatGPT, algorithmic bias, demographic bias, World Values Survey, political attitudes, environmental attitudes, public policy, representativeness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
