Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Open accessAbstract
Abstract Large language models (LLMs) offer new research possibilities for social scientists, but their potential as “synthetic data” is still largely unknown. In this paper, we investigate how accurately the popular LLM ChatGPT can recover public opinion, prompting the LLM to adopt different “personas” and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study (ANES). Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a 3-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Other work has reported lower reliability if GPT runs are several months apart.
“other work has reported lower reliability if GPT runs are several months apart (74).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · cited 1× · see citing works
“…the characterization of public opinion and the creation of synthetic data, connecting with a growing body of research that asks similar questions (Bender et al. 2021; Spirling 2023) and documents similar issues (Abdulhai et al. 2023; Cao et al. 2023; Motoki, Pinho Neto, and Rodrigues 2023;…”
The Origins and Consequences of Affective Polarization in the United States
Iyengar, S., Lelkes, Y., Levendusky, M., Malhotra, N., & Westwood, S. J. (2019) · Annual Review of Political Science · doi:10.1146/annurev-polisci-051117-073034 · Cited by 3,317 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Political Sectarianism in America
Finkel, E. J., Bail, C. A., Cikara, M., Ditto, P. H., Iyengar, S., Klar, S., Mason, L., McGrath, M. C., Nyhan, B., Rand, D. G., et al. (2020) · Science · doi:10.1126/science.abe1715 · Cited by 931 · see citing works
Unwelcome Change: Coming to Terms with Democratic Backsliding
Waldner, D., & Lust, E. (2018) · Annual Review of Political Science · doi:10.1146/annurev-polisci-050517-114628 · Cited by 889 · cited 1× · see citing works
“At the same time, polls are necessary tools to assess and address growing concerns about polarization and democratic backsliding (Graham 2023; Waldner and Lust 2018).”
Affective polarization, local contexts and public opinion in America
Druckman, J. N., Klar, S., Krupnikov, Y., Levendusky, M., & Ryan, J. B. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-01012-5 · Cited by 463 · see citing works
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
(Mis)perceptions of Partisan Polarization in the American Public
Levendusky, M. S., & Malhotra, N. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfv045 · Cited by 356 · see citing works
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
Horton, J., Filippas, A., & Manning, B. (2023) · National Bureau of Economic Research · doi:10.3386/w31122 · Cited by 329 · see citing works
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · see citing works
The Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281 · see citing works
An Evaluation of the 2016 Election Polls in the United States
Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., et al. (2018) · Public Opinion Quarterly · doi:10.1093/poq/nfx047 · Cited by 200 · see citing works
Self-citationWhose Opinions Do Language Models Reflect?
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114 · see citing works
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2308950121 · Cited by 297 · see citing works
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190 · see citing works
Performance and biases of large language models in public opinion simulation.
Qu, Y., & Wang, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03609-x · Cited by 79 · see citing works
- Authors
- Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M.
- Affiliations
- Vanderbilt University (US)
- Corresponding author
- Bisbee, James
- Venue
- Political Analysis
- Publisher
- Cambridge University Press (CUP)
- Published
- 2024-05-17
- Volume
- 32
- Issue
- 4
- Pages
- 401-416
- ISSN
- 1047-1987 (ISSN-L)
- 1476-4989
- OpenAlex
- W4397011890
- Cited by
- 209
- References
- 15
- Citations by year
- 2026: 116 · 2025: 70 · 2024: 13 · 2023: 10
- FWCI
- 174.14
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Political Analysis (publishedVersion, cc-by)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- large language models, synthetic survey data, ChatGPT, public opinion, feeling thermometers, sociopolitical attitudes, American National Election Studies, sampling reliability, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
