CoChat
Sign Up
PreprintCited by 8Ethics and Social Impacts of AISentiment Analysis and Opinion MiningTopic Modeling

Random Silicon Sampling: Simulating Human Sub-Population Opinion Using a Large Language Model Based on Group-Level Demographic Information

Open access
Sun, Seungjong, Lee, Eungu, Nan, Dongyan, Zhao, Xiangying, Lee, Wonbyung, Jansen, Bernard J., Kim, Jang Hyun·2024·arXiv
DOI 10.48550/arxiv.2402.18144arXiv 2402.18144
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models exhibit societal biases associated with demographic information, including race, gender, and others. Endowing such language models with personalities based on demographic data can enable generating opinions that align with those of humans. Building on this idea, we propose "random silicon sampling," a method to emulate the opinions of the human population sub-group. Our study analyzed 1) a language model that generates the survey responses that correspond with a human group based solely on its demographic distribution and 2) the applicability of our methodology across various demographic subgroups and thematic questions. Through random silicon sampling and using only group-level demographic information, we discovered that language models can generate response distributions that are remarkably similar to the actual U.S. public opinion polls. Moreover, we found that the replicability of language models varies depending on the demographic group and topic of the question, and this can be attributed to inherent societal biases in the models. Our findings demonstrate the feasibility of mirroring a group's opinion using only demographic distribution and elucidate the effect of social biases in language models on such simulations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.18144

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

Cited by 8

Research works printed but not linked: 53

  1. Aher, Gati V, Arriaga, Rosa I, & Kalai, Adam Tauman. 2023. Using large language models to simulate multiple humans and replicate human subject studies. In International Conference on Machine Learning, pages 337-371
  2. Aliaga, Alfredo, & Ruilin, R. 2006. Cluster optimal sample size for demographic and health surveys. In 7th International Conference on Teaching Statistics-ICOTS, 7:pages 2-7
  3. Argyle, Lisa P, Busby, Ethan C, Fulda, Nancy, Gubler, Joshua R, Rytting, Christopher, & Wingate, David. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351.
  4. Artz, Benjamin, Kaya, Ilker, & Kaya, Ozgur. 2022. Gender role perspectives and job burnout. Review of Economics of the Household, 20(2), 447-470.
  5. Barocas, Solon, & Selbst, Andrew D. 2016. Big data's disparate impact. California law review, 671-732.
  6. Beck, Tilman, Schuff, Hendrik, Lauscher, Anne, & Gurevych, Iryna. 2023. How (Not) to Use Sociodemographic Information for Subjective NLP Tasks. arXiv preprint arXiv:2309.07034. · arXiv
  7. Binz, Marcel, & Schulz, Eric. 2023. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6), e2218523120.
  8. Black, Dan, Gates, Gary, Sanders, Seth, & Taylor, Lowell. 2000. Demographics of the gay and lesbian population in the United States: Evidence from available systematic data sources. Demography, 37(2), 139-154.
  9. Brand, James, Israeli, Ayelet, & Ngwe, Donald. 2023. Using gpt for market research. Available at SSRN 4395751.
  10. Caron, Graham, & Srivastava, Shashank. 2022. Identifying and manipulating the personality traits of language models. arXiv preprint arXiv:2212.10276. · arXiv
  11. Carpenter, Christopher S. 2020. The direct effects of legal same-sex marriage in the United States: evidence from Massachusetts. Demography, 57(5), 1787-1808.
  12. Cheng, Myra, Durmus, Esin, & Jurafsky, Dan. 2023a. Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models. arXiv preprint arXiv:2305.18189. · arXiv
  13. Cheng, Myra, Piccardi, Tiziano, & Yang, Diyi. 2023b. CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations. arXiv preprint arXiv:2310.11501. · arXiv
  14. Chu, Eric, Andreas, Jacob, Ansolabehere, Stephen, & Roy, Deb. 2023. Language models trained on media diets can predict public opinion. arXiv preprint arXiv:2303.16779. · arXiv
  15. Cobanoglu, Cihan, Cavusoglu, Muhittin, & Turktarhan, Gozde. 2021. A beginner's guide and best practices for using crowdsourcing platforms for survey research: The Case of Amazon Mechanical Turk (MTurk). Journal of Global Business Insights, 6(1), 92-97.
  16. Crissman, Halley P, Berger, Mitchell B, Graham, Louis F, & Dalton, Vanessa K. 2017. Transgender demographics: a household probability sample of US adults, 2014. American journal of public health, 107(2), 213-215.
  17. Deshpande, Ameet, Murahari, Vishvak, Rajpurohit, Tanmay, Kalyan, Ashwin, & Narasimhan, Karthik. 2023. Toxicity in chatgpt: Analyzing persona- assigned language models. arXiv preprint arXiv:2304.05335. · arXiv
  18. Dominguez-Olmedo, Ricardo, Hardt, Moritz, & Mendler-Dünner, Celestine. 2023. Questioning the Survey Responses of Large Language Models. arXiv preprint arXiv:2306.07951. · arXiv
  19. Durmus, Esin, Nyugen, Karina, Liao, Thomas I, Schiefer, Nicholas, Askell, Amanda, Bakhtin, Anton, Chen, Carol, Hatfield-Dodds, Zac, Hernandez, Danny, & Joseph, Nicholas. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388. · arXiv
  20. Fan, Xitao, Thompson, Bruce, & Wang, Lin. 1999. Effects of sample size, estimation methods, and model specification on structural equation modeling fit indexes. Structural equation modeling: a multidisciplinary journal, 6(1), 56-83.
  21. Feng, Shangbin, Park, Chan Young, Liu, Yuhan, & Tsvetkov, Yulia. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. arXiv preprint arXiv:2305.08283. · arXiv
  22. Griffin, Lewis D, Kleinberg, Bennett, Mozes, Maximilian, Mai, Kimberly T, Vau, Maria, Caldwell, Matthew, & Marvor-Parker, Augustine. 2023. Susceptibility to Influence of Large Language Models. arXiv preprint arXiv:2303.06074. · arXiv
  23. Grossmann, Igor, Feinberg, Matthew, Parker, Dawn C, Christakis, Nicholas A, Tetlock, Philip E, & Cunningham, William A. 2023. AI and the transformation of social science research. Science, 380(6650), 1108-1109.
  24. Guadagnoli, Edward, & Velicer, Wayne F. 1988. Relation of sample size to the stability of component patterns. Psychological bulletin, 103(2), 265.
  25. Gupta, Shashank, Shrivastava, Vaishnavi, Deshpande, Ameet, Kalyan, Ashwin, Clark, Peter, Sabharwal, Ashish, & Khot, Tushar. 2023. Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs. arXiv preprint arXiv:2311.04892. · arXiv
  26. Hagendorff, Thilo, Fabi, Sarah, & Kosinski, Michal. 2022. Machine intuition: Uncovering human-like intuitive decision-making in GPT-3.5. arXiv preprint arXiv:2212.05206. · arXiv
  27. Hämäläinen, Perttu, Tavast, Mikke, & Kunnari, Anton. 2023. Evaluating large language models in generating synthetic hci research data: a case study. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-19
  28. Hartmann, Jochen, Schwenzow, Jasper, & Witte, Maximilian. 2023. The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation. arXiv preprint arXiv:2301.01768. · arXiv
  29. Hwang, EunJeong, Majumder, Bodhisattwa Prasad, & Tandon, Niket. 2023. Aligning Language Models to User Opinions. arXiv preprint arXiv:2305.14929. · arXiv
  30. Jansen, Bernard J, Jung, Soon-gyo, & Salminen, Joni. 2023. Employing large language models in survey research. Natural Language Processing Journal, 4, 100020.
  31. Jiang, Hang, Beeferman, Doug, Roy, Brandon, & Roy, Deb. 2022. Communitylm: Probing partisan worldviews from language models. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6818-6826, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  32. Jiang, Hang, Zhang, Xiajie, Cao, Xubo, Kabbara, Jad, & Roy, Deb. 2023. Personallm: Investigating the ability of gpt-3.5 to express personality traits and gender differences. arXiv preprint arXiv:2305.02547. · arXiv
  33. Kalinin, Kirill. 2023. Improving GPT Generated Synthetic Samples with Sampling-Permutation Algorithm. Available at SSRN 4548937.
  34. Keeter, Scott, Hatley, Nick, Kennedy, Courtney, & Lau, Arnold. 2017. What low response rates mean for telephone surveys. Pew Research Center, 15(1), 1- 39.
  35. Kennedy, Courtney, Blumenthal, Mark, Clement, Scott, Clinton, Joshua D, Durand, Claire, Franklin, Charles, McGeeney, Kyley, Miringoff, Lee, Olson, Kristen, & Rivers, Douglas. 2018. An evaluation of the 2016 election polls in the United States. Public Opinion Quarterly, 82(1), 1-33.
  36. Kim, Junsol, & Lee, Byungkyu. 2023. AI-Augmented Surveys: Leveraging Large Language Models for Opinion Prediction in Nationally Representative Surveys. arXiv preprint arXiv:2305.09620. · arXiv
  37. Klar, Samara, & Krupnikov, Yanna. (2016). Independent politics. Cambridge University Press.
  38. Lee, Randy T, Perez, Amanda D, Boykin, C Malik, & Mendoza-Denton, Rodolfo. 2019. On the prevalence of racial discrimination in the United States. PloS one, 14(1), e0210698.
  39. Lee, Sanguk, Peng, Tai-Quan, Goldberg, Matthew H, Rosenthal, Seth A, Kotcher, John E, Maibach, Edward W, & Leiserowitz, Anthony. 2023. Can Large Language Models Capture Public Opinion about Global Warming? An Empirical Assessment of Algorithmic Fidelity and Bias. arXiv preprint arXiv:2311.00217. · arXiv
  40. Leemann, Lucas, Stoetzer, Lukas F, & Traunmüller, Richard. 2021. Eliciting beliefs as distributions in online surveys. Political Analysis, 29(4), 541-553.
  41. MacCallum, Robert C, Widaman, Keith F, Zhang, Shaobo, & Hong, Sehee. 1999. Sample size in factor analysis. Psychological methods, 4(1), 84.
  42. Mayson, Sandra G. 2019. Bias in, bias out. The Yale Law Journal, 2218-2300.
  43. Mills, Stuart, Costa, Samuel, & Sunstein, Cass R. 2023. The Opportunities and Costs of AI in Behavioural Science. Available at SSRN 4490597.
  44. Nawyn, Stephanie J. 2019. Refugees in the United States and the politics of crisis. The Oxford handbook of migration crisis, 163-180.
  45. OpenAI, R. 2023. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2, 13. · arXiv
  46. Ouyang, Long, Wu, Jeffrey, Jiang, Xu, Almeida, Diogo, Wainwright, Carroll, Mishkin, Pamela, Zhang, Chong, Agarwal, Sandhini, Slama, Katarina, & Ray, Alex. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730- 27744.
  47. Panch, Trishan, Mattie, Heather, & Atun, Rifat. 2019. Artificial intelligence and algorithmic bias: implications for health systems. Journal of global health, 9(2). Whose opinions do language models reflect? arXiv preprint arXiv:2303.17548. · arXiv
  48. Santy, Sebastin, Liang, Jenny T, Bras, Ronan Le, Reinecke, Katharina, & Sap, Maarten. 2023. NLPositionality: Characterizing Design Biases of Datasets and Models. arXiv preprint arXiv:2306.01943. · arXiv
  49. Schramowski, Patrick, Turan, Cigdem, Andersen, Nico, Rothkopf, Constantin A, & Kersting, Kristian. 2022. Large pre-trained language models contain human- like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3), 258-268.
  50. Simmons, Gabriel, & Hare, Christopher. 2023. Large Language Models as Subpopulation Representative Models: A Review. arXiv preprint arXiv:2310.17888. · arXiv
  51. Simmons, Gabriel. 2022. Moral mimicry: Large language models produce moral rationalizations tailored to political identity. arXiv preprint arXiv:2209.12106. · arXiv
  52. Törnberg, Petter, Valeeva, Diliara, Uitermark, Justus, & Bail, Christopher. 2023. Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv preprint arXiv:2310.05984. · arXiv
  53. Zhou, Jingyan, Deng, Jiawen, Mi, Fei, Li, Yitong, Wang, Yasheng, Huang, Minlie, Jiang, Xin, Liu, Qun, & Meng, Helen. 2022. Towards identifying social bias in dialog systems: Frame, datasets, and benchmarks. arXiv preprint arXiv:2202.08011. · arXiv

Authors
Sun, S., Lee, E., Nan, D., Zhao, X., Lee, W., Jansen, B. J., & Kim, J. H.
Affiliations
  • Sungkyunkwan University (KR)
  • Hamad bin Khalifa University (QA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.18144
arXiv
2402.18144
OpenAlex
W4392340310
Cited by
8
References
0
Citations by year
2026: 2 · 2025: 3 · 2024: 3
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, social bias, demographic bias, public opinion, demographic subgroups, replicability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review