Synthetic Replacements for Human Survey Data? The Perils of Large Language Models
Open accessAbstract
Large Language Models (LLMs) offer new research possibilities for social scientists, but their potential as "synthetic data" is still largely unknown. In this paper, we investigate how accurately the popular closed-source LLM ChatGPT can recover public opinion, prompting the LLM to adopt different "personas" and then provide feeling thermometer scores for 11 sociopolitical groups. The average scores generated by ChatGPT correspond closely to the averages in our baseline survey, the 2016–2020 American National Election Study. Nevertheless, sampling by ChatGPT is not reliable for statistical inference: there is less variation in responses than in the real surveys, and regression coefficients often differ significantly from equivalent estimates obtained using ANES data. We also document how the distribution of synthetic responses varies with minor changes in prompt wording, and we show how the same prompt yields significantly different results over a three-month period. Altogether, our findings raise serious concerns about the quality, reliability, and reproducibility of synthetic survey data generated by LLMs.
AI snapshot
Written from the paper’s full text.
TL;DR
Bisbee et al. prompt ChatGPT 3.5 Turbo to simulate ANES feeling-thermometer opinions for demographic personas and compare them to matched human respondents. They find aggregate similarity for some groups but synthetic opinions exaggerate polarization/extremity and are artificially precise, raising concerns about replacing human survey data with closed-source LLM outputs.
Key findings
- Aggregate synthetic opinions sometimes closely match humans: for non-Hispanic White personas, average ChatGPT thermometer scores are within 5 units of ANES averages on a 0–100 scale, and are often statistically indistinguishable.
- ChatGPT exaggerates divisions: synthetic estimates of societal antipathy are approximately seven times larger than human ANES respondents across Liberals vs. Conservatives, Democrats vs. Republicans, Whites vs. Blacks, and Christians vs. Muslims; in 29 of 36 race-by-party-by-target group conditions, LLM-derived polarization is more extreme than human responses. Appendix analyses report 6.5 times more extreme with larger samples and 4.88 times more extreme from the neutral 50-point midpoint.
- Synthetic data are artificially precise: the standard deviations of ChatGPT’s synthetic opinions for a given persona are on average only 31% the size of matched human ANES opinion; 95% of profiles with at least two ANES respondents had smaller standard deviations when estimated via ChatGPT.
- Power implications: using ChatGPT estimates, less than 10 partisan respondents would be needed to detect a difference from 2012 affective polarization at 99% power, versus 767 respondents using ANES estimates.
Methods
- Prompted ChatGPT 3.5 Turbo to adopt personas defined by age, race, gender, education, income, and partisanship, producing 1,080 unique demographic profiles; 612 profiles exactly matched ANES respondents, and 3,389 ANES respondents were matched to an equal number of synthetic respondents.
- Compared synthetic feeling-thermometer ratings (0–100) toward groups including Democrats, Republicans, Whites, Blacks, Christians, Muslims, Atheists, and Immigrants to ANES 2016 and 2020 common-file responses; extracted 20 responses per persona per group across temperature settings, with main results at the most “creative” temperature setting (1).
- Analysis compared means, standard deviations, and affective-polarization gaps between synthetic and human responses; results were also validated with a bespoke 2021 survey on affective polarization during Covid-19.
Limitations
- The paper states that because LLMs are proprietary, it is difficult to pinpoint the source of the biases; the authors focus on whether bias exists rather than reverse-engineering why. They also note the ANES could in theory be in the LLM training data and that the opacity of ChatGPT raises ethical issues.
- The paper states that exact matching limits the main analysis to 612 profiles and 3,389 ANES respondents after listwise deletion; larger target-group-by-target-group samples strengthen the substantive conclusions. It also notes that ratio-based polarization measures are sensitive to outliers when the ANES difference is zero, possibly due to social-desirability bias.
- Evident from the text: the study uses only ChatGPT 3.5 Turbo, focuses on the United States, relies on feeling-thermometer items, and does not identify a causal mechanism for the observed synthetic biases.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 36 cited sources checked · 0 supported · 0 flagged
- Findingown data
Among non-Hispanic Whites, the average thermometer score is remarkably similar to the ANES averages in nearly every case, with an overall difference of less than 5 points on a 0-100 scale.
“Among non-Hispanic Whites in particular, the average thermometer score is remarkably similar to the ANES averages in nearly every case, with an overall difference of less than 5 points on a 0-100 scale.”
Quote in the paperSubstantial claim - Findingown data
ChatGPT-generated opinions significantly exaggerate the extremity and certainty of partisan and social divisions.
“synthetic opinions also significantly exaggerate the extremity and certainty of partisan and social divisions.”
Quote in the paperSubstantial claim - Findingown data
There is not an exaggerated antipathy towards out-groups.
“There is not, however, an exaggerated antipathy towards out-groups.”
Quote in the paperSubstantial claim - Findingown data
ChatGPT-generated opinions are similar to human opinion for some groups of US respondents.
“We show that although ChatGPT-generated opinions are similar to human opinion for some groups of US respondents”
Quote in the paperSubstantial claim - Findingown data
The LLM matches the ANES in recovering larger polarization in political groups than in racial and religious groups.
“Comparing polarization between political and social groups by comparing the relationship across the columns of Figure 3 reveals that the LLM matches the ANES in recovering larger polarization in political groups than in racial and religious groups.”
Quote in the paperNot a substantial claim
- Findingown data
Synthetic data are artificially precise, with a standard deviation that is only 31% of the variation found in actual opinions among comparable humans.
“Furthermore, synthetic data are artificially precise, with a standard deviation that is only 31% of the variation found in actual opinions among comparable humans.”
Quote in the paperSubstantial claim - Findingown data
When looking at average opinions toward ideological groups, there are more extreme attitudes toward both liberals and conservatives among Republican and Democrat personas than are found in the ANES data.
“In contrast, when looking at average opinions toward ideological groups (top row), we observe that there are more extreme attitudes toward both liberals and conservatives among Republican and Democrat personas than are found in the ANES data.”
Quote in the paperSubstantial claim - Findingown data
ChatGPT-generated evaluations of social groups are almost always more extreme than the assessments of actual human respondents.
“we find that ChatGPT-generated evaluations of social groups are almost always more extreme than the assessments of actual human respondents.”
Quote in the paperSubstantial claim - Findingnone given
When ChatGPT is asked to adopt the persona of a non-Hispanic White and rate their temperature towards a range of groups, the average synthetic opinions are within 5 units of average actual opinion on a 0 to 100 scale.
“When ChatGPT is asked to adopt the persona of a non-Hispanic White and rate their temperature towards a range of groups, for example, the average synthetic opinions are within 5 units of average actual opinion on a 0 to 100 scale.”
Quote in the paperSubstantial claim - Findingown data
The standard deviations of ChatGPT's synthetic opinions for a given persona profile are, on average, only 31% the size of the standard deviations of matched human opinion from the ANES data.
“The standard deviations of Chat-GPT's synthetic opinions for a given persona profile are, on average, only 31% the size of the standard deviations of matched human opinion from the ANES data.”
- Also stated in: § 1 Research Design · plain
“Substantively, the uncertainty of the LLM's estimates are, on average, only 31% that of the uncertainty found in the responses of the exactly matched demographic profile in the ANES.”
Quote in the paperSubstantial claim - Also stated in: § 1 Research Design · plain
- Findingown data
On average, ChatGPT's estimates of societal antipathy are approximately seven times as large as those from human ANES respondents across the various groups examined (Liberals versus Conservatives, Democrats versus Republicans, Whites versus Blacks, and Christians versus Muslims).
“On average, ChatGPT's estimates of societal antipathy are approximately seven times as large as those from human ANES respondents across the various groups we examine (Liberals versus Conservatives, Democrats versus Republicans, Whites versus Blacks, and Christians versus Muslims).”
Quote in the paperSubstantial claim - Findingown data
ChatGPT sometimes provides responses that are nearly identical to the average responses of respondents interviewed by the ANES.
“we show that ChatGPT sometimes provides responses that are nearly identical to the average responses of respondents interviewed by the ANES.”
Quote in the paperSubstantial claim - Findingown data
When evaluating affective polarization on the basis of target group partisanship, ChatGPT presumes that humans are far more favorable towards their in-group than they are in actuality.
“ChatGPT presumes that humans are far more favorable towards their in-group than they are in actuality.”
Quote in the paperSubstantial claim - Findingown data
In each case, the LLM provides larger differences than exist in human opinion.
“In each case, however, the LLM provides larger differences than exist in human opinion.”
Quote in the paperSubstantial claim - Findingown data
Across profiles with at least 2 human respondents in the ANES data, 95% of ChatGPT estimates have smaller standard deviations than the associated standard deviations found in the ANES data.
“Across profiles with at least 2 human respondents in the ANES data, 95% of ChatGPT estimates have smaller standard deviations than the associated standard deviations found in the ANES data.”
Quote in the paperSubstantial claim - Findingnone given
Among non-White persons and respondents, especially for non-Hispanic Black respondents, larger differences sometimes emerge.
“Among non-White persons and respondents, especially for non-Hispanic Black respondents, however, larger differences sometimes emerge.”
Quote in the paperSubstantial claim - Findingown data
Responses from prompted 'persona' profiles in ChatGPT produce measures of partisan and racial affective polarization that are seven times larger than the average opinion of humans who possess the same attributes as the prompted personas.
“Responses from prompted “persona” profiles in ChatGPT produce measures of partisan and racial affective polarization that are seven times larger than the average opinion of humans who possess the same attributes as the prompted personas.”
Quote in the paperSubstantial claim - Findingown data
Across most personae and across most target groups, ChatGPT's attitudes are more similar to Democrats than to Republicans.
“across most personae and across most target groups, ChatGPT's attitudes are more similar to Democrats than to Republicans.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
The latter (Muslims) has more Republican-leaning attitudes among synthetic ChatGPT persona for every identity excepting women, Hispanics, non-Hispanic Blacks, liberals, and Democrats.
“The latter, in particular, has more Republican-leaning attitudes among synthetic ChatGPT persona for every identity excepting women, Hispanics, non-Hispanic Blacks, liberals, and Democrats.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
The variation approaches zero when reducing ChatGPT's creativity parameter.
“The variation approaches zero when reducing ChatGPT's creativity parameter, as illustrated in the Section 2 of the appendix.”
Quote in the paperNot a substantial claim - Findingown data
Even for 99% power, the ChatGPT estimates imply that less than 10 partisan respondents would be necessary to detect a difference from affective polarization in 2012.
“Even for 99% power, the ChatGPT estimates imply that less than 10 partisan respondents would be necessary to detect a difference from affective polarization in 2012”
Quote in the paperSubstantial claim - Findingown data
The responses provided by ChatGPT personas are overly precise, roughly one-third the standard deviation of the same measures recorded in the ANES.
“the responses provided by ChatGPT personas are overly precise -roughly one-third the standard deviation of the same measures recorded in the ANES”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Across facets, the average LLM exaggeration is roughly 3.3 times that of the ANES estimate.
“Across facets, the average LLM exaggeration is roughly 3.3 times that of the ANES estimate.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Digging deeper reveals extremist results when decomposing aggregated opinion and examining the synthetic opinions of underlying groups.
“digging deeper reveals extremist results when we decompose aggregated opinion and examine the synthetic opinions of underlying groups.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
ChatGPT's standard deviations are consistently between one-quarter and one-third the size of those found among human respondents to the ANES, with an overall average of roughly 0.314.
“ChatGPT's standard deviations are consistently between one-quarter and one-third the size of those found among human respondents to the ANES, with an overall average of roughly 0.314.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Section 4 of the appendix confirms that prompting ChatGPT to adopt the persona of an 'average' voter or citizen produces responses that are closer to those of Democrats than Republicans.
“Section 4 of the appendix confirms that prompting ChatGPT to adopt the persona of an”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
95% of profiles with two or more ANES respondents that expressed a feeling thermometer toward an outgroup had smaller standard deviations when estimated using ChatGPT relative to ANES.
“Overall, 95% of profiles with two or more ANES respondents that expressed a feeling thermometer toward an outgroup had smaller standard deviations when estimated using ChatGPT relative to”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Even at the maximum temperature setting, the ChatGPT estimates were far more precise than those found among ANES respondents.
“even at this maximum temperature setting, the ChatGPT estimates were far more precise than those found among ANES respondents.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Setting ChatGPT's temperature to 1 produces the noisiest and most representative results of actual randomness in human survey responses.
- Findingown data
Reducing the temperature value reduces the average standard deviation across target groups, regardless of aggregation.
“reducing the temperature value (x-axes) reduces the average standard deviation (y-axes) across target groups (rows), regardless of how coarse or how granular our aggregation of the personas is (columns).”
Quote in the paperSubstantial claim- The authors state this as an assumption: Setting ChatGPT's temperature to 1 produces the noisiest and most representative results of actual randomness in human survey responses.
- Findingown data
Of the seven profiles that have more extreme estimates in the ANES data, only four are statistically significant.
“of these seven profiles that have more extreme estimates in the ANES data, only four are statistically significant”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
LLM-based estimates are further from zero, indicating a statistically distinguishable difference in synthetic opinions, in every case where statistically significant differences obtain.
“LLM-based estimates are further from zero -indicating a statistically distinguishable difference in synthetic opinions -in every case where statistically significant differences obtain.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingreasoning
Of these seven profiles, only one (Black Democrats expressing their feelings toward Christians and Muslims) is substantively meaningful.
“only one (Black Democrats expressing their feelings toward Christians and Muslims) is substantively meaningful.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
In 29 of 36 race-by-party-by-target group conditions, LLM-derived measures of affective polarization (whether political parties or ideologies) are more extreme than those found among human respondents.
“In 29 of 36 race-by-party-by-target group conditions, LLM-derived measures of affective polarization -whether denominated in terms of political parties or ideologies -are more extreme than those found among human respondents.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
The only personae where this pattern doesn't hold on average are when prompted to adopt the identity of a conservative American, a Republican American, or a non-Hispanic white American.
“the only personae where this pattern doesn't hold on average are when we prompt it to adopt the identity of a conservative American, a Republican American, or a non-Hispanic white American.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
A majority of personae are more similar to human Democrats across a majority of target groups.
“Majority of personae are more similar to human Democrats across a majority of target groups.”
Quote in the paperNot a substantial claim - Findingown data
In these data, LLM estimates are an average of 6.5 times as extreme as those generated by humans.
“In these data, LLM estimates are an average of 6.5 times as extreme as those generated by humans”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Across all profiles with non-zero human respondents in the ANES, the average LLM exaggeration rises to roughly seven times that of the ANES estimate.
“Across all profiles with non-zero human respondents in the ANES, the average LLM exaggeration rises to roughly seven times that of the ANES estimate.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
On average, LLM estimates were only 40% as variable as their ANES counterparts.
“On average, LLM estimates were only 40% as variable as their ANES counterparts.”
Quote in the paperSubstantial claim- The authors state this as an assumption: Setting ChatGPT's temperature to 1 produces the noisiest and most representative results of actual randomness in human survey responses.
- Findingown data
Broken out by target group, ChatGPT's estimates are always further from 50 on average than the averages found among real humans responding to the ANES.
“Broken out by target group, we see consistent evidence that ChatGPT's estimates are always further from 50 on average than the averages found among real humans responding to the ANES.”
Quote in the paperSubstantial claim - Findingown data
LLM-generated opinions are roughly 40% as variable as humans.
“Our main analyses refer to several summary statistics to characterize the degree to which LLMgenerated opinions are more extreme (roughly 7 times as much as humans) or more precise (roughly 40% as variable as humans).”
Quote in the paperSubstantial claim - Findingown data
LLM-generated opinions are roughly 7 times as extreme as humans.
“Our main analyses refer to several summary statistics to characterize the degree to which LLMgenerated opinions are more extreme (roughly 7 times as much as humans) or more precise (roughly 40% as variable as humans).”
Quote in the paperSubstantial claim - Findingown data
The overall average ratio of the absolute value of the LLM's measure of polarization to that found in the ANES across all respondents and all groups is just under 7.
“The resulting overall average ratio across all respondents and all groups is just under 7”
Quote in the paperSubstantial claim - Findingown data
Figure 3 reveals a systematic bias toward out-group antipathy endemic in the LLM results relative to opinion in the ANES.
“Figure3plots the average differences by partisanship and race, revealing a systematic bias toward out-group antipathy endemic in the LLM results (dotted black bars) relative to opinion in the ANES (solid red bars).”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
The synthetic opinions provided by ChatGPT overstate societal divisions in nearly every case, regardless of which divisions or subgroup characteristics are assessed.
“The synthetic opinions provided by ChatGPT overstate societal divisions in nearly every case, regardless of which divisions or subgroup characteristics are assessed.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
Responses provided by ChatGPT are an average of seven times as polarized as the same measures recorded among actual human respondents.
“Responses provided by ChatGPT are an average of seven times as polarized as the same measures recorded among actual human respondents.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
In aggregate terms, the analysis reveals surprisingly accurate approximations of average sentiment by race.
“While our analysis reveals, in aggregate terms, surprisingly accurate approximations of average sentiment by race”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Findingown data
ChatGPT's estimates are consistently more polarized than those found among human respondents to the ANES.
“As illustrated, ChatGPT's estimates are consistently more polarized than those found among human respondents to the ANES,”
Quote in the paperSubstantial claim - Findingnone given
On average, across all covariate profiles and all target groups, synthetic responses gathered from ChatGPT are 4.88 times more extreme than those recorded among real human respondents to the ANES.
“On average, across all covariate profiles and all target groups, synthetic responses gathered from ChatGPT are 4.88 times more extreme than those recorded among real human respondents to the ANES.”
Quote in the paperSubstantial claim - Premisecitation
Assessments of public opinion based only on the most accessible and numerous groups risk missing important voices in an increasingly diverse polity.
“Assessments of public opinion based only on the most accessible and numerous groups risk missing important voices in an increasingly diverse polity (Brehm [1993], Berinsky [2004]).”
Quote in the paperSubstantial claim0 of 2 sources checked- cites Brehm [1993]
Not checked in the free run.
- cites Berinsky [2004]
Not checked in the free run.
- Interpretationown data
Even at the highest level of 'creativity', ChatGPT exhibits substantial overconfidence in the opinions of its adopted persona toward racial and social groups.
“Additionally, even at the highest level of “creativity”-a tuning parameter that should allow more within-prompt variation in responses-ChatGPT exhibits substantial overconfidence in the opinions of its adopted persona toward racial and social groups.”
Quote in the paperSubstantial claim - Study design and conductreasoning
The fact that the same number of observations are used in each profile for each method means that differences are not due to compositional differences between methods in the number of individuals associated with each profile.
“The fact that the same number of observations are used in each profile for each method means that differences are not due to compositional differences between methods in the number of individuals associated with each profile.”
Quote in the paperSubstantial claim - Study design and conductreasoning
Concerns about the representativeness of the ANES overall are less relevant for the analyses because the study relies on matched comparisons of otherwise similar respondents and personas.
“concerns about the representativeness of the ANES overall is less relevant for our analyses because we rely on matched comparisons of otherwise similar respondents and personas.”
Quote in the paperSubstantial claim - Study design and conductown data
Section 5 of the Supplemental Information demonstrates that similar results obtain when comparing the results to a survey conducted by the authors using a different mode and at a different time.
“To further assuage concerns about using the ANES as a basis for comparison, Section 5 of the Supplemental Information demonstrates that similar results obtain when comparing the results to a survey conducting by the authors using a different mode and at a different time.”
Quote in the paperNot a substantial claim - Interpretationown data
ChatGPT estimates are biased toward out-group antipathy.
“suggesting that ChatGPT estimates are biased toward out-group antipathy.”
Quote in the paperSubstantial claim - Interpretationown data
This indicates far more similarity in responses within a persona's profile than actually exists.
“indicating far more similarity in responses within a persona's profile than actually exists.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Study design and conductown data
The paper validates results using a bespoke survey fielded in 2021 that investigated affective polarization in the context of the Covid-19 pandemic.
“We also validate our results using a bespoke survey fielded in 2021 that investigated affective polarization in the context of the Covid-19 pandemic.”
Quote in the paperSubstantial claim - Interpretationown data
Our results raise real concerns about the ability of LLM-based synthetic opinions to accurately measure human opinion.
“Our results raise real concerns about the ability of LLM-based synthetic opinions to accurately measure human opinion”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Interpretationown data
The paper finds evidence of several troubling patterns that raise serious concerns about the use of LLMs as a substitute for characterizing public opinion.
“we find evidence of several troubling patterns that raise serious concerns about the use of LLMs as a substitute for characterizing public opinion.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Interpretationreasoning
The ChatGPT-implied sample size is an underestimate almost two orders of magnitude less than what would be expected from actual human responses.
“an underestimate that is almost two orders of magnitude less than what we would expect from actual human responses.”
Quote in the paperSubstantial claim - Interpretationown data
Replicating with larger samples confirms that substantive conclusions are strengthened.
“confirming that -if anything -our substantive conclusions are strengthened with these data.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
- Interpretationown data
The same methods validate the generalizability of results to a different period using a different sample from a different polling source.
“validating the generalizability of our results to a different period using a different sample from a different polling source.”
Quote in the paperSubstantial claim
- May combine two claims. 18
- Classified as probably not a substantial claim. 21
May combine two claims.
Findingown dataThe synthetic opinions of Republican personas towards the Republican party and Democratic personas towards the Democratic party are substantially warmer relative to ANES opinions.
“The synthetic opinions of Republican personas towards the Republican party and Democratic personas towards the Democratic party are substantially warmer relative to ANES opinions.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe differences between ChatGPT and ANES affective polarization amount to between half a standard deviation and a full standard deviation of the ANES distribution of attitudes for each partisan group, and between 10 and 20 points on the 0 to 100 thermometer scale.
“These differences are substantively meaningful, amounting to between half a standard deviation and a full standard deviation of the ANES distribution of attitudes for each partisan group, and between 10 and 20 points on the 0 to 100 thermometer scale.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe estimated differences in the LLM are far more precise than human opinion suggests; despite being equally sized by construction, the variation in average differences from the most 'creative' persona profile are far less than the variation we observe in human opinions among individuals matching that profile.
“As above, we also highlight that the estimated differences in the LLM are far more precise than human opinion suggests; despite being equally sized by construction, the variation in average differences from the most “creative” persona profile are far less than the variation we observe in human opinions among individuals matching that profile.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataChatGPT's attitudes toward all target groups are more similar to actual Democrats' attitudes than Republicans, with the exception of attitudes toward Jews, the Republican Party, and Muslims.
“ChatGPT's attitudes toward all target groups are more similar to actual Democrats' attitudes than Republicans, with the exception of attitudes toward Jews, the Republican Party, and Muslims.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
May combine two claims.
Findingown dataThe 2016-2020 pooled ANES values for the effect size and variance were 5.0 and 32.3, respectively.
“The second column records the calculation if we assume an effect size and variance equal to the 2016-2020 pooled ANES values (size 5.0, sd 32.3)”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThese overstatements are substantially more confident than those estimated with the ANES, with standard deviations across every profile that are only 31% the magnitude of those estimated with human respondents.
“these overstatements are substantially more confident than those estimated with the ANES, with standard deviations across every profile that are only 31% the magnitude of those estimated with human respondents.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenThere are many examples where an ANES respondent's attitudes are more polarized than its synthetic counterparts.
“although we note that there are many examples where an ANES respondent's attitudes are more polarized than its synthetic counterparts.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
May combine two claims.
Findingown dataThe ChatGPT estimates for the effect size and variance were 20.6 and 11.7, respectively.
“the third column is the same calculation with our ChatGPT estimates (size 20.6, sd 11.7).”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe generic American is more similar to the generic Democrat than to the generic Republican across almost all outcomes, and significantly more for groups more associated with Democrats.
“the generic American is more similar to the generic Democrat than to the generic Republican across almost all outcomes, and significantly more for groups more associated with Democrats.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
Classified as probably not a substantial claim. May combine two claims.
PremisecitationIncreasing costs, decreasing response rates, and rising concerns about the accuracy of polling have raised questions about the possible biases resulting from coverage or non-response bias in the use of polling and even the viability of polling itself.
“Increasing costs, decreasing response rates [Keeter et al., 2017], and rising concerns about the accuracy of polling [Kennedy et al., 2018, Clinton et al., 2021a, 2022] have raised questions about the possible biases resulting from coverage or non-response bias in the use of polling (e.g., Cavari and Freedman [2022] but see Mellon and Prosser [2021]) and even the viability of polling itself (Meyer”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Keeter et al., 2017
Not checked in the free run.
- cites Kennedy et al., 2018
Not checked in the free run.
- cites Clinton et al., 2021a, 2022
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOthers use sophisticated weighting methods (e.g., multilevel regression poststratification) to leverage observed relationships among respondents who are surveyed to characterize the opinions of those who are not.
“Others use sophisticated weighting methods (e.g., multilevel regression poststratification) to leverage observed relationships among respondents who are surveyed to characterize the opinions of those who are not (Gelman [1997], Lax and Phillips [2009], Ghitza and Gelman [2013], Caughey and Warshaw [2015], Bisbee [2019], Goplerud [2023]).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Gelman [1997]
Not checked in the free run.
- cites Lax and Phillips [2009]
Not checked in the free run.
- cites Ghitza and Gelman [2013]
Not checked in the free run.
Probably not a substantial claim: may not pass S4 (based).
Premisenone givenThe rise of Large Language Models offers the latest possibility of characterizing public opinion without actually polling the public.
“The rise of Large Language Models (hereafter LLMs) offers the latest possibility of characterizing public opinion without actually polling the public.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationSome researchers use non-survey data, most often from social media platforms like Twitter, to characterize public opinion.
“Some use non-survey data -most often from social media platforms like Twitter -to characterize public opinion (e.g., Beauchamp [2017], Tucker [2017], but see Bail [2022]).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Beauchamp [2017]
Not checked in the free run.
- cites Tucker [2017]
Not checked in the free run.
- cites Bail [2022]
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductreasoningTo remove the effect of sample sizes on comparisons, the study compares as many profiles using ChatGPT as there are individuals with the same profile in the 2016 and 2020 ANES common file.
“To remove the effect of sample sizes on our comparisons, we compare as many profiles using ChatGPT as there are individuals with the same profile in the 2016 and 2020 ANES common file.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductnone givenThe prompt phrasing was designed to mimic as closely as possible the specific wording used in the ANES and to generate responses in the same format as on the ANES.
“The phrasing of this prompt was designed to mimic as closely as possible the specific wording used in the ANES and to generate responses in the same format as on the ANES.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationResearchers are increasingly turning to other methods of characterizing public opinion and sentiment, especially for groups that are less numerous or harder to reach.
“Given rising costs and difficulties of interviewing respondents, researchers are increasingly turning to other methods of characterizing public opinion and sentiment -especially for groups that are less numerous or harder to reach (Wang et al. [2015], Ghitza and Gelman [2020], van Klingeren et al. [2021]).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Wang et al. [2015]
Not checked in the free run.
- cites Ghitza and Gelman [2020]
Not checked in the free run.
- cites van Klingeren et al. [2021]
Not checked in the free run.
May combine two claims.
InterpretationreasoningChatGPT portrays a world in which human opinion is both more polarized and more homogeneous within social groups than actual human opinion suggests.
“Simply put, ChaptGPT portrays a world in which human opinion is both more polarized and more homogeneous within social groups than actual human opinion suggests.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationPolls are more needed than ever given increasing concerns about polarization and democratic 'backsliding'.
“At the same time, polls are more needed than ever given increasing concerns about polarization and democratic “backsliding” (Graham [2023], Waldner and Lust [2018]).”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Graham [2023]
Not checked in the free run.
- cites Waldner and Lust [2018]
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisereasoningWorking papers on LLMs express a common optimism about the revolutionary potential of LLMs.
“These contributions differ in the specific applications imagined for the research community (i.e., labeling data [Törnberg, 2023, Gilardi et al., 2023], estimating ideology [Wu et al., 2023], and generating synthetic samples for pilot testing [Argyle et al., 2023, Horton, 2023]), but they all express a common optimism about the revolutionary potential of LLMs.”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Törnberg, 2023
Not checked in the free run.
- cites Gilardi et al., 2023
Not checked in the free run.
- cites Wu et al., 2023
Not checked in the free run.
May combine two claims.
InterpretationcitationThe exaggerated magnitude and undersized variance of ChatGPT responses both pose serious inferential problems, even if silicon sampling were only to be used for pre-analysis study design as some scholars have suggested.
“The exaggerated magnitude and undersized variance of ChatGPT responses both pose serious inferential problems, even if silicon sampling were only to be used for pre-analysis study design as some scholars have suggested [Argyle et al., 2023].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Argyle et al., 2023
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable).
PremisecitationPublic opinion polling is seemingly in a crisis.
“Public opinion polling is seemingly in a crisis (Shapiro [2019]).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Shapiro [2019]
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationFor-profit companies suggest that 'synthetic users' are a suitable replacement for human opinions and experiences when developing and marketing products.
“Perhaps most demonstrative of the runaway interest is the appearance of for-profit companies suggesting that “synthetic users” are a suitable replacement for human opinions and experiences when developing and marketing products (see, for example, https://www.syntheticusers.com/).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites https://www.syntheticusers.com/
Not checked in the free run.
May combine two claims.
PremisecitationVolumes of research have shown that humans online are less thoughtful and more hurtful; less able to reason and quicker to rely on stereotypes and heuristics; more willing to villainize those who disagree with them and less willing to engage in empathy; and more inclined to express themselves to signal group attachments.
“Volumes of research have shown that humans online are less thoughtful and more hurtful (Lapidot-Lefler and Barak [2012], Rowe [2015]); less able to reason and quicker to rely on stereotypes and heuristics (Halpern and Gibbs [2013]); more willing to villainize those who disagree with them and less willing to engage in empathy (Rowe [2015], Rossini [2022]); and -importantly for this paper -more”
Quote in the paperSubstantial claim0 of 3 sources checked- cites Lapidot-Lefler and Barak [2012]
Not checked in the free run.
- cites Rowe [2015]
Not checked in the free run.
- cites Halpern and Gibbs [2013]
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationThermometer scores have been used by social scientists to measure the amount of affect expressed between partisans and other racial and social groupings and are often central to characterizations and discussions about contemporary societal divisions.
“Thermometer scores have been used by social scientists to measure the amount of affect expressed between partisans (Druckman et al. [2021]) and other racial and social groupings and they are often central to characterizations and discussions about contemporary societal divisions (e.g., Iyengar and Westwood [2019], Finkel et al. [2020b], Kalmoe and Mason [2022]).”
Quote in the paperNot a substantial claim0 of 3 sources checked- cites Druckman et al. [2021]
Not checked in the free run.
- cites Iyengar and Westwood [2019]
Not checked in the free run.
- cites Finkel et al. [2020b]
Not checked in the free run.
Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
PremisecitationThe increasing antipathy between groups over time that such measures reveal is widely thought to be among the most serious issues facing contemporary society and governance.
“In fact, the increasing antipathy between groups over time that such measures reveal is widely thought to be among the most serious issues facing contemporary society and governance (Finkel et al. [2020a]).”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Finkel et al. [2020a]
Not checked in the free run.
May combine two claims.
InterpretationreasoningIf the ANES difference happens to be zero, whatever small difference found in the LLM results will be exaggerated.
“If the ANES difference happens to be zero (a likely event given the pressures of social desireability bias in which respondents want to be seen as egalitarian), whatever small difference found in the LLM results will be exaggerated.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenAn ANES difference of zero is a likely event given the pressures of social desirability bias in which respondents want to be seen as egalitarian.
“(a likely event given the pressures of social desireability bias in which respondents want to be seen as egalitarian)”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationown dataThese results confirm and expand upon the demonstrated imperfections that occur when using LLMs to learn about humans.
“These results confirm and expand upon the demonstrated imperfections that occur when using LLMs to learn about humans.”
Quote in the paperNot a substantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
Classified as probably not a substantial claim.
Interpretationnone givenRatios are sensitive to outliers, particularly in the denominator.
“However, ratios are sensitive to outliers, particularly in the denominator.”
Quote in the paperSubstantial claimMay combine two claims.
PremisecitationTraining an LLM requires an enormous corpus and closed-source LLMs typically rely on the Internet to collect this content.
“Training an LLM requires an enormous corpus and closed-source LLMs typically rely on the Internet to collect this content (see, for example, Washington Post [2023]).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Washington Post [2023]
Not checked in the free run.
Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).
PremisecitationSome scholars have argued for rejecting proprietary, closed-source LLMs.
“For this reason, some scholars have argued for rejecting proprietary, closed-source LLMs [Spirling, 2023].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Spirling, 2023
Not checked in the free run.
May combine two claims.
PremisecitationClosed-source LLMs are arguably more accessible and more likely to be used by users like journalists, politicians, and the modal academic.
“closed-source LLMs are arguably more accessible and more likely to be used by users like journalists, politicians, and the modal academic[Cowen, 2022].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Cowen, 2022
Not checked in the free run.
May combine two claims.
Interpretationown dataThe paper contributes another generalized addition: an exaggeration of public differences and the certainty about those differences, as well as the extent to which the perception of those differences is shared among otherwise similar individuals.
“But to this list we contribute another generalized addition: an exaggeration of public differences and the certainty about those differences, as well as the extent to which the perception of those differences is shared among otherwise similar individuals.”
Quote in the paperSubstantial claim- The authors state this as an assumption: The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
Classified as probably not a substantial claim.
PremisecitationLLMs are closed-box.
“Given the closed-box nature of such LLMs(Spirling [2023])”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Spirling [2023]
Not checked in the free run.
Classified as probably not a substantial claim.
PremisecitationIt is unknown whether closed-source LLMs can reproduce human opinion when prompted to adopt a particular persona.
“but it is unknown whether closed-source LLMs can reproduce human opinion when prompted to adopt a particular persona.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [2022], Abdulhai et al.
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationOthers have examined the 'default' persona of prominent LLMs to show that the AI has an ideological bias.
“While others have examined the “default” persona of prominent LLMs to show that the AI has an ideological bias (Santurkar et al. [2023], Rozado [2023]) -a result in line with what we show in Section 4 of the Appendix -we question whether closed-source LLMs are able to accurately recover the opinions of particular groups when specifically prompted to do so.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites Santurkar et al. [2023]
Not checked in the free run.
- cites Rozado [2023]
Not checked in the free run.
The paper relies on this without support.
In the 2012 ANES, the average gap between in-party and out-party assessments among partisans was 47.4.
“when the average gap between in-party and out-party assessments among partisans in the ANES was 47.4.”
The authors state this as an assumption.
Setting ChatGPT's temperature to 1 produces the noisiest and most representative results of actual randomness in human survey responses.
“In theory, these results should be the noisiest and, as such, the most representative of the actual randomness in human survey responses.”
The authors state this as an assumption.
The differences in feeling thermometer scores between groups are valid proxies for affective polarization, partisan sectarianism, racial antipathy, and religious antipathy.
“substantively understood as proxies for either affective polarization (conservatives minus liberals), partisan sectarianism (Republican Party minus Democratic Party), racial antipathy (Whites minus Blacks), and religious antipathy (Christians minus Muslims)”
The paper relies on this without support.
ANES is a nationally representative public opinion poll of U.S. politics.
“Our main results rely on ANES data, one of the most well-known nationally representative public opinion polls of U.S. politics.”
The paper relies on this without support.
Public opinion surveys are designed to recover the true distribution of opinions in the population.
“whereas public opinion surveys are designed to recover the true distribution of opinions in the population”
The paper relies on this without support.
Trade secrets prevent us from knowing the precise recipe that generates responses from ChatGPT.
“Trade secrets prevent us from knowing the precise recipe that generates responses from ChatGPT”
The paper relies on this without support.
The LLM's relationship to humanity is mostly unknown.
“an algorithm whose relationship to humanity is mostly unknown”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GPT 3.5 turbo produces accurate mean estimates of attitudes within a population, but understates variances-exaggerating extreme attitudes.
“At the same time, more recent research indicates GPT 3.5 turbo produces accurate mean estimates of attitudes within a population, but understates variances—exaggerating extreme attitudes (13).”
From Can Generative AI improve social science? (2024) · cites this paper as [13]
LLMs represent some demographic subgroups more accurately than others.
“Studies also indicate LLMs represent some demographic subgroups more accurately than others (13, 15).”
From Can Generative AI improve social science? (2024) · cites this paper as [13]
As LLMs evolve in response to user behavior and ongoing events, this creates a significant challenge for reproducible research with Generative AI.
“Finally, social scientists must consider broader forms of temporal validity (13, 75). As LLMs evolve in response to user behavior in different ways—as well as ongoing events in the world—this will create another significant challenge for those who strive for reproducible research with Generative AI.”
From Can Generative AI improve social science? (2024) · cites this paper as [13]
Studies indicate subtle differences in prompt wording can produce very different output within the same LLM.
“Studies indicate subtle differences in the wording of prompts can also produce very different output within the same LLM (13).”
From Can Generative AI improve social science? (2024) · cites this paper as [13]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Profile-generation code uses four education levels, not three
MajorTable 1 combines three education categories with the other listed traits to produce 1,080 profiles. The supplied code instead separates bachelor's and postgraduate degrees, producing 1,440 combinations, so the stated profile count does not describe the code shown.
“educ = c ( ' high ␣ school ␣ diploma ' , " some ␣ college , ␣ but ␣ no ␣ degree " , " bachelor 's ␣ degree " , " postgraduate ␣ degree " )”
“1,080 unique demographic profiles.10 Age 20, 35, 50, 65 Race non-Hispanic White, non-Hispanic Black, Hispanic Gender Male, Female Education High school diploma, some college but no degree, Bachelor's degree or higher Income $30k, $50k, $80k, $100k, more than $150k Partisanship Democrat, Independent, Republican Table 1: Synthetic profiles”
Cites
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
The Origins and Consequences of Affective Polarization in the United States
Iyengar, S., Lelkes, Y., Levendusky, M., Malhotra, N., & Westwood, S. J. (2019) · Annual Review of Political Science · doi:10.1146/annurev-polisci-051117-073034 · Cited by 3,317 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
The Statistical Crisis in Science
Gelman, A., & Loken, E. (2014) · American Scientist · doi:10.1511/2014.111.460 · Cited by 963 · see citing works
Political Sectarianism in America
Finkel, E. J., Bail, C. A., Cikara, M., Ditto, P. H., Iyengar, S., Klar, S., Mason, L., McGrath, M. C., Nyhan, B., Rand, D. G., et al. (2020) · Science · doi:10.1126/science.abe1715 · Cited by 931 · see citing works
Unwelcome Change: Coming to Terms with Democratic Backsliding
Waldner, D., & Lust, E. (2018) · Annual Review of Political Science · doi:10.1146/annurev-polisci-050517-114628 · Cited by 889 · see citing works
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838 · see citing works
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · see citing works
Effects of anonymity, invisibility, and lack of eye-contact on toxic online disinhibition
Lapidot‐Lefler, N., & Barak, A. (2011) · Computers in Human Behavior · doi:10.1016/j.chb.2011.10.014 · Cited by 709 · see citing works
Halpern, D. M., & Gibbs, J. L. (2012) · Computers in Human Behavior · doi:10.1016/j.chb.2012.10.008 · Cited by 643 · see citing works
What do we measure when we measure affective polarization?
Druckman, J. N., & Levendusky, M. S. (2019) · Public Opinion Quarterly · doi:10.1093/poq/nfz003 · Cited by 615 · see citing works
Meyer, B., Mok, W. K. C., & Sullivan, J. X. (2015) · The Journal of Economic Perspectives · doi:10.1257/jep.29.4.199 · Cited by 556 · see citing works
How Should We Estimate Public Opinion in The States?
Lax, J. R., & Phillips, J. H. (2008) · American Journal of Political Science · doi:10.1111/j.1540-5907.2008.00360.x · Cited by 526 · see citing works
Affective polarization, local contexts and public opinion in America
Druckman, J. N., Klar, S., Krupnikov, Y., Levendusky, M., & Ryan, J. B. (2020) · Nature Human Behaviour · doi:10.1038/s41562-020-01012-5 · Cited by 463 · see citing works
Can AI language models replace human participants?
Dillion, D., Tandon, N., Gu, Y., & Gray, K. (2023) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.04.008 · Cited by 405 · see citing works
Forecasting elections with non-representative polls
Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2014.06.001 · Cited by 371 · see citing works
(Mis)perceptions of Partisan Polarization in the American Public
Levendusky, M. S., & Malhotra, N. (2016) · Public Opinion Quarterly · doi:10.1093/poq/nfv045 · Cited by 356 · see citing works
Civility 2.0: a comparative analysis of incivility in online political discussion
Rowe, I. (2014) · Information Communication & Society · doi:10.1080/1369118x.2014.940365 · Cited by 338 · see citing works
Rossini, P. (2020) · Communication Research · doi:10.1177/0093650220921314 · Cited by 332 · see citing works
More human than human: measuring ChatGPT political bias
Motoki, F., Pinho Neto, V., & Rodrigues, V. (2024) · Public Choice · doi:10.1007/s11127-023-01097-2 · Cited by 329 · see citing works
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
Horton, J., Filippas, A., & Manning, B. (2023) · National Bureau of Economic Research · doi:10.3386/w31122 · Cited by 329 · see citing works
The Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281 · see citing works
The phantom respondents: opinion surveys and political representation
(1993) · Choice Reviews Online · doi:10.5860/choice.31-2316 · Cited by 276 · see citing works
An Evaluation of the 2016 Election Polls in the United States
Kennedy, C., Blumenthal, M., Clement, S., Clinton, J. D., Durand, C., Franklin, C., McGeeney, K., Miringoff, L., Olson, K., Rivers, D., et al. (2018) · Public Opinion Quarterly · doi:10.1093/poq/nfx047 · Cited by 200 · see citing works
Self-citationBreaking the social media prism: how to make our platforms less polarizing
Sufi, M. (2023) · Information Communication & Society · doi:10.1080/1369118x.2023.2258608 · Cited by 187 · see citing works
Deep Interactions with MRP: Election Turnout and Voting Patterns Among Small Electoral Subgroups
Ghitza, Y., & Gelman, A. (2013) · American Journal of Political Science · doi:10.1111/ajps.12004 · Cited by 181 · see citing works
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study
Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., & Hershcovich, D. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.7 · Cited by 163 · see citing works
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, G., Arriaga, R. I., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2208.10264 · Cited by 161 · see citing works
Törnberg, P. (2023) · arXiv · doi:10.48550/arxiv.2304.06588 · Cited by 152 · see citing works
Predicting and Interpolating State‐Level Polls Using Twitter Textual Data
Beauchamp, N. (2016) · American Journal of Political Science · doi:10.1111/ajps.12274 · Cited by 132 · see citing works
Silent Voices: Public Opinion and Political Participation in America
Berinsky, A. J. (2004) · Project Muse (Johns Hopkins University) · doi:10.1515/9781400850747 · Cited by 121 · see citing works
Dynamic Estimation of Latent Opinion Using a Hierarchical Group-Level IRT Model
Caughey, D., & Warshaw, C. (2015) · Political Analysis · doi:10.1093/pan/mpu021 · Cited by 114 · see citing works
Whose Opinions Do Language Models Reflect?
Santurkar, S., Durmus, E., Ladhak, F., Lee, C., Liang, P., & Hashimoto, T. (2023) · arXiv · doi:10.48550/arxiv.2303.17548 · Cited by 114 · see citing works
Why open-source generative AI models are an ethical way forward for science
Spirling, A. (2023) · Nature · doi:10.1038/d41586-023-01295-4 · Cited by 104 · see citing works
The Self-Perception and Political Biases of ChatGPT
Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., Roidl, M., & Pauly, M. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/7115633 · Cited by 98 · see citing works
Measuring Public Opinion with Social Media Data
Klašnja, M., Barberá, P., Beauchamp, N., Nagler, J., & Tucker, J. A. (2017) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190213299.013.3 · Cited by 58 · see citing works
More Human than Human: Measuring ChatGPT Political BiasPreprint
Motoki, F. Y. S., Neto, V. P., & Rodrigues, V. R. D. S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4372349 · Cited by 55 · see citing works
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
Horton, J. J., Filippas, A., & Manning, B. S. (2023) · arXiv · doi:10.48550/arxiv.2301.07543 · Cited by 47 · see citing works
How to train your stochastic parrot: large language models for political texts
Ornstein, J. T., Blasingame, E., & Truscott, J. S. (2025) · Political Science Research and Methods · doi:10.1017/psrm.2024.64 · Cited by 46 · see citing works
Moral Foundations of Large Language Models
Abdulhai, M., Serapio-García, G., Crepy, C., Valter, D., Canny, J., & Jaques, N. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.982 · Cited by 45 · see citing works
Silent Voices: Public Opinion and Political Participation in America
Bloch‐Elkon, Y. (2005) · Political Science Quarterly · doi:10.1002/j.1538-165x.2005.tb01397.x · Cited by 38 · see citing works
BARP: Improving Mister P Using Bayesian Additive Regression Trees
Bisbee, J. H. (2019) · American Political Science Review · doi:10.1017/s0003055419000480 · Cited by 38 · see citing works
Self-citationThe Oxford Handbook of Polling and Polling Methods
(2015) · Oxford University Press eBooks · doi:10.1093/oxfordhb/9780190213299.001.0001 · Cited by 30 · see citing works
Cavari, A., & Freedman, G. (2022) · American Political Science Review · doi:10.1017/s0003055422000399 · Cited by 28 · see citing works
Reluctant Republicans, Eager Democrats?
Clinton, J. D., Lapinski, J. S., & Trussler, M. J. (2022) · Public Opinion Quarterly · doi:10.1093/poq/nfac011 · Cited by 25 · see citing works
Self-citationvan Klingeren, M., Trilling, D., & Möller, J. E. (2020) · Acta Politica · doi:10.1057/s41269-020-00160-w · Cited by 24 · see citing works
The Impact of Survey Non-response on Survey Accuracy
Keeter, S. (2017) · Springer eBooks · doi:10.1007/978-3-319-54395-6_47 · Cited by 12 · see citing works
Correlation with Time Explains the Relationship between Survey Nonresponse and Mass Polarization
Mellon, J., & Prosser, C. (2020) · The Journal of Politics · doi:10.1086/709433 · Cited by 12 · see citing works
Goplerud, M. (2023) · American Political Science Review · doi:10.1017/s0003055423000035 · Cited by 12 · see citing works
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?Preprint
Horton, J. J., Filippas, A., & Manning, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4413859 · Cited by 12 · see citing works
Bisbee, J. H. (2024) · Harvard Dataverse · doi:10.7910/dvn/vpn481 · Cited by 1 · see citing works
Self-citationLarge Language Models Can Be Used to Estimate the Latent Positions of Politicians
Wu, P. Y., Nagler, J., Tucker, J. A., & Messing, S. (2023) · arXiv · doi:10.48550/arxiv.2303.12057
The Self-Perception and Political Biases of ChatGPT
Rutinowski, J., Franke, S., Endendyk, J., Dormuth, I., & Pauly, M. (2023) · arXiv · doi:10.48550/arxiv.2304.07333
Flagged first, then by citations
Research works printed but not linked: 4
- E. J. Finkel, C. A. Bail, M. Cikara, P. H. Ditto, S. Iyengar, S. Klar, L. Mason, M. C. McGrath, B. Nyhan, D. G. Rand, et al. Political sectarianism in america. Science, 370(6516):533-536, 2020b.
- Y. Ghitza and A. Gelman. Voter Registration Databases and MRP: Toward the Use of Large-Scale Databases in Public Opinion Research. Political Analysis, 28 (4):507-531, Oct. 2020. ISSN 1047-1987, 1476-4989. doi: 10.1017/pan.2020. · DOI
- S. Iyengar and S. J. Westwood. The origins and consequences of affective polarization. Annual Review of Political Science, 22(1):129-146, 2019.
- N. P. Kalmoe and L. Mason. Radical American partisanship: Mapping violent hostility, its causes, and the consequences for democracy. University of Chicago Press, 2022.
Cited by
Sorted by citations
Can Generative AI improve social science?
Bail, C. A. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314021121 · Cited by 369 · see citing works
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2308950121 · Cited by 297 · see citing works
Artificial Intelligence and Democracy: A Conceptual Framework
Jungherr, A. M. (2023) · Social Media + Society · doi:10.1177/20563051231186353 · Cited by 217 · see citing works
Using LLMs for Market Research
Brand, J., Israeli, A., & Ngwe, D. (2023) · doi:10.2139/ssrn.4395751 · Cited by 190 · see citing works
Large language models as a substitute for human experts in annotating political text
Heseltine, M., & von Hohenberg, B. C. (2024) · Research & Politics · doi:10.1177/20531680241236239 · Cited by 116 · see citing works
Performance and biases of large language models in public opinion simulation.
Qu, Y., & Wang, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03609-x · Cited by 79 · see citing works
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52 · see citing works
The use of ChatGPT for personality research: Administering questionnaires using generated personas
de Winter, J., Driessen, T., & Dodou, D. (2024) · Personality and Individual Differences · doi:10.1016/j.paid.2024.112729 · Cited by 49 · see citing works
Generative AI Meets Open-Ended Survey Responses: Research Participant Use of AI and Homogenization
Zhang, S., Xu, J., & Alvero, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251327130 · Cited by 48 · see citing works
Quaquebeke, N. V., Tonidandel, S., & Banks, G. C. (2025) · The Leadership Quarterly · doi:10.1016/j.leaqua.2025.101895 · Cited by 41 · see citing works
Davidson, T., & Karell, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251339184 · Cited by 34 · see citing works
Kozlowski, A. C., & Evans, J. A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251337316 · Cited by 34 · see citing works
Machine Bias. How Do Generative Language Models Answer Opinion Polls?
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2025) · Sociological Methods & Research · doi:10.1177/00491241251330582 · Cited by 30 · see citing works
Large language models can predict the results of social science experiments
Ashokkumar, A., Hewitt, L., Ghezae, I., & Willer, R. (2026) · Nature · doi:10.1038/s41586-026-10742-x · Cited by 25 · see citing works
The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations
Broska, D., Howes, M., & van Loon, A. (2025) · Sociological Methods & Research · doi:10.1177/00491241251326865 · Cited by 18 · see citing works
Demonstrations of the Potential of AI-based Political Issue Polling
Sanders, N. E., Ulinich, A., & Schneier, B. (2023) · Harvard Data Science Review · doi:10.1162/99608f92.1d3cf75d · Cited by 25 · see citing works
Balancing Large Language Model Alignment and Algorithmic Fidelity in Social Science Research
Lyman, A., Hepner, B., Argyle, L. P., Busby, E. C., Gubler, J. R., & Wingate, D. (2025) · Sociological Methods & Research · doi:10.1177/00491241251342008 · Cited by 19 · see citing works
Simulating Human Opinions with Large Language Models
Kaiser, C., Kaiser, J., Manewitsch, V., Rau, L., & Schallner, R. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3708319.3733685 · Cited by 12 · see citing works
Public engagement with health and climate change around the world: a Google Trends analysis
Dasandi, N., Mikhaylov, S., Pantera, D. K., & Romanello, M. B. (2025) · The Lancet Planetary Health · doi:10.1016/s2542-5196(25)00029-4 · Cited by 13 · see citing works
Karell, D., Sachs, J. D., & Barrett, R. (2024) · Poetics · doi:10.1016/j.poetic.2024.101966 · Cited by 10 · see citing works
Towards a political economy of algorithmic capitalism
Egan, M. (2025) · Capital & Class · doi:10.1177/03098168251326189 · Cited by 13 · see citing works
Sun, L., An, Y.-H., Song, X., & Liu, J. (2025) · Electronic Commerce Research · doi:10.1007/s10660-025-09990-2 · Cited by 9 · see citing works
Kreuter, F. (2025) · Journal of Official Statistics · doi:10.1177/0282423x251318452 · Cited by 9 · see citing works
Comments on “ AI and the advent of the cyborg behavioral scientist”
Blythe, P. A., Kulis, C., McGraw, A. P., Haenlein, M., Hewett, K., Yoo, K., Wood, S. L., Morwitz, V. G., & Huber, J. C. (2025) · Journal of Consumer Psychology · doi:10.1002/jcpy.1453 · Cited by 7 · see citing works
Vox Populi, Vox AI? Using Large Language Models to Estimate German Vote Choice
von der Heyde, L., Haensch, A., & Wenz, A. (2025) · Social Science Computer Review · doi:10.1177/08944393251337014 · Cited by 12 · see citing works
Using large language models for survey research in communication: opportunities and challenges
Valenzuela, S., Winter, S., & Rivera, S. (2025) · Communication and Change · doi:10.1007/s44382-025-00014-z · Cited by 8 · see citing works
LLMs and Cultural Values: The Impact of Prompt Language and Explicit Cultural Framing
Bulté, B., & Terryn, A. R. (2025) · Computational Linguistics · doi:10.1162/coli.a.583 · Cited by 10 · see citing works
Simulating social perception with large language models: perceptions of China’s common prosperity
Jiang, Z., Huang, B., Ge, J., Lin, C., Xu, Y., & Yu, J. (2025) · Journal of Chinese Governance · doi:10.1080/23812346.2025.2603822 · Cited by 9 · see citing works
Generative AI in Participatory Urban Planning: Synthetic Inhabitants and Experts
Jauhiainen, J. S., Hakanpää, S., Honkasaari, H.-P., Kivilompolo, N., Kurri, M., Lehtiranta, L., & Nurminen, M. (2026) · Land · doi:10.3390/land15030407 · Cited by 7 · see citing works
Hwang, A. H., Bernstein, M. S., Sundar, S. S., Zhang, R., Ribeiro, M. H., Lu, Y., Chang, S. Y., Wu, T., Yang, A., Williams, D. C., et al. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3716299 · Cited by 6 · see citing works
Arti-‘fickle’ intelligence: using LLMs as a tool for inference in the political and social sciences
Argyle, L. P., Busby, E. C., Gubler, J. R., Hepner, B., Lyman, A., & Wingate, D. (2025) · Nature Computational Science · doi:10.1038/s43588-025-00843-4 · Cited by 13 · see citing works
Vogelsmeier, L. V. D. E., de Oliveira, E. A., Misiejuk, K., López‐Pernas, S., & Saqr, M. (2025) · Computers in Human Behavior · doi:10.1016/j.chb.2025.108769 · Cited by 6 · see citing works
Abramson, C. M., Prendergast, T., Li, Z., & Dohan, D. P. (2026) · Annual Review of Sociology · doi:10.1146/annurev-soc-011824-104836 · Cited by 6 · see citing works
Wu, P. Y., Nagler, J., Tucker, J. A., & Messing, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825235 · Cited by 8 · see citing works
How do Generative Language Models Answer Opinion Polls?Preprint
Boelaert, J., Coavoux, S., Ollion, É., Petev, I. D., & Präg, P. (2024) · doi:10.31235/osf.io/r2pnb · Cited by 7 · see citing works
Modeling Trump’s Worldview with Algorithms: Power, Hierarchy, and Transactionalism
Woods, D. (2025) · Chinese Political Science Review · doi:10.1007/s41111-025-00315-0 · Cited by 5 · see citing works
Grizzard, M., Frazer, R., Luttrell, A., Monge, C. K., Matthews, N. L., Francemone, C. J., & Frazer, M. (2025) · Scientific Reports · doi:10.1038/s41598-025-24700-6 · Cited by 5 · see citing works
Science in the era of ChatGPT, large language models and generative AI
Pournaras, E. (2023) · AI critique · doi:10.1515/9783839467664-015 · Cited by 10 · see citing works
Applications of GPT in Political Science Research: Extracting Information from Unstructured Text
Lee, K., Paci, S., Park, J., You, H. Y., & Zheng, S. (2025) · PS Political Science & Politics · doi:10.1017/s1049096525000046 · Cited by 6 · see citing works
Assessing Bias in LLM-Generated Synthetic Datasets: The Case of German Voter BehaviorPreprint
von der Heyde, L., Haensch, A., & Wenz, A. (2023) · doi:10.31235/osf.io/97r8s · Cited by 5 · see citing works
Kaur, A., Aird, A., Borman, H., Nicastro, A., Leontjeva, A., Pizzato, L. A. S., & Jermyn, D. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728339 · Cited by 4 · see citing works
How would AI define neighborhood boundaries? A comparison with human-crowdsourced data
Lin, Y., Bae, C. J., & Talen, E. (2025) · Environment and Planning B Urban Analytics and City Science · doi:10.1177/23998083251369570 · Cited by 4 · see citing works
Generative ai and sociology: analyzing virtual respondent technology
Puzanova, Z. V., Koshoridze, G. G., & Koshoridze, D. G. (2025) · Sociology methodology methods mathematical modeling (Sociology 4M) · doi:10.19181/4m.2025.34.1.6 · Cited by 4 · see citing works
Ethnography and Ethnographic Sensibility in Political Science
Fu, D., Nielsen, R. A., & Schatz, E. A. D. (2026) · Annual Review of Political Science · doi:10.1146/annurev-polisci-041224-084600 · Cited by 4 · see citing works
Digital twins are funhouse mirrors: Five systematic distortions
Peng, T., Brucks, M. S., Gui, G., Merlau, D. J., Fan, G. J., Sliman, M. B., Johnson, E. J., Althenayyan, A., Bellezza, S., Donati, D., et al. (2026) · Science Advances · doi:10.1126/sciadv.aeh8260 · Cited by 4 · see citing works
Artificial Agents in Operations Management ExperimentsPreprint
Kirshner, S. N. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4726933 · Cited by 4 · see citing works
The Use of Large Language Models in HCI: A Critical Analysis of Synthetic Users
Salminen, J. O., Amin, D., Jung, S., & Jansen, B. J. J. (2025) · Augmented Humans International Conference (AHs) · doi:10.1145/3745900.3746108 · Cited by 3 · see citing works
Matthes, J., & Vranken, S. (2025) · Communication and Change · doi:10.1007/s44382-025-00019-8 · Cited by 3 · see citing works
GenAI for qualitative management accounting research: A commentary
Goretzki, L., & Mahlendorf, M. D. (2026) · Management Accounting Research · doi:10.1016/j.mar.2026.100962 · Cited by 3 · see citing works
Evaluating the statistical realism of LLM-generated social science data
Xie, Y., Liang, L., Li, S., Lu, Y., Xiao, Z., Shi, M., Huang, J., Wang, M., & Xie, Y. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2538145123 · Cited by 3 · see citing works
Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research
Madden, E. R. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0022 · Cited by 3 · see citing works
Geographic Limitations of Generative AI Models
Kim, J. (2025) · Springer geography · doi:10.1007/978-3-031-87421-5_20 · Cited by 3 · see citing works
Vox Populi, Vox AI? Using Language Models to Estimate German Public OpinionPreprint
von der Heyde, L., Haensch, A., & Wenz, A. (2023) · doi:10.31235/osf.io/8je9g · Cited by 3 · see citing works
Collaborative Growth: When Large Language Models Meet Sociolinguistics
Nguyen, D. (2025) · Language and Linguistics Compass · doi:10.1111/lnc3.70010 · Cited by 5 · see citing works
Utility of decisions and beliefs based on bounded-confidence opinion dynamics with group conformity
Xiao, K., & Zhang, H. (2025) · Chaos Solitons & Fractals · doi:10.1016/j.chaos.2025.116294 · Cited by 2 · see citing works
GenAI vs. Human Fact-Checkers: Accurate Ratings, Flawed Rationales
Tai, Y. C., Patni, K. N., Hemauer, N., Desmarais, B. A., & Lin, Y. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717896 · Cited by 2 · see citing works
Collectivism and individualism political bias in large language models: A two-step approach
Xiao-bo, S. H. A. N., Teng, Y., Wang, Y., Zhao, H., & Wang, Y. (2025) · Big Data & Society · doi:10.1177/20539517251343861 · Cited by 2 · see citing works
Question form Matters: Examining Trust in Government Through Open and Closed Survey Items
Bernhard-Harrer, J., & Pfaff, K. (2025) · Journal of Survey Statistics and Methodology · doi:10.1093/jssam/smaf014 · Cited by 2 · see citing works
Building Lovemarks with social media marketing: A methodological comparison
Westland, J. C. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104503 · Cited by 2 · see citing works
Affective connotations according to LLMs: implications for meaning measurement and cultural bias
Combs, A., Dametto, D., Blaison, C., Leung, R., Malhotra, A., Schröder, T., Hoey, J., & Smith‐Lovin, L. (2025) · Cognition & Emotion · doi:10.1080/02699931.2025.2568551 · Cited by 2 · see citing works
Llms, Virtual Users, and Bias: Predicting Any Survey Question Without Human Data
Sinacola, E., Pachot, A., & Petit, T. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-94892-3_29 · Cited by 2 · see citing works
A large-scale evaluation of commonsense knowledge in humans and large language models
Nguyen, T. D., Watts, D. J., & Whiting, M. E. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag029 · Cited by 3 · see citing works
Newer, Larger, Better? A Critique of the Unreflective LLM Adoption in Communication Research
Balluff, P., Ho, J. C., Gruber, J. B., Palicki, S.-K., Palmer, A., Rossi, L., Shklovski, I., & Chan, C. (2026) · Political Communication · doi:10.1080/10584609.2026.2618486 · Cited by 2 · see citing works
Wise, A. F., Gresalfi, M. S., & Spencer-Smith, J. B. (2026) · International Journal of Qualitative Methods · doi:10.1177/16094069261435579 · Cited by 2 · see citing works
Computational Text Analysis for Building and Testing Social Theory
Bodell, M. H., Keuschnigg, M., Macanovic, A., & Menshikova, A. (2026) · KZfSS Kölner Zeitschrift für Soziologie und Sozialpsychologie · doi:10.1007/s11577-026-01065-w · Cited by 2 · see citing works
Califano, G., & Caracciolo, F. (2026) · Food Quality and Preference · doi:10.1016/j.foodqual.2026.105930 · Cited by 2 · see citing works
Opinion dynamics: Statistical physics and beyond
Starnini, M., Baumann, F., Galla, T., Garcia, D., Íñiguez, G., Karsai, M., Lorenz, J., & Sznajd-Weron, K. (2026) · Reviews of Modern Physics · doi:10.1103/j1zg-ddqv · Cited by 16 · see citing works
On using large language models to support social research for climate action
Nabavi, E., Maier, H. R., Razavi, S., Hindes, A., Howden, M. S., Grant, W. J., & Raman, S. (2026) · npj Climate Action · doi:10.1038/s44168-026-00375-1 · Cited by 2 · see citing works
Yang, Y., Qi, J., & Ji, X. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2683905 · Cited by 2 · see citing works
Tripp, A. (2025) · Research & Politics · doi:10.1177/20531680251332353 · Cited by 2 · see citing works
Woolsey, C., Gondy Leroy, A., & Maltman, N. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128151 · Cited by 2 · see citing works
The Mixed Subjects Design: Treating Large Language Models as Potentially Informative ObservationsPreprint
Broska, D., Howes, M., & van Loon, A. (2024) · doi:10.31235/osf.io/j3bnt · Cited by 2 · see citing works
LLM-PDM: An LLM Persona-Driven Method for replicating personal mobility preferences at scale
Tzachristas, I., Narayanan, S., & Antoniou, C. (2026) · Communications in Transportation Research · doi:10.26599/commtr.2026.9640004 · Cited by 1 · see citing works
Synthetic Survey Data Generation and Evaluation
Jiang, Y., Liang, S., & Choi, J. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3690624.3709421 · Cited by 1 · see citing works
Against AI Jurisprudence: Large Language Models and the False Promises of Empirical Judging
Pruss, D., & Allen, J. (2025) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1609/aies.v8i3.36695 · Cited by 1 · see citing works
Olson, K. M., & Buskirk, T. D. (2025) · International Journal of Market Research · doi:10.1177/14707853251389789 · Cited by 1 · see citing works
Wang, S. (2025) · AI Magazine · doi:10.1002/aaai.70046 · Cited by 1 · see citing works
Marengo, D., & Longobardi, C. (2025) · Assessment · doi:10.1177/10731911251406405 · Cited by 1 · see citing works
ChatGPT as a research proxy: simulating human attitudes in social science research
Семеновна, Р. А., & Воронин, А. Н. (2026) · Journal of Computational Social Science · doi:10.1007/s42001-025-00452-1 · Cited by 2 · see citing works
Reframing the Performance and Ethics of Empathic AI: Wisdom of the Crowd and Placebos
Thornton, M. A. (2026) · Perspectives on Psychological Science · doi:10.1177/17456916261419718 · Cited by 1 · see citing works
Artificially intelligent opinion polling
Cerina, R., & Duch, R. M. (2026) · Royal Society Open Science · doi:10.1098/rsos.251150 · Cited by 1 · see citing works
Nezgovorova, M. I. (2026) · Sociopolitical sciences · doi:10.33693/2223-0092-2026-16-1-179-186 · Cited by 1 · see citing works
Artificial Intelligence Perspectives in Survey Data Analysis
Kenett, R. S. (2026) · International Statistical Review · doi:10.1111/insr.70040 · Cited by 1 · see citing works
Madhavan, M. M., Satyapriya, Sahu, S., Velayudhan, P. K., Smitha, S. C., Verma, A. P., Pawariya, V., Barua, S., Mahra, G. S., Bishnoi, S., et al. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1769064 · Cited by 1 · see citing works
Lin, Z. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261454231 · Cited by 1 · see citing works
Turchetta, A., Infascelli, L., & Mahoud, M. (2026) · International Journal of Quality and Service Sciences · doi:10.1108/ijqss-11-2025-0304 · Cited by 1 · see citing works
Iwamoto, R., Ishihara, T., & Ida, T. (2026) · Journal of Behavioral and Experimental Economics · doi:10.1016/j.socec.2026.102634 · Cited by 1 · see citing works
Sudmann, A., Echterhölter, A., Ramsauer, M., Retkowski, F., Schröter, J., & Waibel, A. (2023) · AI critique · doi:10.1515/9783839467664 · Cited by 4 · see citing works
Anduiza, E., Breitenstein, S., & Tien, C. P. (2025) · Polity · doi:10.1086/734739 · Cited by 1 · see citing works
Towards More Accurate US Presidential Election via Multi-step Reasoning with Large Language ModelsPreprint
Yu, C., Weng, Z., Li, Z., Hu, X., & Zhao, Y. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4996124 · Cited by 1 · see citing works
Buskirk, T. D., Eck, A., & Timbrook, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5377878 · Cited by 1 · see citing works
Peng, T., Gui, G., Merlau, D. J., Fan, G. J., Sliman, M. B., Brucks, M. S., Johnson, E., Morwitz, V. G., Althenayyan, A., Bellezza, S., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5518418 · Cited by 1 · see citing works
Chrono-Sampling: Generative AI Enabled Time Machine for Public Opinion Data CollectionPreprint
Tamaki, E. R., & Littvay, L. (2024) · doi:10.31234/osf.io/49ags
Семеновна, Р. А., & Воронин, А. Н. (2024) · Research Square · doi:10.21203/rs.3.rs-5425532/v1
What does AI consider praiseworthy?
Peterson, A. (2025) · AI and Ethics · doi:10.1007/s43681-025-00682-z
ZHANG, C., Wei, L., & Jiang, S. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5008330
Revolutionizing Market Research with Artificial Intelligence
Teutloff, J. K. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-6437-6.ch018
Navigating the Sociopolitical Landscape of AI in Smart Cities
Reuter, T. K., & Lian, J. “. (2025) · Oxford University Press eBooks · doi:10.1093/9780198945215.003.0095
Ko, H., & Williams, N. W. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5118151
A Survey of LLM-Based Methods for Synthetic Data Generation and the Rise of Agentic Workflows
Alismail, A., & Lanquillon, C. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-93418-6_9
- Authors
- Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J.
- Affiliations
- Vanderbilt University (US)
- Corresponding author
- Bisbee, James
- Venue
- Political Analysis
- Publisher
- Center for Open Science
- Published
- 2023-05-04
- Volume
- 32
- Issue
- 4
- Pages
- 401-416
- ISSN
- 1047-1987 (ISSN-L)
- OpenAlex
- W4397011890
- Cited by
- 213
- References
- 48
- Citations by year
- 2026: 120 · 2025: 70 · 2024: 13 · 2023: 10
- FWCI
- 176.92
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Political Analysis (publishedVersion, cc-by)
- SocArXiv (OSF Preprints) (submittedVersion)
- SocArXiv (OSF Preprints) (submittedVersion, cc-by-nc)
- Topics
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Sociology and Political Science › Survey Methodology and Nonresponse
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Keywords
- large language models, synthetic survey data, ChatGPT, public opinion, feeling thermometers, sociopolitical attitudes, American National Election Studies, sampling reliability, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
