Can Generative AI improve social science?

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Generative AI may be useful for creating vignettes and/or images to increase the external validity and comparability of primes, or to protect the privacy of real humans whose images might be used in studies.

“Generative AI may be useful for creating such vignettes and/or images—especially with iterative feedback from researchers—to increase the external validity and comparability of those primes, or to protect the privacy of real humans whose images might be used in such studies.”

Probably not a substantial claim: may not pass S3 (load-bearing).

GPT-3 can reproduce several classic studies in behavioral economics.

“Horton, for example, argues synthetic research respondents created using GPT-3 can be used to reproduce several classic studies in behavioral economics (19).”

• cites [19] J. J. Horton, Large language models as simulated economic agents: What can we learn from homo silicus? arXiv [Preprint] (2023). 10.48550/arXiv.2301.07543 (Accessed 20 December 2023). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

GPT 3.5 turbo produces accurate mean estimates of attitudes within a population, but understates variances-exaggerating extreme attitudes.

“At the same time, more recent research indicates GPT 3.5 turbo produces accurate mean estimates of attitudes within a population, but understates variances—exaggerating extreme attitudes (13).”

• cites [13] J. Bisbee, J. Clinton, C. Dorff, B. Kenkel, J. Larson, Synthetic replacements for human survey data? the perils of large language models. socArXiv [Preprint] (2023). 10.31235/osf.io/5ecfa (Accessed 20 December 2023). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims. The quote only partly states this claim.

Generative AI has the potential to improve social science research by advancing its scale, scope, and speed, enabling new forms of inquiry, and improving techniques such as survey research, online experiments, automated content analyses, and agent-based models.

“I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior.”

• Also stated in: § Abstract · hedged

  “In this article, I argue these tools may advance the scale, scope, and speed of social science research—and may enable new forms of scientific inquiry as well.”

May combine two claims.

Silicon samples may be useful for pretesting surveys before they are dispatched to large groups of human respondents, or for imputing missing data.

“Though silicon samples will not soon displace survey research with human respondents, they may still be very useful for pretesting surveys before they are dispatched (at considerable cost) to large groups of human respondents, or imputing missing data (11, 17).”

• cites [11] L. P. Argyle, Out of one, many: Using language models to simulate human samples. Polit. Anal. 31, 337– 351 (2023). sourceNot checked yet

  Not checked in the free run.

• cites [17] J. Kim, B. Lee, AI-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys. arXiv [Preprint] (2023). 10.48550/arXiv.2305.09620 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

GPT-3 can reproduce classic social psychology experiments-including the Milgram experiment-but cannot reproduce the 'wisdom of crowds' phenomenon.

“Similarly, Aher et al. show that GPT-3 can also reproduce classic social psychology experiments—including the infamous Milgram experiment—though they argue it cannot reproduce the “wisdom of crowds” phenomenon (20).”

• cites [20] G. Aher, R. I. Arriaga, A. T. Kalai, Using large language models to simulate multiple humans and replicate human subject studies. arXiv [Preprint] (2023). 10.48550/arXiv.2208.10264 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

LLMs can replicate classic experiments in cognitive science and the study of morality and replicate human behavior in the Prisoner's Dilemma and other behavioral games.

“Still other studies indicate LLMs can replicate classic experiments in cognitive science and the study of morality and replicate human behavior in the Prisoner’s Dilemma and other behavioral games (21–24).”

• cites [21] M. Binz, E. Schulz, Using cognitive psychology to understand GPT-3. Proc. Natl. Acad. Sci. U.S.A. 120, e2218523120 (2023). 36730192 PMC9963545 sourceNot checked yet

  Not checked in the free run.

• cites [22] D. Dillion, N. Tandon, Y. Gu, K. Gray, Can AI language models replace human participants? Trends Cognit. Sci. 27, 597– 600 (2023). 37173156 sourceNot checked yet

  Not checked in the free run.

• cites [23] Q. Mei, Y. Xie, W. Yuan, M. O. Jackson, A turing test of whether AI chatbots are behaviorally similar to humans. Proc. Natl. Acad. Sci. U.S.A. 121, e2313925121 (2024). 38386710 PMC10907317 sourceNot checked yet

  Not checked in the free run.

• cites [24] C. Xie et al., Can large language model agents simulate human trust behaviors? arXiv [Preprint] (2024). 10.48550/arXiv.2402.04559 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

GPT-2 could successfully impersonate humans and write lengthy news stories about international affairs judged to be as credible as those authored by real journalists.

“They found GPT-2 could successfully impersonate humans, and can even write lengthy news stories about international affairs that are judged to be as credible as those authored by real journalists.”

• cites [7] S. Kreps, R. M. McCain, M. Brundage, All the news that’s fit to fabricate: AI-generated text as a tool of media misinformation. J. Exp. Polit. Sci. 9, 104– 117 (2022). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

LLMs exhibit an affirmative bias in yes/no questions.

“Another study indicates LLMs exhibit an affirmative bias in yes/no questions (14).”

• cites [14] V. Dentella, F. Günther, E. Leivada, Systematic testing of three language models. Proc. Natl. Acad. Sci. U.S.A. 120, e2309583120 (2023). 38091290 PMC10743380 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

The progress of AI will require deeper understanding of the social forces that guide human behavior.

“Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.”

Probably not a substantial claim: may not pass S4 (based).

Bias in the data used to train Generative AI can negatively impact social science research.

“I examine how bias in the data used to train these tools can negatively impact social science research—as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Open-source infrastructure is necessary to ensure broad access to high-quality research tools.

“Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Social scientists can address many limitations of Generative AI by creating open-source infrastructure for research on human behavior.

“I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior.”

Classified as probably not a substantial claim. May combine two claims.

Ziems et al. found LLMs perform well at reproducing human expert annotations, particularly in coding data created by political scientists and sociologists; the latest models perform best; accuracy varies by topic; and usage requires human supervision and prompt-engineering.

“Overall, they find LLMs perform well—particularly in coding data created by political scientists and sociologists. Unsurprisingly, they find the latest models perform best. LLMs appear to assign more accurate codes for some topics than others, however, which may be an artifact of the way they were trained.”

• Also stated in: § Can Generative AI Improve Text Analysis?. · plain

  “That such models can reproduce coding decisions of humans without any specific training is encouraging, but Ziems et al. argue usage of LLMs will still require some degree of human supervision, and familiarity with task-specific prompt-engineering.”

• cites [54] C. Ziems et al., Can large language models transform computational social science? arXiv [Preprint] (2023). 10.48550/arXiv.2305.03514 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims. The quote only partly states this claim.

Integrating LLMs and ABMs could open new lines of inquiry, including studying topics and populations that are very difficult to examine in real life and informing observational data collection or social interventions.

“If such issues can be addressed, integrating LLMs and ABMs could open new lines of inquiry. This approach could be employed to study topics that are very difficult to examine in real life (such as violent extremism on social media), or to study populations that are very difficult to access (e.g., violent extremists) (12).”

May combine two claims.

Park et al. created a simulacrum where several dozen GPT 3.5-turbo agents interacted in a fictitious small-town setting, developed daily routines, and demonstrated emergent group properties.

“Park et al. created a simulacrum where several dozen agents—independently powered by multiple instances of GPT 3.5-turbo—interacted with each other in a fictitious small-town setting (6).”

• Also stated in: § Can Generative AI Improve Simulation-Based Research?. · boosted

  “These agents not only developed daily routines as the simulation progressed (e.g., waking up and eating breakfast), but also demonstrated emergent group properties.”

• cites [6] J. S. Park et al., Generative agents: Interactive simulacra of human behavior. arXiv [Preprint] (2023). 10.48550/arXiv.2304.03442 (Accessed 20 December 2023). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The capacity of Generative AI to impersonate humans may improve as model size increases and as researchers prompt LLMs with richer forms of data such as in-depth qualitative interviews or detailed life histories.

“The capacity of Generative AI to impersonate humans may thus improve as model size increases, and as researchers experiment with prompting LLMs with even richer forms of data such as in-depth qualitative interviews or detailed life histories.”

Classified as probably not a substantial claim. May combine two claims.

GPT 3.5-turbo can accurately measure the topic of tweets, the stance or opinions of their authors, and the frames used to organize the message in a narrative manner.

“Gilardi et al. argue GPT 3.5-turbo can accurately measure the topic of tweets, the stance or opinions of their authors, and the “frames” used to organize the message in a narrative manner (51).”

• cites [51] F. Gilardi, M. Alizadeh, M. Kubli, ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. U.S.A. 120, e2305016120 (2023). 37463210 PMC10372638 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Automated agents trained to perform poorly at collaborative tasks can improve the behavior of human team members.

“Conversely, Traeger et al. find automated agents that are trained to perform poorly at collaborative tasks can actually improve the behavior of human team members (29).”

• cites [29] M. L. Traeger, S. Strohkorb Sebo, M. Jung, B. Scassellati, N. A. Christakis, Vulnerable robots positively shape human conversational dynamics in a human–robot team. Proc. Natl. Acad. Sci. U.S.A. 117, 6370– 6375 (2020). 32152118 PMC7104178 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

GPT 3.5-turbo performs better than human workers trained with coding materials on Amazon Mechanical Turk, though such coders are known to be less accurate than those trained directly by researchers in small group settings.

“They find this model performs better than human workers trained with such materials on Amazon Mechanical Turk—though such coders are known to be less accurate than those trained directly by researchers in small group settings.”

• cites [51] F. Gilardi, M. Alizadeh, M. Kubli, ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. U.S.A. 120, e2305016120 (2023). 37463210 PMC10372638 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

LLMs can recreate competitive dynamics within simulated economic and labor markets, the diffusion of information and decision-making within organizations, and crisis response.

“Yet some recent studies indicate LLMs can recreate competitive dynamics within simulated economic and labor markets (42, 43), the diffusion of information and decision-making within organizations, and crisis response (41, 44, 45).”

• cites [41] X. Mou, Z. Wei, X. Huang, Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. arXiv [Preprint] (2024). 10.48550/arXiv.2402.16333 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

• cites [44] N. Ghaffarzadegan, A. Majumdar, R. Williams, N. Hosseinichimeh, Generative agent-based modeling: Unveiling social system dynamics through coupling mechanistic models with generative artificial intelligence. arXiv [Preprint] (2023). https://arxiv.org/abs/2309.11456 (Accessed 19 December 2023).Not checked yet

  Not checked in the free run.

• cites [45] B. Xiao, Z. Yin, Z. Shan, Simulating public administration crisis: A novel generative agent-based simulation system to lower technology barriers in social science research. arXiv [Preprint] (2023). https://arxiv.org/abs/2311.06957 (Accessed 19 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Whether Generative AI can successfully impersonate humans in more complex social settings such as interpersonal conversations is much less clear.

“Whether Generative AI can successfully impersonate humans in more complex social settings such as interpersonal conversations is much less clear.”

May combine two claims.

GPT-3.5 can produce accurate classifications of the ideology of U.S. elected officials by analyzing their public statements, closely approximating the DW-Nominate method but identifying more nuance within moderates.

“Wu et al. demonstrate GPT-3.5 can produce accurate classifications of the ideology of U.S. elected officials by analyzing their public statements (49).”

• Also stated in: § Can Generative AI Improve Text Analysis?. · plain

  “The results closely approximate the popular DW-Nominate method for measuring the ideology of elected officials using roll-call voting, but also identified more nuance within moderates who often vote against the extreme wings of their parties.”

• cites [49] P. Y. Wu, J. Nagler, J. A. Tucker, S. Messing, Large language models can be used to scale the ideologies of politicians in a zero-shot learning setting. arXiv [Preprint] (2023). 10.48550/arXiv.2303.12057 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

Most AI tools are trained using data created by humans, and frequently exhibit a broad range of prejudice and cognitive errors accordingly.

“Most AI tools are trained using data created by humans, and frequently exhibit a broad range of prejudice and cognitive errors accordingly (1, 57–60).”

• cites [1] K. Sayash, A. Narayanan, AI Snake Oil (Princeton University Press, 2024).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [57] R. Benjamin, Race After Technology: Abolitionist Tools for the New Jim Code (Polity, Cambridge, UK/Medford, MA, ed. 1, 2019).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [58] S. Lazar, A. Nelson, AI safety on whose terms? Science 381, 138 (2023). 37440644 sourceNot checked yet

  Not checked in the free run.

• cites [59] E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery, New York, NY, 2021), pp. 610–623. sourceNot checked yet

  Not checked in the free run.

• cites [60] K. Yee, U. Tantipongpipat, S. Mishra, Image Cropping on Twitter. Proc. ACM Hum.-Comput. Interact. 5, 1– 24 (2021).36644216Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

LLMs cannot yet match the accuracy of expert human coders, so they will not soon displace conventional text analysis.

“That LLMs cannot yet match the accuracy of expert human coders means that they will not soon displace conventional text analysis.”

May combine two claims.

Törnberg et al. created a simulated social media platform with five hundred agents calibrated using American National Election Study data; it reproduces known dynamics and simulates alternative newsfeed algorithms that optimize for consensus instead of user engagement.

“For example, Törnberg et al. create a simulated social media platform with five hundred agents whose behaviors are calibrated using data from the American National Election Study (ANES) (39).”

• Also stated in: § Can Generative AI Improve Simulation-Based Research?. · hedged

  “This study both reproduces known dynamics on social media platforms and simulates what might happen if they used alternative newsfeed algorithms that optimize for consensus, instead of user engagement.”

• cites [39] P. Törnberg, D. Valeeva, J. Uitermark, C. Bail, Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv [Preprint] (2023). 10.48550/arXiv.2310.05984 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The introduction of an AI agent makes coordination more difficult for human players and creates less trust among members of the team.

“The authors argue the introduction of the AI agent makes coordination more difficult for human players—and also creates less trust among members of the team.”

• cites [28] F. Dell’Acqua, B. Kogut, P. Perkowski, Super Mario Meets AI: Experimental effects of automation and skills on team performance and coordination (SSRN Scholarly Paper, 2020). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

ABMs are often criticized for failing to capture the full spectrum of human behaviors; conventional agents within ABMs do not use language, interpret social contexts, or engage in conversations with each other.

“Yet ABMs are often criticized for failing to capture the full spectrum of human behaviors (38). For example, conventional agents within ABMs do not use language, interpret social contexts, or engage in conversations with each other (39).”

• cites [38] R. Conte, M. Paolucci, On agent-based modeling and computational social science. Front. Psychol. 5, 668 (2014). 25071642 PMC4094840 sourceNot checked yet

  Not checked in the free run.

• cites [39] P. Törnberg, D. Valeeva, J. Uitermark, C. Bail, Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv [Preprint] (2023). 10.48550/arXiv.2310.05984 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Gao et al. use real-world social media data to calibrate a model they claim successfully predicts the spread of information and emotions about gender discrimination and nuclear energy.

“Similarly, Gao et al. use real-world social media data to calibrate a model they claim successfully predicts the spread of information and emotions about gender discrimination and nuclear energy (40).”

• cites [40] C. Gao et al., S3: Social-network simulation system with large language model-empowered agents. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.14984 (Accessed 19 December 2023). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Integrating LLMs within ABMs may reinvigorate preexisting debates about whether increasing agent complexity is desirable and how LLM performance within ABMs should be evaluated.

“At the same time integrating LLMs within ABMs may reinvigorate preexisting debates about the latter. Scholars often debate whether increasing the complexity of agents is desirable if emergent group dynamics of interest can be created with parsimonious models (33). It is likewise unclear how the performance of LLMs within ABMs should be evaluated.”

May combine two claims.

LLMs appear capable of performing coding tasks in many of the world's most prominent languages and other rudimentary tasks typically assigned to human research assistants such as data coding or data entry.

“LLMs also appear capable of performing coding tasks in many of the world’s most prominent languages as well as other rudimentary tasks typically assigned to human research assistants such as data coding or data entry (as discussed in greater detail in SI Appendix) (56).”

• cites [56] A. Korinek, Language Models and Cognitive Automation for Economic Research (National Bureau of Economic Research, 2023). sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Most LLM responses to surveys are more liberal than the general population and reflect those who are younger and have more education; LLMs are particularly unlikely to perform the responses of those over sixty-five, those who are widowed, or those who identify as Mormon.

“They find most LLM’s responses are more liberal than the general population, and reflect those who are younger and have more education. LLMs are particularly unlikely to perform the responses of those over sixty-five years old, those who are widowed, or those who identify as Mormon.”

• cites [15] S. Santurkar et al., Whose opinions do language models reflect? arXiv [Preprint] (2023). 10.48550/arXiv.2303.17548 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The probabilistic nature of LLMs may complicate scholars' ability to reproduce each other's findings.

“If so, the probabilistic nature of LLMs may complicate scholars’ ability to reproduce each other’s findings, as I discuss in additional detail below.”

May combine two claims.

The correlation between treatment effects observed in 482 studies and responses created by GPT-4 is 0.86, and this correlation holds for both published and unpublished studies and across demographic subgroups.

“Ashokkumar et al. find the correlation between treatment effects observed in 482 studies and responses created by GPT-4 is 0.86. Furthermore, they find this correlation holds for both published and unpublished studies, and across demographic subgroups (25).”

• cites [25] A. Ashokkumar, L. Hewitt, I. Ghezae, R. Willer, Prediction of social science experimental results using large language models (Working Paper, 2024).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Multimodal models that can translate content from images into text suggest these coding capabilities may soon apply to other mediums as well.

“Multimodal models that can translate content from images into text suggest these coding capabilities may soon apply to other mediums as well.”

May combine two claims.

LLMs may enable social scientists to examine corpora of unprecedented size with unforeseen speed, including coding an entire corpus rather than taking a random sample.

“But LLMs may enable social scientists to examine corpora of unprecedented size with unforeseen speed. Rather than taking a random sample of documents, for example, social scientists now have the potential to code an entire corpus in short order.”

Probably not a substantial claim: may not pass S4 (based).

If bias in Generative AI tools can be carefully controlled, it could allow researchers to study its impact in empirical settings.

“If bias in Generative AI tools can be carefully controlled—a major assumption—it could allow researchers to study its impact in empirical settings (for example, a survey respondent evaluating a hypothetical applicant for a job).”

• AssumptionThe authors state this as an assumption: Bias in Generative AI tools can be carefully controlled.

Probably not a substantial claim: may not pass S3 (load-bearing).

One proposal is to create an "accent" for LLMs-giving them a list of words they should use whenever possible-to allow people to retrospectively identify content not generated by humans.

“One proposal is to create an “accent” for LLMs—giving them a list of words they should use whenever possible—to allow people to retrospectively identify content that was not generated by humans (68).”

• cites [68] J. Kirchenbauer et al., A watermark for large language models. arXiv [Preprint] (2023). 10.48550/arXiv.2301.10226 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

Using GPT-4 to code in-depth interviews about sensitive topics may log full-text inside private corporations not beholden to university human-subjects standards, and such data could potentially be sold to other corporations.

“If a researcher uses GPT-4 to code a series of in-depth interviews about a sensitive topic such as intimate partner violence, the full-text of these interviews may be logged inside private corporations that are not beholden to the same standards for protecting human subjects as university researchers. Even worse, such data could potentially be sold to other corporations.”

Probably not a substantial claim: may not pass S4 (based).

GPT-3 could perform responses characteristic of participants in the Milgram experiment.

“Earlier I mentioned that researchers demonstrated that GPT-3 could perform the responses characteristic of participants in the infamous Milgram experiment.”

May combine two claims.

Many Generative AI tools exhibit biases that are offensive and may hallucinate inaccurate information that could be shared by research participants on social media or elsewhere.

“This question is particularly important since many Generative AI tools exhibit biases that are not only offensive (e.g., racism or misogyny), but may also hallucinate inaccurate information that could be shared by research participants on social media platforms, or elsewhere.”

• cites [69] L. Weidinger et al., “Taxonomy of risks posed by language models” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT’22) (Association for Computing Machinery, New York, NY, 2022), pp. 214–229. PMC10661580 37990734 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Ethical questions about Generative AI are less important for supervised use but take on added importance when human participants might have unsupervised conversations with an LLM.

“While these questions may be less important for social scientists using Generative AI in a carefully supervised manner—for example, using DALL-E to generate a picture of a person that might be used in a survey experiment—they take on added importance in situations where human research participants might have conversations with a LLM in an unsupervised manner.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One study suggests LLMs exhibit more rational behavior than humans.

“One study, for example, suggests LLMs exhibit more rational behavior than humans (65).”

• cites [65] Y. Chen, T. X. Liu, Y. Shan, S. Zhong, The emergence of economic rationality of GPT. Proc. Natl. Acad. Sci. U.S.A. 120, e2316205120 (2023). 38085780 PMC10740389 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Generative AI may help increase external validity but may come at the cost of internal validity-the capacity of different groups to reproduce or replicate each other's results.

“Though Generative AI may help researchers increase the external validity of their research designs, this may come at the cost of internal validity—or the capacity of different groups of researchers to reproduce or replicate each other’s results (2, 12).”

• cites [2] A. Spirling, Why open-source generative AI models are an ethical way forward for science. Nature 616, 413 (2023). 37072520 sourceNot checked yet

  Not checked in the free run.

• cites [12] I. Grossmann, AI and the transformation of social science research. Science 380, 1108– 1109 (2023). 37319216 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Implementing watermarks at scale requires each entity developing LLMs to agree to use watermarks and coordinate with each other.

“Each entity that develops LLMs will not only have to agree to use watermarks, but they will also need to coordinate with each other.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Content moderation filters and rigorous prompt testing might mitigate some risks, but given the probabilistic nature of models and changing abuse, such strategies require great care.

“Some of these risks might be mitigated via content moderation filters that are currently available for some LLMs—and through rigorous testing of the prompts used to guide LLMs in research settings. Yet given the probabilistic nature of these models—and the ever-changing ways abuse and harassment can occur in online settings—such strategies will require great care.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Researchers who rely on LLMs for literature reviews, generating research questions, or summarizing large corpora may face problems similar to those seen with Stack Overflow.

“Researchers who rely upon LLMs to perform literature reviews, generate new research questions, or summarize large corpora they are unable to read may face similar problems.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Because LLMs are trained using retrospective data, they may be of limited utility in predicting ethical issues on the horizon, but they may help researchers learn from each other's mistakes.

“Because LLMs are trained using retrospective data, they may be of limited utility in predicting ethical issues on the horizon, but they may nevertheless help researchers learn from each other’s mistakes.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Since model sizes have grown since 2019, social scientists must reflect on presumably much larger environmental costs, even if recent engineering advances have made training more efficient.

“Since the size of Generative AI models has grown considerably since 2019, social scientists must carefully reflect upon the presumably much larger environmental costs of developing such technologies—even if recent engineering advances have made training processes more efficient.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Generative AI might be useful in 'reverse engineering' some types of bias.

“It is further possible that Generative AI might be useful in “reverse engineering” some types of bias.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Studies requiring humans to interact with each other also risk exposing participants to offensive language, misinformation, or abuse.

“On the other hand, studies that require humans to interact with each other also risk exposing research participants to offensive language, misinformation, or abuse.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Lower temperature values might produce more dependable results, but a researcher using LLMs to interact with humans in a field experiment may not want them to become overly repetitive.

“Lower values might produce more dependable results, but a researcher who employs LLMs to interact with humans in a field experiment may not want them to become overly repetitive.”

May combine two claims.

Most LLMs produce different results over time because they are constantly fine-tuned to be more effective or create safeguards.

“Most LLMs also produce different results over time. This is because many of them are constantly being fine-tuned to make them more effective or create new safeguards against bias or illicit behavior.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Milgram showed many respondents were willing to administer lethal shocks out of deference to authority, but the study was widely criticized for creating trauma among participants.

“Milgram showed that many respondents were willing to do so out of deference to authority, but the study was widely criticized for creating trauma among participants.”

May combine two claims.

LLMs are adept at impersonating humans at scale, making malicious use to spread misinformation in the short term very concerning.

“The potential for malicious actors to use Generative AI to spread misinformation in the short term is very concerning because tools such as LLMs are so adept at impersonating humans at scale (67).”

• cites [67] S. Feuerriegel, Research can help to tackle AI-generated disinformation. Nat. Hum. Behav. 7, 1818– 1821 (2023). 37985906 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Government regulation might encourage large companies to coordinate on watermarks, but such coordination would be unable to detect LLMs created by individuals skilled enough to develop smaller models on their own.

“Large companies might be encouraged to do this through government regulation. But such coordination would be unable to detect LLMs created by individuals skilled enough to develop smaller models on their own.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Journals and funding agencies may find themselves overwhelmed by low-quality "junk-science" created by LLMs.

“Journals and funding agencies may find themselves overwhelmed by low-quality “junk-science” created by LLMs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One study suggests carbon emissions of writing and illustrating are lower for AI than for humans.

“One study, for example, suggests the carbon emissions of writing and illustrating are lower for AI than for humans (71).”

• cites [71] B. Tomlinson, R. W. Black, D. J. Patterson, A. W. Torrance, The carbon emissions of writing and illustrating are lower for AI than for humans. Sci. Rep. 14, 3732 (2024). PMC10867074 38355820 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

Guard-rails in Generative AI probably improve safety for public use but may impede social scientists' ability to leverage bias for research purposes.

“Though such guard-rails probably improve the safety of Generative AI tools for public use, they may impede the ability of social scientists to leverage bias for research purposes (12).”

• cites [12] I. Grossmann, AI and the transformation of social science research. Science 380, 1108– 1109 (2023). 37319216 sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The tendency for LLMs to produce heterogeneous results can be partly controlled via temperature parameters, but there are no standards for appropriate values.

“The tendency for LLMs to produce heterogeneous results can be partly controlled via “temperature” parameters that regulate the predictability of model output. At present, however, there are no standards for what values of this parameter are appropriate (72).”

• cites [72] M. Alizadeh et al., Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.02179 (Accessed 19 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

Most open-source models provide more transparency than proprietary counterparts, allowing researchers to better understand and control training and fine-tuning.

“In addition to being free to use, most open-source models provide more transparency than their proprietary counterparts. This not only allows researchers to better understand the processes used to train and fine-tune Generative AI, but it could also allow them to control such processes altogether.”

• AssumptionRests on an assumption: Most open-source models provide more transparency than proprietary counterparts.

Probably not a substantial claim: may not pass S2 (checkable).

Social scientists could build similar models to better control when and how models become biased, especially where corporate values may inhibit research.

“Social scientists could build similar models to better control when and how models become biased—especially in research settings where the values built into models by large corporations may inhibit research.”

May combine two claims.

The Academic Developer Program at Twitter allowed social scientists to collect vast amounts of data; access to much smaller amounts is now prohibitively expensive for most researchers.

“The Academic Developer Program at Twitter that I helped create once allowed social scientists to collect vast amount of data. Access to much smaller amounts of data is now prohibitively expensive for most researchers.”

May combine two claims.

Llama and Gemma are well documented and released alongside scientific papers that can help social scientists evaluate their potential.

“These models are well documented and released alongside scientific papers that can help social scientists evaluate their potential.”

May combine two claims.

One study indicates open-source models perform better than crowd-workers at annotating news articles and tweets, though slightly less well than ChatGPT.

“One study indicates open-source models perform better than crowd-workers at annotating news articles and tweets—even though they perform slightly less well than ChatGPT on these tasks (72).”

• cites [72] M. Alizadeh et al., Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.02179 (Accessed 19 December 2023).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Generative AI has many limitations for social science research, most of which are only beginning to be understood.

“As the sections above describe, Generative AI has many limitations for social science research—most of which we are only beginning to understand.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Accomplishing the agenda of minimizing risks without sacrificing opportunities will require deeper understanding of how Generative AI tools are built and fine-tuned.

“Accomplishing this agenda will require deeper understanding of how Generative AI tools are built and fine-tuned.”

May combine two claims.

Meta has released detailed information about inner workings of Llama, such as numeric weights, enabling researchers to better control fine-tuning and study how LLMs work.

“Meta has even released detailed information about the inner workings of the model such as the numeric weights its uses to respond to prompts. This enables researchers to better control the fine-tuning process, and study how LLMs work.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Transparent public discussion about Generative AI may help researchers evaluate risks such as dissemination of misinformation.

“Transparent, public discussion about Generative AI may also help researchers evaluate some of the other risks above—such as the dissemination of misinformation.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

Understanding how Generative AI can detect and navigate social norms will require a much more sophisticated understanding of how individual agents' behavior is constrained by social networks, institutions, organizations, and other extra-individual factors.

“This will require a much more sophisticated understanding of how the behavior of individual agents is constrained by social networks, institutions, organizations, and other extra-individual factors that are cornerstones of the science of human behavior.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

If capabilities continue to expand at a fraction of current pace, Generative AI may become a fixture within the social scientist's toolkit much sooner than many researchers realize.

“If the capabilities of these tools continue to expand at a fraction of their current pace, Generative AI may become a fixture within the social scientist’s toolkit much sooner than many researchers realize.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Some newer models created by Mistral offer model weights that are "raw," or not fine-tuned.

“Some newer models created by Mistral even offer model weights that are “raw,” or not fine-tuned.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

If an organization for open-source social science tools could be realized, it would improve access to state-of-the-art technology among a broad group of scholars and improve capacity to build disciplinary standards of open science and ethics into research with Generative AI.

“If such an organization could be realized, however, it would not only improve access to state-of-the-art technology among a broad group of scholars, but also improve our capacity to build disciplinary standards of open science and ethics into research with Generative AI.”

May combine two claims.

There is no guarantee that Meta will not restrict access or begin charging researchers, and fees may reproduce inequality within academia.

“There is no guarantee that Meta will suddenly restrict access to its Generative AI tools, or begin charging researchers to access it like their peers. Though researchers at wealthy institutions may be able to afford such fees, many others may not—reproducing inequality within the academic caste system (76).”

• cites [76] A. Clauset, S. Arbesman, D. B. Larremore, Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1, e1400005 (2015). 26601125 PMC4644075 sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Multiple groups of leading scholars and policy experts argue that risks of releasing open-source model weights are marginal compared to information already readily available on the internet.

“Yet multiple groups of leading scholars and policy experts argue such risks are marginal when compared to information that is already readily available on the internet (78, 79).”

• cites [78] R. Bommasani, Considerations for Governing Open Foundation Models. Issue Brief Human, Stanford HAI (Elsevier, 2023).Not checked yet

  Not checked in the free run.

• cites [79] S. Kapoor et al., On the societal impact of open foundation models. arXiv [Preprint] (2024). 10.48550/arXiv.2403.07918 (Accessed 20 December 2023).Not checked yet

  Not checked in the free run.

May combine two claims.

In the early 2010s, many social media companies shared generous data with researchers, enabling new research and collaborations; in recent years, data-sharing at most large social media companies has been discontinued.

“In the early 2010s, many companies shared generous amounts of data with researchers. This enabled entirely new forms of scholarly research, and social scientists and researchers inside industry frequently collaborated with each other and presented their work at conferences. In more recent years, data-sharing practices at most large social media companies have been discontinued (77).”

• cites [77] D. Freelon, Computational research in the post-API age. Polit. Commun. 35, 1– 4 (2018). sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Meta and Google have made two models available to the public: Llama and Gemma.

“For the time being, both Meta and Google have made two of their models available to the public: Llama and Gemma.”