Can Generative AI improve social science?

In the paper

“Generative AI may be useful for creating such vignettes and/or images—especially with iterative feedback from researchers—to increase the external validity and comparability of those primes, or to protect the privacy of real humans whose images might be used in such studies.”

In the paper

“Horton, for example, argues synthetic research respondents created using GPT-3 can be used to reproduce several classic studies in behavioral economics (19).”

What the sources say

• Not checked yetcites [19] J. J. Horton, Large language models as simulated economic agents: What can we learn from homo silicus? arXiv [Preprint] (2023). 10.48550/arXiv.2301.07543 (Accessed 20 December 2023). source

In the paper

“At the same time, more recent research indicates GPT 3.5 turbo produces accurate mean estimates of attitudes within a population, but understates variances—exaggerating extreme attitudes (13).”

What the sources say

• Not checked yetcites [13] J. Bisbee, J. Clinton, C. Dorff, B. Kenkel, J. Larson, Synthetic replacements for human survey data? the perils of large language models. socArXiv [Preprint] (2023). 10.31235/osf.io/5ecfa (Accessed 20 December 2023). source

In the paper

“I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior.”

• Also stated in: § Abstract · hedged

  “In this article, I argue these tools may advance the scale, scope, and speed of social science research—and may enable new forms of scientific inquiry as well.”

In the paper

“Though silicon samples will not soon displace survey research with human respondents, they may still be very useful for pretesting surveys before they are dispatched (at considerable cost) to large groups of human respondents, or imputing missing data (11, 17).”

What the sources say

• Not checked yetcites [11] L. P. Argyle, Out of one, many: Using language models to simulate human samples. Polit. Anal. 31, 337– 351 (2023). source
• Not checked yetcites [17] J. Kim, B. Lee, AI-augmented surveys: Leveraging large language models for opinion prediction in nationally representative surveys. arXiv [Preprint] (2023). 10.48550/arXiv.2305.09620 (Accessed 20 December 2023).

In the paper

“Similarly, Aher et al. show that GPT-3 can also reproduce classic social psychology experiments—including the infamous Milgram experiment—though they argue it cannot reproduce the “wisdom of crowds” phenomenon (20).”

What the sources say

• Not checked yetcites [20] G. Aher, R. I. Arriaga, A. T. Kalai, Using large language models to simulate multiple humans and replicate human subject studies. arXiv [Preprint] (2023). 10.48550/arXiv.2208.10264 (Accessed 20 December 2023).

In the paper

“Still other studies indicate LLMs can replicate classic experiments in cognitive science and the study of morality and replicate human behavior in the Prisoner’s Dilemma and other behavioral games (21–24).”

What the sources say

• Not checked yetcites [21] M. Binz, E. Schulz, Using cognitive psychology to understand GPT-3. Proc. Natl. Acad. Sci. U.S.A. 120, e2218523120 (2023). 36730192 PMC9963545 source
• Not checked yetcites [22] D. Dillion, N. Tandon, Y. Gu, K. Gray, Can AI language models replace human participants? Trends Cognit. Sci. 27, 597– 600 (2023). 37173156 source
• Not checked yetcites [23] Q. Mei, Y. Xie, W. Yuan, M. O. Jackson, A turing test of whether AI chatbots are behaviorally similar to humans. Proc. Natl. Acad. Sci. U.S.A. 121, e2313925121 (2024). 38386710 PMC10907317 source
• Not checked yetcites [24] C. Xie et al., Can large language model agents simulate human trust behaviors? arXiv [Preprint] (2024). 10.48550/arXiv.2402.04559 (Accessed 20 December 2023).

In the paper

“They found GPT-2 could successfully impersonate humans, and can even write lengthy news stories about international affairs that are judged to be as credible as those authored by real journalists.”

What the sources say

• Not checked yetcites [7] S. Kreps, R. M. McCain, M. Brundage, All the news that’s fit to fabricate: AI-generated text as a tool of media misinformation. J. Exp. Polit. Sci. 9, 104– 117 (2022). source

In the paper

“Another study indicates LLMs exhibit an affirmative bias in yes/no questions (14).”

What the sources say

• Not checked yetcites [14] V. Dentella, F. Günther, E. Leivada, Systematic testing of three language models. Proc. Natl. Acad. Sci. U.S.A. 120, e2309583120 (2023). 38091290 PMC10743380 source

In the paper

“Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.”

In the paper

“I examine how bias in the data used to train these tools can negatively impact social science research—as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research.”

In the paper

“Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.”

In the paper

“I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior.”

In the paper

“Overall, they find LLMs perform well—particularly in coding data created by political scientists and sociologists. Unsurprisingly, they find the latest models perform best. LLMs appear to assign more accurate codes for some topics than others, however, which may be an artifact of the way they were trained.”

• Also stated in: § Can Generative AI Improve Text Analysis?. · plain

  “That such models can reproduce coding decisions of humans without any specific training is encouraging, but Ziems et al. argue usage of LLMs will still require some degree of human supervision, and familiarity with task-specific prompt-engineering.”

What the sources say

• Not checked yetcites [54] C. Ziems et al., Can large language models transform computational social science? arXiv [Preprint] (2023). 10.48550/arXiv.2305.03514 (Accessed 20 December 2023).

In the paper

“If such issues can be addressed, integrating LLMs and ABMs could open new lines of inquiry. This approach could be employed to study topics that are very difficult to examine in real life (such as violent extremism on social media), or to study populations that are very difficult to access (e.g., violent extremists) (12).”

In the paper

“Park et al. created a simulacrum where several dozen agents—independently powered by multiple instances of GPT 3.5-turbo—interacted with each other in a fictitious small-town setting (6).”

• Also stated in: § Can Generative AI Improve Simulation-Based Research?. · boosted

  “These agents not only developed daily routines as the simulation progressed (e.g., waking up and eating breakfast), but also demonstrated emergent group properties.”

What the sources say

• Not checked yetcites [6] J. S. Park et al., Generative agents: Interactive simulacra of human behavior. arXiv [Preprint] (2023). 10.48550/arXiv.2304.03442 (Accessed 20 December 2023). source

In the paper

“The capacity of Generative AI to impersonate humans may thus improve as model size increases, and as researchers experiment with prompting LLMs with even richer forms of data such as in-depth qualitative interviews or detailed life histories.”

In the paper

“Gilardi et al. argue GPT 3.5-turbo can accurately measure the topic of tweets, the stance or opinions of their authors, and the “frames” used to organize the message in a narrative manner (51).”

What the sources say

• Not checked yetcites [51] F. Gilardi, M. Alizadeh, M. Kubli, ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. U.S.A. 120, e2305016120 (2023). 37463210 PMC10372638 source

In the paper

“Conversely, Traeger et al. find automated agents that are trained to perform poorly at collaborative tasks can actually improve the behavior of human team members (29).”

What the sources say

• Not checked yetcites [29] M. L. Traeger, S. Strohkorb Sebo, M. Jung, B. Scassellati, N. A. Christakis, Vulnerable robots positively shape human conversational dynamics in a human–robot team. Proc. Natl. Acad. Sci. U.S.A. 117, 6370– 6375 (2020). 32152118 PMC7104178 source

In the paper

“They find this model performs better than human workers trained with such materials on Amazon Mechanical Turk—though such coders are known to be less accurate than those trained directly by researchers in small group settings.”

What the sources say

• Not checked yetcites [51] F. Gilardi, M. Alizadeh, M. Kubli, ChatGPT outperforms crowd workers for text-annotation tasks. Proc. Natl. Acad. Sci. U.S.A. 120, e2305016120 (2023). 37463210 PMC10372638 source

In the paper

“Yet some recent studies indicate LLMs can recreate competitive dynamics within simulated economic and labor markets (42, 43), the diffusion of information and decision-making within organizations, and crisis response (41, 44, 45).”

What the sources say

• Not checked yetcites [41] X. Mou, Z. Wei, X. Huang, Unveiling the truth and facilitating change: Towards agent-based large-scale social movement simulation. arXiv [Preprint] (2024). 10.48550/arXiv.2402.16333 (Accessed 20 December 2023).
• Not checked yetcites [44] N. Ghaffarzadegan, A. Majumdar, R. Williams, N. Hosseinichimeh, Generative agent-based modeling: Unveiling social system dynamics through coupling mechanistic models with generative artificial intelligence. arXiv [Preprint] (2023). https://arxiv.org/abs/2309.11456 (Accessed 19 December 2023).
• Not checked yetcites [45] B. Xiao, Z. Yin, Z. Shan, Simulating public administration crisis: A novel generative agent-based simulation system to lower technology barriers in social science research. arXiv [Preprint] (2023). https://arxiv.org/abs/2311.06957 (Accessed 19 December 2023).

In the paper

“Whether Generative AI can successfully impersonate humans in more complex social settings such as interpersonal conversations is much less clear.”

In the paper

“Wu et al. demonstrate GPT-3.5 can produce accurate classifications of the ideology of U.S. elected officials by analyzing their public statements (49).”

• Also stated in: § Can Generative AI Improve Text Analysis?. · plain

  “The results closely approximate the popular DW-Nominate method for measuring the ideology of elected officials using roll-call voting, but also identified more nuance within moderates who often vote against the extreme wings of their parties.”

What the sources say

• Not checked yetcites [49] P. Y. Wu, J. Nagler, J. A. Tucker, S. Messing, Large language models can be used to scale the ideologies of politicians in a zero-shot learning setting. arXiv [Preprint] (2023). 10.48550/arXiv.2303.12057 (Accessed 20 December 2023).

In the paper

“Most AI tools are trained using data created by humans, and frequently exhibit a broad range of prejudice and cognitive errors accordingly (1, 57–60).”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [1] K. Sayash, A. Narayanan, AI Snake Oil (Princeton University Press, 2024).
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [57] R. Benjamin, Race After Technology: Abolitionist Tools for the New Jim Code (Polity, Cambridge, UK/Medford, MA, ed. 1, 2019).
• Not checked yetcites [58] S. Lazar, A. Nelson, AI safety on whose terms? Science 381, 138 (2023). 37440644 source
• Not checked yetcites [59] E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, “On the dangers of stochastic parrots: Can language models be too big?” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Association for Computing Machinery, New York, NY, 2021), pp. 610–623. source
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [60] K. Yee, U. Tantipongpipat, S. Mishra, Image Cropping on Twitter. Proc. ACM Hum.-Comput. Interact. 5, 1– 24 (2021).36644216

In the paper

“That LLMs cannot yet match the accuracy of expert human coders means that they will not soon displace conventional text analysis.”

In the paper

“For example, Törnberg et al. create a simulated social media platform with five hundred agents whose behaviors are calibrated using data from the American National Election Study (ANES) (39).”

• Also stated in: § Can Generative AI Improve Simulation-Based Research?. · hedged

  “This study both reproduces known dynamics on social media platforms and simulates what might happen if they used alternative newsfeed algorithms that optimize for consensus, instead of user engagement.”

What the sources say

• Not checked yetcites [39] P. Törnberg, D. Valeeva, J. Uitermark, C. Bail, Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv [Preprint] (2023). 10.48550/arXiv.2310.05984 (Accessed 20 December 2023).

In the paper

“The authors argue the introduction of the AI agent makes coordination more difficult for human players—and also creates less trust among members of the team.”

What the sources say

• Not checked yetcites [28] F. Dell’Acqua, B. Kogut, P. Perkowski, Super Mario Meets AI: Experimental effects of automation and skills on team performance and coordination (SSRN Scholarly Paper, 2020). source

In the paper

“Yet ABMs are often criticized for failing to capture the full spectrum of human behaviors (38). For example, conventional agents within ABMs do not use language, interpret social contexts, or engage in conversations with each other (39).”

What the sources say

• Not checked yetcites [38] R. Conte, M. Paolucci, On agent-based modeling and computational social science. Front. Psychol. 5, 668 (2014). 25071642 PMC4094840 source
• Not checked yetcites [39] P. Törnberg, D. Valeeva, J. Uitermark, C. Bail, Simulating social media using large language models to evaluate alternative news feed algorithms. arXiv [Preprint] (2023). 10.48550/arXiv.2310.05984 (Accessed 20 December 2023).

In the paper

“Similarly, Gao et al. use real-world social media data to calibrate a model they claim successfully predicts the spread of information and emotions about gender discrimination and nuclear energy (40).”

What the sources say

• Not checked yetcites [40] C. Gao et al., S3: Social-network simulation system with large language model-empowered agents. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.14984 (Accessed 19 December 2023). source

In the paper

“At the same time integrating LLMs within ABMs may reinvigorate preexisting debates about the latter. Scholars often debate whether increasing the complexity of agents is desirable if emergent group dynamics of interest can be created with parsimonious models (33). It is likewise unclear how the performance of LLMs within ABMs should be evaluated.”

In the paper

“LLMs also appear capable of performing coding tasks in many of the world’s most prominent languages as well as other rudimentary tasks typically assigned to human research assistants such as data coding or data entry (as discussed in greater detail in SI Appendix) (56).”

What the sources say

• Not checked yetcites [56] A. Korinek, Language Models and Cognitive Automation for Economic Research (National Bureau of Economic Research, 2023). source

In the paper

“They find most LLM’s responses are more liberal than the general population, and reflect those who are younger and have more education. LLMs are particularly unlikely to perform the responses of those over sixty-five years old, those who are widowed, or those who identify as Mormon.”

What the sources say

• Not checked yetcites [15] S. Santurkar et al., Whose opinions do language models reflect? arXiv [Preprint] (2023). 10.48550/arXiv.2303.17548 (Accessed 20 December 2023).

In the paper

“If so, the probabilistic nature of LLMs may complicate scholars’ ability to reproduce each other’s findings, as I discuss in additional detail below.”

In the paper

“Ashokkumar et al. find the correlation between treatment effects observed in 482 studies and responses created by GPT-4 is 0.86. Furthermore, they find this correlation holds for both published and unpublished studies, and across demographic subgroups (25).”

What the sources say

• Not checked yetcites [25] A. Ashokkumar, L. Hewitt, I. Ghezae, R. Willer, Prediction of social science experimental results using large language models (Working Paper, 2024).

In the paper

“Multimodal models that can translate content from images into text suggest these coding capabilities may soon apply to other mediums as well.”

In the paper

“But LLMs may enable social scientists to examine corpora of unprecedented size with unforeseen speed. Rather than taking a random sample of documents, for example, social scientists now have the potential to code an entire corpus in short order.”

In the paper

“If bias in Generative AI tools can be carefully controlled—a major assumption—it could allow researchers to study its impact in empirical settings (for example, a survey respondent evaluating a hypothetical applicant for a job).”

In the paper

“One proposal is to create an “accent” for LLMs—giving them a list of words they should use whenever possible—to allow people to retrospectively identify content that was not generated by humans (68).”

What the sources say

• Not checked yetcites [68] J. Kirchenbauer et al., A watermark for large language models. arXiv [Preprint] (2023). 10.48550/arXiv.2301.10226 (Accessed 20 December 2023).

In the paper

“If a researcher uses GPT-4 to code a series of in-depth interviews about a sensitive topic such as intimate partner violence, the full-text of these interviews may be logged inside private corporations that are not beholden to the same standards for protecting human subjects as university researchers. Even worse, such data could potentially be sold to other corporations.”

In the paper

“Earlier I mentioned that researchers demonstrated that GPT-3 could perform the responses characteristic of participants in the infamous Milgram experiment.”

In the paper

“This question is particularly important since many Generative AI tools exhibit biases that are not only offensive (e.g., racism or misogyny), but may also hallucinate inaccurate information that could be shared by research participants on social media platforms, or elsewhere.”

What the sources say

• Not checked yetcites [69] L. Weidinger et al., “Taxonomy of risks posed by language models” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT’22) (Association for Computing Machinery, New York, NY, 2022), pp. 214–229. PMC10661580 37990734 source

In the paper

“While these questions may be less important for social scientists using Generative AI in a carefully supervised manner—for example, using DALL-E to generate a picture of a person that might be used in a survey experiment—they take on added importance in situations where human research participants might have conversations with a LLM in an unsupervised manner.”

In the paper

“One study, for example, suggests LLMs exhibit more rational behavior than humans (65).”

What the sources say

• Not checked yetcites [65] Y. Chen, T. X. Liu, Y. Shan, S. Zhong, The emergence of economic rationality of GPT. Proc. Natl. Acad. Sci. U.S.A. 120, e2316205120 (2023). 38085780 PMC10740389 source

In the paper

“Though Generative AI may help researchers increase the external validity of their research designs, this may come at the cost of internal validity—or the capacity of different groups of researchers to reproduce or replicate each other’s results (2, 12).”

What the sources say

• Not checked yetcites [2] A. Spirling, Why open-source generative AI models are an ethical way forward for science. Nature 616, 413 (2023). 37072520 source
• Not checked yetcites [12] I. Grossmann, AI and the transformation of social science research. Science 380, 1108– 1109 (2023). 37319216 source

In the paper

“Each entity that develops LLMs will not only have to agree to use watermarks, but they will also need to coordinate with each other.”

In the paper

“Some of these risks might be mitigated via content moderation filters that are currently available for some LLMs—and through rigorous testing of the prompts used to guide LLMs in research settings. Yet given the probabilistic nature of these models—and the ever-changing ways abuse and harassment can occur in online settings—such strategies will require great care.”

In the paper

“Researchers who rely upon LLMs to perform literature reviews, generate new research questions, or summarize large corpora they are unable to read may face similar problems.”

In the paper

“Because LLMs are trained using retrospective data, they may be of limited utility in predicting ethical issues on the horizon, but they may nevertheless help researchers learn from each other’s mistakes.”

In the paper

“Since the size of Generative AI models has grown considerably since 2019, social scientists must carefully reflect upon the presumably much larger environmental costs of developing such technologies—even if recent engineering advances have made training processes more efficient.”

In the paper

“It is further possible that Generative AI might be useful in “reverse engineering” some types of bias.”

In the paper

“On the other hand, studies that require humans to interact with each other also risk exposing research participants to offensive language, misinformation, or abuse.”

In the paper

“Lower values might produce more dependable results, but a researcher who employs LLMs to interact with humans in a field experiment may not want them to become overly repetitive.”

In the paper

“Most LLMs also produce different results over time. This is because many of them are constantly being fine-tuned to make them more effective or create new safeguards against bias or illicit behavior.”

In the paper

“Milgram showed that many respondents were willing to do so out of deference to authority, but the study was widely criticized for creating trauma among participants.”

In the paper

“The potential for malicious actors to use Generative AI to spread misinformation in the short term is very concerning because tools such as LLMs are so adept at impersonating humans at scale (67).”

What the sources say

• Not checked yetcites [67] S. Feuerriegel, Research can help to tackle AI-generated disinformation. Nat. Hum. Behav. 7, 1818– 1821 (2023). 37985906 source

In the paper

“Large companies might be encouraged to do this through government regulation. But such coordination would be unable to detect LLMs created by individuals skilled enough to develop smaller models on their own.”

In the paper

“Journals and funding agencies may find themselves overwhelmed by low-quality “junk-science” created by LLMs.”

In the paper

“One study, for example, suggests the carbon emissions of writing and illustrating are lower for AI than for humans (71).”

What the sources say

• Not checked yetcites [71] B. Tomlinson, R. W. Black, D. J. Patterson, A. W. Torrance, The carbon emissions of writing and illustrating are lower for AI than for humans. Sci. Rep. 14, 3732 (2024). PMC10867074 38355820 source

In the paper

“Though such guard-rails probably improve the safety of Generative AI tools for public use, they may impede the ability of social scientists to leverage bias for research purposes (12).”

What the sources say

• Not checked yetcites [12] I. Grossmann, AI and the transformation of social science research. Science 380, 1108– 1109 (2023). 37319216 source

In the paper

“The tendency for LLMs to produce heterogeneous results can be partly controlled via “temperature” parameters that regulate the predictability of model output. At present, however, there are no standards for what values of this parameter are appropriate (72).”

What the sources say

• Not checked yetcites [72] M. Alizadeh et al., Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.02179 (Accessed 19 December 2023).

In the paper

“In addition to being free to use, most open-source models provide more transparency than their proprietary counterparts. This not only allows researchers to better understand the processes used to train and fine-tune Generative AI, but it could also allow them to control such processes altogether.”

In the paper

“Social scientists could build similar models to better control when and how models become biased—especially in research settings where the values built into models by large corporations may inhibit research.”

In the paper

“The Academic Developer Program at Twitter that I helped create once allowed social scientists to collect vast amount of data. Access to much smaller amounts of data is now prohibitively expensive for most researchers.”

In the paper

“These models are well documented and released alongside scientific papers that can help social scientists evaluate their potential.”

In the paper

“One study indicates open-source models perform better than crowd-workers at annotating news articles and tweets—even though they perform slightly less well than ChatGPT on these tasks (72).”

What the sources say

• Not checked yetcites [72] M. Alizadeh et al., Open-source large language models outperform crowd workers and approach ChatGPT in text-annotation tasks. arXiv [Preprint] (2023). https://arxiv.org/abs/2307.02179 (Accessed 19 December 2023).

In the paper

“As the sections above describe, Generative AI has many limitations for social science research—most of which we are only beginning to understand.”

In the paper

“Accomplishing this agenda will require deeper understanding of how Generative AI tools are built and fine-tuned.”

In the paper

“Meta has even released detailed information about the inner workings of the model such as the numeric weights its uses to respond to prompts. This enables researchers to better control the fine-tuning process, and study how LLMs work.”

In the paper

“Transparent, public discussion about Generative AI may also help researchers evaluate some of the other risks above—such as the dissemination of misinformation.”

In the paper

“This will require a much more sophisticated understanding of how the behavior of individual agents is constrained by social networks, institutions, organizations, and other extra-individual factors that are cornerstones of the science of human behavior.”

In the paper

“If the capabilities of these tools continue to expand at a fraction of their current pace, Generative AI may become a fixture within the social scientist’s toolkit much sooner than many researchers realize.”

In the paper

“Some newer models created by Mistral even offer model weights that are “raw,” or not fine-tuned.”

In the paper

“If such an organization could be realized, however, it would not only improve access to state-of-the-art technology among a broad group of scholars, but also improve our capacity to build disciplinary standards of open science and ethics into research with Generative AI.”

In the paper

“There is no guarantee that Meta will suddenly restrict access to its Generative AI tools, or begin charging researchers to access it like their peers. Though researchers at wealthy institutions may be able to afford such fees, many others may not—reproducing inequality within the academic caste system (76).”

What the sources say

• Not checked yetcites [76] A. Clauset, S. Arbesman, D. B. Larremore, Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1, e1400005 (2015). 26601125 PMC4644075 source

In the paper

“Yet multiple groups of leading scholars and policy experts argue such risks are marginal when compared to information that is already readily available on the internet (78, 79).”

What the sources say

• Not checked yetcites [78] R. Bommasani, Considerations for Governing Open Foundation Models. Issue Brief Human, Stanford HAI (Elsevier, 2023).
• Not checked yetcites [79] S. Kapoor et al., On the societal impact of open foundation models. arXiv [Preprint] (2024). 10.48550/arXiv.2403.07918 (Accessed 20 December 2023).

In the paper

“In the early 2010s, many companies shared generous amounts of data with researchers. This enabled entirely new forms of scholarly research, and social scientists and researchers inside industry frequently collaborated with each other and presented their work at conferences. In more recent years, data-sharing practices at most large social media companies have been discontinued (77).”

What the sources say

• Not checked yetcites [77] D. Freelon, Computational research in the post-API age. Polit. Commun. 35, 1– 4 (2018). source

In the paper

“For the time being, both Meta and Google have made two of their models available to the public: Llama and Gemma.”