Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models
Open accessAbstract
On October 14th, 2020, researchers from OpenAI, the Stanford Institute for Human-Centered Artificial Intelligence, and other universities convened to discuss open research questions surrounding GPT-3, the largest publicly-disclosed dense language model at the time. The meeting took place under Chatham House Rules. Discussants came from a variety of research backgrounds including computer science, linguistics, philosophy, political science, communications, cyber policy, and more. Broadly, the discussion centered around two main questions: 1) What are the technical capabilities and limitations of large language models? 2) What are the societal effects of widespread use of large language models? Here, we provide a detailed summary of the discussion organized by the two themes above.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The capabilities of transformer-based language models significantly improve with model size.
“Because the capabilities of these models significantly improve with model size (Brown et al., 2020; Radford et al., 2019; Tamkin et al., 2021)”
From Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models (2023) · cites this paper as Tamkin et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cited by
Sorted by citations
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
The future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045 · see citing works
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
“This could lead to deception and distrust (Tamkin et al., 2021).”
BackgroundTaxonomy of Risks posed by Language Models
Weidinger, L., Uesato, J., Rauh, M., Griffin, C., Huang, P.-S., Mellor, J., Glaese, A., Cheng, M., Balle, B., Kasirzadeh, A., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533088 · Cited by 764 · see citing works
“As one paper concluded, it is difficult to scope all possible (mis-)uses of LMs [180].”
BackgroundInfluentialBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationTowards a standard for identifying and managing bias in artificial intelligence
Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022) · doi:10.6028/nist.sp.1270 · Cited by 548 · see citing works
“Early practice has shown that concerns about the use of LLMs are indeed valid, with preliminary experimental results showing LLMs exhibit significant bias [158, 165, 166].”
BackgroundThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · see citing works
Bouschery, S. G., Blazevic, V., & Piller, F. T. (2023) · Journal of Product Innovation Management · doi:10.1111/jpim.12656 · Cited by 444 · see citing works
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349 · see citing works
“Large language models (LLMs) are sophisticated machine learning algorithms designed to process, understand, and generate humanlike language, enabling key developments in applications such as automated conversation, text analysis, creative writing, and complex problem-solving [1].”
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
ChatGPT in education: global reactions to AI innovations
Fütterer, T., Fischer, C., Alekseeva, A., Chen, X., Tate, T., Warschauer, M., & Gerjets, P. (2023) · Scientific Reports · doi:10.1038/s41598-023-42227-6 · Cited by 242 · see citing works
Predictability and Surprise in Large Generative Models
Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., et al. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533229 · Cited by 207 · see citing works
Self-citationUnderstanding Large-Language Model (LLM)-powered Human-Robot Interaction
Kim, C. Y., Lee, C. P., & Mutlu, B. (2024) · Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction · doi:10.1145/3610977.3634966 · Cited by 162 · see citing works
Societal Biases in Language Generation: Progress and Challenges
Sheng, E., Chang, K.-W., Natarajan, P., & Peng, N. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.330 · Cited by 131 · see citing works
On the creativity of large language models
Franceschelli, G., & Musolesi, M. (2025) · AI & SOCIETY · doi:10.1007/s00146-024-02127-3 · Cited by 113 · see citing works
Li, J., Zong, H., Wu, E., Wu, R., Peng, Z., Zhao, J., Yang, L., Xie, H., & Shen, B. (2024) · BMC Medical Education · doi:10.1186/s12909-024-05738-y · Cited by 104 · see citing works
AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuances
Agarwal, D., Naaman, M., & Vashistha, A. (2024) · arXiv · doi:10.48550/arxiv.2409.11360 · Cited by 99
AI chatbots and (mis)information in public health: impact on vulnerable communities.
Meyrowitsch, D. W., Jensen, A. K., Sørensen, J. B., & Varga, T. V. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1226776 · Cited by 73 · see citing works
Truth, Lies, and Automation: How Language Models Could Change Disinformation
Center for Security and Emerging Technology, Buchanan, B., Lohn, A., Musser, M., & Sedova, K. (2021) · doi:10.51593/2021ca003 · Cited by 74 · see citing works
Kolade, O., Owoseni, A., & Egbetokun, A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25953 · Cited by 66 · see citing works
Riedel, M., Kaefinger, K., Stuehrenberg, A., Ritter, V., Amann, N., Graf, A., Recker, F., Klein, E., Kiechle, M., Riedel, F., et al. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1296615 · Cited by 49 · see citing works
Studying and improving reasoning in humans and machines
Yax, N., Anlló, H., & Palminteri, S. (2024) · Communications Psychology · doi:10.1038/s44271-024-00091-8 · Cited by 40 · see citing works
Computing Power and the Governance of Artificial Intelligence
Sastry, G., Heim, L., Belfield, H., Anderljung, M., Brundage, M., Hazell, J., O'Keefe, C., Hadfield, G. K., Ngo, R., Pilz, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2402.08797 · Cited by 26 · see citing works
Self-citationIn conversation with Artificial Intelligence: aligning language models with human values
Kasirzadeh, A., & Gabriel, I. (2022) · arXiv · doi:10.48550/arxiv.2209.00731 · Cited by 128 · see citing works
“Conversational agents have been shown to perform well on a variety of computational metrics, supporting the emergence of new kinds of capability and opportunity [17, 95].”
BackgroundTesting the use of a large language model (LLM) for performing data quality assessment
MacMaster, S., & Sinistore, J. (2025) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-024-02405-8 · Cited by 14 · see citing works
Meyer, B., Kfuri‐Rubens, R., Schmidt, G., Tariq, M., Riedel, C., Recker, F., Riedel, F., Kiechle, M., & Riedel, M. (2025) · International Journal of Gynecology & Obstetrics · doi:10.1002/ijgo.70251 · Cited by 10 · see citing works
AI-driven abstract generating: evaluating LLMs with a tailored prompt under the PRISMA-A framework
Demi̇r, G. B., Gökmen, Ş., Süküt, Y., Topsakal, K. G., & Görgülü, S. (2025) · BMC Oral Health · doi:10.1186/s12903-025-06982-4 · Cited by 2 · see citing works
- Authors
- Tamkin, A., Brundage, M., Clark, J., & Ganguli, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2102.02503
- OpenAlex
- W3128912454
- Cited by
- 135
- References
- 0
- Citations by year
- 2026: 6 · 2025: 24 · 2024: 42 · 2023: 46 · 2022: 10 · 2021: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, GPT-3, social impact, open research questions, human-centered AI, AI policy
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
