CoChat
Sign Up
PreprintCited by 2Ethics and Social Impacts of AILaw, AI, and Intellectual PropertyOpen Source Software Innovations

Near to Mid-term Risks and Opportunities of Open-Source Generative AI

Open access
Eiras, Francisco, Petrov, Aleksandar, Vidgen, Bertie, de Witt, Christian Schroeder, Pizzati, Fabio, Elkins, Katherine, Mukhopadhyay, Supratik, Bibi, Adel +16 more·2024·arXiv
DOI 10.48550/arxiv.2404.17047arXiv 2404.17047
Open this paper in CoChat — free accountOpen PDF

Abstract

In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential risks and resulted in calls for tighter regulation, in particular from some of the major tech companies who are leading in AI development. This regulation is likely to put at risk the budding field of open-source Generative AI. We argue for the responsible open sourcing of generative AI models in the near and medium term. To set the stage, we first introduce an AI openness taxonomy system and apply it to 40 current large language models. We then outline differential benefits and risks of open versus closed source AI and present potential risk mitigation, ranging from best practices to calls for technical and scientific contributions. We hope that this report will add a much needed missing voice to the current public discourse on near to mid-term AI safety and other societal impact.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2404.17047

Cites

0 citation errors · 5 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The necessarily gradient nature of openness is seen in release methods, where prior work has pointed out tradeoffs between full openness and risk mitigation.

    “The necessarily gradient nature of openness is seen in release methods, where prior work has pointed out tradeoffs between full openness and risk mitigation [16, 53].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [16]

  • Open source AI risks and opportunities are increasingly being studied in the generative AI landscape.

    “Open source AI risks and opportunities are increasingly being studied in the generative AI landscape [16].”

    From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [16]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 37

  1. Agencia de Gobierno. Mesa de diálogo "Inteligencia Artifi- cial: oportunidades y desafíos de una estrategia nacional". Agencia de Gobierno Electrónico y Sociedad de la Infor- mación y del Conocimiento. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R Mortensen, Noah A Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? Tok- enization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing.
  2. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Ma- dian Khabsa. 2023. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884. · arXiv
  3. Rishi Bommasani, Sayash Kapoor, Kevin Klyman, Shayne Longpre, Ashwin Ramaswami, Daniel Zhang, Marietje Schaake, Daniel E. Ho, Arvind Narayanan, and Percy Liang. 2023a. Considerations for governing open founda- tion models. Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. 2023b. Introducing the foundation model transparency index. arXiv preprint arXiv:2310.12941. · arXiv
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
  5. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations.
  6. Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Benjamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, et al. 2024. Black-box access is insufficient for rigorous AI audits. arXiv preprint arXiv:2401.14446. · arXiv
  7. Grant Cooper. 2023. Examining science education in Chat- GPT: An exploratory study of generative artificial intelli- gence. Journal of Science Education and Technology.
  8. Aron Culotta and Nicholas Mattei. 2023. Use open source for safer generative AI experiments. MIT Sloan Manage- ment Review.
  9. Emilio Ferrara. 2023. GenAI against humanity: Nefarious applications of generative artificial intelligence and large language models. arXiv preprint arXiv:2310.00737. · arXiv
  10. Gebru, Jamie Morgenstern, Briana Vecchione, Jen- nifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM, 64(12):86-92.
  11. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. 2023. Llama Guard: LLM-based input-output safe- guard for Human-AI conversations. arXiv preprint arXiv:2312.06674. · arXiv
  12. Jeevan Jaisingh, Eric WK See-To, and Kar Yan Tam. 2008. The impact of open source software on the strategic choices of firms developing proprietary software. Journal of Management Information Systems.
  13. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. arXiv preprint arXiv:2004.09095. · arXiv
  14. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright Violations and Large Lan- guage Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
  15. Ajay Kumar and Tom Davenport. 2023. How to make generative ai greener. Harvard Business Review.
  16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question an- swering research. Transactions of the Association for Computational Linguistics.
  17. Andreas Liesenfeld, Alianda Lopez, and Mark Dingemanse. 2023. Opening up ChatGPT: Tracking openness, trans- parency, and accountability in instruction-tuned text gen- erators. In Proceedings of the 5th International Confer- ence on Conversational user interfaces.
  18. Hila Lifshitz-Assaf and Frank Nagle. 2021. The digital economy runs on open source. here's how to protect it. Harvard Business Review.
  19. Tegwen Malik, Laurie Hughes, Yogesh K Dwivedi, and San- dra Dettmer. 2023. Exploring the transformative impact of generative AI on higher education. In Conference on e-Business, e-Services and e-Society.
  20. Alex Mei, Anisha Kabir, Sharon Levy, Melanie Subbiah, Emily Allaway, John Judge, Desmond Patton, Bruce Bim- ber, Kathleen McKeown, and William Yang Wang. 2022. Mitigating covertly unsafe text within natural language systems. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022.
  21. Alex Mei, Sharon Levy, and William Wang. 2023. ASSERT: Automated safety scenario red teaming for evaluating the robustness of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023.
  22. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with hu- man feedback. Advances in Neural Information Process- ing Systems, 35:27730-27744.
  23. James W Paulson, Giancarlo Succi, and Armin Eberlein. 2004. An empirical study of open-source and closed- source software products. IEEE Transactions on Software Engineering, 30(4):246-256.
  24. Aleksandar Petrov, Emanuele La Malfa, Philip HS Torr, and Adel Bibi. 2023. Language model tokenizers intro- duce unfairness between languages. Neural Information Processing Systems (NeurIPS).
  25. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  26. Guido Schryen and Rouven Kadura. 2009. Open source vs. closed source software: towards measuring security. In Proceedings of the 2009 ACM Symposium on Applied Computing, pages 2016-2023.
  27. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, et al. 2022. LAION-5B: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 25294.
  28. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. 2023. Beyond the imitation game: Quantify- ing and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615. · arXiv
  29. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. arXiv preprint arXiv:1906.02243. · arXiv
  30. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
  31. Roberto Verdecchia, June Sallou, and Luís Cruz. 2023. A systematic review of Green AI. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, page e1507.
  32. Georg Von Krogh and Sebastian Spaeth. 2007. The open source software phenomenon: Characteristics that pro- mote research. The Journal of Strategic Information Systems, 16(3):236-253.
  33. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sus- tainable AI: Environmental implications, challenges and opportunities. Proceedings of Machine Learning and Systems, 4:795-813.
  34. Kangxi Wu, Liang Pang, Huawei Shen, Xueqi Cheng, and Tat-Seng Chua. 2023. LLMDet: A third party large lan- guage models generated text detection tool. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2113-2133.
  35. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36.
  36. Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury. 2023a. Synthetic lies: Un- derstanding ai-generated misinformation and evaluating algorithmic and human solutions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-20.
  37. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrik- son. 2023. Universal and transferable adversarial at- tacks on aligned language models. arXiv preprint arXiv:2307.15043. · arXiv

Cited by

Sorted by citations

Related

Authors
Eiras, F., Petrov, A., Vidgen, B., de Witt, C. S., Pizzati, F., Elkins, K., Mukhopadhyay, S., Bibi, A., Csaba, B., Steibel, F., Barez, F., Smith, G., Guadagni, G., Chun, J., Cabot, J., Imperial, J. M., Nolazco-Flores, J. A., Landay, L., Jackson, M., Röttger, P., Torr, P. H. S., Darrell, T., Lee, Y. S., & Foerster, J.
Affiliations
  • University of Oxford (GB)
  • Kenyon College (US)
  • Louisiana State University (US)
  • Institute for Technology & Society (BR)
  • University of California, Berkeley (US)
  • University of Virginia (US)
  • University of Luxembourg (LU)
  • National University (PH)
  • Berklee College of Music (US)
  • University of Notre Dame (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2404.17047
arXiv
2404.17047
OpenAlex
W4396506709
Cited by
2
References
0
Citations by year
2025: 1 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Archivio istituzionale della Ricerca - Bocconi (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • UK Research and Innovation
  • Royal Academy of Engineering
  • University of Bath
  • King Abdullah University of Science and Technology
  • Engineering and Physical Sciences Research Council (EP/Y028481/1)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Law, AI, and Intellectual Property
  • Physical Sciences › Computer Science › Computer Science Applications › Open Source Software Innovations
Keywords
large language models, AI regulation, AI safety, risk mitigation, responsible AI, societal impact of AI

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review