CoChat
Sign Up
PreprintCited by 23Software Engineering ResearchEthics and Social Impacts of AISafety Systems Engineering in Autonomy

A Hazard Analysis Framework for Code Synthesis Large Language Models

Open access
Khlaaf, Heidy, Mishkin, Pamela, Achiam, Joshua, Krueger, Gretchen, Brundage, Miles·2022·arXiv
DOI 10.48550/arxiv.2207.14157arXiv 2207.14157
Open this paper in CoChat — free accountOpen PDF

Abstract

Codex, a large language model (LLM) trained on a variety of codebases, exceeds the previous state of the art in its capacity to synthesize and generate code. Although Codex provides a plethora of benefits, models that may generate code on such scale have significant limitations, alignment problems, the potential to be misused, and the possibility to increase the rate of progress in technical fields that may themselves have destabilizing impacts or have misuse potential. Yet such safety impacts are not yet known or remain to be explored. In this paper, we outline a hazard analysis framework constructed at OpenAI to uncover hazards or safety risks that the deployment of models like Codex may impose technically, socially, politically, and economically. The analysis is informed by a novel evaluation framework that determines the capacity of advanced code generation techniques against the complexity and expressivity of specification prompts, and their capability to understand and execute them relative to human ability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2207.14157

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • To obtain a high degree of confidence in a system, a positive safety case providing quantifiable guarantees is needed, using either or both empirical and theoretical arguments.

    “We argue that to obtain a high degree of confidence in a system we need a positive safety case providing quantifiable guarantees, using either or both empirical and theoretical arguments (also see Bloomfield et al. (2019); Khlaaf et al. (2022) for a similar argument).”

    From Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems (2024) · cites this paper as Khlaaf et al. (2022)

    The discussion of mitigations (p. 18) argues that verification and monitoring are needed to ensure hazards are sufficiently controlled, which is similar to the citing authors’ call for evidence of safety. It does not call for a positive safety case or quantifiable guarantees based on empirical or theoretical arguments; the risk-assessment discussion (Section 3, p. 9) says quantitative analysis is not always currently possible. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 29

  1. [2][2] Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. arXiv preprint arXiv:2101.05783 (2021). - · arXiv
  2. [3][3] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A General Language Assistant as a Laboratory for Alignment. arXiv preprint arXiv:2112.00861 (2021). - · arXiv
  3. [5][5] Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency . 610-623. -
  4. [6][6] Iwo Błądek, Krzysztof Krawiec, and Jerry Swan. 2018. Counterexample-driven genetic programming: heuristic program synthesis from formal specifications. Evolutionary computation 26, 3 (2018), 441-469. -
  5. [7][7] Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of" bias" in nlp. arXiv preprint arXiv:2005.14050 (2020). - · arXiv
  6. [9][9] Marc Brockschmidt, Miltiadis Allamanis, Alexander L. Gaunt, and Oleksandr Polozov. 2019. Generative Code Modeling with Graphs. ArXiv abs/1805.08490 (2019). - · arXiv
  7. [10][10] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020). - · arXiv
  8. [11][11] Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, et al. 2020. Toward trustworthy AI development: mechanisms for supporting verifiable claims. arXiv preprint arXiv:2004.07213 (2020). - · arXiv
  9. [12][12] Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, and Yisong Yue. 2021. Neurosymbolic Programming. Foundations and Trends® in Programming Languages 7, 3 (2021), 158-243. https://doi.org/10.1561/2500000049 - · DOI
  10. [13][13] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. (2021). arXiv:2107.03374 [cs.LG] - · arXiv
  11. [14][14] Michael R Clarkson, Bernd Finkbeiner, Masoud Koleini, Kristopher K Micinski, Markus N Rabe, and César Sánchez. 2014. Temporal logics for hyperproperties. In International Conference on Principles of Security and Trust . Springer, 265-284. -
  12. [16][16] Samuel Drews, Aws Albarghouthi, and Loris D'Antoni. 2019. Efficient Synthesis with Probabilistic Constraints. In Computer Aided Verification , Isil Dillig and Serdar Tasiran (Eds.). Springer International Publishing, Cham, 278-296. -
  13. [17][17] Sumit Gulwani. 2011. Automating String Processing in Spreadsheets Using Input-Output Examples. SIGPLAN Not. 46, 1 (jan 2011), 317-330. https://doi.org/10.1145/1925844.1926423 - · DOI · Paper page
  14. [18][18] Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh. 2017. Program Synthesis. Foundations and Trends® in Programming Languages 4, 1-2 (2017), 1-119. https://doi.org/10.1561/2500000010 - · DOI
  15. [19][19] Brendan Hall, Jan Fiedor, and Yogananda Jeppu. 2020. Model Integrated Decomposition and Assisted Specification (MIDAS). In INCOSE International Symposium , Vol. 30. Wiley Online Library, 821-841. -
  16. [20][20] Thomas Helmuth and Lee Spector. 2015. General program synthesis benchmark suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation . 1039-1046. -
  17. [21][21] Thomas Helmuth and Lee Spector. 2015. General Program Synthesis Benchmark Suite. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation (Madrid, Spain) (GECCO '15) . Association for Computing Machinery, New York, NY, USA, 1039-1046. https://doi.org/10.1145/2739480.2754769 - · DOI
  18. [22][22] John R Koza, David Andre, Martin A Keane, and Forrest H Bennett III. 1999. Genetic programming III: Darwinian invention and problem solving . Vol. 3. Morgan Kaufmann. -
  19. [23][23] Leslie Lamport. 1980. "Sometime" is Sometimes "Not Never": On the Temporal Logic of Programs. In Proceedings of the 7th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (Las Vegas, Nevada) (POPL '80) . Association for Computing Machinery, New York, NY, USA, 174-185. https://doi.org/10.1145/567446.567463 - · DOI
  20. [24][24] Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871 (2018). - · arXiv
  21. [26][26] Nancy G Leveson. 2016. Engineering a safer world: Systems thinking applied to safety . The MIT Press. -
  22. [27][27] Alistair Mavin and Philip Wilkinson. 2010. Big ears (the return of" easy approach to requirements engineering"). In 2010 18th IEEE International Requirements Engineering Conference . IEEE, 277-282. -
  23. [28][28] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency . 220-229. -
  24. [29][29] Michael O'Neill and Lee Spector. 2020. Automatic programming: The open issue? Genetic Programming and Evolvable Machines 21, 1 (2020), 251-262. -
  25. [30][30] Edward Pantridge, Thomas Helmuth, Nicholas Freitag McPhee, and Lee Spector. 2017. On the difficulty of benchmarking inductive program synthesis methods. In Proceedings of the Genetic and Evolutionary Computation Conference Companion . 1589-1596. -
  26. [31][31] Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021. An Empirical Cybersecurity Evaluation of GitHub Copilot's Code Contributions. CoRR abs/2108.09293 (2021). arXiv:2108.09293 https://arxiv.org/abs/2108. 09293 - · arXiv
  27. [33][33] Irene Solaiman and Christy Dennison. 2021. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. CoRR abs/2106.10328 (2021). arXiv:2106.10328 https://arxiv.org/abs/2106.10328 - · arXiv
  28. [34][34] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from Language Models. arXiv preprint arXiv:2112.04359 (2021). - · arXiv
  29. [35][35] Frank F Xu, Bogdan Vasilescu, and Graham Neubig. 2021. In-IDE Code Generation from Natural Language: Promise and Challenges. arXiv preprint arXiv:2101.11149 (2021). · arXiv

Cited by

Sorted by citations

Authors
Khlaaf, H., Mishkin, P., Achiam, J., Krueger, G., & Brundage, M.
Affiliations
OpenAI (United States) (US)
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2207.14157
arXiv
2207.14157
OpenAlex
W4288805334
Cited by
23
References
0
Citations by year
2026: 1 · 2025: 11 · 2024: 4 · 2023: 7
Type
posted content
Language
en
Open access
Licence
creativecommons.org/publicdomain/zero/1.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
Keywords
hazard analysis, large language models, code generation, codices, AI safety, risk assessment, program synthesis, evaluation framework, prompt complexity, human-AI comparison

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review