CoChat
Sign Up
PreprintCited by 1Safety Systems Engineering in AutonomyEthics and Social Impacts of AIDigitalization, Law, and Regulation

A Structured Approach to Safety Case Construction for AI Systems

Open access
Lee, Sung Une, Zhu, Liming, Shamsujjoha, Md, Dong, Liming, Lu, Qinghua, Chen, Jieshan, Briand, Lionel·2026·arXiv
DOI 10.48550/arxiv.2601.22773arXiv 2601.22773
Open this paper in CoChat — free accountOpen PDF

Abstract

Safety cases, structured arguments that a system is acceptably safe, are becoming central to the governance of AI systems. Yet, traditional safety-case practices from aviation or nuclear engineering rely on well-specified system boundaries, stable architectures, and known failure modes. Modern AI systems, such as generative and agentic AI, are the opposite. Their capabilities emerge unpredictably from low-level training objectives, their behaviour varies with prompts, and their risk profiles shift through fine-tuning, scaffolding, or deployment context. This study examines how safety cases are currently constructed for AI systems and why classical approaches fail to capture these dynamics. This study introduces comprehensive taxonomies for AI-specific claim types (assertion-based, constraint-based, capability-based), argument types (demonstrative, comparative, causal/explanatory, risk-based, and normative), and evidence families (empirical, mechanistic, comparative, expert-driven, formal methods, operational/field data, and model-based). It then proposes a reusable safety-case template, each of which follows a predefined structure of claims, arguments, and evidence tailored for AI systems. Each template is illustrated by end-to-end patterns that address distinctive challenges, such as evaluation without ground truth, dynamic model updates, and threshold-based risk decisions. The result is a systematic, composable, and reusable approach to constructing and maintaining safety cases that are credible, auditable, and adaptive to the evolving behaviour of generative and frontier AI systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.22773

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Whenever inspection, approval, rollback, or policy enforcement matters, externalized artifacts have clear advantages over opaque parametric behavior.

    “Whenever inspection, approval, rollback, or policy enforcement matters, externalized artifacts have clear advantages over opaque parametric behavior [Li et al., 2026b, Lazaros et al., 2026, Lee et al., 2026, Fernandez, 2026, Zhu and Lu, 2026].”

    From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lee et al., 2026

    The “Operational and field data” section identifies audit trails and approval records as governance artifacts linking operational data to oversight, and the “Argument functions and reasoning logics” section discusses rollback controls (pages unavailable). These passages support the value of inspectable records and controls, but do not establish the claimed comparative advantage over opaque parametric behavior. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The layered configuration model means the same base agent can operate under different policy regimes depending on deployment context, without changes to the model or skill artifacts.

    “This layered model means that the same base agent can operate under different policy regimes depending on its deployment context, without any change to the model or the skill artifacts it loads [Anthropic, 2026, Lee et al., 2026].”

    From Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering (2026) · cites this paper as Lee et al., 2026

    Section 7.2 discusses adapting safety-case templates to different regulatory regimes, not configuring the same base agent for different policy regimes. Searches for layered agent configuration and skill artifacts found no passage saying this can be done without changing the model or loaded skills. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 153

  1. S. Burton, "A causal model of safety assurance for machine learning," arXiv preprint arXiv:2201.05451, 2022. · arXiv
  2. D. Tola and P. G. Larsen, "A co-simulation based approach for developing safety-critical systems," in Proceedings of the 18th International Overture Workshop, vol. 65, 2021.
  3. B. Herd, J.-V. Zacchi, and S. Burton, "A deductive approach to safety assurance: Formalising safety contracts with subjective logic," in International Conference on Computer Safety, Reliability, and Security. Springer Nature Switzerland Cham, 2024, pp. 213-226.
  4. Y. Jia, C. Verrill, K. White, M. Dolton, M. Horton, M. Jafferji, and I. Habli, "A deployment safety case for ai-assisted prostate cancer diagnosis," Computers in biology and medicine, vol. 192, p. 110237, 2025.
  5. E. Denney and G. Pai, "A formal basis for safety case patterns," in International Conference on Computer Safety, Reliability, and Security. Springer, 2013, pp. 21-32.
  6. J. Krook, Y. Selvaraj, W. Ahrendt, and M. Fabian, "A formal-methods approach to provide evidence in automated- driving safety cases," arXiv preprint arXiv:2210.07798, 2022. · arXiv
  7. N. Hayama, Y. Yamagata, H. Nishihara, and Y. Matsuno, "A gsn-based requirement analysis of the eu AI regulation," in International Conference on Computer Safety, Reliability, and Security. Springer Nature Switzerland Cham, 2025, pp. 183-196.
  8. P. Bishop and R. Bloomfield, "A methodology for safety case development," in Safety and Reliability, vol. 20, no. 1. Taylor & Francis, 2000, pp. 34-42.
  9. Z. Porter, I. Habli, J. McDermid, and M. Kaas, "A principles-based ethics assurance argument pattern for AI and autonomous systems," AI and Ethics, vol. 4, no. 2, pp. 593-616, 2024.
  10. O. Odu, A. B. Belle, S. Wang, and K. K. Shahandashti, "A prisma-driven bibliometric analysis of the scientific literature on assurance case patterns," arXiv preprint arXiv:2407.04961, 2024. · arXiv
  11. K. K. Shahandashti, A. B. Belle, T. C. Lethbridge, O. Odu, and M. Sivakumar, "A prisma-driven systematic mapping study on system assurance weakeners," Information and Software Technology, vol. 175, p. 107526, 2024.
  12. M. Gyllenhammar, G. R. d. Campos, and M. Törngren, "A safety argument fragment towards safe deployment of performant automated driving systems," in International Conference on Computer Safety, Reliability, and Security. Springer, 2025, pp. 197-210.
  13. E. Wozniak, C. Cârlan, E. Acar-Celik, and H. J. Putzer, "A safety case pattern for systems with machine learning components," in International Conference on Computer Safety, Reliability, and Security. Springer, 2020, pp. 370-382.
  14. T. Korbak, J. Clymer, B. Hilton, B. Shlegeris, and G. Irving, "A sketch of an AI control safety case," arXiv preprint arXiv:2501.17315, 2025. · arXiv
  15. R. Wei, S. Foster, H. Mei, F. Yan, R. Yang, I. Habli, C. O'Halloran, N. Tudor, T. Kelly, and Y. Nemouchi, "Access: Assurance case centric engineering of safety-critical systems," Journal of Systems and Software, vol. 213, p. 112034, 2024.
  16. S. Burton and B. Herd, "Addressing uncertainty in the safety assurance of machine-learning," Frontiers in computer science, vol. 5, p. 1132580, 2023.
  17. B. Kaiser, M. Soden, R. Diefenbach, and E. Holz, "An agile approach to safety cases for autonomous systems through model-based engineering and simulation," in Proc. 33rd Saf. Crit. Syst. Symp, 2025.
  18. M. D. Buhl, J. Pfau, B. Hilton, and G. Irving, "An alignment safety case sketch based on debate," arXiv preprint arXiv:2505.03989, 2025. · arXiv
  19. F. R. Ward and I. Habli, "An assurance case pattern for the interpretability of machine learning in safety-critical systems," in International Conference on Computer Safety, Reliability, and Security. Springer, 2020, pp. 395-407.
  20. J. Clymer, J. Weinbaum, R. Kirk, K. Mai, S. Zhang, and X. Davies, "An example safety case for safeguards against misuse," arXiv preprint arXiv:2505.18003, 2025. · arXiv
  21. S. Nair, J. L. De La Vara, M. Sabetzadeh, and L. Briand, "An extended systematic literature review on provision of evidence for safety certification," Information and Software Technology, vol. 56, no. 7, pp. 689-717, 2014.
  22. U. D. Ferrell and A. H. A. Anderegg, "Applicability of ul 4600 to unmanned aircraft systems (uas) and urban air mobility (uam)," in 2020 AIAA/IEEE 39th Digital Avionics Systems Conference (DASC). IEEE, 2020, pp. 1-7.
  23. J. P. C. de Araujo, B. V. Balu, E. Reichmann, J. Kelly, S. Kugele, N. Mata, and L. Grunske, "Applying concept- based models for enhanced safety argumentation," in 2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2024, pp. 272-283.
  24. S. Barrett, P. Fox, J. Krook, T. Mondal, S. Mylius, and A. Tlaie, "Assessing confidence in frontier AI safety cases," arXiv preprint arXiv:2502.05791, 2025. · arXiv
  25. R. Kaur, R. Ivanov, M. Cleaveland, O. Sokolsky, and I. Lee, "Assurance case patterns for cyber-physical systems with deep neural networks," in International Conference on Computer Safety, Reliability, and Security. Springer, 2020, pp. 82-97.
  26. T. Chowdhury, "Assurance case templates: Principles for their development and criteria for their evaluation," Ph.D. dissertation, McMaster University, 2021.
  27. R. Bloomfield and J. Rushby, "Assurance of AI systems from a dependability perspective," arXiv preprint arXiv:2407.13948, 2024. · arXiv
  28. E. Denney and G. Pai, "Assurance-driven design of machine learning-based functionality in an aviation systems context," in 2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC). IEEE, 2023, pp. 1-10.
  29. A. Wardziński and A. Jarzębowicz, "Automated generation of modular assurance cases with the system assurance reference model," Formal Aspects of Computing, vol. 36, no. 4, pp. 1-29, 2024.
  30. O. Odu, A. B. Belle, S. Wang, S. Kpodjedo, T. C. Lethbridge, and H. Hemmati, "Automatic instantiation of assurance cases from patterns using large language models," Journal of Systems and Software, vol. 222, p. 112353, 2025.
  31. C. Cârlan, L. Gauerhof, B. Gallina, and S. Burton, "Automating safety argument change impact analysis for machine learning components," in 2022 IEEE 27th Pacific Rim International Symposium on Dependable Computing (PRDC). IEEE, 2022, pp. 43-53.
  32. S. Diemert, E. Cyffka, N. Anwari, O. Foster, T. Viger, L. Millet, and J. Joyce, "Balancing the risks and benefits of using large language models to support assurance case development," in International Conference on Computer Safety, Reliability, and Security. Springer, 2025, pp. 209-225.
  33. C. Cârlan, "Checkable safety arguments-a modeling framework supporting the maintenance of safety arguments consistent with system development artifacts," Ph.D. dissertation, Technische Universität München, 2025.
  34. Y. Idmessaoud, D. Dubois, and J. Guiochet, "Confidence assessment in safety argument structure-quantitative vs. qualitative approaches," International Journal of Approximate Reasoning, vol. 165, p. 109100, 2024.
  35. M. Sivakumar, "Design and automatic generation of safety cases of ml-enabled autonomous driving systems," Master's thesis, York University, 2024.
  36. M. Sivakumar, A. B. Belle, J. Shan, O. Odu, and M. Yuan, "Design of the safety case of the reinforcement learning-enabled component of a quanser autonomous vehicle," in 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW). IEEE, 2024, pp. 57-67.
  37. E. Mirzaei, C. CARLAN, C. Thomas, and B. Gallina, "Design-time specification of dynamic modular safety cases in support of run-time safety assessment," in Proceedings of the 30th Safety-Critical Systems Symposium (SCSC), 2022.
  38. R. Hawkins, "Developing compelling safety cases," arXiv preprint arXiv:2502.00911, 2025. · arXiv
  39. E. Asaadi, E. Denney, J. Menzies, G. J. Pai, and D. Petroff, "Dynamic assurance cases: a pathway to trusted autonomy," Computer, vol. 53, no. 12, pp. 35-46, 2020.
  40. S. Ramakrishna, "Dynamic safety assurance of autonomous cyber-physical systems," Ph.D. dissertation, Vander- bilt University, 2022.
  41. C. Cârlan, F. Gomez, Y. Mathew, K. Krishna, R. King, P. Gebauer, and B. R. Smith, "Dynamic safety cases for frontier ai," arXiv preprint arXiv:2412.17618, 2024. · arXiv
  42. M. Borg, J. Henriksson, K. Socha, O. Lennartsson, E. Sonnsjö Lönegren, T. Bui, P. Tomaszewski, S. R. Sathyamoorthy, S. Brink, and M. Helali Moghadam, "Ergo, smirk is safe: a safety case for a machine learning component in a pedestrian automatic emergency brake system," Software quality journal, vol. 31, no. 2, pp. 335-403, 2023.
  43. C. Burr and D. Leslie, "Ethical assurance: a practical approach to the responsible design, development, and deployment of data-driven technologies," AI and Ethics, vol. 3, no. 1, pp. 73-98, 2023.
  44. J. Roßbach, O. De Candido, A. Hammam, and M. Leuschel, "Evaluating ai-based components in autonomous railway systems: A methodology," in German Conference on Artificial Intelligence (Künstliche Intelligenz). Springer, 2024, pp. 190-203.
  45. F. Ikhwantri and D. Marijan, "Explainable compliance detection with multi-hop natural language inference on assurance case structure," arXiv preprint arXiv:2506.08713, 2025. · arXiv
  46. M. Sivakumar, A. B. Belle, J. Shan, and K. K. Shahandashti, "Exploring the capabilities of large language models for the generation of safety cases: the case of gpt-4," in 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW). IEEE, 2024, pp. 35-45.
  47. I. Sljivo, B. Gallina, J. Carlson, and H. Hansson, "Generation of safety case argument-fragments from safety contracts," in International Conference on Computer Safety, Reliability, and Security. Springer, 2014, pp. 170-185.
  48. M. Sivakumar, A. B. Belle, J. Shan, and K. K. Shahandashti, "Gpt-4 and safety case generation: An exploratory analysis," arXiv preprint arXiv:2312.05696, 2023. · arXiv
  49. M. Chelouati, A. Boussif, J. Beugin, and E.-M. El Koursi, "Graphical safety assurance case using goal structuring notation (gsn)-challenges, opportunities and a framework for autonomous trains," Reliability Engineering & System Safety, vol. 230, p. 108933, 2023.
  50. I. Sljivo, E. Denney, and J. Menzies, "Guided integration of formal verification in assurance cases," in Interna- tional Conference on Formal Engineering Methods. Springer, 2023, pp. 172-190.
  51. R. Hawkins and P. Ryan Conmy, "Identifying run-time monitoring requirements for autonomous systems through the analysis of safety arguments," in International Conference on Computer Safety, Reliability, and Security. Springer, 2023, pp. 11-24.
  52. G. Despotou, S. White, T. Kelly, and M. Ryan, "Introducing safety cases for health it," in 2012 4th International Workshop on Software Engineering in Health Care (SEHC). IEEE, 2012, pp. 44-50.
  53. M. J. Squair, "Issues in the application of software safety standards," in ACM International Conference Proceed- ing Series, vol. 162, 2006, pp. 13-26.
  54. A. Sabuncuoglu, C. Burr, and C. Maple, "Justified evidence collection for argument-based AI fairness assurance," in Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 2025, pp. 18-28.
  55. A. Agrawal, S. Khoshmanesh, M. Vierhauser, M. Rahimi, J. Cleland-Huang, and R. Lutz, "Leveraging artifact trees to evolve and reuse safety cases," in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1222-1233.
  56. Y. Fujiwara, T. Tuchida, R. Miyata, H. Washizaki, and N. Ubayashi, "Llm-based automated mitigation and assurance case generation against threats to AI systems," in 2025 IEEE Conference on Artificial Intelligence (CAI). IEEE, 2025, pp. 906-911.
  57. O. Odu, A. B. Belle, and S. Wang, "Llm-based safety case generation for baidu apollo: Are we there yet?" in 2025 IEEE/ACM 4th International Conference on AI Engineering-Software Engineering for AI (CAIN). IEEE, 2025, pp. 222-233.
  58. S. Burton, L. Gauerhof, and C. Heinzemann, "Making the case for safety of machine learning in highly automated driving," in International Conference on Computer Safety, Reliability, and Security. Springer, 2017, pp. 5-16.
  59. R. Dassanayake, M. Demetroudi, J. Walpole, L. Lentati, J. R. Brown, and E. J. Young, "Manipulation attacks by misaligned ai: Risk analysis and safety case framework," arXiv preprint arXiv:2507.12872, 2025. · arXiv
  60. C.-L. Lin, W. Shen, S. Drager, and B. Cheng, "Measure confidence of assurance cases in safety-critical domains," in Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results, 2018, pp. 13-16.
  61. J. Murdoch, G. Clark, A. Powell, and P. Caseley, "Measuring safety: applying psm to the system safety domain," in Proceedings of the 8th Australian workshop on Safety critical systems and software-Volume 33, 2003, pp. 47-55.
  62. M. A. Langford, K. H. Chan, J. E. Fleck, P. K. McKinley, and B. H. Cheng, "MoDALAS: addressing assurance for learning-enabled autonomous systems in the face of uncertainty," Software and systems modeling, vol. 22, no. 5, pp. 1543-1563, 2023.
  63. L. Nascimento, A. L. de Oliveira, R. Villela, E. F. Silva, R. Wei, R. Hawkins, and T. Kelly, "Model-based security assurance cases for open and adaptive cyber-physical systems," in International Conference on Advanced Information Networking and Applications. Springer, 2025, pp. 326-340.
  64. A. Retouniotis, Y. Papadopoulos, I. Sorokos, D. Parker, N. Matragkas, and S. Sharvia, "Model-connected safety cases," in International Symposium on Model-Based Safety and Assessment. Springer, 2017, pp. 50-63.
  65. D. H. Becht, "Moving towards goal-based safety management," in Proceedings of the Australian System Safety Conference-Volume 133, 2011, pp. 19-26.
  66. C.-H. Cheng, C.-H. Huang, and G. Nührenberg, "nn-dependability-kit: Engineering neural networks for safety- critical autonomous driving systems," in 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD). IEEE, 2019, pp. 1-6.
  67. C. Preschern, N. Kajtazovic, A. Höller, and C. Kreiner, "Pattern-based safety development methods: overview and comparison," in Proceedings of the 19th European Conference on Pattern Languages of Programs, 2014, pp. 1-20.
  68. N. Annable, M. Lawford, R. F. Paige, and A. Wassyng, "Principled safety assurance arguments," in International Conference on Computer Safety, Reliability, and Security. Springer, 2025, pp. 18-32.
  69. M. Sivakumar, A. B. Belle, J. Shan, and K. K. Shahandashti, "Prompting gpt-4 to support automatic safety case generation," Expert Systems with Applications, vol. 255, p. 124653, 2024.
  70. A. Di Sandro, S. Kokaly, R. Salay, and M. Chechik, "Querying automotive system models and safety artifacts with mmint and viatra," in 2019 ACM/IEEE 22nd international conference on model driven engineering languages and systems companion (MODELS-c). IEEE Computer Society, 2019, pp. 2-11.
  71. Y. Dong, W. Huang, V. Bharti, V. Cox, A. Banks, S. Wang, X. Zhao, S. Schewe, and X. Huang, "Reliability assessment and safety arguments for machine learning components in system assurance," ACM transactions on embedded computing systems, vol. 22, no. 3, pp. 1-48, 2023.
  72. L. Buysse, I. Habli, D. Vanoost, and D. Pissoort, "Safe autonomous systems in a changing world: Operationalising dynamic safety cases," Safety Science, vol. 191, p. 106965, 2025.
  73. S. Ballingall, M. Sarvi, and P. Sweatman, "Safety assurance concepts for automated driving systems," SAE International Journal of Advances and Current Practices in Mobility, vol. 2, no. 2020-01-0727, pp. 1528-1537, 2020.
  74. H. Fujino, N. Kobayashi, and S. Shirasaka, "Safety assurance case description method for systems incorporating off-operational machine learning and safety device," in INCOSE International Symposium, vol. 29. Wiley Online Library, 2019, pp. 152-164.
  75. C. Paterson, R. Hawkins, C. Picardi, Y. Jia, R. Calinescu, and I. Habli, "Safety assurance of machine learning for autonomous systems," Reliability Engineering & System Safety, vol. 264, p. 111311, 2025.
  76. S. Burton, I. Kurzidem, A. Schwaiger, P. Schleiss, M. Unterreiner, T. Graeber, and P. Becker, "Safety assurance of machine learning for chassis control functions," in International Conference on Computer Safety, Reliability, and Security. Springer, 2021, pp. 149-162.
  77. S. Burton, C. Hellert, F. Hüger, M. Mock, and A. Rohatschek, "Safety assurance of machine learning for perception functions," in Deep Neural Networks and Data for Automated Driving: Robustness, Uncertainty Quantification, and Insights Towards Safety. Springer International Publishing Cham, 2022, pp. 335-358.
  78. T. P. Kelly and J. A. McDermid, "Safety case construction and reuse using patterns," in Safe Comp 97: The 16th International Conference on Computer Safety, Reliability and Security. Springer, 1997, pp. 55-69.
  79. C. Cârlan, B. Gallina, and L. Soima, "Safety case maintenance: a systematic literature review," in International Conference on Computer Safety, Reliability, and Security. Springer, 2021, pp. 115-129.
  80. A. Goemans, M. D. Buhl, J. Schuett, T. Korbak, J. Wang, B. Hilton, and G. Irving, "Safety case template for frontier ai: A cyber inability argument," arXiv preprint arXiv:2411.08088, 2024. · arXiv
  81. R. Bloomfield, G. Fletcher, H. Khlaaf, L. Hinde, and P. Ryan, "Safety case templates for autonomous systems," arXiv preprint arXiv:2102.02625, 2021. · arXiv
  82. M. D. Buhl, G. Sett, L. Koessler, J. Schuett, and M. Anderljung, "Safety cases for frontier ai," arXiv preprint arXiv:2410.21572, 2024. · arXiv
  83. B. Hilton, M. D. Buhl, T. Korbak, and G. Irving, "Safety cases: A scalable approach to frontier AI safety," arXiv preprint arXiv:2503.04744, 2025. · arXiv
  84. J. Clymer, N. Gabrieli, D. Krueger, and T. Larsen, "Safety cases: How to justify the safety of advanced AI systems," preprint arXiv:2403.10462, 2024. · arXiv
  85. S. Diemert, L. Millet, J. Groves, and J. Joyce, "Safety integrity levels for artificial intelligence," in International Conference on Computer Safety, Reliability, and Security. Springer, 2023, pp. 397-409.
  86. Y. Jia, T. Lawton, J. Burden, J. McDermid, and I. Habli, "Safety-driven design of machine learning for sepsis treatment," Journal of Biomedical Informatics, vol. 117, p. 103762, 2021.
  87. S. Jahan, A. Marshall, and R. Gamble, "Self-adaptation strategies to maintain security assurance cases," in 2018 IEEE 12th International Conference on Self-Adaptive and Self-Organizing Systems (SASO). IEEE, 2018, pp. 180-185.
  88. Y. Matsuno, F. Ishikawa, and S. Tokumoto, "Tackling uncertainty in safety assurance for machine learning: con- tinuous argument engineering with attributed tests," in International Conference on Computer Safety, Reliability, and Security. Springer, 2019, pp. 398-404.
  89. I. Habli, R. Hawkins, C. Paterson, P. Ryan, Y. Jia, M. Sujan, and J. McDermid, "The big argument for AI safety cases," arXiv preprint arXiv:2503.11705, 2025. · arXiv
  90. R. Salay, K. Czarnecki, H. Kuwajima, H. Yasuoka, V. Abdelzad, C. Huang, M. Kahn, V. D. Nguyen, and T. Nakae, "The missing link: Developing a safety case for perception components in automated driving," SAE International Journal of Advances and Current Practices in Mobility, vol. 5, no. 2022-01-0818, pp. 567-579, 2022.
  91. F. Wolny, S. Vock, T. Holoyad, and R. Adler, "The need for systematic approaches in risk assessment of safety-critical ai-applications in machinery," in European Safety and Reliability & Society for Risk Analysis Europe Conference ESREL SRA-E, 2025.
  92. M. Wagner and C. Carlan, "The open autonomy safety case framework," arXiv preprint arXiv:2404.05444, 2024. · arXiv
  93. N. Leveson, "The use of safety cases in certification and regulation," Journal of System Safety, 2011, aeronautics and Astronautics/Engineering Systems, MIT.
  94. M. Loba, N. F. Salem, M. Nolte, A. Dotzler, D. Ludwig, and M. Maurer, "Toward a harmonized approach- requirement-based structuring of a safety assurance argumentation for automated vehicles," arXiv preprint arXiv:2505.03709, 2025. · arXiv
  95. M. Bagheri, J. Lamp, X. Zhou, L. Feng, and H. Alemzadeh, "Towards developing safety assurance cases for learning-enabled medical cyber-physical systems," arXiv preprint arXiv:2211.15413, 2022. · arXiv
  96. M. Balesni, M. Hobbhahn, D. Lindner, A. Meinke, T. Korbak, J. Clymer, B. Shlegeris, J. Scheurer, C. Stix, R. Shah et al., "Towards evaluations-based safety cases for AI scheming," arXiv preprint arXiv:2411.03336, 2024. · arXiv
  97. E. Stensrud, T. Skramstad, J. Li, and J. Xie, "Towards goal-based software safety certification based on prescriptive standards," in 2011 First International Workshop on Software Certification. IEEE, 2011, pp. 13-18.
  98. Z. Chen, Y. Deng, and W. Du, "Trusta: Reasoning about assurance cases with formal methods and large language models," Science of Computer Programming, vol. 244, p. 103288, 2025.
  99. P. Koopman, "Ul 4600: What to include in an autonomous vehicle safety case," Computer, vol. 56, no. 05, pp. 101-104, 2023.
  100. J. McDermid, Y. Jia, and I. Habli, "Upstream and downstream AI safety: Both on the same river?" arXiv preprint arXiv:2501.05455, 2024. · arXiv
  101. K. K. Shahandashti, A. B. Belle, M. M. Mohajer, O. Odu, T. C. Lethbridge, H. Hemmati, and S. Wang, "Using gpt-4 turbo to automatically identify defeaters in assurance cases," in 2024 IEEE 32nd International Requirements Engineering Conference Workshops (REW). IEEE, 2024, pp. 46-56.
  102. T. Myklebust, T. Stålhane, G. D. Jenssen, and I. Wærø, "Autonomous cars, trust and safety case for the public," in 2020 Annual Reliability and Maintainability Symposium (RAMS). IEEE, 2020, pp. 1-6.
  103. J. Bragg and I. Habli, "What is acceptably safe for reinforcement learning?" in International Conference on Computer Safety, Reliability, and Security. Springer, 2018, pp. 418-430.
  104. T. P. Kelly et al., "Arguing safety: a systematic approach to managing safety cases," Ph.D. dissertation, University of York York, UK, 1999. [Online]. Available: https://www.jstor.org/stable/44699541
  105. R. Bloomfield and P. Bishop, "Safety and assurance cases: Past, present and possible future-an adelard perspective," in Making Systems Safer: Proceedings of the Eighteenth Safety-Critical Systems Symposium, Bristol, UK, 9-11th February 2010. Springer, 2009, pp. 51-67. [Online]. Available: https://doi.org/10.1007/978-1-84996-086-1_4 · DOI
  106. P. Königs, "The negativity crisis of AI ethics," Synthese, vol. 206, p. 277, 2025. [Online]. Available: https://doi.org/10.1007/s11229-025-05378-9 · DOI
  107. M. D. Buhl, G. Sett, L. Koessler, J. Schuett, and M. Anderljung, "Safety cases for frontier ai," 2024. [Online]. Available: https://arxiv.org/abs/2410.21572 · arXiv
  108. C. Cârlan, F. Gomez, Y. Mathew, K. Krishna, R. King, P. Gebauer, and B. R. Smith, "Dynamic safety cases for frontier ai," 2024. [Online]. Available: https://arxiv.org/abs/2412.17618 · arXiv
  109. J. Clymer, J. Weinbaum, R. Kirk, K. Mai, S. Zhang, and X. Davies, "An example safety case for safeguards against misuse," 2025. [Online]. Available: https://arxiv.org/abs/2505.18003 · arXiv
  110. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., "On the opportunities and risks of foundation models," 2022. [Online]. Available: https://arxiv.org/abs/2108.07258 · arXiv
  111. U. MoD, "Safety management requirements for defence systems part 1 requirements," Ministry of Defence, Defence Standard 00-56 Issue, vol. 4, 2007. [Online]. Available: https://segoldmine.ppi-int.com/taxonomy/term/1166
  112. I. Habli, R. Hawkins, C. Paterson, P. Ryan, Y. Jia, M. Sujan, and J. McDermid, "The big argument for AI safety cases," 2025. [Online]. Available: https://arxiv.org/abs/2503.11705 · arXiv
  113. R. Bloomfield, G. Fletcher, H. Khlaaf, L. Hinde, and P. Ryan, "Safety case templates for autonomous systems," 2021. [Online]. Available: https://arxiv.org/abs/2102.02625 · arXiv
  114. A. Goemans, M. D. Buhl, J. Schuett, T. Korbak, J. Wang, B. Hilton, and G. Irving, "Safety case template for frontier ai: A cyber inability argument," 2024. [Online]. Available: https://arxiv.org/abs/2411.08088 · arXiv
  115. T. Korbak, J. Clymer, B. Hilton, B. Shlegeris, and G. Irving, "A sketch of an AI control safety case," 2025. [Online]. Available: https://arxiv.org/abs/2501.17315 · arXiv
  116. S. Nair, J. L. De La Vara, M. Sabetzadeh, and L. Briand, "An extended systematic literature review on provision of evidence for safety certification," Information and Software Technology, vol. 56, no. 7, pp. 689-717, 2014. [Online]. Available: https://doi.org/10.1016/j.infsof.2014.03.001 · DOI
  117. J. Clymer, N. Gabrieli, D. Krueger, and T. Larsen, "Safety cases: How to justify the safety of advanced AI systems," 2024. [Online]. Available: https://arxiv.org/abs/2403.10462 · arXiv
  118. A. R. Wasil, J. Clymer, D. Krueger, E. Dardaman, S. Campos, and E. R. Murphy, "Affirmative safety: An approach to risk management for high-risk ai," 2024. [Online]. Available: https://arxiv.org/abs/2406.15371 · arXiv
  119. M. Balesni, M. Hobbhahn, D. Lindner, A. Meinke, T. Korbak, J. Clymer, B. Shlegeris, J. Scheurer, C. Stix, R. Shah, N. Goldowsky-Dill, D. Braun, B. Chughtai, O. Evans, D. Kokotajlo, and L. Bushnaq, "Towards evaluations-based safety cases for AI scheming," 2024. [Online]. Available: https://arxiv.org/abs/2411.03336 · arXiv
  120. T. Chowdhury, "Assurance case templates: Principles for their development and criteria for their evaluation," Ph.D. dissertation, McMaster University, 2021. [Online]. Available: http://hdl.handle.net/11375/26941
  121. T. P. Kelly and J. A. McDermid, "Safety case construction and reuse using patterns," in International Conference on Computer Safety, Reliability and Security, Safe Comp 1997, York, UK, September 7-10, 1997. Springer, 1997, pp. 55-69. [Online]. Available: https://doi.org/10.1007/978-1-4471-0997-6_5 · DOI
  122. E. Wozniak, C. Cârlan, E. Acar-Celik, and H. J. Putzer, "A safety case pattern for systems with machine learning components," in International Conference on Computer Safety, Reliability, and Security, ser. Lecture Notes in Computer Science, vol. 12235. Springer, 2020, pp. 370-382. [Online]. Available: https://doi.org/10.1007/978-3-030-55583-2_28 · DOI
  123. R. Kaur, R. Ivanov, M. Cleaveland, O. Sokolsky, and I. Lee, "Assurance case patterns for cyber-physical systems with deep neural networks," in International Conference on Computer Safety, Reliability, and Security, ser. Lecture Notes in Computer Science, vol. 12235. Springer, 2020, pp. 82-97. [Online]. Available: https://doi.org/10.1007/978-3-030-55583-2_6 · DOI
  124. Z. Porter, I. Habli, J. A. McDermid, and M. H. L. Kaas, "A principles-based ethics assurance argument pattern for AI and autonomous systems," AI Ethics, vol. 4, no. 2, pp. 593-616, 2024. [Online]. Available: https://doi.org/10.1007/S43681-023-00297-2 · DOI
  125. L. Dong, Q. Lu, and L. Zhu, "Agentops: Enabling observability of llm agents," 2024. [Online]. Available: https://arxiv.org/abs/2411.05285v2 · arXiv
  126. J. Chen, S. Ma, Q. Lu, S. U. Lee, and L. Zhu, "Maria: A framework for marginal risk assessment without ground truth in AI systems," 2025. [Online]. Available: https://arxiv.org/abs/2510.27163 · arXiv
  127. Q. Hua, L. Ye, D. Fu, Y. Xiao, X. Cai, Y. Wu, J. Lin, J. Wang, and P. Liu, "Context engineering 2.0: The context of context engineering," 2025. [Online]. Available: https://arxiv.org/abs/2510.26493 · arXiv
  128. B. Hilton, M. D. Buhl, T. Korbak, and G. Irving, "Safety cases: A scalable approach to frontier AI safety," 2025. [Online]. Available: https://arxiv.org/abs/2503.04744 · arXiv
  129. E. Asaadi, E. Denney, J. Menzies, G. J. Pai, and D. Petroff, "Dynamic assurance cases: a pathway to trusted autonomy," Computer, vol. 53, no. 12, pp. 35-46, 2020. [Online]. Available: https://doi.org/10.1109/MC.2020.3022030 · DOI
  130. S. Ramakrishna, "Dynamic safety assurance of autonomous cyber-physical systems," Ph.D. dissertation, Vanderbilt University, 2022.
  131. L. Buysse, I. Habli, D. Vanoost, and D. Pissoort, "Safe autonomous systems in a changing world: Operationalising dynamic safety cases," Safety Science, vol. 191, p. 106965, 2025. [Online]. Available: https://doi.org/10.1016/j.ssci.2025.106965 · DOI
  132. M. Shamsujjoha, J. Grundy, L. Li, H. Khalajzadeh, and Q. Lu, "Developing mobile applications via model driven development: A systematic literature review," Information and Software Technology, vol. 140, p. 106693, 2021. [Online]. Available: https://doi.org/10.1016/j.infsof.2021.106693 · DOI
  133. M. Shamsujjoha, Q. Lu, D. Zhao, and L. Zhu, "Swiss cheese model for AI safety: A taxonomy and reference architecture for multi-layered guardrails of foundation model based agents," in 2025 IEEE 22nd International Conference on Software Architecture (ICSA). IEEE, 2025, pp. 37-48. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICSA65012.2025.00014 · DOI
  134. M. Petticrew and H. Roberts, Systematic reviews in the social sciences: A practical guide. John Wiley & Sons, 2006. [Online]. Available: https://doi.org/10.1002/9780470754887 · DOI · Paper page
  135. I. D. Raji and R. Dobbe, "Concrete problems in AI safety, revisited," 2023. [Online]. Available: https://arxiv.org/abs/2401.10899 · arXiv
  136. M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong et al., "Toward trustworthy AI development: Mechanisms for supporting verifiable claims," 2020. [Online]. Available: https://arxiv.org/abs/2004.07213 · arXiv
  137. J. Babcock, J. Kramár, and R. Yampolskiy, "The agi containment problem," in International Conference on Artificial General Intelligence. Springer, 2016, pp. 53-63. [Online]. Available: https://doi.org/10.1007/978-3-319-41649-6_6 · DOI
  138. R. Hawkins, T. Kelly, J. Knight, and P. Graydon, "A new approach to creating clear safety arguments," in Advances in Systems Safety: Proceedings of the Nineteenth Safety-Critical Systems Symposium, Southampton, UK, 8-10th February 2011. Springer, 2010, pp. 3-23. [Online]. Available: https://doi.org/10.1007/978-0-85729-133-2_1 · DOI
  139. N. G. Leveson, Engineering a safer world: Systems thinking applied to safety. The MIT Press, 2016. [Online]. Available: https://doi.org/10.7551/mitpress/8179.001.0001 · DOI · Paper page
  140. A. B. Arrieta, N. Díaz-Rodríguez, J. D. Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, R. Chatila, and F. Herrera, "Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai," Information Fusion, vol. 58, pp. 82-115, 2020. [Online]. Available: https://doi.org/10.1016/j.inffus.2019.12.012 · DOI · Paper page
  141. R. Salay, M. Angus, and K. Czarnecki, "A safety analysis method for perceptual components in automated driving," in 2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2019, pp. 24-34. [Online]. Available: https://doi.org/10.1109/ISSRE.2019.00013 · DOI
  142. J. Rushby, "On the interpretation of assurance case arguments," in JSAI International Symposium on Artificial Intelligence. Springer, 2015, pp. 331-347. [Online]. Available: https://doi.org/10.1007/978-3-319-50953-2_23 · DOI
  143. B. Herd, J.-V. Zacchi, and S. Burton, "A deductive approach to safety assurance: Formalising safety contracts with subjective logic," in International Conference on Computer Safety, Reliability, and Security. Springer, 2024, pp. 213-226. [Online]. Available: https://doi.org/10.1007/978-3-031-68738-9_16 · DOI
  144. N. Basir, E. Denney, and B. Fischer, "Deriving safety cases from automatically constructed proofs," in 4th IET International Conference on System Safety 2009. Incorporating the SaRS Annual Conference. IET, 2009, pp. 1-6. [Online]. Available: https://doi.org/10.1049/cp.2009.1535 · DOI
  145. A. E. Oztekin and J. T. Luxhøj, "An inductive reasoning approach for building system safety risk models of aviation accidents," Journal of Risk Research, vol. 13, no. 4, pp. 479-499, 2010. [Online]. Available: https://doi.org/10.1080/13669870903484344 · DOI
  146. D. B. Leake, Case-Based Reasoning: Experiences, lessons and future directions. MIT press, 1996. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/524680 · DOI
  147. S. Burton, "A causal model of safety assurance for machine learning," 2022. [Online]. Available: https://arxiv.org/abs/2201.05451 · arXiv
  148. S. Burton and B. Herd, "Addressing uncertainty in the safety assurance of machine-learning," Frontiers in Computer Science, vol. 5, p. 1132580, 2023. [Online]. Available: https://doi.org/10.3389/fcomp.2023.1132580 · DOI
  149. E. Denney, G. Pai, and I. Habli, "Towards measurement of confidence in safety cases," in 2011 International Symposium on Empirical Software Engineering and Measurement. IEEE, 2011, pp. 380-383. [Online]. Available: https://doi.org/10.1109/ESEM.2011.53 · DOI
  150. E. A. Act, "Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence (artificial intelligence act)," Official Journal of the European Union (OJ L 1689, 12.7.2024), 2024. [Online]. Available: https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng
  151. N. Leveson, "The use of safety cases in certification and regulation," Journal of System Safety, 2011, aeronautics and Astronautics/Engineering Systems, MIT.
  152. N. Hayama, Y. Yamagata, H. Nishihara, and Y. Matsuno, "A GSN-based requirement analysis of the EU AI regulation," in International Conference on Computer Safety, Reliability, and Security. Springer, 2025, pp. 183-196. [Online]. Available: https://doi.org/10.1007/978-3-032-02018-5_14 · DOI
  153. L. Zhu and Q. Lu, "Verifiability-first AI engineering in the era of AIware: Aconceptual framework, design principles, and architectural patterns for scalable verification," SSRN, 2025. [Online]. Available: http://dx.doi.org/10.2139/ssrn.6031534 · DOI

Cited by

Sorted by citations

Authors
Lee, S. U., Zhu, L., Shamsujjoha, M., Dong, L., Lu, Q., Chen, J., & Briand, L.
Affiliations
  • Commonwealth Scientific and Industrial Research Organisation (AU)
  • University of Ottawa (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2601.22773
arXiv
2601.22773
OpenAlex
W7127093612
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Engineering › Safety, Risk, Reliability and Quality › Safety Systems Engineering in Autonomy
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
safety case, AI safety, generative AI, agentic AI, argument types, types of evidence, prompt sensitivity, model updating, AI governance

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review