CoChat

SafeCodeRL: Security-Constrained Multi-Agent Reinforcement Learning for Trustworthy LLM-Generated IoT/CPS Software

Open access

The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 15 of the 32 sources its claims cite were checked for support.

Abstract

SafeCodeRL integrates multi-agent collaboration with constrained reinforcement learning for trustworthy LLM-generated IoT/CPS software.

Overview

CoChat Summary

## TL;DR

SafeCodeRL is a multi-agent reinforcement learning framework for LLM-generated IoT/CPS software that combines planning, code generation, security auditing, sandbox testing, and critic-driven routing with explicit security constraints. Across code and security benchmarks plus a 25-task IoT/CPS suite, it improves functional performance while reducing high-risk vulnerabilities.

## Key findings

  • On generic benchmarks with GPT-4o as the backbone, multi-agent baselines reached 88.6–90.2% HumanEval Pass@1 and 11.6–13.1% SWE-bench resolution, but still had high vulnerability rates of 17.6–21.9% on SecurityEval and 16.4–19.8% on CyberSecEval.
  • Compared with the strongest generic multi-agent baseline (AgentCoder), SafeCodeRL with GPT-4o improved HumanEval Pass@1 from 90.2% to 91.4%, SWE-bench resolution from 13.1% to 14.8%, and reduced SecurityEval VR from 17.6% to 5.5% and CyberSecEval VR from 16.4% to 6.2%.
  • In the IoT/CPS case study, SafeCodeRL with GPT-4o achieved the highest Secure Pass@1 of 89.3%; the DeepSeek-V3 configuration reached 84.7% Secure Pass@1 and 4.9% vulnerability rate.
  • Efficiency results report that SafeCodeRL used 10,500 tokens versus 35,200 tokens for DeepSeek-R1, and four multi-agent iterations took 21.8 s versus 45.2 s for a single DeepSeek-R1 pass.

## Methods

  • The paper formulates secure code generation as a Constrained Markov Decision Process (CMDP) with a Planner, Code, Security, Test, and Critic agent loop; the agents exchange structured feedback for task decomposition, code synthesis, vulnerability auditing, sandbox validation, and routing.
  • The evaluation uses HumanEval (164 tasks), SWE-bench (2294 tasks), SecurityEval, CyberSecEval, and a custom 25-task IoT/CPS suite spanning telemetry APIs, gateway diagnostics, firmware/configuration handlers, secure aggregation, and sensor parsing.
  • Training uses SFT warm-start on filtered HumanEval and APPS, then PPO-style reinforcement learning with dense functional reward, security penalty, Lagrangian relaxation, and hard shielding; experiments were run on eight NVIDIA A100 GPUs and repeated five times with independent seeds.

## Limitations

  • The paper states that the IoT/CPS study is scenario-level and software-only: it uses mocked sensor streams, virtual device paths, and sandboxed gateway commands, not physical sensors, embedded firmware, MQTT/CoAP deployments, hardware-in-the-loop testbeds, or live industrial control loops.
  • It notes that the framework relies on external analyzers such as CodeQL and Docker sandboxes, which add rollout overhead; this is an explicit limitation stated in the Discussion.
  • The paper says hard Shielding may reject uncommon but legitimate implementation patterns in resource-constrained embedded environments; this is also explicitly stated.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s26113502

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Motivation for security-constrained LLM-generated IoT/CPS software.(The asterisk in the embedded SQL snippet “SELECT * FROM …” is the SQL wildcard for all columns, not a footnote symbol).
Figure 1 Motivation for security-constrained LLM-generated IoT/CPS software.(The asterisk in the embedded SQL snippet “SELECT * FROM …” is the SQL wildcard for all columns, not a footnote symbol).CC BY · PubMed Central
Figure 2 IoT/CPS-oriented overview of the SafeCodeRL framework, including task-context modeling, the five-agent closed loop, constraint-aware optimization, shielding, sandbox execution, and the training pipeline. Solid arrows denote the main generation–feedback flow, dashed arrows denote possible/optional transitions, and the star marks the target (final-goal) node in the Router Distillation module.
Figure 2 IoT/CPS-oriented overview of the SafeCodeRL framework, including task-context modeling, the five-agent closed loop, constraint-aware optimization, shielding, sandbox execution, and the training pipeline. Solid arrows denote the main generation–feedback flow, dashed arrows denote possible/optional transitions, and the star marks the target (final-goal) node in the Router Distillation module.CC BY · PubMed Central
Figure 3 Performance and security trends of SafeCodeRL across iterations. (a) Pass@1 trends across refinement iterations. (b) Vulnerability-rate trends across refinement iterations.
Figure 3 Performance and security trends of SafeCodeRL across iterations. (a) Pass@1 trends across refinement iterations. (b) Vulnerability-rate trends across refinement iterations.CC BY · PubMed Central
Figure 4 IoT/CPS case-study comparison against single-agent, self-refinement, and multi-agent code-generation baselines. (a) Security-correctness Pareto view. (b) Functional Pass@1 and Secure Pass@1 with consistent bar encodings; the red label marks the best overall configuration. (c) Residual vulnerability rate, where lower is better.
Figure 4 IoT/CPS case-study comparison against single-agent, self-refinement, and multi-agent code-generation baselines. (a) Security-correctness Pareto view. (b) Functional Pass@1 and Secure Pass@1 with consistent bar encodings; the red label marks the best overall configuration. (c) Residual vulnerability rate, where lower is better.CC BY · PubMed Central
Figure 5 (a) Cost-effectiveness and (b) latency analysis of SafeCodeRL compared to state-of-the-art baselines.
Figure 5 (a) Cost-effectiveness and (b) latency analysis of SafeCodeRL compared to state-of-the-art baselines.CC BY · PubMed Central
Figure 6 Effectiveness of dynamic constraint adjustment and action routing in SafeCodeRL. (a) Five-seed average convergence of the normalized security reward (Rsec) with and without the dynamic Lagrangian multiplier; negative values indicate that vulnerability costs exceed the clean-code reward. (b) Action distribution executed by the Critic Routing module, demonstrating a balanced allocation between code generation, testing, and security verification.
Figure 6 Effectiveness of dynamic constraint adjustment and action routing in SafeCodeRL. (a) Five-seed average convergence of the normalized security reward (Rsec) with and without the dynamic Lagrangian multiplier; negative values indicate that vulnerability costs exceed the clean-code reward. (b) Action distribution executed by the Critic Routing module, demonstrating a balanced allocation between code generation, testing, and security verification.CC BY · PubMed Central
Figure 7 Suppression effect of different critic baselines on VR.
Figure 7 Suppression effect of different critic baselines on VR.CC BY · PubMed Central
Figure 8 Distribution of Shield-triggered catastrophic action categories (N=1420 total interceptions). Each slice is annotated with both the absolute count and the relative percentage. This figure visualizes the operational scope of the hard Shield rather than broader CWE coverage. CWE-78: OS command injection (320, 22.5%), CWE-89: SQL injection (660, 46.5%), CWE-22: Path traversal (440, 31.0%).
Figure 8 Distribution of Shield-triggered catastrophic action categories (N=1420 total interceptions). Each slice is annotated with both the absolute count and the relative percentage. This figure visualizes the operational scope of the hard Shield rather than broader CWE coverage. CWE-78: OS command injection (320, 22.5%), CWE-89: SQL injection (660, 46.5%), CWE-22: Path traversal (440, 31.0%).CC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 2 Not valid · 8 Not checked · 114 Own results and interpretations

68 more claims · 1 with their cited sources checked

Create a free account to see all 132 claims and to verify the 9 references still open.

See all claims — free account
  • It may state two separate claims as one. 47
  • Probably not a substantial claim. 21

Create a free account to see these claims.

Citations

Cites

Research works printed but not linked: 13

  1. Sandoval, G., Pearce, H., Nys, T., Karri, R., Garg, S., and Dolan-Gavitt, B. (2023, January 9–11). Lost at C: A user study on the security implications of large language model code assistants. Proceedings of the 32nd USENIX Security Symposium, Anaheim, CA, USA.
  2. Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017, January 6–11). Constrained policy optimization. Proceedings of the International Conference on Machine Learning (ICML), Sydney, Australia.
  3. He, J., Vero, M., Krasnopolska, G., and Vechev, M. (2024, January 21–27). Instruction Tuning for Secure Code Generation (SafeCoder). Proceedings of the 41st International Conference on Machine Learning (ICML), Vienna, Austria.
  4. Le, 2022, CodeRL: Mastering code generation through pretrained models and deep reinforcement learning, Adv. Neural Inf. Process. Syst., 35, 21314
  5. Newsome, J., and Song, D. (2005, January 3–4). Dynamic Taint Analysis for Automatic Detection, Analysis, and Signature Generation of Exploits on Commodity Software. Proceedings of the 12th Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, USA.
  6. Merkel, 2014, Docker: Lightweight Linux Containers for Consistent Development and Deployment, Linux J., 239, 2
  7. Stooke, A., Achiam, J., and Abbeel, P. (2020, January 13–18). Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. Proceedings of the 37th International Conference on Machine Learning (ICML), Virtual.
  8. Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., and Song, D. (2021, January 6–14). Measuring Coding Challenge Competence with APPS. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, Virtual.
  9. Jimenez, C.E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2024, January 7–11). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. Proceedings of the The Twelfth International Conference on Learning Representations (ICLR), Vienna, Austria.
  10. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., and Ruan, C. (2024). DeepSeek-V3 Technical Report. arXiv.
  11. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., and Bi, X. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.
  12. Snell, C., Lee, J., Xu, K., and Kumar, A. (2024, January 11). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. Proceedings of the The Twelfth International Conference on Learning Representations (ICLR) Workshop, Vienna, Austria.
  13. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. (2024, January 7–11). Let’s Verify Step by Step. Proceedings of the The Twelfth International Conference on Learning Representations (ICLR), Vienna, Austria.

Cited by

See all 2

Metadata

Authors
Wang, Z., Chen, J., Wei, Z., Lin, L., & Tong, G.
Affiliations
  • University of Shanghai for Science and Technology (CN)
  • Beijing University of Technology (CN)
Corresponding author
Tong, Guoxiang
Venue
Sensors
Publisher
MDPI AG
Published
2026-06-02
Volume
26
Issue
11
Pages
3502
ISSN
1424-8220 (ISSN-L)
OpenAlex
W7163126220
Cited by
2
References
18
Citations by year
2026: 2
FWCI
16.63
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
LLM-generated code, code generation security, multi-agent reinforcement learning, cyber-physical systems, vulnerability detection, SQL injection, command injection, proximal policy optimization, AI-assisted software development

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.