NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Open accessAbstract
NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails (or rails for short) are a specific way of controlling the output of an LLM, such as not talking about topics considered harmful, following a predefined dialogue path, using a particular language style, and more. There are several mechanisms that allow LLM providers and developers to add guardrails that are embedded into a specific model at training, e.g. using model alignment. Differently, using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications - these are user-defined, independent of the underlying LLM, and interpretable. Our initial results show that the proposed approach can be used with several LLM providers to develop controllable and safe LLM applications using programmable rails.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing safety mechanisms such as sandboxing, input/output filtering, and static rule-based guardrails provide a necessary but insufficient first line of defense.
“Existing safety mechanisms-such as sandboxing, input/output filtering, and static rule-based guardrails (Inan et al., 2023; Rebedea et al., 2023) -provide a necessary but insufficient first line of defense.”
From VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation (2025) · cites this paper as Rebedea et al., 2023
Section 7.1 says programmable rails are not a perfect or stand-alone safety solution and should complement embedded rails; Appendix G.2.1 finds combined input and output moderation more robust than either alone. This supports the claim that guardrails and filtering help but are insufficient by themselves. The paper does not address sandboxing or characterize the listed mechanisms collectively as a necessary first line of defense (section 7.1 and Appendix G.2.1; page numbers unavailable). (LLM, conf=0.84)
Experimental — accuracy not yet measuredA number of LLMOps tools exist for users to implement code-based guardrails.
“In fact, a number of LLMOps tools exist for users to implement such code-based guardrails [19, 22, 42].”
From Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (2024) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 29
- Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when its lying. arXiv preprint arXiv:2304.13734. · arXiv
- Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. · arXiv
- Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073. · arXiv
- Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated ma- chine reading comprehension dataset. arXiv preprint arXiv:1611.09268. · arXiv
- Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. 2017. Rasa: Open source language understanding and dialogue management. arXiv preprint arXiv:1712.05181. · arXiv
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
- Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375. · arXiv
- Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. · arXiv
- Vojtěch Hudeček and Ondřej Dušek. 2023. Are llms all you need for task-oriented dialogue? arXiv preprint arXiv:2304.06556. · arXiv
- Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-547.
- Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2023. Ex- ploiting programmatic behavior of llms: Dual-use through standard security attacks. arXiv preprint arXiv:2302.05733. · arXiv
- Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2021. Benchmarking natural language understanding services for building conversational agents. In Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Systems, pages 165- 183. Springer.
- Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hal- lucination detection for generative large language models. arXiv preprint arXiv:2303.08896. · arXiv
- Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
- Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, and Jason Weston. 2023. Leveraging implicit feedback from deployment data in dialogue. arXiv preprint arXiv:2307.14117. · arXiv
- Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116. · arXiv
- Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813. · arXiv
- Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419-3448, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Lee Boyd-Graber, and Lijuan Wang. 2022. Prompting gpt-3 to be reliable. In The Eleventh International Conference on Learn- ing Representations. Spotify. ANNOY library. https://github.com/ spotify/annoy. Accessed: 2023-08-01.
- Makesh Narsimhan Sreedhar and Christopher Parisien. 2022. Prompt learning for domain adaptation in task- oriented dialogue. In Proceedings of the Towards Semi-Supervised and Reinforced Task-Oriented Di- alog Systems (SereTOD), pages 24-30, Abu Dhabi, Beijing (Hybrid). Association for Computational Lin- guistics.
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Boxin Wang, Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong, Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. 2023. Shall we pretrain autoregressive language models with retrieval? a comprehensive study. arXiv preprint arXiv:2304.06762. · arXiv
- Yau-Shian Wang and Yingshan Chang. 2022. Toxicity detection with generative prompt-based inference. arXiv preprint arXiv:2205.12390. · arXiv
- Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? arXiv preprint arXiv:2307.02483. · arXiv
- Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
- Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023. Sgp-tod: Building task bots effortlessly via schema-guided llm prompting. arXiv preprint arXiv:2305.09067. · arXiv
- Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, et al. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.10625. · arXiv
- Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrik- son. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. · arXiv
Cited by
Sorted by citations
Hallucination is Inevitable: An Innate Limitation of Large Language Models
Xu, Z., Jain, S., & Kankanhalli, M. (2024) · arXiv · doi:10.48550/arxiv.2401.11817 · Cited by 180 · see citing works
“Guardrails and fences can be formally programmed [53] to explicitly affect LLMs’ behaviours.”
BackgroundWho Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A., & Arawjo, I. (2024) · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3654777.3676450 · Cited by 151 · see citing works
“Here we leverage a similar algorithmic approach to SPADE, but embed it inside an LLM-assisted user interface for evaluator prototyping, EvalGen, that also assists with criteria generation, measuring alignment with human preferences, and visualizing results.”
Large language models for preventing medication direction errors in online pharmacies
Pais, C., Liu, J., Voigt, R., Gupta, V., Wade, E., & Bayati, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-02933-8 · Cited by 94 · see citing works
Fostering effective hybrid human-LLM reasoning and decision making
Passerini, A., Gema, A. P., Minervini, P., Sayin, B., & Tentori, K. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1464690 · Cited by 35 · see citing works
ContextQ: Generated Questions to Support Meaningful Parent-Child Dialogue While Co-Reading
Dietz Smith, G., Prasad, S., Davidson, M. J., Findlater, L., & Shapiro, R. B. (2024) · Proceedings of the 23rd Annual ACM Interaction Design and Children Conference · doi:10.1145/3628516.3655809 · Cited by 29 · see citing works
Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models
Hu, Z., Wang, C., Shu, Y., Paik, H.-Y., & Zhu, L. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671932 · Cited by 27 · see citing works
Malla: Demystifying Real-world Large Language Model Integrated Malicious Services
Lin, Z., Cui, J., Liao, X., & Wang, X. (2024) · arXiv · doi:10.48550/arxiv.2401.03315 · Cited by 13 · see citing works
“…of guardrails, which are designed to monitor and control LLM inputs and outputs and deployed by LLM providers (e.g., the OpenAI Moderation Endpoint [106,111] and Llama Guard [100]) or third parties (e.g., NeMo Guardrails [120], the OpenChatKit Moderation Model [71], and Guardrails AI [37]).”
MethodGuardrails for Large Language Models: A Review of Techniques and Challenges
Akheel, S. A. (2025) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/syed-arham-akheel/536 · Cited by 10 · see citing works
Evaluation of NeMo Guardrails as a Firewall for User–LLM Interaction
Azambuja, A. J., Guilherme, M., Castro, J. V. F. D., Lima, J. P. D. O., Oliveira, L. B., & Soares, A. D. S. (2026) · Future Internet · doi:10.3390/fi18050252 · Cited by 2 · see citing works
How I learned to stop worrying and love learned OS policies
Saxena, D., Chen, J. K., Yadalam, S., Ro, Y., Dwivedula, R., Campbell, E. H., Akella, A., Rossbach, C. J., & Swift, M. M. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730384 · Cited by 2 · see citing works
From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Ahn, J., & Kim, M. (2026) · arXiv · doi:10.48550/arxiv.2607.08028 · Cited by 1 · see citing works
“We take up harness engineering —an emerging practice of wrapping LLM agents in a code-owned control layer [8–10]—and develop it into a measurable, contract-based method that relocates product behavior from prompts into explicit, versioned contracts [11, 12].”
Self-supervised Meta-Heuristic Mapping - a Framework for Adaptive Human-AI GovernancePreprint
Cheraitia, M. (2024) · doi:10.31219/osf.io/5tbhw · Cited by 1 · see citing works
VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
Miculicich, L., Parmar, M., Palangi, H., Dvijotham, K. D., Montanari, M., Pfister, T., & Le, L. T. (2025) · arXiv · doi:10.48550/arxiv.2510.05156 · Cited by 2 · see citing works
Vir, R., Shankar, S., Chase, H., Hinthorn, W., & Parameswaran, A. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.213 · Cited by 1 · see citing works
Runtime Risk Detection and Control for AI Agents and LLM ApplicationsPreprint
Gour, D. K., & Agrawal, S. (2026) · Research Square · doi:10.21203/rs.3.rs-10877573/v1
- Authors
- Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., & Cohen, J.
- ORCID
- Rebedea, Traian
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.10501
- OpenAlex
- W4387725416
- Cited by
- 8
- References
- 0
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- guardrails, large language models, dialogue systems, runtime control, dialogue management, output control, harmful content filtering, LLM safety, open-source software toolkit
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
