End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks

“Furthermore, if we use TRPO for the RL algorithm, then the control law Proof.”

“The existence of a CBF implies that there exists a deterministic controller u CBF : S → A such that the set C is forward invariant for system (1) (Agrawal and Sreenath 2017; Ames et al. 2017).”

“Previous approaches to safe reinforcement learning include reward-shaping , policy optimization with constraints (Gaskett 2003; Moldovan and Abbeel 2012; Achiam et al. 2017; Wachi et al. 2018), or teacher advice (Abbeel and Ng 2004; Abbeel, Coates, and Ng 2010; Tang et al. 2010).”

“This method has been successfully applied to continuous control tasks (Duan et al. 2016; Lillicrap et al. 2015) where controllers have learned to stabilize complex robots (after many policy iterations).”

“[4], navigation [5], [6], and autonomy [7], [8] have been extensively studied; safe reinforcement learning [9], [10] in”

“…and safe learning-based control that studies how to learn optimal policies without violating constraints and without knowing the system (Fisac et al. 2018; Aswani et al. 2013; Wabersich and Zeilinger 2018; Garcıa and Fernández 2015; Cheng et al. 2019; Zanon and Gros 2019; Fulton and Platzer 2018).”

“tems received considerable attention, primarily motivated by safety-critical control applications, such as in autonomous cyber-physical systems and robotics [1,7,11,20,2,34].”

“within a specified set, CBFs have proven to be effective both in correcting some potentially unsafe control action [3], [4] and as a constraint in optimization-based control [5], [6].”

“[4], [5], [6], [7], [8], [9].”

“The authors in [7] also use Gaussian process to estimate the model but focusing on the safety during the training process.”

“For example in [5], [6], [9], the authors assume a Gaussian”

“That is different from the traditional view in areas such as safe reinforcement learning, where one typically wants to ensure that the agent does not visit unsafe regions during training [34].”

“…class of approaches incorporates Lagrangian constraint regularization into the policy updates in policy-gradient algorithms (Achiam et al. 2017; Ray, Achiam, and Amodei 2019; Tessler, Mankowitz, and Mannor 2018; Dalal et al. 2018; Cheng et al. 2019; Zhang, Vuong, and Ross 2020; Chow et al. 2019).”

“In [26], CBFs are instead used to guide the learning process.”

“Examples of algorithms that adopt this approach encompass differentiable optimization layers [2], [4], [5], [6], [7], [11], Neural Frank-Wolfe Policy Optimization (NFWPO) [8], and α-projection [12].”

“approaches are based on either Lyapunov functions [7], [24], [25], Hamilton-Jacobi reachability [26], Control Barrier Functions [27], [28], constraint linearization [29], [30] or planning [31], [32].”

“For example, the authors of [14] and [15] leverage GPs to obtain probabilistic safety guarantees, but use the standard deviation of the GP in a worst-case fashion.”

“Inspired by [25], we propose the framework SRL-TR 2 , which utilizes an MPC controller to restrain the RL controller.”

“Nevertheless, it is difficult to find the safe projection, for example, projecting the output to a safe set on the action space [26], [53], [54], or adding a safe layer to the policy network [33].”

“This makes them often not suitable for safety-critical applications [35], [36], [37].”

“Recently, it was demonstrated that control barrier functions can be used in reinforcement learning to ensure safety in continuous control tasks [25].”

“On the other hand, unlike safe RL approaches based on the use of control barrier functions (Cheng et al., 2019; Sloth et al., 2012), our RAG is formulated based on discrete-time models from the start and thereby can be more directly applied in a digital setting (e.”

“Methods such as [3], [5], [4] require good nominal dynamics models and can only tolerate disturbances to some extent.”

“…learning-based research can be divided into three main categories: (1) learning safe control policies with known feasible energy function [17], [18], [22]; (2) learning to synthesize energy functions with known dynamic models or controller [21], [23], [28], [29], [30]; and (3) joint synthesis…”

“5 Department of Computer Science, University of Manchester, United Kingdom. controller [8], [10], [13], a model predictive control (MPC) controller [14], etc.”

“For safe exploration in the learning process under uncertain dynamics, the work in [20] employed a Gaussian Process (GP) to approximate the unknown part of the dynamics from the dataset by learning a mean estimate μd(s) and an uncertainty σ(2)d(s) in tandem with the policy update with probability confidence intervals…”

“…for complete knowledge of the system dynamics in the CBF approach and thus to partially bridge the gap between safety and learning-based control design, Gaussian process (GP) has been used in [7], [8] in which a nominal model is assumed to be known and the disturbances are learned by GP.”

“With these control-based approaches to compute a a safe set U at a specific state, some RL studies projects the policy output into U achieve safety of current state [7, 9].”

“The performance constraint of the LACS problem motivates us to adopt safe RL, which aims to learn a policy that maximizes the expected return while ensuring (or encouraging) the satisfaction of some safety constraints [25].”

“Similar to [7], [8], we use a Gaussian process (GP) to estimate the unknown dynamics d from data.”

“Recently, the control barrier functions (CBFs) have raised great attention that can be suitably employed for the state-variables constraints formulation for optimization problems with known nominal model [13], [14], [15].”

“Typical Safe DRL approaches [13], [14] address the safety of the model introducing constraints in the training phase or well-defined reward functions, but they are not designed to formally guarantee the correct behavior of the network.”

“Recent approaches on this problem [8, 25, 27, 38, 41] are either computationally expensive or require explicit, strong assumptions about the model of agentenvironment interactions.”

“…advancements, existing interactive solutions are not designed to satisfy explicit safety constraints, and either couple the often opposing objectives of safety and performance (e.g. in collision-avoiding RL [16]), or need to rely on post-processing to ensure safety (e.g. via safety filters [17]).”

“Although DECBFs have been successfully implemented in discrete-time applications [12], [13], they suffer from similar drawbacks as continuous-time CBFs: finding a DECBF for arbitrary constraint sets is challenging, and resorting to candidate DECBFs can lead to infeasibility issues in the DECBF-based…”

“Current efforts use it to design a modified controller for stability [26] or to enhance the efficiency of the learning process with Gaussian assumption [27].”

“A first approach to mention is Safe RL, where a CBF is jointly learned during training to guide the policy to implicitly encode the underlying safety mechanism [22].”

“The practice of collecting data from a dynamical system for a learning model while keeping the system safe is called safe learning, and has been investigated both from the perspective of both robust control [1], [2] and reinforcement learning [3]–[6].”

“Control Barrier Functions (CBF) [1] and the associated idea of controller shielding [2], [3], [4], [5], [6] have become important tools in the design of safety-critical control systems, especially those that incorporate learning-enabled components such as Neural Networks (NNs).”

“Robust control methods are able to model constraints on the short-term horizon and ensure their long-term satisfaction through constraints on terminal costs [26]–[28].”

“Another line of work utilizes ideas from control theory and model-based approach [Cheng et al., 2019, Berkenkamp et al., 2017, Taylor et al., 2019, Zeng et al., 2020].”

“There has been rising interest in integrating CBF with RLtoguaranteesystemsafety[23],[24].”

“For the first approach, usually, a penalty is given in case a correction is necessary [16], [17], [25], [26].”

“In other words, only the best action is selected when measuring this quantum state, as shown in Equation (28).”

“CBF-based approaches have also been used to certify the safety of reinforcement learning (RL) algorithms [8], [9], [10], [11], [12], [13].”

“Few works have been dedicated to the safe multiagent policy learning [31], [32], drawing inspiration from cost-shaping-based RL [27], [28].”

“Moreover, the effects of action correction from the optimization module are nontrivial in RL as it may break the Markov assumption [12, 13].”

“)is is just a soft constraint and cannot guarantee complete safety since RL is likely to take unsafe behaviors without a hard constraint [42].”

“The infeasible region of π is defined as The above-defined feasible regions are related to the concept of “safe set” in CBF literature [13], [14], [19], which is used to refer to the forward invariant set of CBF.”

“The absence of a model of the system also tends to limit the sample efficiency of these methods [3], [18].”

“A safe RL-CBF framework has been developed to guarantee safety and improve exploration in [28].”

“Based on this mathematical formulation, some common approaches to ensuring safety for RL-based con-trol include - using control barrier functions (CBF) [7,8], model predictive control (MPC) [9,10,11], temporal logic (TL) specifications [12,13], and using Gaussian processes [14,15], to name a few.”

“[12] proposed that other types of utility, including risk, should be within certain given boundaries, taking such constraint-system states as fence functions and strategy space constraints or boundaries.”

“Cheng et al. [40] proposed an end-to-end safe RL algorithm based on the teacher-advice framework, where a control barrier function (CBF) was used to build the teacher model.”

“Control barrier functions (CBFs) have been used to enforce safety limits within a safe set, rather than a static limit [31,32] .”

“We refer to [8,14,17,16,18] for some interesting approaches.”

“An RL algorithm combined with CBF guarantees safety while learning the set of explorable policies as shown in [16].”

“Several works are based on barrier certificates and adversarial examples [14,38] or Lyapunov functions [25].”

“However, the major issue of these approaches is that safety is not guaranteed during initial learning interactions [12].”

“In [17], the author uses CBF to guide the reinforcement learning (RL) algorithm to explore a safe policy set.”

“As discussed above, other approaches to assuring safety in reinforcement learning include shielding [1,4,51,30,24] and constrained or safe RL [16,20,12,44,21,36,25,22].”

“…optimization problem is typically solved under which the constraints act as a shield to certify the safety of the RL actions before being deployed [11]: where the constraint is a barrier certification constraint that guarantees invariance of the set described by C s = { x : h ( x ) ≥ 0 } .”

“Safety : [23] introduces a framework that combines model-free reinforcement learning with model-based CBF controllers and learned system dynamics to ensure safety during exploration.”

“The authors in [11]–[13] exploited control barrier function (CBF) where neural networks (NN) and learning are used.”

“Recent works have demonstrated the success of CBF- and/or CLFbased method along with learning algorithms [10], [11], [16], [17].”

“Compared with the Lyapunov-based methods (Berkenkamp et al. 2017; Chow et al. 2018), barrier function-based approaches (Cheng et al. 2019) relax the condition to guarantee the forward invariant property in the safe set and significantly increase the feasibility.”

“A CBF-based guiding control with reinforcement learning, including DDPG and TRPO, was proposed to achieve safe and efficient learning [18].”

“This approach has garnered widespread adoption in recent years [17], [18], [19], [20].”

“Many such algorithms have been proposed in recent years, for example by using robust MPC [15], control barrier functions (CBFs) [16] and Hamilton-Jacobi reachability analysis [17], [18].”

“This approach of using the backup safe policy to override the tasking-solving policy is known as the least restrictive control law or shielding in Alshiekh et al. [6], Fisac et al. [46] and also examined in Cheng et al. [28], Leung et al. [72].”

“CBFs have been used to develop a new kind of CBF-based SRL method to achieve vehicle speed control [33], trajectory tracking control [34] and platoon control [35], and have shown superiority with lower collision risks and faster training speed than traditional DRL-based methods [36].”

“Therefore, these algorithms are usually used in specifically structured state-based problems or single-agent scenarios [18] [19].”

“CBFs for safe reinforcement learning were introduced in [35].”

“…have been developed for safe RL, such as constrained RL (Yu et al., 2019), control barrier functions based compensating control with model-free RL (Cheng et al., 2019), using reachability analysis in (Akametalu et al., 2014) or Lyapunov stability verification in (Berkenkamp et al., 2017) to…”