Yesmina Jaafra
email: yesmina.jaafra@expleogroup.com

Christophe Bohn
email: christophe.bohn@irt-systemx.fr

Lucas Schott
email: lucas.schott@irt-systemx.fr

Faouzi Adjed
email: faouzi.adjed@irt-systemx.fr

Frédéric Pelliccia
email: frederic.pelliccia@irt-systemx.fr

Mehdi Rezzoug
email: mehdi.rezzoug@irt-systemx.fr

Maria Chiara Leva

Edoardo Patelli

Luca Podofillini

Simon Wilson

On Improving the Robustness of Reinforcement Learning Policies against Adversarial Attacks

Keywords: Deep reinforcement learning, Safe exploration, Observation perturbation, Adversarial training, Reward shaping, Autonomous vehicles

With deep neural networks as universal function approximators, the reinforcement learning paradigm has been adopted in several commonplace services such as autonomous vehicles, aircrafts and domestic assistance, which is raising new safety requirements. Indeed, a deep reinforcement learning agent obtains its states through observations, which may contain natural accuracy errors or malicious adversarial noises. Since the observations may diverge from the true environment states, they can lead the agent into taking risky suboptimal decisions. This vulnerability is well-known in computer vision literature where it has been emphasized via adversarial attacks. In terms of defense, various techniques have been proposed, including heuristic and certified methods, mainly to improve the robustness of deep neural networks-based classifiers. It is therefore necessary to propose solutions adapted to this learning challenge faced by reinforcement learning agents. In this paper, we propose two defense mechanisms based on reward shaping and adversarial training as a countermeasure against attacks on environment observations. The results reported from experiments conducted on autonomous vehicles controlled by reinforcement learning policies demonstrate that our approach successfully provide sufficient information to effectively learn the task in the context of highly perturbed environments. Furthermore, the defense mechanisms improve the robustness and generalization capacities of the learning models decreasing risky decisions in the presence of adversarial attacks.

Introduction

Among Machine Learning (ML) paradigms, Reinforcement Learning (RL) provides an effective framework for controllers to learn through information collected in real time and to behave appro-priately without relying on a perfect model of the environment. With deep neural networks (DNNs) as universal function approximators, Deep Reinforcement Learning (DRL) has been applied to Cyber Physical Systems, where the search space 2

Yesmina Jaafra and Christophe Bohn becomes intractable for tabular algorithms. These complex physical entities interacting with the real world are progressively used in commonplace services, such as autonomous vehicles, aircrafts , and domestic assistance [START_REF] Jaafra | Context-aware autonomous driving using meta-reinforcement learning[END_REF]). Given the involvement of costly equipment and human safety, deploying DRL algorithms requires rigorous assessments before their deployment in order to prevent high-risk situations. More specifically, it has been demonstrated that DRL is vulnerable to adversarial attacks, where an imperceptible perturbation incorporated in the networks input causes inconsistencies in the behavior of the algorithm. Such perturbations in the input space of DNN, may result in a significant variation of the predicted output as shown by [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF]. This well-known problem in computer vision literature, has been recently detected for neural network policies generated by state-of-the-art DRL algorithms . Indeed, when an RL agent obtains its current state via observations, the latter may contain uncertainty that originates from unavoidable equipment inaccuracies or malicious perturbations leading to catastrophic failures.

In terms of defenses, various techniques have been proposed recently, mainly assigned to DNNbased classifiers. Heuristic defenses are experimentally validated and operate without theoretical guarantees. Currently, the most common heuristic defense is adversarial training, which integrates adversarial samples into the training phase . Other heuristic defenses mainly achieve input transformations and denoising to alleviate the perturbation in the feature domains. On the contrary, certified defenses are provable methods providing theoretical error-rate guarantee reflecting their lowest accuracy under a well-defined type of attacks [START_REF] Raghunathan | Semidefinite relaxations for certifying robustness to adversarial examples[END_REF].

Motivated by these safety concerns, we consider in this paper the problem of improving the robustness of DRL models applied to autonomous control systems. To this end, we propose the following contributions: (i) Develop a safe reward function to shape and guide agents training using Safety of The Intended Functionality (SOTIF) standard (ii) Train an agent through an online DRL algorithm in presence of adversarial attacks generated by state-of-the-art techniques adapted to RL settings (iii) Present a case study on autonomous vehicles equipped with DNN policies that process environment observations to produce control actions.

The rest of this paper is organized as follows. In Section 2, we introduce the required background knowledge and related work on DRL, reward shaping and adversarial attacks. Section 3 gives a description of our methodology to implement the proposed defense mechanisms and section 4 presents and discusses the experiments results. Finally, we draw conclusions and perspectives in section 5.

Background and related work

In this section, we discuss the fundamentals of the DRL process and its application. Then, we recall the concept of reward shaping. Finally, we present a taxonomy of adversarial ML attacks and their recent implementations.

Deep Reinforcement Learning

The mathematical formulation of RL derives from Markov Decision Process (MDP) in terms of the state, action, reward, and dynamics of the system. At each time step, the agent observes the current state s t and performs an action a t based on its current policy π. After the action is executed, the agent observes its reward r t and next state s t+1 .

More formally, an RL task T i is defined according to the tuple (S, A, p, r, γ, H) where S is the set of states, A is the set of actions, p(s t+1 |s t , a t ) is the state transition distribution, r is a reward function, γ is the discount factor and H the horizon. A RL setting aims at learning a policy π of parameters θ that maps each state s to an optimal action a maximizing the return of the agent trajectories R t = t+H-1 k=t γ k-t r k+1 . The discounted return stated above allows the definition of a state value function V π (s) = E [R t |s t = s] and a state-action value function Q π (s, a) = E [R t |s t = s, a t = a] to measure, respectively, the current state and state-action returns estimated under the policy π. In high dimensional environments, the estimation of these value functions Robust Deep Reinforcement Learning 3 becomes intractable. Through non-linear approximation, DRL proposes to control RL agents with function approximations based on DNN to learn the optimal policy or the value/reward functions.

There are three main approaches to solving RL problems. In value-based RL algorithms such as Q-learning, a value function is approximated to select the best action according to the maximum value attributed to each state and action pair. On the other hand, policy-based methods directly optimize a parameterized policy without using a value function. They use instead gradient descents like in the family of REINFORCE algorithms. Actor-critic (AC) methods combine the advantages of the two previous approaches by learning both a policy and a value function in order to reduce variance and accelerate learning [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]).

Reward Shaping

A crucial role is played by reward functions to build driving policies in large-scale applications. However, learning in such sparse settings is complicated and slow. A powerful technique for scaling up RL approaches to handle complex tasks is to transform domains knowledge into complementary rewards. The combination of the original and new rewards is known as reward shaping, inspired by the concept of operant conditioning from psychology discipline. It guides the RL agents to learn faster and more efficiently

The review of shaping methods reveals that, despite the current dominance of reward shaping, the concept itself extends perfectly beyond designing a powerful reward scheme and applies to any supervised transformation of the learning task including environment dynamics, internal parameters and the action space [START_REF] Marthi | Automatic shaping and decomposition of reward functions[END_REF]. Nevertheless, it is the reward scheme that evolves in most modern shaping scenarios, where the learning agent is rewarded for meeting additional subgoals.

More practically, in order to improve the return assignment by making the correct behavior apparent at early stages of training, we apply a shaping function F acting similarly to the native reward function r. At each transition, F oper-ates an assessment of the trajectory, and returns a corresponding reward value yielding a more supportive environment to the RL agent. In the most general form, namely the additive form [START_REF] Randløv | Learning to drive a bicycle using reinforcement learning and shaping[END_REF], the new environment is defined as a transformation of the original MDP to a shaped MDP with supplementary rewards (r + F ). Besides this approach, other important works of reward shaping include the Potentialbased reward shaping (PBRS) [START_REF] Ng | Policy invariance under reward transformations: Theory and application to reward shaping[END_REF] and its variants, the potential-based advice (PBA), the dynamic PBRS and the dynamic potential-based advice (DPBA) [START_REF] Harutyunyan | Expressing arbitrary reward functions as potential-based advice[END_REF]. They express F as the difference of potential function ϕ defined over a source s and a destination state

s ′ : F = γϕ(s ′ ) -ϕ(s).

Adversarial attacks

Among security attacks, the malicious input generated by inserting crafted perturbations into the original input is identified as an adversarial example. Formally, given f (•) a DNN classifier, the adversarial example x is created by adding an imperceptible perturbation δ to the initial example x. The perturbation δ is computed by iteratively approximating the optimization problem until the resulting adversarial example is classified in targeted class c where

x = x + arg min δx ||δ|| until f (x + δ) = c.
The attacks on DRL can be divided into four categories based on the functional components of the DRL process: reward, action, state or model spaces. In this study, we are interested in adversarial attacks on DRL state observations. For example, [START_REF] Huang | Adversarial attacks on neural network policies[END_REF] provided a first attempt to evaluate the robustness of deep reinforcement learning policies through attacks based on fast gradient sign method. The experiments show a significant decrease in the accuracy of the DRL algorithms. [START_REF] Lin | Tactics of adversarial attack on deep reinforcement learning agents[END_REF] considered a more complicated case where the adversary is allowed to attack only a subset of time steps, and used a generative model to predict the future states and actions in order to formulate the misleading actions.

The countermeasures proposed to deal with adversarial attacks on DRL include many strate-Yesmina Jaafra and Christophe Bohn gies based on adversarial training, randomization schemes, denoising methods, and provable defenses [START_REF] Ren | Adversarial attacks and defenses in deep learning[END_REF]. In our work, we will rely on adversarial training as defense mechanism. It attempts to improve the robustness and the generalization of DNN policies outside of the standard training manifold by learning a better distribution. Researchers such as [START_REF] Kos | Delving into adversarial attacks on deep policies[END_REF] proposed to re-train the model with perturbations generated by adding noises to states and rewards where the attacker is considered to be competing in a game with the agent. [START_REF] Pinto | Robust adversarial reinforcement learning[END_REF] configured interactions between both the agent and the attacker as a zero-sum minimax objective function where the agent improves its policy by trying to win the attacker.

Approach

In this section we propose our approach to deal with adversarial attacks on DRL. The strategy adopted to build on the framework robustness includes adversarial training and the design of a safer reward function.

Task environment

First, we consider an MDP environment implemented in the 2D open-source simulator for autonomous driving Highway-v0 [START_REF] Leurent | An environment for autonomous driving decision-making[END_REF]. The agent task consists in driving a car on an infinite 4 lanes unidirectional highway. The egovehicle controlled by the agent is inserted in the traffic flow with the exo-vehicles that follow the Intelligent Driver Model and MOBIL model [START_REF] Kesting | General lane-changing model mobil for car-following models[END_REF]. The setting implemented to collect the environment states consists of the transversal position of the agent y ego and its velocity v ego . In order to account for interactions between the traffic participants, we use relative data for exo-vehicles. Hence, the z i are the positions of the exo-vehicles relative to the agent in the longitudinal direction of the road and v i their relative velocities. Furthermore, we identify the exo-vehicles according to their topological relation with the ego-vehicle. In this regard, bl, f l, b, f , br, and f r represent respectively the closest exo-vehicles to the agent, in the back-left, front-left, back, front, back-right, and front-right positions.

The goal of the agent is to drive as fast and as long as possible while avoiding the accidents. The episode ends when the agent has a collision with another vehicle or reaches a time limit. The ego-vehicle can be controlled with a finite discrete set of tactical decisions implemented by low-level controllers: no-action, right/left change lane, accelerate/decelerate.

Reward shaping for safe driving

The basic reward function implemented in Highwayenv is expressed in 1. It prompts the agent to reach high speed mainly by avoiding collisions.

R v = max -1, V ego - n i=0 Vexo i n V max - n i=0 Vexo i n (1)
Nevertheless, the speed-based reward function fails to enhance the agent performance in terms of collisions number and episodes duration, notably in high-density traffic scenarios. Since the focus of this work is shed on RL safety, we propose to shape the basic reward function by extending it to a safe design reward in relation with Safety Of The Intended Functionality (SOTIF) and defined in ISO Standards under the reference ISO/PAS 21448:2019 a . SOTIF offers a proper understanding of road vehicles safety requirements granting the absence of unreasonable risk caused by the hazardous behavior of the intended functionality. More formally, let C 1 and C 2 denote two vehicles represented in a follower-leader topology. We define three functions P (x), V (x) and A(x) to describe the position, the velocity and the acceleration of each vehicle, respectively. The three following operational metrics have been considered: Time Inter Vehicles (TIV). Heavy traffic on highways requires the optimization of inter-distances between the follower and leader vehicles in order to provide safety solution in transport. This distance allows to react in case of critical situation taking into consideration the reaction time of the driver and the braking distance of the car. Therefore, it's possible to derive the safety time between two cars C 1 and C 2 as T IV = |P (C2)-P (C1)|

V (C1)

. Time To Collision (TTC). TTC has proven to be a https://www.iso.org/standard/70939.html a cue for decision-making in traffic and a pertinent metric for rating the severity of a conflict. It is defined as the time span left until a collision between two vehicles occurs if the course and speed difference are maintained and no evasive action is taken . The TTC is expressed as T T C = |P (C2)-P (C1)| V (C1)-V (C2) . Braking Time (BT). It is a main component of the stopping distance in addition to the Driver Reaction Time. The braking distance can be defined as the distance the car will travel once the driver has reacted and applied the brakes. The BT is then specified as BT = V (C2)-V (C1)

A(C1)

. A new operational reward R o is designed by integrating the safety measures described above:

R o = min(r f , r f t , r bt ) if change lane r f b else, (2)
where r f , r f t and r bt are functions of TTC and TIV defining, respectively, the risk from exovehicles in front-same lane, front-target lane and before-target lane. On the other hand, r f b is a function of braking time of the ego-vehicle (same lane). The safe reward function is obtained by shaping the speed-based reward function with the following minimum bounding combination: R = min (R v , R o ).

Adversarial training

Our second contribution to improve the robustness of RL agent policies consists in adopting an approach focusing on adversarial training techniques. In this paper, we implement a Proximal Policy Optimization algorithm (PPO) Schulman et al. ( 2017), however, any other actor-critic method could be retained. The critic component yields an estimation of the value function V π (s t ) of the state s t represented by the observation x t and following the current policy π learned by the actor network. We consider in this work a gradient-based whitebox method inspired by [START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF]. It aims at attacking the actor network of the RL agent by crafting perturbed observations x to replace the actual observations x returned by the environment, and then allow the agent to decide the action a = π(x). In the case of discrete action spaces, an attack is effective if the agent modifies its decision. In the context of a DNN policy, we operate in two steps. First, we propose to use the gradient of loss with respect to every component of the input (i.e. Jacobian matrix) to extract the sensitivity direction. Then a saliency map is computed to select the dimension which generates the maximum error using as little perturbations as possible.

More formally, we aim at crafting a perturbation δ x of x that minimizes the probability of the optimal action predicted by the actor using the Jacobian matrix of the probability function learned by the actor policy π. Let's consider its logit outputs (x) where aj (x) expresses the probability of the action a j in the policy output given an input x and a j ∈ A the different possible actions. Therefore, determining the appropriate perturbation to attack the observation x consists in solving the following optimization problem:

arg min δx ||δ x || s.t. π(x) ̸ = π(x + δ x ).
Furthermore, since it's preferable to change all input features by no more than a small quantity, the perturbation is bounded by a parameter ε defining the budget that the adversary is allowed to introduce in the input. Building an adversarial example x from a given input x requires calculating a perturbation map H(.) with respect to the input features x i , based on the jacobian ∇ x (x):

H(i) = ( aj ̸ =a d ,aj ∈A ∂ aj (x) ∂x i ) -( ∂ a d (x) ∂x i )
(3) where a d = arg max a∈A π(a|x) is the action corresponding to the highest probability computed by π in x. The Jacobian matrix identifies how the elements of the environment observation affect the logit outputs of different discrete actions. Precisely, the proposed algorithm iteratively perturbs the feature x i with the highest value H(i) to affect the logit outputs significantly. The proposed method creates an adversarial observation x which reduces the probability of the selected action a d and increases the probability of all other actions. This process is repeated until π(x) ̸ = π(x).

Simulation experiment

In this section, we first describe the simulation configuration used to run experiment scenarios Yesmina Jaafra and Christophe Bohn for the RL agent in Highway-v0 simulator. Then we depict and interpret the results of experiments conducted to evaluate the defense mechanisms.

Experimental design

The experimentation is based on the PPO algorithm, which performs comparably to state-of-theart RL approaches while being much simpler to implement and tune. The agent is trained in the 2D open-source simulator for autonomous driving Highway-v0 described in section 3.1. The perturbations applied on the dynamics of the environment imply the velocities and positions of the exovehicles as introduced in section 3.3. For the experimental setup, hyper-parameters tuning is performed following a holdout validation methodology which consists in holding out part of the training set used to evaluate several candidate models and then selecting the best one (for more details, cf. [START_REF] Géron | Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems[END_REF].

To quantify the safety and effectiveness of the proposed robust RL approach in both training and testing process, we use three metrics that evaluate the average cumulative return, the episode length and the average velocity of the ego-vehicle. The metrics are computed under 10 rollouts average with a duration of training and testing respectively equal to two millions and fifty thousand steps per experiment. The agent purpose is to reach the highest possible speed avoiding collisions. Each episode terminates when the agent has a collision with another vehicle or reaches a time limit. The results are reported in the form of graphs for training to apprehend the evolution and convergence of learning, and boxplot for testing for a more global and final vision of the performance.

The experiments evaluating the proposed defense mechanisms involve a set of models and scenarios that are defined in the following. Unsafe model. Policy of the RL agent that has been trained using the basic reward function (R v ). Safe model. Policy of the RL agent that has been trained using the safe reward function (R). Attack Attack. Main scenario where the safe agent is trained and tested in the presence of adversarial attacks. NoAttack Attack. Witness scenario that specifies a safe agent trained with no adversarial attacks and tested with attacks. Attack NoAttack. Complementary scenario to assess the generalization capacities when the safe agent is trained with adversarial attacks and tested with no attacks. NoAttack NoAttack. Standard scenario where the safe agent is trained and tested with no attacks.

Results

The evaluation of our contributions consists in analyzing to what extent can the safe reward function and the adversarial training improve the policy of the RL agent in the presence of perturbed observations. First, we evaluate the global performance of the safe reward function in experiment scenarios without attack. Training results are depicted in figure 1 where we can see through the curves of average episodic length and traveled distance that the safe function improves drastically the performances of the agents, compared to the basic reward function.

Training with safe reward function

We can also state that the safe model reaches higher episode length values denoting an improved capacity of collision avoidance. This interpretation is compliant with the reward shaping effort fostering more safety in the interaction between the neighboring vehicles as described in section 3.2. The testing results presented in figure 2 confirm the conclusions stated above. Furthermore, we observe that the mean velocity of the safe agent (90.1 km/h) is slightly lower than the unsafe one (99.9 km/h) but is still very appropriate to the highway context and comparatively higher than exo-vehicles average speed (70.3 km/h). This means that the safe policy has been able to operate Robust Deep Reinforcement Learning 7

an efficient tradeoff between higher velocity and safer driving decreasing the risk of collisions. In figure 3(a), we assess the efficiency of the safe reward defense mechanism in the training environment over eight experiments with a different intensity of disturbance ε applied in each one. The major finding is that the safe model is more robust and resistant to the increase of the parameter ε. Indeed, the metrics curves of the unsafe model are almost flattened starting from ε = 0.1. In figure 3(b), we expand the y-axis (returns) of the scenario run with an attack intensity ε = 0.5. While the safe model is showing a typic case of convergence, the unsafe model curve points out a failure of the learning process. This is probably due to the fact that the basic reward function R v is unable to provide sufficient information to learn the problem in the context of highly perturbed observations.

Performance of adversarial training

The evaluation of the second defense mechanism is depicted in figure 4, where we consider policies performance in four different scenarios emphasizing the contribution of adversarial training in improving RL agents robustness and generalization capacities. Some interpretations can be given in this respect. In the following, the figures are presented in terms of episode reward (in first position) and episode length (between brackets). Robustness. The boxplots related to scenarios Attack Attack and NoAttack Attack show that the agent trained and tested with adversarial attacks is substantially behaving better than the attacked agent trained in standard conditions. This defense mechanism offers 85%(100%) of room for improvement. Thus, we can confirm that the adversarial training prevents drastic failures of RL agents in presence of attacks against environment observations. Practically, the safety aspect is visible in the decreasing of collision risk expressed by higher expectations of episode length. Generalization. This aspect is analyzed in 2 steps. First, let's compare NoAttack NoAttack and NoAttack Attack scenarios. In the case of standard training, the policy performance is decreasing when tested in an environment different from the training one. Concretely, the model trained without attack loses 47% of average reward (50% of episode length) when it's tested in an attacked environment. This can be explained by the inherent feature of deep learning policies which are sensitive to significant change in data distribution [START_REF] Lake | Building machines that learn and think like people[END_REF]. On the other hand, we raise almost the same performance of Attack Attack and Attack NoAttack scenarios reflected by a small improvement of 9% (3%) when training with attacks then testing in standard conditions. Hence, the defense mechanism of adversarial training not only enhances the robustness, but also the generalization capacities of RL policies. Yesmina Jaafra and Christophe Bohn

Conclusion

Despite its great advances, DRL is vulnerable to adversarial attacks, which prevents its deployment in real-life critical systems. This problem has directed our concern to develop defense mechanisms based on reward shaping and adversarial training. The results reported for a case study conducted on autonomous vehicles are promising. The reward shaping has successfully provided sufficient information to accelerate training convergence in the context of perturbed environment. Furthermore, the adversarial training has specifically fostered the robustness and generalization capacities of the obtained models in the of attacks. An important direction of future work is to implement learnable adversaries, which we assume to be more harmful and susceptible to demonstrate higher resilience to defense mechanisms. We also intend to enhance the naïve application of adversarial training by reformulating the defense strategy as a modified MDP.

Fig. 1 .

 1 Fig. 1. Safe Vs Unsafe models trained without attack: Average traveled distance across episodes (left), average episode length (right).

Fig. 2 .

 2 Fig. 2. Boxplots of Safe Vs Unsafe models tested without attack, from left to right: average episode length, average cumulative episode reward and average velocities across episodes.

Fig. 3 .

 3 Fig. 3. Adversarial training: (a) Safe Vs Unsafe models for all ε values (b) Safe Vs Unsafe models for ε = 0.5.

Fig. 4 .

 4 Fig. 4. Boxplots of robustness test scenarios: average cumulative episode reward (left), average episode length (right).