Armagan, Ludovic, Maëlic, Thomas, Yoann, Quentin, Joris, Julien, Carlos Alam Gabriel

Thomas Chaffre

Gilles Le Chenadec

Karl Sammut

Estelle Chauveau

Benoit Clement

Direct Adaptive Pole-Placement Controller using Deep Reinforcement Learning: Application to AUV Control ⋆

Keywords: Adaptive control, Pole-placement, Deep reinforcement learning, Underwater vehicle Underwater Vehicle, Adaptive Control, Deep Learning, Lyapunov Stability

Je tiens tout d'abord à remercier mon père Julien et ma mère Frédérique qui m'ont énormément soutenu lors de ces trois dernières années, et de façon générale tout au long de ma vie. Ils ont toujours sur m'écouter, me redonner le moral, et me conseiller lors des moments difficiles. Merci d'avoir supporté mes périodes de stress et d'angoisse, de m'aider à prendre du recul et d'avoir directement et indirectement participé à ma thèse. Sans leur soutien mon expérience de thèse aurait été totalement différente. Je tiens également à remercier ma soeur Julie et son mari Dimitri qui m'auront également soutenu durant toutes ces années. Merci pour votre présence, nos échanges et votre joie de vivre qui m'a si souvent permis de relativiser face aux épreuves de la vie. L'affection et les encouragements constants de ma famille m'ont permis de traverser avec confiance cette étape de ma vie.

Je tiens ensuite à remercier mes encadrants de thèse, Benoit Clement, Karl Sammut, Gilles Le Chenadec, Paulo Santos et Estelle Chauveau, sans qui cette thèse n'aura pas eu lieu. Durant ces trois dernières années j'ai pu bénéficier d'un encadrement et d'une disponibilité sans failles de leur part. Je vous remercie d'avoir cru en moi et de m'avoir donné cette opportunité de m'exprimer et de me développer intellectuellement. J'ai pu grâce à vous apprendre tout ce qu'implique le travail de recherche et pour cela je vous en suis extrêmement reconnaissant. Travailler avec vous durant ces années fut un réel plaisir et je garderai en mémoire nos longues discussions. Je vous suis reconnaissant pour votre bienveillance, votre soutient appuyé lorsque j'en avais besoin et pour m'avoir accordé une pleine liberté dans mes recherches.

Je tiens tout particulièrement à remercier Tony Kyriacou qui aura tout fait pour que mon séjour Australien à Flinders University se passe le mieux possible. Tony est une personne formidable qui m'a permis de me sentir comme à la maison et ce dès mon premier jour au campus. Tony m'a également beaucoup aidé pour mes démarches administratives et pour toutes ces raisons je tenais encore à le remercier.

Je tiens à remercier Jonathan Wheare et Andrew Lammas, ingénieurs de recherche à Flinders University, qui m'ont aidé à mettre en place nos validations expérimentales. Je les remercie pour leur temps et leur patience au fil de ces mois d'expérimentations pendant lesquels ils ont dû s'adapter à de nombreux imprévus. Sans eux, nous n'aurions pas pu obtenir ces résultats. J'ai pu à vos côtés me rendre compte à quel point la robotique mobile est une tâche compliquée et qu'il faut s'attendre à tous les scénarios avec ce type de véhicule.

Merci à l'équipe de l'IRL CROSSING d'Adelaide avec qui j'ai pu travailler et où j'ai pu me ressourcer lorsque le mal du pays se faisait sentir. Merci JeanPhilippe et Cédric ainsi que toutes les personnes de votre équipe pour ces moments.

Et pour finir, la vie se continuant en dehors du laboratoire, je tiens à remercier mes amis, merci

Summary

Autopilots for unmanned systems are usu ally designed based on the feedback pro vided by velocity and orientation sensors. In the case of autopilot systems for au tonomous underwater vehicles (AUVs), the main objective in the design is to compen sate for waves and currentinduced disturb ing forces acting on their body. Existing AUV autopilots are however only able to compensate for lowfrequency components of seainduced disturbances. It seems nat ural to assume that the AUV performance could be improved by taking the nature of the disturbances into account in the design of the autopilot.

Adaptive control provides what seems to be an ideal framework for this end. The objec tive of this technique is to adjust automat ically the control parameters when facing unknown or timevarying processes such that the desired performance threshold is met. Developed in the late 1950s, adap tive control frameworks have been consid erably expanded and used in various fields, their application has been facilitated by the rapid progress in microelectronics and the increasing interaction between laboratories and companies, from aerospace to mar itime industries. As a result, adaptive con trollers started to be widely adopted in the industry in the early 1980s. It was estab lished at that time that robust designs with fixed parameters are too limited to handle complex regimes. The study of adaptive controllers for AUV maneuvering is associ ated with various challenges, and the focus of this thesis was the external disturbances including:

Unknown dynamics: the uncertainty asso ciated with describing precisely the states of waves or currents is high. This, to gether with its dynamic nature, prevents lin ear feedback control methods from achiev ing optimal performance of the plant. This becomes more critical in the presence of changes in weather conditions that impose a multiplicative factor in the component of the induced forces. The disturbance period will also vary with the speed of the vehicle and its orientation relative to the waves.

Nonlinearity: the controller response at some operating points must be overly con servative to satisfy the specification at other operating points. This is difficult to achieve for fixed parameters obtained through local linearization, that do not encompass the en tire regime envelope.

In this thesis, we considered the case where the AUVs have limited observabil ity of the process and therefore the afore mentioned uncertainties are not measured by the system. A class of adaptive control methods, known as learningbased adap tive controllers, have been developed to tackle some of these limitations. This fam ily of solutions uses modelfree optimiza tion methods capable of compensating for the unknown part of a process while also maintaining optimal control of its known part using traditional modelbased control struc tures. Among the various modelfree meth ods, deep reinforcement learning is cur rently leading the field. They exploit strong statistical tools that provide control systems the ability to automatically learn and im prove from experience without being explic itly told how to.

The objective of this thesis was to formalize a novel learningbased adaptive control us ing deep reinforcement learning and adap tive poleplacement control. In addition, we proposed a novel experience replay mecha nism that takes into account the characteris tic of the biological replay mechanism. The methods were validated in simulation and in real life, demonstrating the benefits of combining both theories against using them separately.

Introduction

Context and motivations

Autonomous vehicles are becoming increasingly prevalent in our daytoday activities. From cars, trains, warehouse robots to medical delivery quadcopters, the field of autonomous vehicles is blooming. This de velopment was driven by an urge to strengthen productivity, accuracy, operational efficiency but also to improve human operators' and users' safety. While this tendency is seen globally, there is also evidence of an unequal development in underwater applications. Despite the comparable needs in applications, includ ing offshore platforms inspection, marine geoscience, harbor and coastal surveillance and underwater mine countermeasures, the majority of uncrewed underwater vehicles remain remotely operated or with limited autonomy abilities.. The problem of autonomy is even more pronounced in the context of smallsize underwater vehicles. The latter are paradoxically denoted as AUVs for Autonomous Underwater Vehicles, not on the basis of improved autonomy, but rather by dint of the historical need for physical connection to a surface vessel that has been removed. AUVs are required to operate over large regions (from deep oceans to coastal and riverine regions), lengthy periods of time (extending from several hours to days before the possibility for human intervention) and to perform complex tasks such as search and rescue [START_REF] Anderson | Workhorse AUV -A costsensible new Autonomous Underwater Vehicle for Surveys/Soundings, Search & Rescue, and Research[END_REF], underwater manipulation [START_REF] Marani | Underwater autonomous manipulation for intervention missions AUVs[END_REF], pipeline and facility inspection operations [START_REF] Gilmour | Field resident AUV systems -Chevron's long term goal for AUV development[END_REF], underice exploration [START_REF] Barker | Scientific Challenges and Present Capabilities in Underwater Robotic Vehicle Design and Navigation for Oceanographic Exploration UnderIce[END_REF], target following [START_REF] Sun | Target following for an autonomous underwater vehicle using regularized ELM based reinforcement learning[END_REF], etc. Although there has been rapid development of aerial and terrestrial autonomous vehicles, there has been relatively slow development of AUVs, leading to interrogations as to what could be the cause of this contrast.

Until the late 1940s, it was widely believed that fixed control design was sufficient for processes of limited uncertainties. It was only with the development of autopilots by NASA for their highperformance aircraft, such as the X15 in the early 1950s, that it was recognized that constantgain linear feedback control could not provide satisfactory performance over the entire flight regime. Such aircraft operate over a wide range of altitudes and speeds with their internal dynamics changing as fuel is consumed. A more sophisticated control system able to maintain its performance over multiple operating conditions was therefore needed. Great efforts were therefore made into the development of autopilots for the X15, which led to the emergence of what is known today as Adaptive Control. The objective of adaptive control is to grant the control law some flexibility to adjust its response based on process variation. The Gain Scheduling technique [START_REF] Dydek | Adaptive Control and the NASA X153 Flight Revisited[END_REF] was then proposed as a solution for the control of hypersonic aircraft, becoming shortly after a standard tool in the field. Following this successful application, adaptive control has since attracted significant attention from both academic and industry communities. Nevertheless, the deployment of autonomous vehicles in the maritime domain was still limited. Adaptive control methods require some a priori knowledge of the processes and the system to be controlled and often require expensive simulation efforts. This first generation of methods is denoted as modelbased [START_REF] Kálmán | Design of a SelfOptimizing Control System[END_REF] [START_REF] Bellman | Adaptive Control Processes A Guided Tour (Reprint from 1961)[END_REF][MBT59] accordingly due to the need for a model of the process dynamics to be designed. The difference observed in maritime vehicles can be explained in part due to the limited measurement abilities on board the vehicles and our little to no understanding of the natural phenomena taking place underwater and disturbing the vehicles. This lack of prior knowledge is even more problematic in the case of AUVs, which in addition face several challenges [START_REF] Hassanein | Modelbased adaptive control system for autonomous underwater ve hicles[END_REF], including:

1. Unknown dynamics: the uncertainty associated with describing precisely the states of waves or cur rents is high. This, together with their dynamic nature, prevents linear feedback control methods from achieving optimal performance of the plant. This becomes more critical in the presence of changes in weather conditions that impose a multiplicative factor in the component of the induced forces. The disturbance period will also vary with the speed of the vehicle and its orientation relative to the waves.

2. Nonlinearity: the controller response at some operating points must be overly conservative in order to satisfy the specification at other operating points. This is hardly possible for fixed parameters obtained through local linearization, which does not encompass the entire regime envelope.

3. Thruster efficiency: a fullyactuated vehicle can often become underactuated when its speed varies. This is especially true for hoveringtype AUVs which use thrusters in place of steering fins to achieve maneuverability at low speeds. As the forward speed increases, the effectiveness and efficiency of lat eral thrusterinduced movements are drastically reduced, making it impossible for the vehicle to account for pure lateral motions. 4. System reliability: if the performances of one or more thrusters become increasingly less effective, the control system should be able to detect this and engage a new control algorithm specially designed to accommodate the failures and, if possible, to complete the mission.

The design of autopilots under these circumstances seems laborious. As a result, the vast majority (95%) of AUV control systems are based on fixed PID controllers, despite their poor performance against process variation [START_REF] Yuh | Design and Control of Autonomous Underwater Robots: A Survey[END_REF]. A new class of adaptive controllers is desired to reduce or remove this need for prior knowledge. When no information on the process is available, modelfree adaptive control can be used. These methods aim at describing complex processes from sensory data only, rather than from first principle laws. The first such theories appeared in French literature in the early 1920's [START_REF] Leblanc | Sur l'electrification des chemins de fer au moyen de courants alternatifs de fre quence elevee[END_REF]. The concept was then formalized under what is currently known as Extremum Seeking Control (ESC) [START_REF] Suttner | Extremum seeking control with an adaptive dither signal[END_REF]. The goal of ESC is to maximize a control objective function without any knowledge of its extremum. Since its initial development, research in modelfree adaptive control theory has gained pace, exhibiting a reliable level of maturity with good analysis and understanding of its main properties. One of the main applications of ESC has been in reaction processes, including fuel flow control, combustion process control, and wind or solar energy conversion.

However, the lack of guaranteed transient performance remains a significant drawback in the application of modelfree methods in AUV control. This is due to the safety of the platforms which is most important given the cost of deploying such vehicles. This led to the development of an entirely new theory of adaptive control, denoted as learningbased. It is based on the idea that by combining both modelbased and model free theories, one designer can compensate for both the known and unknown parts of the process model. The concept behind this theory is to use a modelbased control structure to maintain optimal control of the known part of a process, and to compensate for the unknown part by learning to adapt to process variations in a modelfree fashion. Following the recent breakthrough in datadriven control [START_REF] Brunton | DataDriven Science and Engineering: Machine Learning, Dynamical Systems, and Control[END_REF] [Ben18] and the successful development of modern machine learning techniques [START_REF] Richard | Reinforcement learning an introduction Second edition[END_REF] with improvements in computing power (e.g. faster GPUs, enhanced Parallel computing, etc), we have seen a resurgence of interest in the field of learningbased adaptive control.

In particular, such methods based on deep reinforcement learning (DRL) have shown promising achieve ments. This is possible, especially thanks to the use of deep neural networks to extract physical insights through sensory data (empowered by progress in data collection with highfidelity simulations, faster com puters, etc). However, as stated in [START_REF] Sünderhauf | The limits and potentials of deep learning for robotics[END_REF], the extension of DRL techniques for robotic tasks raises serious questions. Compared to other applications, robots have to interact with a dynamic environment where rele vant information about the system is not always accessible or tractable over time.

In their uptodate thorough investigations [START_REF] Dulacarnold | An empirical investigation of the challenges of realworld reinforce ment learning[END_REF], G. DulacArnold et al. presented realworld reinforce ment learning challenges that are still not resolved, these include Satisfying Environmental Constraints, High Dimensional Continuous State and Action Spaces or MultiObjective Reward Functions. In addition, the no tions of stability (in terms of Lyapunov stability) in the DRL framework are more deeply investigated [START_REF] Garcia | A comprehensive survey on safe reinforcement learning[END_REF] but the lack of formalism is still often highlighted by the control community as a significant concern. The purpose of this thesis is to explore to what extent modelfree control techniques combined with the classical theory of dynamical systems can be, at least, part of the solution to these challenges. This is the objective of the learningbased control area [START_REF] Benosman | Learningbased adaptive control for nonlinear systems[END_REF] which aims to combine the advantages of the aforementioned paradigms (modelbased and modelfree) into one hybrid control scheme.

An adaptive controller, being by nature nonlinear, is more complicated and thus more computationally ex pensive than a fixedgain controller. Therefore, before choosing to use an adaptive controller it is important to study whether or not the problem could be solved by a constantgain controller. One way in deciding if an adaptive controller is justified is outlined in Figure 1.1 depending on the characteristic of the disturbances. Following this procedure, we discuss next why adaptive control is mandatory in our use case of AUV control.

Open vs closedloop control

Although the act of control seems active, there are many examples of control in everyday life that is not active but rather passive. An example is the hull of boats that passively causes the water around the body to behave in a favorable way to reduce drag. If we can reach the desired behavior with passive control, then we just have to design beforehand the vehicle's body properly and there will not be energy expenditure. Passive control, however, is typically not enough and often some form of active control is required. Active control essentially means that we are injecting energy into the system to actively manipulate its behavior. There exist many solution methods for active control. The most common form of active control is called openloop where, as illustrated in Figure 1.2, we reverse design the controlled system and invert the dynamics in order to determine exactly what is the optimal control input u to get the desired output y. The principal downside of openloop control is that we are always putting energy into the system and the moment we stop, the system becomes unsafe. What we can do instead is to take sensor measurements of what the system is actually doing, then somehow build a controller, and feed that back into the input signal so that can manipulate the system. This allows much more subtle control of the system with very low energy input. The basic idea of closedloop feedback control is that by measuring the system output, we can often do much better than just feeding in preplanned control input, and this is the entire subject of this thesis.

Figure 1.2:

Control and dynamics are two sides of the same coin that are present in every aspect of our life. Control can be either passive or active, and allows us to ensure that our system of interest behaves as desired. The most common form of active control is known as closedloop feedback control where the system output is measured and used to compute the next control input. As illustrated here, the closedloop feedback controller is much more efficient at handling uncertainties and disturbances compared to its openloop counterpart.

In order to show how we can reduce the effect of disturbances, let's now illustrate the benefits of feedback with a simple example. Consider an AUV facing a current disturbance in its environment and we want to control its speed. Consider now that we are given a nominal model for the system that is a very crude model of the AUV such as:

y = 2u. (1.1)
We do not give a unit to this value (1.1) but let's just say we measured, in a test tank without disturbance, that by increasing the input to the thrusters by one unit, we obtain an increment of two units of velocity. Let's see now how openloop control would solve here a tracking problem where we want the AUV to maintain a reference velocity value r.

If we want the system output y to be equal to r, with openloop control u ol we will have:

u ol = r 2 . (1.2)
This (1.2) is a very bad controller for many reasons, namely uncertainties in the model and disturbances.

In fact, let's imagine that the model would actually be y = u (possibly because the vehicle's thrusters are old, the sensors are slightly out of calibration, and maybe the drag has changed, etc, since the vehicle was originally modeled). Most likely, the vehicle is no longer fully efficient and y is actually just equal to u, so it is only half as responsive as our nominal model (1.1). In this case, the openloop controller (1.2) may not achieve the desired velocity r, but instead, only achieve r 2 . If the actual system differs from the nominal model, the openloop controller has no way of correcting that. Moreover, if we add the disturbance, the true model yT of the AUV becomes:

yT = u + d, ( 1.3)
with d a scalar representing the disturbance. In this case, the openloop controller (1.2) is only tracking 50% of the reference velocity and all of the disturbance passes right through to the actual velocity y. Openloop control does not take into account the uncertainties and disturbances at all and therefore fails at controlling the AUV in this context.

On the other hand, as illustrated in Figure 1.2, we can close the loop with some controller K. The idea is to measure the actual velocity of the AUV (so we can tell if it is speeding up or down), feed that back, and subtract it from the reference velocity (the difference is an error signal ε). Feedback control then consists in choosing K to make ε small. This is denoted as proportional feedback control because K is just a number. The closedloop feedback controller is thus defined as:

u cl = Kε = K(r -y), (1.4)
and the resulting model of the closedloop feedback control is:

y cl = P u + d. (1.5)
We can solve y cl as a function of r and d as: where the left term tells us how well the output velocity y cl matches the reference velocity, and the right term tells us how much disturbance gets reduced by the control. For recall, the nominal model is P = 2 (1.1), but the true AUV system has P = 1. Notice here (1.6), we want to have the left term equal to 1 which means that we have a really good reference tracking. We want the right term to be equal to 0 because if we have disturbance, we want whatever feedback multiplying it to be as small as possible. We can directly see that the best way to achieve that is to consider a very big value of K. For instance, with K = 100, the left term in (1.6) is very close to 1, the actual velocity is only off from the reference velocity by about 1%, and we have reduced disturbance by about a factor of a 101. In contrast, the openloop controller was off by 50% because the model was bad, and the disturbance was not reduced at all. This is an example where we can handle model uncertainties and disturbances by taking measurements and feeding them back in proportional control.

y
There exist various techniques to choose the value of K and optimal control theory [START_REF] Kwakernaak | Linear Optimal Control Systems[END_REF][Ber95] [START_REF] Doyle | Robust and optimal control[END_REF] is the most common method traditionally used for this purpose.

Nevertheless, when facing process variation, the resulting fixed value of K is not enough to satisfy control performance over a wide spectrum of operating conditions. When the physics of the process is reasonably well known, it is possible to determine suitable values of K for different operating conditions by linearizing the models. System identification is an alternative to physical modeling, however, both approaches do require a significant engineering effort in addition to knowing the a priori process model.

In this thesis, we consider the case where these approaches can not be used because we do not have access to a model process and the disturbances are not measured. In particular, we are focusing solely on external disturbances that are sea currents. In the following, we illustrate this limit and therefore the need for adaptation.

Limits of fixed optimal control

Effect of process variation: The standard approach to control system design has been to develop a linear model for some operating conditions and to design a closedloop feedback controller having constant param eters. This approach has been remarkably successful as we have been able to see that such controllers are intrinsically insensitive to modeling errors and disturbances. However, in practice, there are many different sources of process variations, and the underlying reasons for those variations are not fully understood. We will consider here two simple examples to illustrate the difficulties rising from process variations. Consider a closedloop feedback controller with a Proportional and Integral term, a nonlinear AUV, and the third order process G(s) illustrated in Figure 1.3. Let's consider the characteristic of the AUV as follows:

v = f (u) = u 4 , u ≥ 0.
(1.7) When linearizing the system around a steadystate operating point, we can see that the incremental gain of the AUV is f ′ (u) (i.e. the loop gain is proportional to f ′ (u)). This means that the system can perform well at one operating point and poorly at another. This is illustrated in Figure 1.4 where we tuned the controller parameters using optimal control theory to have limited oscillation at the operating condition Uc = 0.01. For higher values, the controller becomes unstable as shown in Figure 1.4d. 4 and G 0 (s) = 1/(s + 1) 3 .

T i = 1, the AUV characteristic is f (u) = u

Effect of disturbance variation:

The second issue to be explored in this thesis is the effect of variations in current disturbance characteristics. We propose to model this disturbance as an additional disturbing force as illustrated in the block diagram displayed in Figure 1.5. Again, we tuned the controller parameters using optimal control theory for the same operating condition Uc = 0.01 and we plot the controller response against the different values of sea current denoted by d. As illustrated in Figure 1.6, the optimal values of K and Ti that were obtained for the operating condition Uc = 0.01 and d = 0.01 are no longer satisfactory when the value of the disturbance change. Over a certain value of d, the controller is even diverging from the steady state as shown in Figure 1.6d. It seems intuitive that we can recover from these process variations by adjusting the controller parameters according to the operating conditions with adaptive control. However, in the AUV case, we do not have direct access to disturbance measurements and we do not have access to a model of its effect on the AUV. Thus we can not perform this tuning procedure beforehand to find optimal values of the control parameters based on the process variations.

Now that the challenge of AUVs control in our context is demonstrated, we present next the problem and the research objectives of this thesis.

Problematic and research objectives

As previously discussed, the design of an adaptive controller for AUVs in our context is associated with a number of challenges. The principal problem is that the AUV does not have a sensor to directly measure the current and wave disturbances. While a downwardlooking doppler velocity logger (DVL) can be used to measure a current below the vehicle, this only works when the DVL has a bottom lock, i.e, the seabed is within acoustic range of the DVL. As a consequence, we have limited sensorial ability (we only have access to the IMU feedback), and therefore the disturbance is completely ignored in the process modeling, preventing us from using standard modelbased adaptive control methods. The overall objective of this thesis is to propose a learningbased adaptive control system able to compensate for the current disturbance despite not measuring it directly. This design is associated with several research questions that we aim to study in this thesis, including:

• What is the impact of process observability on the design of modelbased controller for AUVs?

• What are the most effective strategies to use the deep reinforcement learning as a modelfree tuning method in adaptive control systems?

• How do the combination of modelbased and modelfree theories compare in regularization perfor mance and robustness to disturbance as opposed to when exploited separately?

• What effect does the experience replay mechanism have on the learning dynamics and the gener alization ability of the resulting policy?

• What are the similarities and differences in the simtoreal transfer of control parameters and con trol inputs?

The number of AUV applications is increasing fast and control designers are obliged to address more and more complex tasks while being limited in the complexity of the control laws they can use due to restricted powerconservative onboard computational resources coupled with the issue of insufficient knowledge of un derwater dynamics. To cope with this situation, the design and development of a learningbased adaptive control system conceived on the basis of a modelbased control structure adjusted by a modelfree opti mization algorithm would allow (1) to compensate for the known part of the process with strong stability components, and (2) to effectively compensate for unobservable external disturbance. The principal objec tive of this thesis is to design a learningbased adaptive control system that could adapt to changes in process dynamics and current disturbance characteristics. The research objective is presented, and we discuss next the proposed structure of the thesis which aims to answer the aforementioned questions.

Structure of the thesis

By way of introduction, the first section presents the context of the thesis with the particularities of AUV applications. Then, the general problem of AUV control is introduced with the objectives of the thesis. Then, the research questions are defined and finally, the research methodology is described.

Section 2 presents the technical background regarding the building elements of AUV adaptive control. The first part is dedicated to the technical elements of adaptive control. We propose a classification of solu tions methods based on the dependence on the process model resulting in three classes of adaptive control methods. A succinct description of each of these methods is provided with a greater emphasis given to learningbased methods. This part ends with a discussion of the limitations of these solutions to our problem of AUV control. In the second part, we present the background elements of reinforcement learning theory. Classification of solutions methods is provided in terms of the nature of the action sampling methods. Then a complete description of the maximum entropy deep policy gradient method is provided, which is used as a central building block of the proposed control system. This part ends with an analysis of the limitations of deep reinforcement learning (DRL). Finally, this second section ends with a description of the simulation tool used to train the aforementioned algorithms.

In Section 3, the research steps that lead to the proposed control system design are presented. It starts with the first part comprising a literature review of related works in learningbased adaptive control of AUVs. Then, in the second part, we present the preliminary studies that were motivated by this literature review which include: the study of a modelfree and a learningbased adaptive control and stability analysis. Finally, the section ends with the proposed learningbased adaptive control system.

In section 4, the proposed control system is evaluated and validated under simulation. It is applied to different AUV applications and improved designs are provided throughout the section. It ends with some additional findings and insights that will lead to the design of the simtoreal transfer methodology.

Section 5 is devoted to the experimental validation of the proposed method. It starts with a description of the experiment protocol and settings. Then, the design of the control system is presented along with an adjusted domain randomization procedure. The section ends with an analysis of the experimental results.

Background

This section presents the technical elements necessary to derive the method proposed in this thesis and to support our choice of approaches. First, we explain the need for adaptation when facing process variation and how to design such a control system using adaptive control theory. A classification of solution methods is provided based on the dependence on the process model resulting in three classes of methods: model based, modelfree, and learningbased. In the latter, there exist many machine learning techniques to use as a modelfree optimization procedure, and this thesis focuses on the use of deep reinforcement learning. Therefore, we here present the theoretical elements of reinforcement learning and deep policy gradient meth ods, in particular. To conclude, because deep reinforcement learning methods need to interact greatly with the environment, we finally present the different simulation tools we used to train our algorithms.

Adaptive control 2.1.1 The adaptive control problem

Interest in industrial adaptive control systems started in the 1950s with the emergence of flight controllers [START_REF] Dydek | Adaptive Control and the NASA X153 Flight Revisited[END_REF]. It was clear at that time that in order to adapt to process variation, one has to exploit any knowledge available to adapt the control system response to changes in operating conditions. To illustrate the adaptive control problem, consider the dynamics of a system described by the nonlinear differential equations of state:

ẋ = f (t, x, u, p), t ∈ R + , y = h(t, x, u), (2.1)
where x ∈ R nx is the state vector; u ∈ R nu is the vector of control inputs; p ∈ P ⊂ R np is a vector of unknown parameters that is an element of an a priori known set P; y ∈ R ny is the vector of system outputs; f and h are smooth functions. The set of outputs and control inputs available at time t is: Γt = {y(t), . . . , y(0), u(t), . . . , u(t)}, t ∈ R + , Γ0 = 0.

(

The control performance index historically takes the following form:

J = 1 t t 0
e 2 (τ )dτ, e(t) = w(t)y(t), (2.3) where w(t) is the desired setpoint. The process is controlled by a controller with adjustable parameters. The general adaptive control problem consists in finding the control policy u(t) = ut(Γt) ∈ Ωt that minimizes the performance index (2.3) for the system described by (2.1) with Ωt representing the domain in the space R nu where the entire admissible control values are defined. The fundamental hypothesis considered in adaptive control theory [START_REF] Page | Adaptive Control Processes: A Guided Tour[END_REF] states that there is a design procedure that makes it possible to determine a controller which satisfies some design criteria if the process and environment are known:

Hypothesis: For any possible values of a plant model's parameters, there is a controller with a fixed structure and complexity such that the specified performances can be achieved with appropriate values of the controller parameters.

Accordingly, the task of adaptation is to search for appropriate values of the controller parameters. The adaptive control problem can be summarized as the design of a method capable of adjusting the controller parameters when the characteristics of the process and environment are unknown or changing. This ad justment can be done in two fashions: by direct adaptive control, in which the controller parameters are determined without any estimation of the aforementioned uncertainties; by indirect adaptive control where the process model, and potentially the disturbance characteristics, are first determined. This emphasizes that a priori knowledge about the process and system is required in the adaptive control theory in order to specify achievable performances, to determine the structure and the complexity of the controller as well as the choice of a proper adjustment method. Traditionally, adaptive control methods are classified based on the nature of the process model (i.e. linear vs nonlinear, discrete vs continuous,…). Conversely, we de cided to classify the methods based on their dependence on the process model, which fits better the main purpose of this work. Therefore, three classes of adaptive control methods can be identified: modelbased [Ste80; DAL10; Kre73; Lan84; Par81; Käl+79], modelfree [Leb22; Rot00; KW00; Nes09; Biz20; Sol+20a] and learningbased [WH09; Spo+01; LVV12; Wan+06; KRP06; BA13] approaches. We will present each of these solution methods in the following.

Classification of solution methods

Modelbased adaptive control

When the physics of the system is known, it is possible to determine a suitable adaptive controller based on a mathematical modelization of its dynamics. For such systems, one can make an approximation near equilibrium points and then derive a simple differential model using Newton's second law. Adaptive controllers designed entirely in accordance with this kind of representation are denoted as modelbased.

One of the first and most straightforward modelbased adaptive methods proposed is called Gain Scheduling.

It consists in finding suitable scheduling variables that effectively characterize the process (on the basis of knowledge about the physical system) and tuning the controller parameters directly based on these variables. This is possible due to the fact that, in many situations, measurable variables can be found that correlate well with changes in the process dynamics. Originally used to adapt changes in the process gains only (hence the name), gain scheduling was first developed to design autopilots for highperformance aircraft [Ste80; DAL10]. As can be seen in Figure 2.1, this method is composed of a linear controller whose parameters are changed as a function of operating conditions in a "preprogrammed" way. After the scheduling variables are determined, the controller parameters are computed at a number of operating conditions based on the system's performance. Gain Scheduling can be seen as a nonlinear mapping from process parameters to controller parameters [START_REF] Krener | On the Equivalence of Control Systems and the Linearization of Nonlinear Systems[END_REF] that is usually framed as a function or a table lookup. Application of Gain Scheduling to maneuvering industrial ships can be found in [VKM96; WD05]. The major drawback of Gain Scheduling is that there is no feedback from the closedloop system to com pensate for an incorrect schedule. Its design can be highly timeconsuming since the controller parameters must be determined for many operating conditions, and the associated performance needs to be verified by extensive simulations (that are not as straightforward to conduct in the underwater domain as they are for aerial simulations). In addition, in the context of AUVs, we are limited in the number of sensors available, and most of the time, we can not measure most of the scheduling variables of interest. For this reason, Gain Scheduling can not be considered for the adaptive control of AUVs.

Figure 2.2:

Illustration of the resulting table lookup obtained when using 2 scheduling variables. The space of operating regimes (either discrete or continuous) is divided into regions associated with a particular value of the controller parameters. Depending on the value of the scheduling variables α and β, we switch from one configuration to another. Particular attention must be given to the intersection of operating conditions where nonlinearities exist.

For problems where the performance specification is formulated in terms of a reference model, Gain Schedul ing might not be applicable because of the increasing number of possible scheduling variables and the nonlin earities taking place at the intersection of regimes. The ModelReference Adaptive System (MRAS) has been developed to tackle this type of problem. It uses a desired reference model to tune the controller parameters by comparing its output ym to the actual output of the plant y in such a way as to minimize the output error and to maintain the feedback signals bounded. The principal challenge with MRAS is to determine the adjustment rule in order to obtain a stable system that brings the error to zero. The adjustment mechanism presented in the original formulation of MRAS is called the MIT rule dθ dt = -γ × ∂e ∂θ , [START_REF] Osburn | New developments in the design of adaptive control systems[END_REF], where e = yym is the model error and θ is a control parameter. The quantity ∂e ∂θ is known as the sensitivity derivative of the error with respect to θ and γ is the adaptation rate. This adjustment can be regarded as a gradient scheme to minimize the squared error. Various adjustment mechanisms have been proposed for MRAS [Lan84; Par81] and its stability analysis has been thoroughly conducted using Lyapunov theory [Mor79; NV80; GM87]. The MRAS framework has been demonstrated to be suitable for the control of various types of maritime vehicles, from surface tankers [START_REF] Källström | Adaptive autopilots for tankers[END_REF] to smallsize AUVs [START_REF] Sarhadi | Model reference adaptive PID control with antiwindup compensator for an autonomous underwater vehicle[END_REF]. MRAS requires the determination of the sen sitivity derivative that cannot be obtained for an unknown process (unless several assumptions are made). Adaptive methods have been extensively used following these modelbased schemes when a major part of the process model is available. Nevertheless, the limits of modelbased adaptive methods have also been well outlined [START_REF] Anderson | Failures of adaptive control theory and their resolution[END_REF], restricting their application to processes of limited uncertainty.

In Figure 2.3 a block diagram of a modelreference adaptive system (MRAS) is provided, which is composed of an inner ordinary feedback loop enclosing the process with the controller and an outer loop that includes the parameter adjustment mechanism. The adjustment mechanism is obtained either by using gradient methods or by applying stability theory. However, this knowledge of the gradient is not always available.

Modelfree adaptive control

Adaptive methods that do not rely on any mathematical model of the system are called modelfree. These methods aim at describing complex processes from sensory data only, rather than from first principle laws. The first such theories appeared in the French literature in the early 1920's [START_REF] Leblanc | Sur l'electrification des chemins de fer au moyen de courants alternatifs de fre quence elevee[END_REF]. The concept was then formalized under what is currently known as Extremum Seeking Control (ESC). The goal of ESC is to max imize a control objective function J(•) without any knowledge of its extremum y ⋆ . We present now a simple ESC procedure applied to the system (2.1) also known as perturbationbased ESC (PBESC). Consider that the control objective J(•) is to maximize the system output y. For simplicity, let's consider a largely simplified case where J(•) is a static paraboloid function of u as illustrated in Figure 2.4 and that there exists a unique

x ⋆ such that y ⋆ = h(x ⋆ ) is the extremum of the h(•) function. We assume here that (2.1) is SISO, for which we can fairly design an optimal feedback control law as: u = α(x, θ), (2.4) where θ ∈ R is a scalar parameter. The closedloop form of system (2.1) considering (2.4) is:

ẋ = f (x, α(x, θ)).
(2.5)

A straightforward first order PBESC system can be derived as: ẋ = f (x, α(x, θ + a sin(ωt))), θ = kh(x)b sin(ωt), (2.6) where (k, a, b, ω) are tuning parameters. As shown in Figure 2.4, this algorithm contains only one integrator and can be summarized as:

1. Inject some sinusoidal signal on the current control input û.

2. Measure the sinusoidal perturbation on the resulting J(•).

3. Integrate the multiplied signal to the control input û.

4. Repeat from step 1.

Because of the shape of J(•), the control û can either be in phase (when it is on the left of the optimum) or out of phase (when it is on its right). Thus, the integration ensures that we move toward u ⋆ . This simple PB ESC can be applied to processes that are subject to variations and uncertainties where the function J(•) is not static. The PBESC algorithm can track the optimal u * only if the disturbances and process parameters are changing slowly compared to the sinusoidal probing. It was shown using averaging and perturbation theories that this process can (locally) converge, under some assumption (of local optimality and smoothness of J) toward a neighborhood of u * [START_REF] Rotea | Analysis of multivariable extremum seeking algorithms[END_REF]. The PBESC algorithm is a modelfree adaptive control scheme because the parameter θ of the control law u is adjusted solely based on the system feedback.

PBESC is essentially a local optimizer and reallife dynamical systems exhibit J(•) functions with multiple peaks and discontinuities. In this case, the nonlinear version of ESC can be used [START_REF] Krstić | Stability of extremum seeking feedback for general nonlinear dynamic systems[END_REF] which consists mostly of the addition of lowpass and highpass filters. The convergence analysis of ESC has also been conducted showing that convergence to a global maximum in the presence of multiple local maxima can indeed be achieved with proper tuning of the control parameters [Nes09; Biz20]. ESC algorithms have also been applied to tune PID controllers [KK06; KK05]. There exist various other optimization techniques to find extrema (e.g. Hill Climbing, Nelder-Mead method or Simulated annealing). Today, the most commonly used modelfree algorithm is Reinforcement Learning [BT96; SB18]. The hypothesis underlying Reinforcement Learning is that, by trying multiple control inputs at random, a controller can eventually (by trial and error) build a predictive model of the system on which it is operating. The optimal control policy is hence obtained through this experimental process that leads to the wellknown exploration versus exploitation tradeoff. Some Reinforcement Learning methods use this trial and error scheme not to learn the statetooptimal actions mapping, but rather to learn the model of the process. This model can then be used for planning future actions. Over the past decades, modelfree adaptive control theory has reached a reliable level of maturity with good analysis and understanding of its main properties. Thenceforth, great efforts have been made in order to take advantage of the modelbased design, with its stability characteristics, and add to it the advantage of modelfree learning, with its fast convergence and robustness to uncertainties. The field of learningbased adaptive control was later developed having this ambition as the goal.

Learningbased adaptive control

Realworld systems are in general nonlinear and their motion equations, parameters and system measure ments are affected by uncertainty. A more realistic scheme is to consider that the process model is partially available. In learningbased adaptive controllers, modelfree algorithms are used to mitigate this lack of a complete description of the process by finding (learning) an approximate representation of the unknown parts, or by fitting (tuning) the best control parameters for a target behavior. The dynamics represented in (2.1) can be rewritten as the sum of known (f1) and unknown (f2) parts of the process: ẋ(t) = f1(t, x, u) + f2(t, x, p), y(t) = h(t, x, u), (2.7) where classical modelbased control methods can be used to efficiently control f1, and f2 can be approxi mated by modelfree learning algorithms. In other words, learningbased control methods take advantage of the fast convergence and robustness to the uncertainty of learning algorithms to approximate an unknown performance function, while applying modelbased control laws to obtain optimal performance of the system. as formalized below, where a statespace model of (2.1) is described in the Brunovsky form as:

ẋ1 = x2, ẋ2 = f (•) + u, ( 2.8)
where f (•) = f1(x(t), u(t), t) + f2(x(t), p(t), t), f1 is a known function, f2 is an unknown function, both functions are smooth over the state variables x = (x1, x2) T , and u the control signal. The unknown part of the model (f2) is estimated by a NN as: f2 = Ŵ T S(x(t)), (2.9) where Ŵ = ( ŵ1, . . . , ŵN ) T ∈ R N is the estimated vector of synapse weights of the neural network node and S(x) = (s1(x), . . . , sn(x)) T is the regressor vector, with si, i = 1, . . . , N . Given the reference model:

ẋref 1 = x ref 2 , ẋref 2 = f ref (x), (2.10
e1 = x1 -x ref 1 , e2 = x2 -v, v = -c2e1 + x ref 2 , v = -c2(-c2e1 + e2) + f ref (x ref )
, (c1, c2 > 0), Ẇ = Γ(S(e)e2σ Ŵ ), (σ > 0 and Γ T > 0). (2.11) It can be observed that u is now a function of the Brunovsky form (2.8) (i.e., modelbased information), and the remaining part is based on the neural network estimates of f2 (modelfree estimation), thus the name learningbased adaptive controller. Following this postulate, many ExtremumSeekingbased learningbased methods have been proposed in the literature, such as [XB15; HA13]. Other optimization methods such as Genetic algorithms [Dav90; Gol08] and Evolution Strategies [Wie+08; Con+18] have also been proposed in the learningbased framework. More recently, in NeuralNetwork (NN) learningbased control design, the unknown part of the model can be estimated by a NN, whose weights are obtained using some modelfree optimization procedure. Among the various Machine Learning techniques, a prominent candidate for that end is Reinforcement Learning (RL). In the next section, we present the background elements of RL and we will describe how it can be used for AUVs' learningbased adaptive control.

Reinforcement Learning

Reinforcement Learning (RL) as it is referred to today, is a study, and a problem of intelligence attempting to discover what could drive agents (natural or artificial) to behave intelligently in such a variety of ways. As pictured in Figure 2.5, RL is often framed in the form of an agent that interacts with an environment through actions, and one particular thing that the environment generates is called the reward signal which tells how well the agent is doing in the environment. The objective of RL is simple, that is to make rational decisions that are decisions maximizing a measure of utility, portrayed by the reward. Rational decisionmaking can be seen, through a particular lens, as the encapsulation of the artificial intelligence problem. Any problem that we would want a machine to do can likely be represented as a rational decisionmaking problem (like classifying images, controlling a chemical plant, or deciding what movies to recommend on Netflix) and RL provides a framework to design solution methods for this class of problem. For this reason, great efforts were made in scaling RL, by both the academic and industrial communities. RL has now proved its efficiency in many applications in science and engineering with the remarkable advantage that is, it enables agents to maximize their cumulative rewards through online exploration and interactions with unknown (or partially unknown) and uncertain environments, which is regarded as a variant of datadriven adaptive optimal control theory. Nowadays, RL methods are coupled with deep machine learning methods, allowing their use in applications where no solutions were previously available. The goal of this Section 2.2 is to provide an overview of the aforementioned notions that are the building block of modern RL and why it is suitable for the considered control problem. The fundamentals of RL will be presented, and some theorems and proofs might be described later in the document and will be cited throughout when appropriate. RL is defined as a class of solutions methods for learning how to interact with the environment from raw experience. At the heart of RL, we start with an agent and an environment. The term agent implies some agency, as the agent gets to take action by following its behavior (policy). The agent is said to be able to capture a snapshot of itself (state) within the environment. In order to improve its performance, the agent can learn either from interactions generated by its current policy (OnPolicy) or from interactions generated by any policies (OffPolicy). These past interactions would need to be held in some kind of memory unit (Replay Buffer) which we would often appraise to refresh our beliefs. From these past interactions, the agent has to extract and build meaningful understandings (model) of the environment response (value function). Using this knowledge, the agent can then take the best action to take (optimal policy). RL is a framework allowing us to learn the optimal policy in a modelfree setting, where the model of the process is partially or fully unknown. For this reason, using RL in the context of underwater robotic applications is a pertinent choice given that underwater vehicles have a limited observation ability of the environment and have to compensate for various disturbing forces that are often not measurable while having limited computational resources.

Key concepts of RL

Let's first define a set of key concepts required for the understanding of RL.

The agent is acting in what we call an environment. The environment can be represented by different means, such as an actual physical space where the agent is evolving, or by something more abstract such as a set of rules. In all cases, how the environment reacts to certain actions is defined by a model, which may be known or not. In RL, the model is a descriptor of the environment that can be used to learn of infer how the environment would react to and provide feedback to the agent. The agent is defined at every time in one of many states of the environment s ∈ S and chooses to take some actions among a predefined set a ∈ A, whose execution make the agent transit to another state s ′ ∈ S. Following this transition, the environment generates, in addition to a new state, a reward signal r ∈ R which transcribes numerically to how good this choice of actions was with respect to the goal of future reward maximization.

The action selection procedure is represented by what is called the policy π(s). It is a mapping from the state s to actions a and can then be either deterministic or stochastic. For the goal of reward maximization, each state is associated with a value, denoted as statevalue V (s), which assesses the expected amount of future reward accessible from the considered state and by acting accordingly to the policy. In other words, the statevalue quantifies how good a state is. In general, both the policy and statevalue are what we try to learn in RL.

The interaction between the agent and the environment involves a sequence of actions, states, and rewards observed at the time, t = 1, 2, . . . , T . During the learning phase, the agent is expected to collect knowledge about the environment, build the optimal policy accordingly to its current understanding, and make decisions on which actionmaking strategy to take so as to efficiently learn the best policy. Now, we can label the state, action, and reward at the timestep t as st, at, and rt respectively. In RL, the sequence of interaction is fully described by an episode (also denoted as trial or trajectory) which ends at the terminal state sT :

s1, a1, s ′ 1 , r1, s ′ 1 , a2, s ′ 2 , r2, . . . , sT . (2.12)
The model is a description of the environment, from which we can learn or infer how it will react to the agent's actions. The model consists of two parts: the transition probability function P and the reward function R. A transition is defined as follows: from a state s, the agent takes the actions at, transiting to a new state s ′ and obtaining r. A transition step is therefore framed as the tuple s, a, s ′ , r . The transition probability function P accounts for the probability of transit from state s to state s ′ after taking action a while generating the reward r. Formally, this function is defined using the symbol P (for probability distribution) as follows:

P (s ′ , r|s, a) = P[St+1 = s ′ , Rt+1 = r|St = s, At = a]. (2.13)
The statetransition function can then be defined as a function of P (s ′ , r|s, a):

P a ss ′ = P (s ′ , r|s, a) = P[St+1 = s ′ , Rt+1 = r|St = s, At = a] = r∈R P (s ′ , r|s, a).
(2.14)

Similarly, the reward function predicts the reward generated by the actions:

R(s, a) = E[Rt+1|St = s, At = a] = r∈R r s ′ ∈S P (s ′ , r|s, a). (2.15)
Following the above definition of the transition probability function, the policy, which is a mapping from the state s to actions a, is defined as:

π(s) = a (Deterministic) π(s|a) = Pπ[A = a, S = s] (Stochastic). (2.16)
In order to measure how good it is for an agent to be at a given state, the statevalue function estimates the expected future cumulative reward obtained starting from the considered state. This quantity is denoted as the agent return Gt, and is equal to the total sum of discounted future rewards going forward:

Gt = Rt+1 + γRt+2 + • • • = ∞ k=0 γ k R t+k+1 .
(2.17)

The parameter γ, known as the discount factor, is incorporated in the return such as 0 < γ < 1. As the discount factor is less than 1, this factor penalizes far future rewards by shrinking their value toward 0. The addition of this parameter has two effects:

• Discounting provides mathematical convenience by bounding the infinite summation. By considering such a design, we do not need to worry about infinite loops in the statetransition graph.

• Depending on the value of γ, we can control how far in the future we look in the resulting statevalue function. With a γ ∼ 0, the agent is shortsighted and puts way more emphasis on the shortterm in his decisionmaking. With a γ ∼ 1, the decisionmaking takes into account very far predictions, which do not have any immediate benefits. The value of the discount factor is thus a parameter to tune depending on the agent, environment, and desired performance. The discount factor has another objective, which is to reduce the variance of the return cf Eq. (2.17). There is uncertainty about the far future state of the environment because the further we look into the future, the more stochasticity we accumulate and the more variance the return will have. Under these conditions, estimating the return can be very difficult and the choice of discount factor should therefore be influenced by the envi ronment's complexity. In addition, in Figure 2.6 we can see that for each value of γ, after a certain threshold of step number, the generated reward will have little to no impact on the discounted return. For example, with γ = 0.90 (i.e. red line in Figure 2.6), this threshold is approximately equal to 50. Let's now consider that the length of the episode exceeds this threshold. The resulting actions will then be the best ones for only a portion of the episode. Thus, the agent is biased because what we believe are optimal actions, are mostly optimal only over this reduced period of time, and thus could be suboptimal (or even far from it) with regards to the entire episode. Potentially, better future states could be accessible from different early actions.

For this reason, the discount factor is a parameter to tune also according to the length of the episode. One designer should ensure that the entire trajectory will have a notable impact on the computation of the return. At the same time, the discount factor should not be too close to 1 so as to make the return estimation feasible. The design of these parameters (i.e. whether or not to set the discount factor based on the episode length or the other way around) is a choice to make based on the environment, the task, and the desired usage of the resulting policy.

The more we look into the future when estimating the QValue function (i.e. γ → 1), the more stochasticity we accumulate and the more variance the estimate will have. Although a discount factor lower than 1 notably stabilizes the learning process, it should not be too small in order to avoid a shortsighted agent. In our application, the control system of the AUV needs to take into account future states of the process because due to the disturbance, what we believe to be optimal with respect solely to the current timestep (i.e. γ = 0) can be far from it or even dangerous for the platform in the long term.

The statevalue function can now be defined as the expected return starting from the current state:

Vπ(s) = Eπ[Gt|St = s]. (2.18)
The statevalue function measures only the quality of a given state. As the goal is to take the best action possible, we would like to include the policy in this evaluation through what we could call an "actionvalue" function. The quality of a pair of state and action (s, a) is denoted as the Qvalue function ("Q" for quality) and defined as:

Qπ(s, a) = Eπ[Gt|St = s, At = a]. (2.19)
The Qvalue function (2.19) measures how good it is for the agent to follow the current policy starting from the evaluated state. Since we follow the target policy π, we can recover the statevalue function using the probability distribution over the possible actions:

Vπ(s) = a∈A Qπ(s, a)π(a|s). (2.20)
Finally, as the goal of RL is to learn the best policy possible, we will be interested in the optimal value of the aforementioned functions. In this document, the optimal value functions will be denoted as V * (s) and Q * (s, a) and are defined as:

V * (s) = max π Vπ(s), Q * (s, a) = max π Qπ(s, a). (2.21)
Accordingly, the optimal policy π * achieves optimal value functions:

π * = arg max π Vπ(s), π * = arg max π Qπ(s, a). (2.22)
In the context of adaptive control of AUVs, the RLbased optimal policy (2.22) can consists of estimating at each timestep the best control inputs to apply or the best controller parameters to use for the objective of minimizing the regularization error. This is possible despite the unobservable current disturbance as the StateValue function (2.20) captures and encircles all of the things in which an agent interacts with an envi ronment, including the aforementioned uncertainties. In the following, we will present how we can formalize mathematically this optimization procedure in order to apply it to our AUV application.

Markov decision process

Mathematicians have provided different tools to frame the problem of RL, and historically, the Markov Decision Process (MDP) describes a framework to solve RL. MDPs are discretetime stochastic control processes and almost all RL problems can be formalized as MDPs (essentially all processes where we can have feedback and thus derive a reward signal). As pictured in Figure 2.7, we can see MDPs as extensions of Markov Chains, the difference being the addition of actions (i.e. choice) and reward (i.e. optimization objective). A stochastic control process is considered an MDP if each state of the process holds the Markov Property. A state St+1 is said to be Markovian if and only if:

P[St+1|St] = P[St+1|S1, . . . , St]. (2.23)
MDPs have been studied for decades, and various solutions methods have been proposed to solve them. In the following, we will introduce the methods that are used in our proposed methods. MDPs are expressed as the tuple S, A, T, R , in which:

S is the set of possible states;

A is the set of actions that can be executed by the agent; T is the transition function that defines the probability of reaching a successor state s ′ ∈ S, from the application of action a ∈ A in a state s ∈ S; R is the reward function.

Bellman equations

If an optimization problem can be decomposed into various subproblems, Dynamic Programming (DP) [Ber95; Put94] can be used to solve it. In this case, the Bellman equations [START_REF] Bellman | On the Theory of Dynamic Programming[END_REF] can be used as a substitute for the model of the process. It is the fundamental mathematical tool used to solve MDPs. It allows us to decompose a dynamic optimization problem into a sequence of subproblems, for which we can prove that an optimal so lution exists. Here, these equations allow us to express the relationship between the value of a state and the value of the successor state. In practice, the Bellman equations allow us to rewrite the statevalue function as the immediate reward plus the discounted future value function:

V (s) = E[Gt|St = s], = E[Rt+1 + γRt+2 + γ 2 Rt+3 + . . . |St = s], = E[Rt+1 + γ(Rt+2 + γRt+3 + . . . )|St = s], = E[Rt+1 + γGt+1|St = s], = E[Rt+1 + γV (St+1|St = s)].
(2.24)

Similarly, the Qvalue function can be decomposed as:

Q(s, a) = E[Rt+1 + γV (St+1)|St = s, At = a], = E[Rt+1 + γEa∼πQ(St+1, a)|St = s, At = a]. (2.25)
We know that from a given state s, we can choose actions from multiple possibilities determined by the stochastic policy π(a|s), and each of these actions is associated with a Qvalue. By multiplying the possible actions with the Qvalue function and summing them, we can derive an indicator of how good it is to be in that given state, which is also known as the Bellman Expectation equation:

Vπ(s) = a∈A π(a|s)Qπ(s, a). ( 2.26)
From this equation, we can assess the value of a state by multiplying the possible actions with the action value function and summing them. Similarly, for a list of possible actions, there is a list of possible next states s ′ which are associated with a statevalue function V (s ′ ), a transition probability function P a ss ′ of where the agent could end up based on the actions, and a reward R a s generated by taking the action. By summing the reward and the transition probability function associated with the statevalue function, we can now derive an indicator of how good it is to take the action, given a state:

Qπ(s, a) = R(s, a) + γ s ′ ∈S P a ss ′ Vπ(s ′ ). ( 2
Vπ(s) = a∈A π(s|a)(R(s, a) + γ s ′ ∈S P a ss ′ Vπ(s ′ )), and
Qπ(s, a) = R(s, a) + γ s ′ ∈S P a ss ′ a ′ ∈A π(a ′ |s ′ )Qπ(s ′ , a ′ ).
(2.28)

In order to solve an RL problem that is framed as an MDP, we are interested in the optimal values of the Bellman Expectation Equations (2.28) rather than computing the expectation following a policy:

V * (s) = max a∈A Q * (s, a), Q * (s, a) = R(s, a) + γ s ′ ∈S P a ss ′ V * (s ′ ).
(2.29)

Again, by substitution, we can derive what are known as Bellman Optimality Equations:

V * (s) = max a∈A (R(s, a) + γ s ′ ∈S P a ss ′ V * (s ′ )),
and

Q * (s, a) = R(s, a) + γ s ′ ∈S P a ss ′ max a ′ ∈A Q * (s ′ , a ′ ).
(2.30)

The field of RL then consists mainly in designing solution methods that determine the Bellman Optimal Equa tions (2.30). In Section 2.2.4, we proposed to classify solutions methods based on the nature of the inter actions with the environment, generating the rewards that are mandatory to estimate the above functions (2.30). We propose to classify methods in such a manner because we are mainly interested in mobile robotic applications. In order to interact with their environment, these types of agents have to directly explore their surroundings. Depending on the system and environment, exhaustive interactions (such as required in gen eral by RL methods to be efficient) might not be possible or efficient because: the policy (especially at an early stage of training) can be dangerous for the robotic platform because of hardware vulnerability; the operating cost can be too high to consider thorough exploration of the environment due to time restrictions; the process uncertainty can be so large that the Markov property does not hold anymore. With the proposed classifica tion, we can easily identify which class of solution methods to favor according to the primary knowledge of the robot and operating conditions.

To summarize, the Bellman equations can be used as a substitute for the model of MDPs and RL is a tech nique to solve a set of Bellman equations. When determined, the adaptive control problem of AUVs which can be framed as an MDP can therefore be solved using these functions despite the model of the associated MDP not being a priori known. Accordingly, RL allows us to obtain a solution to the adaptive control prob lem of AUV that is not possible when using modelbased adaptive control theory solely. Next, we present a classification of solution methods for estimating these functions.

Classification of solution methods

Reinforcement Learning is a very ambitious problem definition as it is trying to capture and encircle all of the things in which an agent interacts with an environment. Maybe the complexity of the environment is so great that we can not even imagine how to build a system that would understand how to take action just from a stream of observations. To even start thinking about how to solve such a hard problem, the first step we can take as human beings is to decompose (when it is possible) that big hard problem into pieces that work together to solve that hard problem. By taking a look at the decomposition that might be inside the agent's head, we can ask what form that decomposition could take. This decomposition commonly consists of some building blocks and solution methods in RL are different choices of using them, and some of the most common pieces that people use when they are trying to put solution methods together are:

• is there a model in the system, that is something that is explicitly trying to predict what will happen in the environment?

• whether or not that solution has a value function, that means is it trying to predict, explicitly, how much reward it will get in the future?

• or does it have a representation of a policy, that means something that is deciding how to pick actions, is the decisionmaking process explicitly represented? These three pieces are the most common building blocks, and as illustrated in Figure 2.8 with the three classes of RL methods and their attribute, solution methods in RL are different choices of whether or not to use them. The fundamental idea of this decomposition is to ask how could we solve problems, that can be decomposed into smaller problems, where we are trying to figure out how to take actions just from this stream of observation. The first step of this decomposition is to say that we have to learn, the system has to learn for itself. Learning is required because it is good at achieving good performance in large and complex environments.

This step gives rise to all the other pieces because now we might ask what should we be learning, and what learning does even mean? In this context, learning might mean that we are trying to update the parameters of some system which is then the thing that actually picks the actions. Those parameters could be representing anything, a Value function, a model, or a policy. In that sense, there is a lot of commonality between these building blocks in whatever is being represented there, the thing which is being learned with the ultimate goal of maximizing the reward. Solution methods in reinforcement learning are inherently different choices of whether or not to use them. In this thesis, we use RL methods that exploit each of these building blocks.

For this reason, a concise presentation of these concepts is provided in Sections 2.2.4, 2.2.4 and with an extended focus in Section 2.2.4 on the class of solution methods denoted as Policy Gradient that is the main component of the contributions of this thesis.

Modelbased reinforcement learning

When the model P (s ′ , r|s, a), see Eq.(2.13), is fully known, the RL problem can be framed as a planning problem. In modelbased methods, the rewards are not the results of the interaction with the environment but they are given by the model that represents the dynamics of the associated MDP. This can be seen as a deterministic optimization procedure, where we know for each action where the agent will transit in the environment and the associated statevalue, and the most common approach for this is called Dynamic Programming (DP) [Ber95; Put94]. This method can be used when the model is known to solve either: a prediction problem (i.e. policy evaluation); or a control problem (i.e. policy improvement). The policy evaluation consists in assessing how good a policy π is, given an MDP. The idea is to start with an initial state value function V1 with a value of 0 and then to update it to V2 using the Bellman backup (2.24). This process is repeated many times, until convergence to Vπ, the statevalue function associated with the evaluated policy:

V1 → V2 → • • • → Vπ. (2.31)
The V (s) function is updated by using synchronous backups and all states at every step with:

Vt+1(s) = Eπ[r + γVt(s ′ )|St = s] = a∈A π(s|a) s ′ ∈S,r∈R P (s ′ , r|s, a)(r + γVπ(s ′ )). (2.32)
This is illustrated in Figure 2.9 with the backup diagram of a DP procedure where from a state st, all following states are considered in the StateValue estimate Eq. (2.32). The policy improvement consists in generating a better policy π ′ . In fact, with this evaluation (2.32), we have assessed the performance of a given policy π but have not found the best ones for our environment. To improve the policy, we can act greedily with respect to the determined statevalue function. This is possible using a onestep look ahead to determine the action which maximizes the Qvalue function: π ′ (s) = arg max a∈A Qπ(s, a). By acting greedily we obtain a better policy π ′ > π:

Qπ(s, a) = E[Rt+1 + γVπ(St+1)|St = s, At = a] = s ′ ∈S,r∈R P (s ′ , r|s, a)(r + γVπ(s ′ )). (2.33)
In order to obtain the optimal policy, one can intuitively understand that we will need to combine both policy evaluation and policy improvement. This is exactly the goal of what is denoted as Generalized Policy Iteration (GPI) algorithm which iteratively alternates between these tasks:

π0 evaluation -------→ Vπ 0 improvement ---------→ π1 evaluation -------→ Vπ 1 . . . improvement ---------→ π * evaluation -------→ Vπ * (2.34)
With GPI, the statevalue function is approximated repeatedly so as to be as close as possible to the true value of the current policy, and at the same time, the policy is improved repeatedly to reach optimality. We can ensure that this iterative process converges toward the optimal policy, in other words, that the value of the improved policy V ′ π is better than the previous one Vπ because: Modelbased RL can be applied essentially when we have full knowledge of the considered process (e.g. when we can write down as equations the exact evolution of the process dynamics at every timestep). In this case, the model of the MDP (2.13) is known and can be used within the GPI method. However, in our application, the AUV is facing sea current disturbances that are not measured. The effect of this disturbing force will also vary depending on the AUV's relative orientation to the sea current which we don't know how to model. The control of AUVs is also associated with various uncertainties that we are yet able to model, including water temperature and salinity affecting the vehicle buoyancy, thruster, and propeller power loss which can make the vehicle underactuated or body wrench disturbances. Modelbased RL is thus not applicable to our application because it is impossible here to determine beforehand the MDP's transition probability function (2.13) given the aforementioned uncertainties.

Qπ(s, π ′ (s)) = Qπ(

Valuebased reinforcement learning

AUVs have to evolve in highly uncertain environments where we are still not able to model the undergoing natural phenomena. The model P (s ′ , r|s, a) of the MDP is therefore unknown and modelbased RL can not be used. In this context, we can still solve the RL problem by using Bellman value functions as a substitute for the model without the need to model the environment dynamics. In fact, the statevalue function allows us to estimate the agent return as the discounted cumulative reward γ k Rt+1. This is illustrated in Figure 2.10 with the backup diagram of MC where starting from a state st, all the future states until the terminal state T are considered in the return estimate. The empirical return is then estimated as:

V (s) = E[Gt|St = s].
V (s) = T t=1 [St = s]Gt T t=1 [St = s] , ( 2.36)
where [St = s] is the binary characteristic function of a subset of a set. This approximation procedure can be easily extended to the Qvalue function as:

Q(s, a) = T t=1 [St = s, At = a]Gt T t=1 [St = s, At = a]
.

(2.37)

The most common MCbased method (2.10) is denoted as Markov Chain MonteCarlo (MCMC) and can be summarized as follows:

1. Improve the policy greedily with respect to the current Qvalue function: π(s) = arg max a∈A Q(s, a).

2. Generate a new episode (until termination) with the new policy π (using ϵgreedy strategy).

3. Estimate the Qvalue function using the samples from the new episode with Therefore, the MCMC method (similar to GPI) iterates between evaluation and improvement:

qπ(s, a) = T t=1 ([St = s, At = a] T -t-1 k=0 γ k R t+k+1 ) T t=1 [St = s, At = a] . ( 2
π0 evaluation -------→ Q∼qπ Qπ 0 improvement ---------→ π∼greedy(Q) π1 evaluation -------→ Q∼qπ Qπ 1 . . . improvement ---------→ π∼greedy(Q) π * evaluation -------→ Q∼qπ Qπ * . (2.39)
The key difference between DP and MC methods is that we are changing how we are evaluating the policy and estimating the V (•) or Q(•) function. As illustrated in Figure 2.9, the sampling return from DP is different from the Bellman equations because we just use the lookahead to the next state st+1, and use it to update our value estimate of the current state st. In other words, we update our value estimate only for visited states. Contrary to DP, we don't need to know the states ahead of time, we can just discover them as we interact with the environment, which makes MC much more applicable when the action space dimension is large.

MC methods learn from complete episodes, they can only be applied to episode MDPs. In addition, since we are averaging the value estimate over the full episode, MC methods are often subject to high variance. This is due to the fact that samples collected within the same episode are highly correlated, and updating our estimate using such samples tends to concentrate our updates to specific parts of our estimate, driving us away from the real value of the Qvalue function. The real value of a very good action along the episode will be reduced when averaged and similarly, the value of poor action will be enhanced. For this reason, MCbased methods tend to require a large number of iterations before converging to a satisfying solution.

When using MCbased methods such as MCMC, in order to improve the policy the agent needs to visit the exact same states multiple times. We need to interact with the environment a lot. This is not possible in our considered application. Despite having access to the simulation of underwater vehicles and environments, the amount of simulated data remains notably small given the considered continuous action and state spaces that are by definition untrackable. Thus, it is impossible to perform an exhaustive exploration of the underlying spaces or to go back to promising states, which results in the failure of MCbased methods.

To reduce the variance of MCbased Valuebased methods, we would like to update our value function using uncorrelated samples. This is exactly the purpose of Temporal Difference (TD) learning [Sut+09; SM11; SMW16]. The TD learning methods are modelfree methods similar to MCbased methods, but with the advantage of being able to learn the value function from incomplete episodes. This is possible by performing the bootstrapping trick which consists in updating targets values with regards to existing estimates rather than exclusively relying on actual rewards and complete returns (i.e. MCbased methods). In other words, with TD learning we update a guess toward a guess. The principal idea in TD learning is to update the statevalue function towards an estimated discounted return V (s) = Rt+1 + γV (St+1) (known as TD target). This is illustrated in Figure 2.11 with the backup diagram of TD where from a state st, only the next state is considered in the estimation of the return. The TD learning methods are associated with a number of parameters: Gamma (γ) is the discount factor previously introduced in Eq. (2.17); Lambda (0 < λ < 1) is the credit assignment variable controlling how deep we look into the Markov Chain; and Alpha (α) is the learning rate, controlling how much of the error we accept and adjust our estimate towards.

TD learning methods can further be divided into two groups: onpolicy and offpolicy. The first one refers to methods where the policy is improved using samples generated by the same policy while offpolicy methods use samples generated by any policies to improve the current one. The initial onpolicy method to appear in the literature is called SARSA (which expands to State, Action, Reward, State, Action), an iterative process similar to GPI. With SARSA, the Qvalue function is updated using bootstrapping (i.e. no need for episode termination) and samples are generated by the same policy according to the TD error:

Q(St, At) On-policy ← ------- SARSA Q(St, At) + α(Rt+1 + γQ(St+1, At+1) -Q(St, At)).
(2.40)

The SARSA algorithm can be summarized as follow:

1. Initialize t = 0 and set an initial state S0. (d) Update the Qvalue function as:

Q(St, At) ← Q(St, At) + α(Rt+1 + γQ(St+1, At+1) -Q(St, At)).

Repeat from step 2.

With SARSA, for each timestep, we choose the next action according to the current policy. If we always take action according to the same policy, there will be no performance improvement possible (i.e. we are purely doing exploitation). Therefore, the capacity of SARSA to improve the agent return lies in the exploration ability of the policy used to generate the samples. In practice, the ϵgreedy strategy is used, allowing the agent to take random actions along the trajectory. In addition, as we update our Qvalue estimate, the resulting new policy might produce different action a ′′ t from that same state St. For this reason, and because the efficiency of the ϵgreedy strategy drastically reduces against continuous state and action spaces, SARSA requires an enormous amount of iteration to converge as it needs to visit each possible state multiple times to improve the policy. To strengthen the sample efficiency of this type of approach, we would want to use samples from any policies to update our Qvalue estimate. This is exactly the objective of QLearning which update the Qvalue function by assuming the use of the optimal policy. This is known as offpolicy TD learning and we describe this process in Section 2.2.4. Within one episode QLearning works as:

(St, At) ← Q(St, At) + α(Rt+1 + γ maxa∈A Q(St+1, a) -Q(St, At)).
5. Set t = t + 1 and repeat from step 2.

Because we use bootstrapping in TD learning, the difference among the solution methods becomes a tradeoff between bias and variance, which has to be made according to the characteristics of the MDP:

• The return Gt = Rt+1 + γRt+2 + • • • + γ T -1 RT (from MCMC) is an unbiased estimate of Vπ(St).
• The TRUE TD target Rt+1 + γVπ(St+1) is also an unbiased estimate of Vπ(St) (but we don't have a heuristic to tell us the true value...).

• The bootstrap TD target Rt+1 + γV (St+1) is a biased estimate of Vπ(St) (...so we can only substitute in our best estimate so far).

• Using bootstrap, we introduce bias in our estimate and in our target because of γV (St+1) = γVπ(St+1).

• TD target is much lower variance than the return because:

the return depends on many random actions, transitions, and rewards, while the TD target depends on one random action, transition, and reward.

We have been able to see that the MCbased return is unbiased while the TD target is biased. This means that the MCbased asymptotic prediction error V E(WMC ) is typically smaller. Nevertheless, the TD learning asymptotic prediction error V E(WT D ) can be bounded as follows:

V E(WT D ) ≤ 1 1 -γ V E(WMC ) = 1 1 -γ min W V E(W ) (2.41)
where γ is the discount factor introduced in Section 2.2.1. For example, if γ = 0.90 it roughly corresponds to V E(WMC ) being 10 times smaller than V E(WT D ), but at most. This, in addition to the fact that TD learning converges faster, is why TD learning is leading the field of RL. A value of γ = 0.99 is traditionally used to ensure an asymptotic error at most 1% bigger than the MCbased one, despite the TD estimate being biased. The methods described so far allow us to build an estimate of the QValue function. The actions are then determined either by planning or by pooling. These methods are advantageous in the case of processes with limited uncertainties such as chess where: we know exactly at each timestep the state of the board and the set of possible moves of the opponent, and we know that new pieces can not appear on the board and the result of each individual move. On the other hand, a little change in the environment that was not taken into account in the QValue function estimate will result in the total failure of these methods. In fact, these methods are not satisfying when facing process variation because as they only rely on the QValue function estimate for decisionmaking, the resulting solution is valid only on the exact same set of states. This is never guaranteed in our AUV application. The vehicle is evolving in a dynamic environment where it is facing unobservable process variation and the state and action spaces are continuous. For these reasons, we would rather not rely only on the QValue estimate in our decisionmaking (as it will never be able to encompass all of these uncertainties) but instead learn in addition the behavior policy directly. This class of solution methods is denoted as Policy Gradient. The learned policy π θ (at|st) estimates the best action to take given a state. If the policy is learned, despite not being able to fully explore the state of space, the resulting actions will encompass the dynamics of the environment because it is represented by a parameterized function.

Facing unseen states, the learned policy function would estimate meaningful actions (as the policy function estimate would be defined on a continuous domain and uncorrelated states are still similar to some extent given that they are still associated with the same physical process). On the other hand, with modelbased and valuebased methods, in this case, the resulting actions could essentially be meaningless because with these techniques we only learn to fit one particular environment, and any change in the environment will not be reflected in the model. In other words, with modelbased and valuebased methods we can not recover from unseen process variation because they are not encompassed in the QValue estimation. We can expect that, against unseen states, the resulting actions to be at best random. The decisionmaking obtained when learning the policy explicitly is less sensitive to process variation because since we sample the actions from a policy directly, the resulting solution maintains a physical meaning which makes it most likely to be better than the one sampled from the QValue estimate (i.e. if we learning a policy to control an AUV against a sea current that always comes from one direction when facing another current, the policy is more likely to estimate a proper action compare to modelbased and valuebased methods because there, change in the current disturbance are totally not taken into consideration in the action sampling). For this reason, Policy Gradient methods have been privileged in this thesis.

Policy Gradient

The class of solutions methods presented so far aim to compute (with modelbased methods) or learn (in the case of valuebased methods) the Qvalue function and then select actions accordingly. Instead, policy gradient methods target modeling and optimizing the policy behavior π directly (i.e. the decisionmaking).

The policy is traditionally [START_REF] Sutton | Policy Gradient Methods for Reinforcement Learning with Function Approximation[END_REF] represented by a parameterized function with respect to θ, π θ (a|s). The value of the reward function J(θ) depends on this policy and thus we can use various algorithms to optimize θ which achieves the best reward. The reward function is defined as the expected return and the parameters θ are optimized with the goal of maximizing the reward function. In discrete space, it is defined with S1 the initial state as

J(θ) = Vπ θ (S1) = Eπ θ [V1],
or in continuous space as:

J(θ) = s∈S d π θ (s)V π θ (s) = s∈S d π θ (s) a∈A π θ (a|s)Q π θ (s, a), (2.42)
where d π θ (s) is the stationary distribution of Markov chain for π θ . For recall, the stationary distribution theorem (sometimes called the Fundamental Theorem of Markov Chains) states that for a very long random walk along within a Markov chain, the probability to end up at some vertex v is independent of where we started the random walk [START_REF] Van Ravenzwaaij | A simple introduction to Markov Chain Monte-Carlo sampling[END_REF]. All of these probabilities in sum are called the stationary distribution of the random walk and are unambiguously determined by the Markov chain. Policy Gradient approaches consist in moving the parameter, θ, of the current behavior policy π toward the direction suggested by the gradient of the reward function ∇ θ J(θ) to find the best θ that maximizes the return. However, computing the gradient of (2.42) is not straightforward. The reward function depends on both the current policy (that is directly determined by π θ ) and the stationary distribution of states following the target policy (that is indirectly determined by π θ ). Given that most of the time the model (with respect to modelbased RL theory) is unknown, it is difficult to estimate the effect on the state distribution by a policy update. The Policy Gradient theorem [START_REF] Richard | Reinforcement learning an introduction Second edition[END_REF] provides an effective reformulation of (2.42) as:

∇ θ J(θ) = ∇ θ s∈S d π θ (s) a∈A Q π θ (s, a)π θ (a|s) = ∝ s∈S d π θ (s) a∈A Q π θ (s, a)∇ θ π θ (a|s). (2.43)
Using gradient ascent, we can find the best parameters θ that produce the highest return. The proof of the Policy Gradient theorem (2.43) is provided in the appendix. The policy gradient lays the theoretical founda tion of many policy gradient methods as the resulting gradient (2.43) has no bias, but high variance. With this formulation, the parameters θ are updated so as to increase the probability to take the evaluated actions proportionally to their associated Qvalue function. This update essentially makes good trajectories (as mea sured by the Qvalue function) more likely and bad ones less likely. However, this is not always guaranteed. Let's imagine that all rewards are large positive numbers, so maybe the best trajectory is associated with a value of 1 million and ten, and the worst trajectory is associated with a value of 1 million and one. With this vanilla formulation (2.43), we will not have the aforementioned update behavior. We will actually have all trajectories becoming more likely as illustrated in Figure 2.12, where the probability to take each action is increased (proportionally to their associated QValue) despite action 1 being the best one. This is not the desired policy update behavior and in practice when using this formulation (2.43), it will not work. For this reason, the development of policy gradient methods was first focused on reducing this variance while keeping the bias unchanged. Intuitively, we would want to make trajectories more likely if they are better than the average one and make them less likely otherwise. This is equivalent to taking our rewards and subtracting the average reward (called baseline and noted b when subtracted) from them, which will result in exactly the aforementioned update behavior. This procedure is actually very common in statics and is possible only if the expectation of the baseline is null. In fact, we know that for a given samplebased estimator, if we can subtract off something holding zero in expectation, then the resulting estimate will have less variability between groups of samples because we have subtracted a baseline that we know is zero, but might not be zero for a finite number of samples. We can easily prove that subtracting a baseline is valid as follows:

E[∇ θ log π θ (τ )b] = π θ (τ )∇ θ log π θ (τ )bdτ, = ∇ θ π θ (τ )bdτ → using the identity: π θ (τ ) = ∇ θ log π θ (τ ) = ∇ θ π θ (τ ) = b∇ θ π θ (τ )dτ = b∇ θ 1 = 0.
(2.44)

This demonstration (2.44) shows that subtracting a baseline is unbiased in expectation. By doing this, the resulting estimator is still correct and in fact will result in a more accurate policy gradient. By using causality (i.e. sum rewards only from the current timestep until the end) and a baseline, these tricks made policy gradient go from an algorithm that never works to an algorithm that sometimes works. In our case, subtracting the average reward will reduce the variance of the estimator of the reward function gradient. This means that for the same number of samples, we will get a more accurate estimate. In practice, the common baseline is the statevalue function V π = a∈A Q π (s, a)π(a|s). By doing this, the parameters are updated proportionally to their associated difference between the Qvalue and statevalue functions, which is denoted in the field of RL as Advantage function A(s, a) = Q(s, a) -V (s). The gradient of the reward functions becomes:

∇ θ J(θ) = Eπ [(Q π (s, a) -V (s))∇ θ log π θ (a|s)] = Eπ [A(s, a)∇ θ log π θ (a|s)] . (2.45)
This formulation (2.45) is the principal component of the majority of policy gradient methods given that esti mating the Qvalue function implies that the statevalue function is available (i.e. it can be used as a baseline in every case). The value of the Advantage function is positive if the evaluated Qvalue is higher than the expected one, otherwise, it is negative. As illustrated in Figure 2.13 with the new associated gradient, the trajectories that are better than the average will be made more likely which is exactly the desired update behavior described earlier. the estimator variance. Again, the values used here were chosen only so as to illustrate that when using the Advantage function, the resulting policy gradient update is much more effective.

The first policy gradient method to appear in the literature is called REINFORCE and exploits this baseline trick to reinforce the good actions and push down the probabilities of lower actions until we reach the optimal policy. The original REINFORCE algorithm (also known as MonteCarlo policy gradient) [START_REF] Williams | Simple statistical gradientfollowing algorithms for connectionist reinforce ment learning[END_REF] relies on an estimated return by MonteCarlo methods using episodes samples to update the policy parameters θ. In REINFORCE, a trajectory will be defined as a sequence τ = State, Action, Rewards, . . . ST . A trajectory is more flexible than an episode because we do not have a length restriction, meaning that the algorithm search for optimal policies for both episodic and continuing tasks. The return Gt of a trajectory is measured again as the sum of discounted future reward (2.17). The algorithm can then be summarized as follows:

1. Initialize θ at random.

2. Use π with parameters θ to generate a trajectory τ = S1, A1, R2, S2, A2, . . . , ST .

3. For t = 1, 2, . . . , T :

(a) Estimate the return Gt.

(b) Update the parameters as: θ ← θ + αγ t Gt∇π(At|St, θ).

The REINFORCE algorithm only learns the parameters θ and computes the expected return directly from the generated trajectory (thus the term MonteCarlo policy gradient). In order to reduce the resulting variance of this MonteCarlo return estimate, we could instead also learn the value function using TD learning. When the policy and the value function are both learned, the resulting class of solutions methods is named Actor Critic algorithms, and this is the principal RL method used in this thesis because they allow us to use TD learning (which we show is faster to learn to compare to MC and does not need the MDP model contrary to DP) and to learn a policy as well as the value functions (which is more robust to process variations as discussed at the end of Section 2.2.4. As illustrated in Figure 2.14, the ActorCritic architecture is composed of two components:

• A Critic aims at learning the parameters w of a value function estimate (depending on the algorithm it can be the statevalue or the Qvalue function).

• An Actor whose parameters are optimized in the direction suggested by the Critic represented as the policy π θ .

The ActorCritic algorithm can then be summarized as: where αw, α θ are learning rates for the policy and value function updates respectively. Both REINFORCE and the previous ActorCritic methods are onpolicy because the training samples are generated by the same policy that we try to optimize for. This means that we must collect additional samples, by actually running the policy in the environment every time we modify the policy parameters. In fact, as illustrated in the REINFORCE algorithm above, the gradient is an expected value under π θ (τ ):

∇ θ J(θ) = E τ ∼π θ (τ ) [∇ θ log π θ (τ )r(τ )]. (2.46)
This causes troubles because that means that every time has to compute the gradient, we have to sample from π θ (τ ). At every step of training, we have to generate new samples and throw out the old samples. Because of that, the exploration ability is low and a large number of iterations is required for convergence. We would prefer to derive an offpolicy approach where:

1. we do not require full trajectories to update the policy, allowing us to reuse any past episodes; 2. the sample collection can follow any policy, different from the target policy, improving exploration. Now let's derive an offpolicy formulation. The policy that is used to interact with the environment is a known policy, denoted as β(a|s). The reward function is then defined as the sum of rewards over the state distribution defined by that policy:

J(θ) = s∈S d β (s) a∈A Q π (s, a)π θ (a|s) = E s∼d β a∈A Q π (s, a)π θ (a|s) (2.47)
where d β (s) is the stationary distribution of the policy β, d β (s) = limt→∞ P (St = s|S0, β), and Q π is the QValue function estimated with regard to the target policy π (different from β). To achieve the offpolicy goal, we essentially want to use samples from any policy to estimate the gradient of the target policy. This is possible using Importance Sampling (IS). It is a method to estimate an expected value of some function under a distribution, given only samples from a different distribution. Let's say we want to estimate the expected value of f (x) under a distribution p(x). According to IS, we can write:

E x∼p(x) [f (x)] = p(x)f (x)dx = q(x) q(x) p(x)f (x)dx = q(x) p(x) q(x) f (x)dx = E x∼q(x) [ p(x) q(x) f (x)].
(2.48)

The IS formulation Eq. (2.48) means that if we have samples from q(x), and we would like to estimate an ex pectation under p(x), we just have to multiply the samples by p(x)/q(x) which is called an importance weight. Samples that are more likely under p(x) and less likely under q(x) are deemed as being more important, thus this importance weight is a correction allowing us to get the expectation of the desired distribution. We can now apply this idea to estimate expectation under π θ (τ ) using only samples from any different policy π(τ ):

J(θ) = E τ ∼π(τ ) [ π θ (τ ) π(τ ) r(τ )] π θ (τ ) = p(s1) T t=1 π θ (at|st)p(st+1|st, at) π θ (τ ) π(τ ) = p(s1) T t=1 π θ (at|st)p(s t+1|s t ,a t ) p(s1) T t=1 π(at|st)p(s t+1|s t ,a t ) = T t=1 π θ (at|st) T t=1 π(at|st) (2.49)
Similarly, we can derive an offpolicy formulation using IS. The goal is to estimate the value of some new parameters θ ′ :

J(θ ′ ) = E τ ∼π θ (τ ) [ π θ ′ (τ ) π θ (τ r(τ )].
(2.50)

Using the IS formulation (2.48) and the identity π θ (τ )∇ θ log πθ(τ ) = ∇ θ π θ (τ ), we can write the gradient of (2.50) as:

∇ θ ′ J(θ ′ ) = E τ ∼π θ (τ ) [ ∇ θ ′ π θ ′ (τ ) π θ (τ ) r(τ )] = E τ ∼π θ (τ ) [ π θ ′ (τ ) π θ (τ ) ∇ θ ′ log π θ ′ (τ )r(τ )] (2.51)
If we do not want to just recover the onpolicy gradient, and we actually want an offpolicy algorithm, it means that we have to take multiple gradient steps without generating new samples. We will then have to use the following importance weight which is just the ratio of the products of the action probabilities:

π θ ′ (τ ) π θ (τ ) = T t=1 π θ ′ (at|st) T t=1 π θ (at|st) (2.52)
For policy gradient methods where we have θ * = arg max θ J(θ) and J(θ) = E τ ∼π θ (τ )[r(τ )] , the resulting offpolicy gradient is derived as:

∇ θ ′ J(θ ′ ) = E τ ∼π θ (τ ) π ′ θ (τ ) π θ (τ ) ∇ θ ′ log π θ ′ (τ )r(τ ) , (when θ = θ ′ ) = E τ ∼π θ (τ ) ∐ T t=1 π θ ′ (at|st) π θ (at|st) T t=1 ∇ θ ′ log π θ ′ (at|st) T t=1
r(st, at) (by adding causality)

= E τ ∼π θ (τ ) T t=1 ∇ θ ′ log π θ ′ (as|st) ∐ t t ′ =1 π θ ′ (a t ′ |s t ′ ) π θ (a t ′ |s t ′ ) T t ′ =t r(s t ′ , a t ′ ) ∐ t ′ t ′′ =t π θ ′ (a t ′′ |s t ′′ ) π θ (a t ′′ |s t ′′ ) .
(2.53)

It is very desirable to delete the importance weight on the future rewards represented in red in Eq. (2.53).

The resulting gradient is biased as equality is not hold anymore when doing that, but it turns out to still be a valid gradient. Getting rid of those additional importance weights is highly advantageous as it results in not having to multiply so many numbers together. However, we can't get rid of importance weights from timestep 1 until timestep t, which means those importance weights quickly go to 0 or infinity. In order to alleviate that problem, we can rewrite the objective a bit differently. Instead of writing it as an expectation over trajectories, we can write it as an expectation over stateaction marginals (as long as they are sampled from the marginal this is a correct estimator). Fortunately, if we use this approximated gradient, we still guarantee the policy improvement and eventually achieve the true local minimum which is justified in the proof [START_REF] Degris | OffPolicy ActorCritic[END_REF].

The offpolicy gradient is now given by:

∇ θ J(θ) ≈ 1 N N i=1 T t=1
∇ θ log π θ (ai,t|si,t) Qi,t.

(2.54)

With this formulation (2.54), we can optimize the target policy by using samples from any policies.

The offpolicy formulation of the policy gradient (2.54) is the central component of the RL methods used in this thesis. It allows us to improve a target policy by using samples generated by any other policies. This procedure has two main benefits. Firstly, the offpolicy gradient allows an improved exploration ability as any policy can be used to explore the environment and the associated interactions can still be used for the TD learning procedure. In other words, we are not constrained to explore again the environment as soon as an update is performed. This results in a much more sampleefficient algorithm (compared to onpolicy methods) as we can now hold in memory these diverse interactions and perform the updates at any desired rate. The stateoftheart method to achieve this type of offpolicy TD learning is called Experience Replay and it will be presented in Section 2.2.6. Secondly, as we can use samples generated by any policies, a nonlinear function approximator can be effectively used to estimate the values and policy functions. In fact, this kind of function approximator requires the use of uncorrelated samples so as to be optimized properly and not get stuck in local optima. In Section 2.2.7 we present how ANNs can be used as function approximators in the ActorCritic algorithm and how they can be optimized using the Gradient Descent method.

The explorationexploitation tradeoff

The deep policy gradient methods presented earlier are all based on the standard estimate of the policy gradient (2.45). In this case, the distribution representing the action distribution is updated such as increasing the probability of actions that hold a QValue higher than the average one and reducing it otherwise. In other words, policy gradient methods do not learn what good actions are, but rather which actions are better than the others. Therefore, the improvement ability of these algorithms directly depends on the capacity to take actions that are different from the ones suggested by the policy. This process is known as exploration while following the current policy is known as exploitation. This problem is known as the explorationexploitation tradeoff and is a critical topic in RL: an agent needs relevant experiences to learn a good policy, but it also needs a good policy to obtain those experiences. We would like our agent to find the best actions as fast as possible, but committing to solutions too quickly and without enough exploration can be dangerous as it can lead to local minima or total failure.

Depending on the nature of the MDP, there exist optimal or suboptimal solutions to the explorationexploitation problem which guarantee the convergence to the optimal actions. We list below methods that can be used successfully where the action and state spaces are discrete:

• Epsilongreedy: the agent executes random actions randomly with some probability ϵ < 1 while fol lowing the policy most of the time with probability 1ϵ. This is the most used exploration strategy thanks to its simplicity. However, with epsilongreedy, we can keep exploring known actions, which can be in the worst case bad actions. To avoid such inefficient exploration, one idea is to reduce the parameter ϵ in time toward 0 when the optimal actions have been explored. Therefore, the final policy is almost always suboptimal, and it is difficult to determine beforehand when the parameter ϵ should be annealed. The following methods aims at automatically adjust the amount of exploration by keeping track of the occurrence of actions and their associated value.

• Upper Confidence Bounds (UCB): this method relies on a basic idea that is to favor the exploration of actions with a strong potential to have an optimal value. With UPB, this potential is measured by the upper bound confidence bound of the reward value Ût(a), so that the true value is below with bound Qt(a) ≤ Qt(a) -Ût(a) with high probability. The upper bound Ût(a) is a function of Nt(a) a large number of trials which ensure a smaller bound Ût(a). This is possible using Hoeffding's Inequality allowing us to not assign any prior knowledge on how the distributions look like. The agent selects the greediest action to maximize the upper confidence bound: a U CB t = arg max a∈A Qt(a) + Ût(a).

• Bayesian UBC: in UCB we don't consider any prior on the shape of the reward distribution and therefore we rely on Hoeffding's Inequality to have a very general estimation by considering every possibility. If we could have the shape of this distribution beforehand, we could have a better estimation. If for example, we expect the distribution to be a Gaussian, we can set the upper bound confidence as 95% confidence interval by setting Ut(a) to be twice the standard deviation.

• Boltzmann exploration: we would like to use all the information encompass in our QValue function estimate. Instead of taking the optimal action or a random one, Boltzmann exploration [START_REF] Cesabianchi | Boltzmann Exploration Done Right[END_REF] in volves choosing an action with weighted probabilities. Thus, the softmax function is used over the QValue function estimate. Therefore, the action which the estimate believe is optimal is more likely, but not guaranteed, to be chosen. The advantage of Boltzmann's exploration is that information about the value of other actions is taken into consideration in the relative weight. This way the agent ignores actions that are estimated to be largely suboptimal while giving more attention to potentially promising, but not necessarily ideal actions. In practice, a parameter τ is introduced. It is annealed over time and it controls how spread is the softmax distribution so that all actions are considered equally at the beginning and sparsely distributed by the end of training. The action sampling process is then defined as:

π(a|ht) = exp{[Qt(a)/τ ]} n i=1 exp{[Qt(i)/τ ]} . ( 2

.55)

• Thompson sampling: the agent keeps track of a belief over the probability of optimal actions and samples from this distribution. The action sampling process is here defined as: The convergence to the optimal policy of the previous methods is guaranteed only when the action and state spaces are discrete. When dealing with continuous spaces, the benefits of these methods do not hold anymore as the probability to visit the exact same state twice is in this case null. Other methods have to be used for improved exploration when using ANNs. The problem of exploration is a complete field itself (as illustrated by the "NoisyTV" problem [START_REF] Burda | Exploration by Random Network Distillation[END_REF]) and extensive work is currently done toward the development of efficient methods. In the following, we will only focus on exploration methods for deep policy gradient methods. When the action and state spaces are continuous, the explorationexploitation problem becomes untrackable and the true optimal solutions can not be obtained. In our case of AUV applications, the number of training trials, despite being mostly simulated, is closer to being finite than infinite. Therefore, given a relatively small amount of interactions, we need to explore effectively these spaces in order to build an appropriate policy. The exploration strategies can then be decomposed into two groups: direct and indirect exploration. With direct exploration, the policy outputs are modified before being applied. The most simple way of doing that remains the epsilongreedy strategy described earlier. However, epsilongreedy is not effective to explore continuous space (where there is an infinite set of possibilities), an improved practice consists instead in applying noise to the actions estimated by the policy network. The action sampling is then defined as:

π(a|ht) = P[Qt(a) > Q(a ′ ), ∀a ′ = a|ht] = E R|h t [(a =
at = π(at|st) + N (0, σ), (2.57)
where N (0, σ) is a normal noise of mean 0 and standard deviation σ. Depending on the dynamics of the pro cess, the noise standard deviation requires careful tuning as a toosmall value can result in poor exploration and failure at performance improvement, while a too high value can result in high variance and restrain the agent from completing the task. Similar to Boltzmann's exploration [START_REF] Cesabianchi | Boltzmann Exploration Done Right[END_REF], the noise standard deviation could also be adapted over the course of training. In indirect exploration, the idea is to introduce noise be fore the action sampling process. This type of exploration is more appropriate for robotic applications as the resulting actions will be correlated to the agent's perception of the environment, and therefore they will be related to the dynamics of the process rather than being purely random and uncorrelated. The actions result ing from epsilongreedy are totally random and uncorrelated to the agent's state, while with noise induced for example in the state, observation, or even the parameter space, the resulting actions will be correlated to the environment dynamics. One approach consists in changing the overall objective function, exactly as performed within the maximum entropy reinforcement learning framework presented in Section 2.2.8. By adding the entropy term in the policy loss function, the agent is forced to explore more equally and take more diverse actions. Inspired by Thompson sampling, Bootstrapped DQN [START_REF] Osband | Deep Exploration via Bootstrapped DQN[END_REF] introduces a notion of un certainty in Qvalue approximation in the DQN algorithm by using the bootstrapping method. Bootstrapping is to approximate a distribution by sampling with replacement from the same population multiple times and then aggregating the results. The idea is to have multiple Qvalue estimators trained in parallel but each only consumes a bootstrapped subsampled set of data and each has its own corresponding target network.

All the Qvalue heads share the same backbone network. At the beginning of one episode, one Qvalue estimator is sampled uniformly and acts for collecting experience data in this episode. Then a binary mask is sampled from the masking distribution m ∼ M and decides which estimator can use this data for training.

Nevertheless, this kind of indirect exploration is still restricted because the induced noise and uncertainty rely on the training data. We can inject some prior information independent of the data by using parameter noise [START_REF] Plappert | Parameter Space Noise for Exploration[END_REF]. Parameter noise adds adaptive noise to the parameters of the policy neural network policy, rather than to its action space. It injects randomness directly into the parameters of the agent, altering the types of decisions it makes such that they always fully depend on what the agent currently senses. The technique is a middle ground between evolution strategies (where you manipulate the parameters of your policy but don't influence the actions a policy takes as it explores the environment during each rollout) and deep reinforcement learning approaches like TRPO, DQN, and DDPG (where we add noise on the action).

In practice, random Gaussian noise N (0, σ) is added to the parameters of the policy network at the beginning of each episode (then kept during the rollout) as:

σ k+1 = ασ k , if d(π, π) < δ, 1 α σ k , otherwise.
(2.58)

The noise standard deviation σ is adapted according to a distance measure d(•) between the nonperturbed π and perturbed policy π which is defined in [START_REF] Plappert | Parameter Space Noise for Exploration[END_REF] as:

d(π, π) = 1 N N i=1 Es[(π(s)i -π(s)i) 2 ], (2.59)
where the metric Es[•] is estimated over a large number of samples from the Replay Buffer and N is the dimension of the action space. We can directly see that by setting δ = σ (2.58), it is equivalent to an action space noise N (0, σ). Now that we have seen how to interact with the environment to address the exploration exploitation tradeoff, we will present in the next section how to use these generated data to improve our value estimates and policy.

Experience replay

The concept of Experience Replay (ER) [START_REF] Lin | Selfimproving reactive agents based on reinforcement learning, planning and teaching[END_REF] employs the agent's past experience to improve its current behavior. It aims to artificially make the agent's experience look Independent and Identically Distributed (IID). This is highly desirable in order to not concentrate the updates to a limited area of the desired functions.

Given that an agent's experience at the timestep t is defined as the quadruplet et = (st, at, rt, st+1), the ER method consists in storing (at each timestep) the experience et in a memory unit D = {e1, . . . , et} of fixed size, also known as the replay buffer. Then, the neural networks are trained by performing minibatch gradient descent of past experiences randomly sampled from the replay buffer. The estimators are hence trained on IID samples that are generated by various trajectories and policies and they are, therefore, more representative of the true function. As illustrated in Figure 2.2.6, in offpolicy policy gradient methods such as the ActorCritic algorithm, the policy used for data collection is different from the one that generated the samples used for offpolicy TDLearning. The usage of ER can be summarized as follow: (c) Update the policy parameters as: θ ← θ + α θ Q(s, a; w)∇ θ log π(a|s; θ).

(d) Compute the TD error at timestep t as: Gt:t+1 = rt + γQ(s ′ , a ′ ; w) -Q(s, a; w).

(e) Update the parameters w according to this TD error: w ← w + αwGt:t+1∇wQ(s, a; w).

(f) Move to the next transition a ← a ′ and s ← s ′ .

There are multiple benefits to using experience replay. By sampling at random, we increase the probability that our updates to the neural network will have less variance. In Onpolicy TD learning, most of the sam ples used for gradient descent were correlated and similar. Updating with similar samples concentrates the updates we make to our neural network to a limited area of our function, and it potentially overemphasizes the magnitude of the updates. If we sample uniformly at random from a very large buffer, on the other hand, chances are our updates to the network will be better distributed all across, and therefore more represen tative of the true function. Using a replay buffer also gives the impression our data is IID, so optimization methods will be better behaved. Samples will seem independent and identically distributed because we will be sampling from multiple trajectories and policies at once.

Deep Policy Gradient

In robotic applications, we use sensors to measure the environment, and we want to use the measured data to perform informed decisions based on what the sensors observe. Therefore, the goal is to build a policy function that takes as input the sensor data and returns the best actions. There is a large number of methods to find patterns in sensor data. For example, we might just use linear regression to determine how data is trending over time. Or we may look for when the signal drops below some threshold to indicate that whatever the sensor was observing is no longer in view. Nevertheless, the complexity of these methods, and how difficult it is to develop, scales with the complexity of the pattern that we are trying to find, as well as how those patterns vary from one observation to another. Using rulebased approaches, the defining features are difficult to describe.

In addition, the policy, value functions, and model can all be represented by a parametric function. For discrete spaces, in theory, we can memorize Q * (•) for all stateaction pairs, like an enormous lookup table. However, it quickly becomes computationally infeasible when the state and action space are large, or simply impossible when they are continuous. Moreover, when using ANNs as function approximators for the policy, we can consider stochastic actors that are better at rejecting disturbance compare to their deterministic counterparts. This is possible thanks to the reparameterization trick that will be presented in Section 2.2.8.

For these reasons, a common practice consists in using function approximators (i.e. machine learning mod els) to learn them from experience. Among the various methods of function approximation, artificial neural networks (ANNs) are leading the field. When multiple layers are used, people typically refer to this as deep reinforcement learning. In this section, we will thus focus on deep policy gradient methods which use ANNs to approximate the value and policy functions.

Artificial neural networks

Deep reinforcement learning can be applied to solve very large problems such as:

• Backgammon [Tes94]: 10 20 states

• Go [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF]: 10 170 states

• Helicopter [Ng+03; Ng+04]: continuous state space

• Robots [START_REF] Shixiang | Deep reinforcement learning for robotic manipulation with asyn chronous offpolicy updates[END_REF]: limited view of the world So far, we have only considered lookup tables to represent the value function where every state s has an entry v(s) or every stateaction pair (s, a) has an entry Q(s, a). The resulting problem with large MDP, which we want to tackle with ANNs, is obviously that there are so many states in the lookup tables that can not fit the memory in the computer. We can not build a table that covers a game like Go for instance. Additionally, and sometimes even more importantly, it is too slow to try to learn for each state individually. This is essentially the problem of generalization, as whenever we observe a state, we want to learn about that state, but we also want to learn about all the similar states because if it is a very big problem or continuous space, we will never see the exact same state twice. So in this case, if we would learn about each state individually, we would never learn anything, or we would never learn anything that we can use in the latter states because they would all look like brand new states. Another challenge is that individual states are often not fully observable. The perception of the world by robots is limited by their sensors and we want to somewhat deal with that.

There are many function approximators such as ANN, Decision Tree, Nearest neighbor, Fourier / Wavelet bases or Coarse coding, but reinforcement learning has specific properties:

• The agent's experience is not Independent and Identically Distributed (IID). Typically, successive timesteps are correlated.

• The agent's policy affects the data it receives which affects the nature of the function we are learning.

• The value function V (s) can be nonstationary. Policy Gradient methods are a very clear example of that as we change the policy repeatedly because we want to build a policy that works better, which means each time we are trying to estimate a different value function. Depending on the algorithm it can still be stationary, for instance when using bootstrapping with TD Learning.

• Feedback is delayed. In the online case, we might execute an action, and immediately update using TD Learning, but it is not clear that it is always the best thing to do. Sometimes we want to wait for a few updates as in MonteCarlo methods, which is not always trivial to do.

For these reasons, differentiable (nonlinear) function approximation (ANNs) is preferred in the field of RL and often performs best. Let's now describe how such a function approximator can be optimized using the Gradient Descent approach.

Let's consider J(θ) to be a differentiable function of parameter vector θ. The gradient of J(θ) is given by:

∇ θ J(θ) =      ∂J(θ) ∂θ1 . . . ∂J(θ) ∂θn      (2.60)
The goal will be to find a minimum of J(θ) (if it is considered as a loss) by moving the parameters in the direction of the negative gradients to perform Gradient Descent:

∆θ = - 1 2 α∇ θ J(θ), (2.61)
where α is a stepsize parameter.

For illustration, now let's see how we can approximate values by using stochastic gradient descent in order to use ANNs as function approximators. The goal is then to find θ that minimizes the difference between V θ (s) and Vπ(s):

J(θ) = Eπ (Vπ(S) -V θ (S)) 2 , (2.62)
with Gradient Descent the update is derived as:

∆θ = - 1 2 α∇ θ J(θ), = αEπ [(Vπ(S)) -V θ (S)∇ θ V θ (S)] .
(2.63)

In this particular formulation (2.63), the only thing that is random is the state, because we are using here the true value function which of course is not something we can do in practice but we do it to just define the loss. With Stochastic Gradient Descent, the update is given by:

∆θt = α(Vπ(St) -V θ (St))∇ θ V θ (St). (2.64)
In this case (2.64), this sampled gradient is still not something that is available because although we have instantiated the states, the true function is still not instantiated, which we do not have. Noted that here we assumed the policy is fixed and does not depend on the value function, therefore these updates are only considering policy evaluation and do not cover policy improvement.

To make this approach more concrete, we present now the loss functions used to design a deep Actor Critic method. The QValue function does not require an estimation of the StateValue function, but as some standard methods still estimate them together for stability purposes, we will define their associated loss function. First, the StateValue function is estimated by an ANN parameterized by Ψ. In order to reduce ActorCritic value overestimation [HGS16; FHM18], the StateValue function is estimated using the minimum of two different QValue estimates represented by two ANNs parameterized by Υ1 and Υ2, respectively. We use TD learning to estimate these functions, thus Ψ are optimized to minimize the TD error:

JV (Ψ) = V πµ Ψ (st) -min Q πµ Υ 1 (st, at), Q πµ Υ 2 (st, at) . (2.65)
Similarly, the parameters Υi of the ith QValue function estimator are optimized to minimize the TD error:

JQ(Υi) = Q πµ Υ i (st, at) -r(st, at) + γ × V πµ Ψ ′ (st+1) , (2.66)
where Ψ ′ are the parameters of the bootstrapped statevalue function, denoted as the target value (defined later in Section 2.2.8) and γ is the discount factor 2.2.1. Finally, the parameters θ of the policy π θ are updated as follows:

θ ← θ + α min[Q πµ Υ 1 (st, at), Q πµ Υ 2 (st, at)]∇ θ log π θ (at|st), (2.67)
where α is a learning rate. When considering these loss functions (2.65)(2.66)(2.67), the ANNS are optimized exactly for the purpose of estimating the Bellman's equations (2.26)(2.27) and the vanilla policy gradient (2.43) respectively. Using experience replay, the samples are no longer correlated and we have multiple samples in the batch, so we no longer need to perform these singlesample SGD updates.

Maximum entropy RL

When an agent is exploring the environment, it picks the next actions to execute based on its current belief (i.e. QValue estimate). Without enough exploration, an agent can easily get stuck in local minima and never experience satisfying trajectories. This behavior is illustrated in Figure 2.16 below. In this example, the goal of the agent is, from a starting position represented in green, to reach the target position (represented by the life lifebuoy) that is accessible only by rounding the container ship to the right. Now let's imagine that for some reason due to randomness, the agent has initially passed more often and deeper into the left path. At this time, the Qvalue estimate in the left region will then be the highest, because, despite the goal not being reachable from there, we can more easily get closer to it (especially if the reward is for example a function of the Euclidean distance to the target). Thus, the policy gradient (2.45) will encourage the agent to further explore the left path, such as the right path is not explored anymore, which can happen quickly.

Figure 2.16:

In maximum entropy reinforcement learning the policy is forced to explore the space of QValue more equally. This results in improved robustness to process variation as the agent is forced to explore suboptimal strategies until the optimal longterm one is found.

We would like a framework allowing us to systematically explore both paths such that the target is reached at least once. In order to improve the exploration balance between the two paths, we would be interested in transforming the QValue distribution so as lower values (and negative ones in particular) will still induce some exploration by the policy gradient. A simple transformation performing exactly this is the exponential transformation (illustrated by the blue curve in Figure 2.16). As the policy is represented by an ANN, we can optimize our policy exactly for this goal by minimizing the Kullback-Leibler (KL) divergence [START_REF] Joyce | KullbackLeibler Divergence[END_REF] between the policy distribution and the exponential of the Qvalue distribution:

min π DKL(π(•|s0)|| exp{Q(s0, •)})) (2.68)
Nonetheless, if we write the definition of the KL divergence, up to an additive constant, this objective function (2.68) is then equal to the expectation under π of Q(•) minus log of π:

JMaxEnt(π|s0) = max π Eπ[Q(s0, a0) -log π(a0|s0)].
(2.69)

When we look at the objective function (2.69), we can simply decompose the first term as the reward and the second term is just the entropy of the policy:

JMaxEnt(π|s0) = max π Eπ[ t r(st, at) + αH(π(•|st)) s0], (2.70)
where the term in green is known as the Shannon entropy measure, weighted by a temperature parameter α. This formulation (2.70) looks similar to the original Reinforcement Learning objective (2.17) [START_REF] Richard | Reinforcement learning an introduction Second edition[END_REF] except that it has now an entropy term in there. This function is called the maximum entropy objective JMaxEnt(π|s0). Solution methods using the objective function (2.70) with that addition term are denoted as maximum entropy reinforcement learning method. There exist different ways to measure a distribution entropy, but Shannon's entropy is commonly used as it is easy to compute. The term "entropy" in thermodynamics and information theory both captures increasing randomness. The maximum entropy objective function is quite interesting as we are asking the agent to be as random as possible while also maximizing the reward. If we optimize this objective function (2.70), the agent will look for all the different ways to maximize the reward. It will undoubtedly explore both left and right passages until it figures out which one is really better and leads to the target waypoint. In the following, we present an ActorCritic algorithm that takes into account this entropy term.

Soft ActorCritic

Let x be a random variable with probability mass or density function P . The entropy H of x is computed from its distribution P according to:

H(x) = Ex∼P [-log (x)].
(2.71)

In practice, following this expression (2.71) the entropy term is explicitly incorporated in the StateValue function V (st) as:

V (st) = E[Q(st, at) + αH(πµ(.|st))], = E[Q(st, at) -α log πµ(at|st)], (2.72)
where α is controlled either directly or indirectly and affects the stochasticity of the resulting policy because it serves the role of the temperature of the energybased optimal policy [START_REF] Haarnoja | Reinforcement Learning with Deep EnergyBased Policies[END_REF]. With this modification, the expected return of the agent does not encompass only the reward but also the entropy of the policy. The parameters Ψ of the StateValue function are then optimized to minimize the TD error:

JV (Ψ) = V πµ Ψ (st) -(min Q πµ Υ 1 (st, at), Q πµ Υ 2 (st, at) -log πµ(.|st)). (2.73)
Similarly, the parameters Υi of the ith QValue function estimator are optimized to minimize the TD error:

JQ(Υi) = Q πµ Υ i (st, at) -r(st, at) + γ × V πµ Ψ ′ (st+1) , (2.74)
Finally, the parameters µ of the Policy network are then optimized in order to minimize the expected Kullback Leibler (KL) divergence between the current policy and the exponential of the QValue function that is nor malized by a function Z Υ according to [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] as:

Jπ(µ) = Es t ∼D DKL πµ(•|st) Q * (st, •) Z Υ (st) , ( 2.75)
where,

Q * (st, at) = exp min Q πµ Υ 1 (st, at), Q πµ Υ 2 (st, at) , (2.76)
where Z Υ (st) is a partition function used to normalize the resulting distribution. While Z(•) is generally intractable, it does not contribute to the gradient with respect to the new policy and therefore can be ignored. When using the distribution expressed in Eq. (2.76) as a target for the policy shown in Eq. (2.75), the agent is forced to explore actions proportionally to their associated exponential QValues. This positive transformation allows a smarter explorationexploitation tradeoff as negative QValues will be transformed into small but positive ones, forcing the policy to make progress along suboptimal strategies. The overall algorithm is called Soft ActorCritic (SAC) [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] which provides an unbiased estimator of the gradient of (2.75) as: [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] for the mathematical proof). The parameter α in (2.72) controls the relative weight of the entropy term against the QValue in the StateValue function. As illustrated in Figure 2.17, for small α magnitudes, the policy becomes nearly uniform, and consequently fails to exploit the reward signal, resulting in substantial degradation of performance. For large α magnitudes, the policy learns quickly at first, but the policy then becomes nearly deterministic, leading to poor local minima due to a lack of adequate exploration.

∇µJπ(µ) =
With the right reward scaling, the agent balances exploration and exploitation, leading to faster learning and better asymptotic performance. Next, we will present an improvement of the SAC algorithm which consists in adjusting automatically the temperature parameter α.

Figure 2.17:

Illustration from [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] of the sensitivity of the nominal SAC algorithm on the AntV1 task from the OpenAI gym benchmark suite [START_REF] Brockman | OpenAI Gym[END_REF].

Automatic entropy adjustment

The optimal reward scale is difficult to determine beforehand because the entropy term in the objective func tion (2.70) can greatly vary across tasks and during training as the policy becomes better. An improvement of the SAC algorithms consists of adapting the temperature term so as to maintain the desired entropy value:

max π 0 ,...,π T E[ T t=0
r(st, at)] s.t. ∀t, H(πt) ≥ H0.

(2.78)

The expected return E[ T t=0 r(st, at)] can be decomposed into a sum of rewards at all the timesteps. Be cause the policy πt at time t has no effect on the policy at the earlier timestep, πt-1, we can maximize the return at different steps backward in time:

max π 0 E[r(s0, a0)] + max π 1 E[...] + max π T E[r(sT , aT )]
1st maximization second but last maximization last maximization (2.79) where we consider γ = 1 (this is essentially a DP process as described in Section 2.2.4). The practical algorithm was proposed in [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF] and is presented now: The QValue function is estimated exactly as proposed in the first version of SAC by minimizing the TD error (2.74). On the other hand, the StateValue function is now not explicitly represented by a neural network, but it is implicitly defined through the QValue functions and the policy (as no differences are observed when comparing both methodologies):

V (s) ≃ E (s t ∼D,a t ∼πµ) min i∈1,2 Q πµ Υ i (st, at) -α log πµ(at|st) .
(2.80)

Using the reparameterization trick provided in Section 2.2.8, the Gaussian policy is learned by minimizing the following cost function:

Jπ(µ) = Es t D,a t πµ [α log πµ(at|st) -min i∈1,2 Q πµ Υ i (st, at)]. (2.81)
This procedure Eqs. (2.80)(2.81) is the same as the first version of SAC but with an explicit, dynamic tem perature parameter α. To learn α, we have to minimize the dual objective (2.78), which is possible by using dual gradient descent. Instead of optimizing with respect to both variables (i.e. reward and desired entropy), the authors [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF] proposed to perform incomplete optimization that alternates between taking a single gradient step on each objective successively. By doing this, the resulting loss function can now be used with minibatch gradient descent. Therefore, the gradients for α are computed with the following objective function:

J(α) = Es t D,a t πµ -α log πµ(at|st) -αH . (2.82)
Finally, in order to make sure that α (2.82) is nonnegative, in practice we parameterize αt = exp{βt} and we optimize βt instead. This results in a maximum entropy policy gradient method where the reward scale does not need to be tuned. The target entropy is more intuitive to tune and is traditionally set to H = -dim(u) which means that for each degree of freedom, the entropy term in the objective function will hold a relative weight of 1% against the reward term. In order to update the policy, we need to draw a sample by computing a deterministic function of the state, policy parameters, and independent noise. This is possible using the reparameterization trick that we present next. The performance is more sensitive to the reward scale with a fixed temperature parameter (left) compare to when it is adjusted automatically (right).

Reparameterization trick

In order to optimize the policy for both versions of the SAC algorithm, we need to use the reparameterization trick. It allows us to express the expectation over actions into an expectation over the noise. By doing so, the probability distribution representing the action distribution has no dependence on the policy parameters. The reparameterization is defined as:

Ea∼π µ [Q πµ (s, a) -α log πµ(a|s)] = E ξ∼N [Q πµ (s, ãµ(s, ξ)) -α log πµ(ãµ(s, ξ)|s)] . (2.83)
To derive the policy loss, the final step is to substitute the QValue function with one of the function approxi mators. In the case of SAC, unlike TD3 and DDPG, the minimum of two QValue approximators is exploited. The policy is thus optimized according to:

max µ E s∼D,ξ∼N min j=1,2 Q Υ i (s, ãµ(s, ξ)) -α log πµ(ãµ(s, ξ)|s) , (2.84)
which is almost exactly the same as the DDPG and TD3 policy optimization procedure, except for the mini mum of the double QValue approximators, the stochasticity of the policy, and the entropy term.

By considering the maximum entropy reinforcement learning framework, we can build a more robust control system for the AUV. The control system will be forced during training to complete the considered task in various ways. This results in more adaptability in operation when facing a new set of states and thus not all solutions are available anymore. The exponential transformation of the space of QValue ensures an improved exploration of the continuous action and state spaces and reduces the chances of getting stuck in local optima. In this thesis, we use this maximum entropy reinforcement learning framework as it allows us to build an AUV control system that is more robust to process variation. Indeed, the agent is forced during training to solve the task in various ways, rather than not committing to a single appropriate solution, allowing it to be more flexible in evaluation when facing unseen scenarios. This strategy is preferred to the original RL objective that solely takes into account the reward because since we can not perform an exhaustive exploration of the action and state spaces, it is better to learn a policy that intrinsically holds some adaptation abilities (i.e. Soft ActorCritic) rather than trying to only maximize the expected return (e.g. DDPG [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF], TD3 [START_REF] Fujimoto | Addressing Function Approximation Error in ActorCritic Methods[END_REF], TPRO [START_REF] Schulman | Trust Region Policy Optimization[END_REF], PPO [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF], …). In addition, Deep Policy Gradient methods are generally very sensitive to the choice of hyperparameters (e.g. learning rate, minibatch size, target update procedure, …). The Soft ActorCritic on the other hand has shown to work properly on many environments and with the exact same hyperparameters and ANNs architecture [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF][Haa+18b] [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF] which is why we choose it as the core RL algorithms for our proposed methods.

The TD learning procedure seems similar to supervised learning as the considered loss functions Eqs. (2.73) and (2.74) are composed of variables that are either stored in the Replay Buffer or that can be sampled from the estimators directly (i.e. the target values are known). However, the major difference is that these losses are functions of each other which remove all of the convergence guarantees that are well established in the context of supervised learning. This is known as the moving target problem and we present next how it is tackled in the context of Deep Reinforcement Learning.

Target values

In supervised learning, target values are the labels on the dataset that by definition are static. The weights of the ANN are then updated for the purpose of modeling an optimal nonlinear mapping function between the input vector and the corresponding target values vector. As defined earlier, in ActorCritic methods, the value estimators are optimized in order to minimize the TD errors presented in Section 2.2.8 cf Eqs. (2.73) and (2.74). The value estimates are then used to update the Policy network so that optimal actions are taken (according to the associated optimal values Q * and V * that are the best return we can obtain). The target value in Eq. (2.74) is composed of the reward r(st, at) generated at the evaluated state st for a given policy, that is stored in the Replay Buffer and thus is fixed, as well as of the expected future cumulative reward of the agent represented by V (st+1) which is bootstrapped using our current estimate of the StateValue function. Therefore, contrary to the supervised learning case, a part of the target values are not static. At each gradient update, this target value will change given that the weights of the critic network have been updated. In fact, each estimator Eqs. (2.73), (2.74), and (2.75) are functions of the other (directly or indirectly). At each update step, they are improved, changing the shape of the other functions. The value estimates change at each iteration, making the previous estimates invalid after each gradient update:

V π t (st+1) = V π t+1 (st+1) Q π t (st, at) = Q π t+1 (st, at).
(2.85)

The TDLearning procedure (2.73)(2.74) is different from Gradient Descent in the sense that the target value periodically changes, transforming the loss landscape. This generates training behaviors that are unstable and divergent. To tackle this problem, it is common practice to have a separate copy of the considered value network, denoted as the "target" network, whose parameters are moving slowly or are fixed over a predefined amount of iteration steps. Here, we define a Target StateValue network V Ψ ′ (s) that is then used to compute the QValue TD error (2.74) as:

JQ(Υi) = Q πµ Υ i (st, at) -r(st, at) + γ × V πµ Ψ′ (st+1) , (2.86)
where the parameters Ψ ′ slowly track the value of Ψ. As illustrated in Figure 2.19, by having a target network we reduce the "chasing your own tail" [START_REF] Mnih | Humanlevel control through deep reinforcement learning[END_REF] problem by artificially creating small supervised learning problems. This is illustrated in Figure 2.19, where, by fixing the target value (i.e. label), the model is able to smoothly converges toward it before it is updated. This procedure improves the chances for convergence, not to the optimal values (as this can not be the case with respect to nonlinear function approximation), but convergence in general, while reducing substantially the chances of divergence. Two strategies for target network update are defined, both leading to stable learning dynamics: hard update and soft update.

1. Hard update: it consists of freezing completely the target network weights and copying directly the weights of the chosen value estimators in the target network every Γ gradient step. The advantage of this procedure is that by choosing a proper Γ, we can ensure a minimal amount of residual TD error, making the overall learning process more stable as we ensure the convergence guarantees of gradient based supervised learning. The drawback is that it significantly increases the training time as the policy is not trained on uptodate values. In addition, the value of Γ has to be tuned according to the process complexity.

2. Soft update: it consists of slowly copying the weights using an exponential moving average with ∆ a smoothing constant. The advantage of this procedure is that the policy update is not lagging for up to Γ timesteps as the target values move (slowly) at each update. The drawback is that residual TD error can increase over the course of training, increasing learning instability. The range of proper value for ∆ is notably smaller compare to Γ and nowadays the value ∆ = 0.005 is the standard.

The target value trick allows us to make the TD learning procedure actually converges when using ANNs. As shown in Figure 2.19, without any target network, deep TD learning diverges which results in the failure of RL algorithms. The soft update method with the standard value ∆ = 0.005 is traditionally used in AUV applications as it ensures limited residual TD error. Nevertheless, it has also been proved [START_REF] Fujimoto | Addressing Function Approximation Error in ActorCritic Methods[END_REF] that Value estimates diverge through overestimation when the policy is poor, and the policy will become poor if the value estimate itself is inaccurate. For this reason, the policy network should be updated at a lower frequency than the value network, to first minimize error before introducing a policy update. This is known as the Delayed Target Update trick (proposed in [START_REF] Fujimoto | Addressing Function Approximation Error in ActorCritic Methods[END_REF]) and consists in updating the Critic every step while only updating the Actor and target Critic network every d iterations, with d > 1. By sufficiently delaying the policy updates we limit the likelihood of repeating updates with respect to an unchanged Critic. The less frequent policy updates that do occur will use a Value estimate with lower variance, which results in higherquality policy updates.

Limits

Deep reinforcement learning has been used intensively over the past few years. It is attracting a lot of attention from both academia and industry because it provides a framework to learn the solution to a rational decision making problem, and most of today's problems of interest can essentially be described as such problems.

Recent breakthroughs in deep neural networks make it, even more, attractive as no a priori expert knowledge is required anymore. Nevertheless, the deployment of DRLbased agents is still limited. Their applications seem to be restricted to games and their performance varies from one implementation to another. This is not only due to the stochastic nature of these algorithms, but more importantly because of their fundamental elements [START_REF] Ghosh | Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability[END_REF] making it hard to build an agent that generalizes well enough to be useful. In this section, we propose to discuss these principal components of DRL which are known to restrain the usage of these methods.

Reward design is hard

The performance of the Reinforcement Learning methods is directly related to the reward function. Some might even say that the reward function is, for better or worse, the unique piece of intelligence in the Rein forcement Learning framework [START_REF] Silver | Reward is enough[END_REF]. Its importance has been studied for years [DL04; DK12] but the field is still lacking formal design rules. Writing a reward function is not hard, what is hard is designing a reward function that encourages the desired behavior while still being learnable. Unfortunately, shaped rewards can bias learning which can lead to behaviors that don't match what we want. To design a good reward function, we need to understand well the process and the environment because, for many reward functions, different behavior can generate the exact same output. Let's illustrate this with the following example.

Consider a vehicle whose objective is to reach an objective position of coordinates in a 2dimensional space OP = [Xo, Yo]t from a starting position SP = [Xs, Ys]t. In order to design a reward function to teach the vehicle to perform this task, we first can think of a metric to use as a cost function so as when minimized, the target is reached. An intuitive candidate is the Euclidean distance DL2(t) computed at timestep t between the vehicle position VP = [Xv, Yv]t and the target position as:

DL2(t) = i=dim(V P ) i=1 e 2 i (t), (2.87)
with ei = OP i -VP i the error between the vehicle and target position for the ith dimension of the 2dimensional reference frame.

We know that when this distance measure is equal to 0, it means that the vehicle and the target position are superposed in space, thus the objective is reached. A straightforward reward function would consist in designing a reward that is a function of this distance as:

r(st) = -DL2(t). ( 2

.88)

With this reward function (2.88), the closer the vehicle is to the waypoint and the higher the reward will be.

The resulting optimal policy will thus reduce the distance at each timestep until it is equal to 0, resulting in the success of the task. However, when taking a deeper look at this simple reward function as illustrated by the resulting spatial distribution of rewards in Figure 2.20, we can see in Figure 2.20 that different states can generate exactly the same reward. In Figure 2.20 we represent the spatial distribution of rewards according to the reward function (2.88). The problem here is that because the reward is a function of the distance to the waypoint (as (2.88) is essentially a measure of the Euclidean distance) there is an infinite set of states that return the exact same reward value. This is illustrated by the continuous yellow line in Figure 2.20. In fact, each point on this continuous line is associated with the same Euclidean distance to the setpoint. Therefore, despite being intuitive at first, this reward function (2.88) can be problematic because very different trajectories can lead to the same value. This can make it very difficult for the QValue estimator to determine which trajectory is really the best for the longterm objective.

One way to tackle this problem is for example to consider the derivative of the distance drate = DL2(t -1) -DL2(t) in the reward function as follows:

reward(t) = -DL2(t) + drate. (2.89)
With this new reward function (2.89), each time the actions reduce the distance between the vehicle and the target position, a positive reward is generated and add to the previous reward (2.88), otherwise a negative one is sent. By doing that, two points on the yellow line in Figure 2.20 will be associated with the same value if and only if they are associated with the same value of drate which is drastically less likely to be the case compared to with the first reward function Eq. (2.88). This will make it easier for the QValue estimator to differentiate trajectories from each other. Depending on the controlled system, the process itself, the characteristics of the environment, or even the presence or not of reward signals, designing a suitable reward function can or not be feasible. There is not yet a uniform theory for the design of these objective functions. It remains an open question as to either if the reward is enough to drive behaviors that exhibits intelligent abilities [START_REF] Silver | Reward is enough[END_REF] in spite of being the central piece of reinforcement learning algorithms. In our applications of AUVs adaptive control, a natural candidate for the reward function is the sum of errors on each DoF. We will see in this thesis that a proper reward design will be required in order to reach satisfying performance. The second component which limits the use of DRL is the Experience Replay mechanism. In fact, DRL uses the agent's past experience to improve its function approximators and we will see next how it can influence badly the performance of the resulting policy.

Experience matters

In order to improve itself, Policy Gradient methods are required to experience good trajectories. In accordance with the policy gradient (2.45), the policy improves solely if trajectories that are better (i.e. associated with a higher QValue) than the mean one are experienced. Therefore, using these samples in the maximum entropy policy update (2.77) is primordial. As presented in Section 2.2.6, the ER technique is the fundamental technique to choose those samples. Nevertheless, even with this standard formulation cf Section 2.2.6, there is a great number of parameters that are usually ignored despite having an impact on the learning performance, including:

• The replay buffer size: the total number of transitions that the replay buffer can store. When its maximum size is reached, the replay buffer is accessed in a firstinfirstout fashion. The bigger the replay buffer size, the more the data will look like it is IID, which in turn improves the gradient update quality. However, if the replay buffer is too big, an important transition will have much less chance of being used to update the policy, which could impair the learning process. In contrast, if the replay buffer is too small, the learned policy can be the result of an overfitting process on recent transitions, which precludes performance improvement.

• The age of a transition: the number of gradient steps taken by the agent since the transition was generated. This value can be seen as a measure of the extent to which the transitions stored in the Replay Buffer are offpolicy, as it tells us how different the current policies are from those stored in the buffer. The age of the oldest policy stored increases with respect to the buffer size.

• The replay ratio: the number of gradient updates per environment transition. It can be viewed as a measure of the frequency at which the agent is learning using existing data versus learning from collecting new experiences.

The size of the replay buffers, however, can impact negatively the learning performances [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF]. There are two competing methods that can be used to solve this issue: the Combined ER (CER) [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF] and the Prioritized ER (PER) [START_REF] Schaul | Prioritized Experience Replay[END_REF]. CER consists of adding the latest transition performed to the minibatch pooled over the replay buffer, whereas with PER important transitions, as measured by their associated TD error (2.73)(2.74), are given a higher probability to be used in the gradient updates. Using CER, however, the last transition will undoubtedly be sampled and instantly affect the policy.

Nevertheless, even with CER, a drop in performance was observed for certain sizes of replay buffer, at some point in the training (even when tuning the learning rate). This behavior was related to the process itself rather than to the aforementioned parameters [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF]. As written in [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF], "CER is a workaround …and future effort should focus on developing a new principled algorithm to fully replace ER."

A recent detailed analysis of ER is provided in [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF], where an analysis of the effects of the aforemen tioned parameters are presented. Several conclusions on how the parameters can affect the learning dy namics are drawn. These conclusions can be summarised as follows:

• Increasing the replay capacity while fixing the age of the oldest policy improves the performance be cause it lowers the chances of overfitting to a small subset of (state, actions).

• As the agent trains, it spends more time in higher quality regions of the environment (as measured by rewards), thus learning to better estimate the return in such regions leads to further gains in perfor mance.

• Increasing the buffer size with a fixed replay ratio has varying improvements. The replay ratio stays constant when the buffer size is increased because of both the replay capacity and the age of the oldest policy increase. If one of these two factors is independently modulated, the replay ratio will change.

In practice, these elements may be difficult to control independently and may also have different effects on the data distribution itself depending on the precise architecture of the agent. These issues highlight the entanglement that exists between these different properties of the experience replay mechanism at the level of practical algorithmic adjustments and motivates further study into how these properties can be disentangled. This direction of research is particularly important with regard to obtaining agents which can effortlessly scale with increased availability of data. Finally, the combination of Bellman Equations with deep neural networks itself is problematic because it induces overestimation in value estimation. We present next how this problem arises and why it is inherent to DRL algorithms.

The distribution shift problem in RL

In deep learning, and in regression, in particular, we optimized function approximation so as to match the training data. The performance of the model is then evaluated on data that were not observed during training.

In reinforcement learning, the model (i.e. the agent) is often trained in simulators that can in theory provide an infinite amount of data. The agent is then evaluated by interacting with another version of the environment (which can be explicitly a region of the environment that was not explored during training, another the same environment but with different disturbances or uncertainties, or even other tasks). As illustrated in Figure 2.22, there is no guarantee that the solution obtained from the training set will hold its performance on another set of states, that is the real world. This problem is also known as the distribution shit problem.

Let's now illustrate this problem mathematically. As proposed in [START_REF] Kumar | Stabilizing OffPolicy QLearning via Bootstrapping Error Reduction[END_REF], consider an empirical risk mini mization problem where want to minimize some loss:

θ ← arg min θ E x∼p(x),y∼p(y|x) [(f θ (x) -y) 2 ], (2.90)
where we are essentially doing regression on a data distribution f (x) → y, and the training samples are coming from p(x), and the goal is to minimize the empirical risk. We can ask, given a new x * , is my learned f (x * ) going to be correct? This is a more tricky question that we can think of. What we can say is that if the model is not overfitting, then the expected value of the loss is low:

E x∼p(x),y∼p(y|x) [(f θ (x) -y) 2 ] is low. E x∼ p(x),y∼p(y|x) [(f θ (x) -y) 2 ] is not low, in general p(x) = p(x). (2.91) What if x * is sampled from p(x) (i.e.
x * ∼ p(x))? Even there the error on x * is not necessarily low, it is low in expectation, but x * could be that one point where we got unlucky. Usually, we do not worry about this in deep learning, because we are using neural networks, and deep neural networks generalize well. This is true in most cases, including supervised and unsupervised learning. However, what if we do not just pick x * randomly, what if we pick it to maximize f θ (x):

x * ← arg max x f θ (x). (2.92)
This would be very dangerous as illustrated in cases, when we pick the maximum value of the function, we might pool out exactly that point where f (x) overestimates to a larger degree (i.e. the largest positive error). This is similar to an adversarial example, in a particular direction.

As illustrated in Figure 2.23, even if we think that our model has generalized very well, if we pick x * by maximizing our model, then we can always trick it toward a wrong value. This is very problematic, and it turns out that this maximization is exactly what we do in QLearning. For recall, the Bellman equation:

Q(s, a) ← r(s, a) + max a ′ Q(s ′ , a ′ ), (2.93)
can be written differently as:

Q(s, a) ← r(s, a) + E a ′ ∼πnew [Q(s ′ , a ′ )], (2.94)
where instead of writing it as a max, we can write it as an expected value under some distribution πnew, where πnew is that arg max policy (making both equations equivalent). When we do QLearning, the right side of Eq. ( 2.94) is used as the target and the objective is to minimize the error on that target value:

min Q E (s,a)∼π β (s,a) [(Q(s, a) -y(s, a)) 2 ] (2.95)
We expect a good accuracy when π β (a|s) = πnew(a|s). But how often does that happen? The whole point of training is to find a πnew that improves over π β , and even worse we are going to pick πnew = arg max x E a∼π(a|s) [Q(s, a)] that is exactly that maximization depicted in Figure 2.23 that is so problematic because since we are using deep neural networks to approximate the QValue function, as we are only interested in its maximum value (which is associated to the optimal action according to Bellman Equations), there is a great probability that the error on the maximum of the estimate is very high. This is why we see the overestimation in QLearning [START_REF] Kumar | Stabilizing OffPolicy QLearning via Bootstrapping Error Reduction[END_REF] because essentially that maximization is finding these adversarial examples which produce large error values, and because we are iterating this, those errors accumulate more and more until we have massive QValue overestimation. This is basically the challenge of distribution shift in reinforcement learning because as overestimation can easily happen in the QValue framework, the resulting policy tends to overestimate its future gains in a new environment, which makes it very difficult to effectively transfer such agents in the real world.

In this section, we have discussed the different limits of DRL methods by analyzing various variables and parameters that directly affect the performance of the resulting policy. Another component that can greatly affect the performance of the agent is the quality of the training data. In fact, as DRL methods require a large amount of data to converge, they are traditionally trained under simulation and if not realistic or representative enough of the agent and its environment, DRL methods can also fail at learning at all. In the following, we will present the simulation tools we used in this thesis.

Simulation framework for deep reinforcement learning 2.3.1 ROS

In this thesis, the experimental platform are using mainly the Robot Operating System (ROS) framework [START_REF] Quigley | ROS: an opensource Robot Operating System[END_REF] for sensors communication and actuators control. ROS is the most commonly used opensource metaoperating system for robots. It comes with a large number of offtheshelf tools, denoted as packages, developed by the community. ROS comes with a large number of offtheshelf tools, denoted as packages, developed by the community. From computer vision to advanced control, these packages are designed in such a manner that only the highlevel variables are required to reuse the packages, facilitating code and knowledge reuse. ROS has been used for many years now, both in academia and industry and because there exist a lot of great tutorial on how to use it, we will not present ROS in its totality.

Gazebobased AUV simulation

The UUV Simulator package

Deep Policy Gradient methods need a large amount of interaction with the environment in order to con verge to a satisfying behavior. Their initial behaviors are mostly random and thus risky for robotic platforms.

For this reason, it is common practice to perform their training in a simulator at first, which in theory, can provide an infinite amount of training data. In this thesis, Gazebo [START_REF] Koenig | Design and use paradigms for Gazebo, an opensource multirobot simulator[END_REF] was chosen as the simulation environment for training our agents because it is based on ROS which is the main framework used on the robotic platforms considered for the experimental validation. For underwater environments simulation, we used the ROS package called UUV Simulator [START_REF] Manhães | UUV Simulator: A Gazebobased package for underwater intervention and multirobot simulation[END_REF]. It is a Gazebobased library of AUVs and underwa ter environments allowing us to run personalized missions as illustrated in Figure 2.24 where an AUV faces an underwater wreck. The UUV Simulator [START_REF] Manhães | UUV Simulator: A Gazebobased package for underwater intervention and multirobot simulation[END_REF] can simulate several current and wave disturbances, thruster dynamics and body wrench disturbances. When incorporated into the simulations, the induced forces have a realistic physical impact on the robots and fluid dynamics. The sea current disturbance (which is the main focus of this study) is modeled as a uniform force acting over the Gazebo environment. This force is represented by a linear velocity, vc (in m.s -1 ), a horizontal hc and a vertical angle jc (measured in radians). These current variables can be changed at any step in the simulations through ROSbased callbacks or directly through Python/C++ scripts. The simulated RexROV2 platform is equipped with an IMU which feed backs its linear velocities and orientation (Euler angles). These variables are accessible through ROS topics, which are essentially data pipelines to access the simulation variables. Our software architecture consists in using the simulation metadata to train the learning algorithms considered in this work. To that end, we fix the simulation's realtime factor at 1, thus the training time is equivalent to what could be experienced on a physical platform.

In order for RL algorithms to learn efficiently, the data inside the state vector should allow it to find the corre lation between the actions and the environment changes: the difference between successive states should be large enough. This can be ensured by setting what we define as Sampling rate which is the rate at which a state vector is sampled from the environment after the execution of an action. A good practice consists in synchronizing it with the slowest sensor of the system which ensures the fastest sampling rate with no potential loss of information in the state vector (since at least the slowest sensor will be updated, it ensures that each variable of the state vector has changed since the last sampling).

The Rexrov2 platform

The RexROV2 platform provided in UUV Simulator [START_REF] Manhães | UUV Simulator: A Gazebobased package for underwater intervention and multirobot simulation[END_REF] is based on the ROV Minerva platform. Com plete modeling of this vehicle (including its kinematic, kinetic, and thruster model as well as its control allo cation) can be found in [START_REF] Berg | Development and Commissioning of a DP system for ROV SF 30k[END_REF]. The RexROV2 can be modeled using the general equations of motion for a marine craft, which can be written in the vectorial form according to [START_REF] Fossen | Nonlinear Modelling And Control Of Underwater Vehicles[END_REF] as:

η = J Θ (η)ν, M ν + C(ν)ν + D(ν)ν + g(η) = δ + δ cable , (2.96)
where η and ν are the position and velocity vectors respectively, δ is the control force vector and δ cable is the vector describing the umbilical forces from the cable attached to the ROV. The RexROV2 is an ROVtype platform provided in UUV Simulator. It is propelled by 6 thrusters (complete details on its equation of motions are provided in [Ber12; McC16; Yan+15]. The control vector u is obtained by transforming the actuator force vector:

δ = T(α)Ku, (2.97)
where T(α) ∈ R n×r is the thrust allocation matrix; K is the thrust coefficient matrix; δ is the control force vector in n DOF and u ∈ R r is the actuator input vector. UUV Simulator can be used to derive a vector of thruster contributions for every DOFs (for clarity, sin(•) and cos(•) are denoted as s• and c• below):

Ti =        Surge Sway Heave Roll Pitch Yaw        =        cθcϕ sθcϕ sϕ -Zsθ + Y sϕ -Zcθ + Xsϕ -Y cθ + Xsθ        . ( 2

The Bluerov2 platform

As shown in Figure 2.27, the Blue Robotics BlueROV2 Heavy platform is a smallsize ROV that is mostly used for observation applications. It includes four horizontal and four vertical thrusters of type T200 that are placed as illustrated in Figure 2.27. Similarly to the RexRov2 platform, the Bluerov vehicle can be modelled according to [START_REF] Fossen | Nonlinear Modelling And Control Of Underwater Vehicles[END_REF] as:

η = J Θ (η)ν, M ν + C(ν)ν + D(ν)ν + g(η) = δ + δ cable , (2.99)
where the above parameters have been described in Section 2.3.2. Exhaustive modeling of the vehicle will not be provided in this thesis as the complete description of the vehicle was previously performed by our team from Flinders University in [START_REF] Wu | 6DoF Modelling and Control of a Remotely Operated Vehicle[END_REF]. Nevertheless, some assumptions have to be made on the vehicle in order to be able to consider the proposed control designs:

• The Bluerov operates at relatively low speeds (i.e. less than 2 m.s -1 ), thus the lift forces can be neglected.

• The Bluerov is assumed to have portstartboard symmetry and foreafter symmetry. The center of gravity (CoG) is therefore assumed to be located in the symmetry planes.

• The Bluerov is assumed to be hydrodynamically symmetrical about 6DoFs. Thus, the motions between the DoFs of the vehicle in hydrodynamic can be decoupled.

Again, we have access to a thruster allocation matrix that allows us to control the Bluerov in the 6 degrees of freedom (Surge, Sway, Heave, Roll, Pitch, and Yaw). A complete description of the vehicle and its control parameterization is proposed in [START_REF] Wu | 6DoF Modelling and Control of a Remotely Operated Vehicle[END_REF]. Green and blue thrusters indicate counterclockwise propellers respectively (right). Source: BlueRobotics.

Summary

In this section, the technical background elements necessary to design the proposed learningbased adaptive control system have been presented. First, the elements on adaptive control theory were described with a focus on the AUV applications. We proposed a classification of solution methods based on their dependence on the process model which leads to three classes of adaptive control methods: modelbased, modelfree, and learningbased. We have provided a description of each of these methods and we have shown why a learningbased adaptive controller is the best choice for the problem considered in this thesis which is AUV control under unobservable current disturbance. We discussed why reinforcement learning is relevant here to design such a control system and the limits of adaptive control theory are provided at the end of the associated section.

Then, the technical background elements of reinforcement have been presented. We proposed this time classification of solutions methods based on the nature of the decisionmaking process which lead to three classes of reinforcement learning methods: modelbased, valuebased, and policy gradient methods. We provided a description of each of these methods and we outlined why deep policy gradient methods are the more relevant choice to adjust the control parameters. The limits of reinforcement learning theory are discussed at the end of the associated section.

Finally, the simulation tools and the vehicle modelization is provided with a description of the ROSbased package that we used to simulate AUV applications. The general block diagram of the desired learning based adaptive controller is illustrated in Figure 2.28. Simulated data are used to optimize a policy whose objective is to estimate the best control parameters for a given state of the process. This policy is learned using OffPolicy TD Learning as depicted by the presence of the Experience Replay in the offline learning loop. During online control, the control structure adapts its parameters based on process feedback.

In the next section, we described the proposed novel learningbased adaptive controller, in order to fill the gaps in Figure 2.28, along with all the preliminary studies that lead to its design.

Proposals for a novel learningbased adaptive control system

The exercise of going through a Ph.D. is a particular acting exercise where one person is trying to focus on one problem, that it does think is interesting enough, and that will be able to make a reasonable amount of advance to it in a given time frame. In practice, it turns out very rapidly that the problem of interest has already been studied for decades, or even longer, and ideas that were once believed to be groundbreaking turn out to have been already disproved experimentally. From this return to reality, a reasonable first step is to study the related works of the community to identify what seems to be achievable and where contributions can be made. In this section, we present this procedure with first an analysis of related works in AUVs learning based adaptive control. Then, based on the resulting findings, we present the preliminary studies that lead to the final design of the proposed learningbased adaptive control system.

Related works in AUVs learningbased adaptive control

The Deep Deterministic Policy Gradient algorithm (DDPG) [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] was used in [START_REF] Yu | Deep reinforcement learning based optimal trajectory tracking control of au tonomous underwater vehicle[END_REF] to learn the optimal trajectory tracking control of AUVs. This control problem consists in keeping the error e = xx d between the actual trajectory x and the target x d at zero. The authors concentrate on trajectory tracking in the (x, y) plan.

The ideal trajectory is defined as the Euclidean distance to the desired path. The error to be minimized is measured as the Euclidean distance between the current and the desired trajectory. The DDPG architecture is composed of critic and actor neural networks. In this architecture, the actor chooses which action to take and the critic tells it how good this choice was. The actor parameters are then updated using this information to improve its future decisionmaking processes. The authors defined a loss function to update the parameters of the actornetwork which include Lyapunov stability components [START_REF] Bacciotti | Liapunov functions and stability in control theory[END_REF]. They proved that reaching the minimum value of the loss results in the optimal desired behavior, making the error converge to zero in a stable fashion (with respect to Lyapunov stability theory). The performance of the proposed approach was tested on straightline and curved trajectories, showing that the DDPG algorithm was able to effectively solve the task in both cases. This approach was compared to a fixed gain PID and the results indicate that the learningbased controller exhibits better performance in terms of tracking error. However, as the stability components are incorporated in an indirect way (i.e. by an additional term in the actor loss function) there are no formal guarantees that the system will remain stable at all times.

In [START_REF] Wang | Reinforcement Learningbased Adaptive Trajectory Planning for AUVs in Under ice Environments[END_REF], the DDPG algorithm was used to learn adaptive trajectory planning for multiple AUVs in under ice environments. This work also considered an additional objective in terms of satisfying constraints related to kinematics, communication range, and sensing area. This is a challenging environment, where there is great uncertainty related to the current flow under the ice. The water temperature as well as partially sub merged ice structures (against which acoustic signals bounce) can substantially impact the sensor feedback, directly influencing the control performance. In order to solve this challenge, the authors proposed a new cost function together with an experience replay mechanism. The cost function used to update the Policy network includes terms that represent the field uncertainty, the cost of the trajectory, and the constraints induced by the trajectory. By making the reward a function of the control constraints, the associated optimal behavior policy takes into account these restrictions during the decisionmaking process. The traditional experience replay technique is also modified to store past experiences of the agent that specifically satisfy the commu nication range and sensing area constraints in distinct replay buffers. The gradient updates are then applied using transitions from these replay buffers only. This procedure helps the agent to identify positive actions (with respect to these constraints) more rapidly since the associated behaviors will have a much higher prob ability to be selected during gradient descent. This resulted in a robust behavior policy that chooses actions that comply with the aforementioned constraints. Simulation results showed that this approach was able to achieve a performance matching that of a benchmark method which assumes perfect knowledge of the field hyperparameters. Again, the stability components were incorporated in an indirect fashion.

Learningbased adaptive control was investigated in [START_REF] Kristoffer Borgen Knudsen | Deep Learning for Station Keeping of AUVs[END_REF] for the station keeping of an AUV under un known currents. They used the DDPG algorithm to control the position of a BlueROV2 platform in surge

x and sway y combined to a PD control law that regulated the AUV position in heave z and orientation in roll ϕ, pitch θ and yaw ψ. The DRL algorithm was used [START_REF] Kristoffer Borgen Knudsen | Deep Learning for Station Keeping of AUVs[END_REF] to learn a PD control law as a function of the vehicle position and velocity at previous time steps. The training was performed within the ROSbased Gazebo simulator [START_REF] Quigley | ROS: an opensource Robot Operating System[END_REF]. The evaluation was conducted on a real platform in an indoor water tank. The authors stated that the agent's performance was satisfactory after 600 episodes. The resulting behavior pol icy was then used on a real platform to solve various tasks including a DP 4corner test scenario, which is a benchmark test for validating dynamic positioning (DP). It was observed that the agent is able to complete the stationkeeping task even when facing this unknown operating condition. The experimental evaluation consisted of three scenarios: two different desired pose definitions and a 4corner test. The first scenario consists in changing one error state while in the second scenario, both error states are changed at the same time. The 4corner test consists in performing station keeping at the 4 corners of a rectangular trajectory. These experiments proved that the agent is able to complete the task under real conditions. The performance was, however, slightly worse in the real environment compared to the simulated one, especially for the most challenging task of a DP 4corner test.

More recently, Deep Imitation Learning (DIL) [Liu+18; Pen+18] and another Deep Policy Gradient algorithm named Twin Delay Deep Deterministic Policy Gradient (known as TD3) [START_REF] Fujimoto | Addressing Function Approximation Error in ActorCritic Methods[END_REF], were combined in [START_REF] Chu | Motion control of unmanned underwater vehicles via deep imitation rein forcement learning algorithm[END_REF] for the design of a learningbased controller for an AUV (the combination of DIL and DRL is denoted as DIRL).

The goal of this work was to leverage imitation learning (IL) to speed up the training of DRL methods such as DDPG in order to facilitate the simtoreal transfer of RLbased agents. The idea of IL is to use some expert agent to generate examples of appropriate behaviors that are then used to perform the pretraining procedure of the ANNs (in a supervised fashion). Then, the networks can be finetuned using the normal DRL framework under a reduced number of episodes. Depending on the task, expert knowledge can be provided by a human operator (remotely controlling the robotic platform). This knowledge can also be obtained by using some Robust or Optimal control designs (derived from modelbased theory using the a priori knowledge on the process and plant) to regulate the controlled system. In both cases the essence of the work is the same, that is to build a data set out of experiences generated by an expert agent and to use it for prior offline learning.

Here, the PID structure was also considered but this time as the expert agent [START_REF] Chu | Motion control of unmanned underwater vehicles via deep imitation rein forcement learning algorithm[END_REF]. The PID controller is used to construct a set of state st and thrust vectors Ft as expert demonstration data. The overall training procedure is defined as follows: until the replay buffer reaches a specific size, the training is performed on the expert data ((st, Ft) from the PID controller) in a supervised scheme; when the replay buffer size exceeds this threshold, the Policy network is further trained in an RL procedure with the TD3 algorithm. Their method is hence denoted as ILTD3. Two control tasks were considered: 1) constant depth and attitude control and 2) depth trajectory tracking control. For the first task, a total of 400 episodes were performed and it was observed that after only 100 episodes of supervised learning, the behavior cloning was almost complete.

In addition, the authors trained the TD3 and DDPG algorithms for the same problem (i.e. with the same loss functions). They showed that ILTD3 is the fastest to converge and the most stable during training. They compared ILTD3 under simulation to the original PID controller with and without current disturbances.

Results showed that, in the case of no disturbances, both methods were able to solve the task (ILTD3 exhibited faster response and lower overshoot but at the cost of a much higher thrusters solicitation than the PID algorithm). In the case of current disturbances, the PID controller failed at solving the task. Thanks to prior expert knowledge, the ILTD3 was able to capture actions associated with highvalue rewards despite the early stage of the training. This, according to the authors, leads to further gains. The advantage of their method was moreover demonstrated with reallife tank experiments on the BlueROV2 platform. The Policy network, which was trained under simulation only, was able to provide satisfying control ability when transferred in the real world.

This initial literature study allowed us to identify the advantages and drawbacks of different designs of such control systems. We were able to observe that two trends are dominating the field of adaptive control of AUVs: direct and indirect approaches. In the first case, the parameters of the controller are adjusted directly using some Machine Learning techniques. In the second case, the adjusted control parameters are the result of an optimization problem where the state and/or unknown parameters of the process are estimated and then used to compute the associated optimal parameters. In most cases, these approaches are applied to classic modelbased control structures such as the PD or PID control laws. The objective is then to adjust the parameters of these control structures, namely their gains, according to process variation and using deep reinforcement learning. The principal RL algorithms used in the literature are the TD3 and the DDPG algorithms. These deep policy gradient methods (See section 2.2.7) build deterministic actors and do not take into account the entropy term from the maximum entropy reinforcement learning framework (presented in Section 2.2.8). Most of these works use the original experience replay mechanism detailed in Section 2.2.6 except [START_REF] Wang | Reinforcement Learningbased Adaptive Trajectory Planning for AUVs in Under ice Environments[END_REF] where they proposed to select the past experience of the agent based on different control constraints and to store them in different replay buffers accordingly. By using only these particular samples to update the actor, the resulting policy displays a more robust behavior with respect to these constraints. Based on these findings, the next step of the thesis consisted in performing a number of preliminary studies on the design of RLbased control systems for AUVs with the objective of tackling the abovementioned challenges.

Preliminary studies

In this section, we present some preliminary studies that guided the design of the learningbased adaptive controller proposed in this thesis.

Modelbased vs modelfree adaptive control of AUV under current disturbance

Following the analysis of the related works presented in the previous Section 3.1, the next step of this thesis consisted in evaluating the first design of an adaptive controller based on deep reinforcement learning. In collaboration with Yoann SOLA, another Ph.D. student at ENSTA Bretagne, we proposed an endtoend modelfree adaptive controller based on the SAC algorithm. The purpose of this study was to compare this approach to a purely modelbased PID controller and the results were presented [START_REF] Sola | Evaluation of a DeepReinforcementLearningbased Controller for the Control of an Autonomous Underwater Vehicle[END_REF] at the IEEE Global Oceans Conference 2020. In the following, we summarize the main results of this paper, and its preprint version can be found in Appendix C.

Task description

In this study [START_REF] Sola | Evaluation of a DeepReinforcementLearningbased Controller for the Control of an Autonomous Underwater Vehicle[END_REF], we address the control problem of target rallying by an AUV. The goal is to stabilize the vehicle at a given position and bearing angle.

ow = atan2(yw -yv, xw -xv) -2π if(ϕw -ϕv) > π atan2(yw -yv, xw -xv) + 2π else. (3.1)
The task of target rallying can be achieved if the distance measure to the target is minimized:

dt = (xv -xw) 2 + (yv -yw) 2 + (zv -zw) 2 ≤ d reached , (3.2)
where dt is the Euclidean distance measure, and d reached is the threshold value that we want the distance sig nal dt to be lower than. This class of control objective is used in various AUV missions, such as autonomous docking or underwater inspection, where a conservative regulation of the AUV position is required.

Design of the modelbased controller

We used as evaluation platform the RexROV2 vehicle provided by UUV Simulator presented in Section 2.3.2. Using the thruster allocation matrix from [START_REF] Manhães | UUV Simulator: A Gazebobased package for underwater intervention and multirobot simulation[END_REF], we can directly control the vehicle in the surge, sway, heave, roll, pitch and yaw as described in Section 2.3.2. Thus, we can use a PIDtype control law (i.e. modelbased) to control each of these DoFs to perform the target rallying mission. The PID control input is defined as:

ui(t) = kpei(t) + ki ei(t) + k d ėi(t). (3.3)
The modelbased controller consists in using the PID control law for all 6 DoFs with the target value for the roll and pitch being 0 and for the yaw angle being the bearing angle ow defined in Eq. (3.1). In order to evaluate the performance benefits of the proposed approach, we use a fixed but optimal version of the PID controller (3.3). The parameters of the controller, namely its gains kp, ki, and k d , have been optimized and are fixed during operation. Thus, this controller will be denoted in the following as a PID controller. The gains of the resulting PID controller (provided by the UUV Simulator) were tuned using the modelbased optimization method called SMAC [START_REF] Hutter | Sequential ModelBased Optimization for General Algorithm Configuration[END_REF]. Details on the gains can be found on the UUV Simulator website and the complete list of the controller parameters are provided below in Table 3

Design of the modelfree controller

As presented in Section 2.1.2, in modelfree control we aim at maximizing an objective function without any prior information on the system and process models. For this reason, we propose a modelfree controller which consists in using Deep Reinforcement Learning to learn how to control the vehicle by applying control inputs T directly on the vehicle thrusters, thus without the thrust allocation matrix presented in Section 2.3.2. The objective of the learning agent is to build a stochastic predictive model, using the SAC algorithm, that maps the thrusters inputs (2.97) directly from the current state:

π θ : S ⊂ R 20 → A ⊂ R 12 x = [st] T → [λi, µi] . (3.4)
Thus, the outputs of the Policy network are the 6 pairs of (λi, µi). Since we are building a stochastic actor, the control inputs applied to the ith thruster of the AUV are denoted as Ti ∈ [-240, +240] and are modeled by a Gaussian distribution Ni(Ti) defined as:

N (Ti) = (2πµi) -1/2 exp - 1 2µi (x -λi) 2 , ( 3.5)
where λi ∈ R and µi ∈ R + are the mean and variance of N (Ti) that are estimated by the Policy network. In the following, this controller will be denoted as an RL controller.

State vector

Since the proposed approach here only focuses on the highlevel guidance and lowlevel control parts of a GNC system for an AUV, the navigation considerations are not tackled. Therefore we consider that the AUV has access to good estimates of its pose, its linear and angular velocities, and its tracking errors with respect to the waypoint. The environment state st observed by the SAC algorithm at time t is given as follows:

st = [ x ; Θ; v ; Ω ; ϕe ; xe ; ut-1 ] T , ( 3.6)
where

• x = [ x, y, z ]
is the position vector of the AUV in Cartesian coordinates,

• Θ = [ ψ, θ, ϕ ] is its orientation vector expressed with Euler angles (respectively roll, pitch and yaw),

• v is its linear velocity vector (the temporal derivative of x),

• Ω is the angular velocity vector (the temporal derivative of Θ),

• ϕe is the error between its current yaw and the desired yaw leading directly towards the waypoint,

• xe is the error between its current position and the desired position, corresponding to the waypoint position,

• and ut-1 is the vector of the inputs of the actuators computed at the previous step, at time t -1.

Reward function

In order to perform a waypoint tracking mission, we designed the following reward function rt (3.7). It was inspired partly by [START_REF] Carlucho | Adaptive lowlevel control of autonomous underwater vehicles using deep reinforcement learning[END_REF], where the reward function takes into account lowlevel variables such as linear and angular velocities and their respective references. We adapted this work for a higher level of control, taking directly into account the position of the AUV and its reference. Therefore the guidance and control parts of the GNC system of the AUV are both provided by the SAC algorithm.

rt =      r toward if dt -dt-1 > 0 r backward else rwaypoint if dt < ϵ r limit if z / ∈ [zmin, zmax] (3.7)
where rt is the reward received by the agent at time t, dt is the current relative distance between the AUV and the waypoint to reach, zmin and zmax are the authorized limits for the vertical movement z of the AUV, and ϵ is a strictly positive real number.

Each term appearing in (3.7) represents a specific feature of the global desired behavior of the AUV:

• r toward is a variable reward given when the distance dt is decreasing, which means that the AUV moves toward the waypoint. It is defined as follows:

r toward = λ1(dt -dt-1) -λ2 Ω , ( 3.8)
where λ1 and λ2 are positive weighting terms. The term weighted by λ1 rewards large movements toward the waypoint, while the term weighted by λ2 penalizes strong angular speeds, and promotes indirectly a softer use of the actuators.

• r backward is a constant negative reward given when the distance dt is increasing, which means that the AUV moves backward the waypoint.

• rwaypoint is a constant positive reward given to the agent when the AUV reaches the waypoint, which leads to a terminal state, ending the current episode.

• r limit is a constant negative reward given to the agent when the vertical movement of the AUV exceeds the limits defined by [zmin, zmax], which leads to a terminal state, ending the current episode.

We chose the value of all the parameters in order to give the signals r toward and r backward a magnitude of around 10, as recommended in [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]. The complete list of hyperparameters used to design the DRL controller is provided in Table 3.2. To make the training more realistic, and to avoid overfitting, we will describe next how we randomize several parameters of the environment during training.

Training

The training of the RL controller consists in performing a total of 1200 training episodes. This amount of episodes represents approximately 4 hours of realtime training, after which convergence in terms of success rate was reached (i.e. the variance of rewards was small enough, for us, to consider the policy sufficiently good to stop the training). A training episode is defined as follows: at the beginning of the episode, we initialize the vehicle at a position x = [0, 0, -20] (in meters and relative to the frame attached to the Gazebo world center) with a random orientation Θ, where ψ = θ = 0 and ϕ ∈ [0; 360] (in degrees and with respect to the center of mass of the vehicle body). The waypoint is then placed at a random position located at a maximum euclidean distance of 50 meters from the AUV initial position whose coordinates are used as a setpoint. Then, the RL controller is launched and the episode ends either when the target is reached, when the vehicle depth exceeds a predefined threshold, or when the maximum episodic step size is reached.

The sensors measurements included in our state vector incorporate added noise such as: for each xi ∈ s, xi = xi + σi with σi randomly sample from the uniform distribution U[0.05; 0.1], except for the past actions where σi is there randomly sample from the uniform distribution U [0.01; 0.05] (because the amplitude of the actions is notably smaller than the other variables). We also added fluctuating sea currents to our underwater simulated environment. The current velocity cv ∈ [0; 1] (in m.s -1 ) and angles, (c ha ; cva) ∈ [-0.5; 0.5] (for horizontal and vertical angles respectively in radians), are randomly modified every 100 time steps during training and evaluation.

Evaluation

The evaluation consists of using both controllers for a series of 500 simulated episodes each with the same environment parameters, including: AUV initial position, target positions, and sea currents characteristics. We measure their performance in terms of success rate (percentage of episodes where the AUV reaches the waypoint), temporal mean, and standard deviation (SD) of the distance error dδ to the ideal trajectory (the direct path from the initial position of the AUV to the waypoint location), but also in terms of thrusters usage under the form of the temporal mean of the Euclidean norm of the input vector u at each step (with each element of u being a signal between -240 and +240).

Discussion

In table 3.3 we provide the evaluation results with the success rate, the mean distance to the target, the standard deviation of the distance to the target, and the mean value of the sum of control inputs. We can see that despite being capable of completing the task, both controllers display very different behaviors as illustrated in Figure 3.2 where we plot the 2D trajectory performed by each controller for the same target and sea current disturbance. In terms of success rate, the PID controller is doing better than the RL controller with 10% more success. When taking a look at the mean and standard deviation of the distance to the optimal trajectory (that is the line passing through the starting point and the target), we can also observe that the PID controller is doing better in both metrics. This trend is illustrated in Figure 3.1a where we plot the evolution of the euclidean distance to the target for the same episode characteristics. We believe that this is due to the modelbased part that is included in the PID controller. In fact, with the PID control law Eq. (3.3), as described in Section 1.1, the closedloop control is continuously monitoring the error value based on feedback measurements (i.e. modelbased information). Therefore, in order to reduce the error on a given DoF, the PID controller knows, through the thruster allocation matrix, which combination of thrusters to use in order to regulate that particular DoF.

We can see that the RL controller has lower thruster usage, and thus indirectly in power consumption. This is illustrated in Figure 3.1b where we plot the evolution of the norm of the sum of the control input vector, again for the same episode. The difference in power consumption can be explained by the characteristics of the controllers. In fact, the PID controller incorporates the correlation between the DOFs of the RexROV 2 platform, whereas the RL controller does not. This means that to move in a specific direction, the PID con troller knows exactly which combination of thrusters to use. The RL controller, on the other hand, estimates directly the inputs to the actuators and tends to use fewer thrusters than the PID controller to perform the same movement. This leads to smaller power consumption but poorer control performances than the PID controller.

Eventually, we can observe that despite reaching the goal, the trajectory of both controllers is not close to the optimal one. This is due to the current disturbance that is not implicitly included in both control methods: the PID control law takes into account only the state of the vehicle (its position and bearing angle) while the RL controller does not include the characteristics of the current disturbance in the state vector Eq. (3.6) because in practice we do not have a sensor to measure it. However, we saw that the closedloop feedback controller is able to compensate for disturbances by means of adaption (see Sections 1.1). It sounds intuitive that we could benefit from both paradigms by using Deep Reinforcement Learning to learn how to adapt the parameters of the closedloop feedback controller with what is denoted as a learningbased adaptive controller (see Section 2.1.2). Instead of estimating directly the control inputs to apply to the thruster, our idea is to use deep reinforcement learning to adjust the parameter of the modelbased control structure (e.g. the PID control law) in a modelfree manner. In the following, we present a second preliminary study that is our first attempt at designing such a control system. This work was in collaboration with ONERA, the French National Aerospace Laboratory, as part of my MSc. graduation internship performed in that institution on the use of deep reinforcement learning for the control of a terrestrial vehicle [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Re inforcement Learning of DepthBased Robot Navigation[END_REF].

Modelfree vs learningbased adaptive control of MAV under wind disturbance

Following the work described in Section 3.2.1, and in accordance with previous work in collaboration with ONERA [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Re inforcement Learning of DepthBased Robot Navigation[END_REF], we proposed a first design of a learningbased adaptive controller using deep reinforce ment learning. The purpose of this study was to compare this approach to a purely modelbased and a purely modelfree counterpart of the same control structure in order to identify the benefits of the proposed approach. The objective is to observe the effect of merging together modelbased and modelfree theories on both the control and learning performance. Despite the main focus of this thesis being AUVs, we apply this methodology on an aerial drone as it is the principal system of interest of ONERA. Later, we will present how to improve and adapt this method for an AUV. The results were published [Cha+22] as a chapter of the Lecture Notes in Electrical Engineering book series (LNEE, volume 793). In the following, we summarize the main results of this paper, and its preprint version can be found in Appendix C, with the related work of learningbased adaptive control in the aerial domain. We performed this study under simulation with Gazebo again and by using the ROS package called RotorS [START_REF] Furrer | RotorS-A Modular Gazebo MAV Simulator Framework[END_REF] described in Appendix B.

Task description

In this study [Cha+22], we address the same control problem of target rallying as described in Section 3.

Design of the modelfree adaptive controller

Compared to the related work presented in the paper [Cha+22] available in Appendix C, we propose to only treat the parameters adjustment task. We will now present the design of a modelfree adaptive control strategy initially proposed in the first preliminary study (see Section 3.2.1) but this time for the application of a target rallying mission by a MAV under unknown wind gusts. The methodology follows the same line of thought as the previous study [START_REF] Sola | Evaluation of a DeepReinforcementLearningbased Controller for the Control of an Autonomous Underwater Vehicle[END_REF]. The objective of the learning agent is to build a predictive model, using the SAC algorithm, that directly maps the hexacopter control inputs defined in B. In fact, with the RotorS package [START_REF] Furrer | RotorS-A Modular Gazebo MAV Simulator Framework[END_REF], the MAV can be controlled in terms of vertical thrust force, and roll and pitch orientation. Therefore, the modelfree adaptive controller consists in building a predictive model that maps a state vector directly to the MAV control inputs:

π θ : S ⊂ R 114 → A ⊂ R 6 x = [st] T → [λi, µi] . (3.9)
With the modelfree adaptive controller, the outputs of the Policy network are the 3 pairs (λi, µi) of mean and standard deviation. The control inputs [T ϕr , T θr , TT ] applied to the MAV is modeled by Normal distributions defined as:

N (Ti) = (2πµi) -1/2 exp{- 1 2µi (x -λi) 2 }, (3.10)
where (T ϕr , T θr ) ∈ [-π 6 ; + π 6 ] are roll and pitch angle inputs, and TT ∈ [m×g; m×(g+3.0)] (with m = 1.544Kg, g = 9.81m.s -1 ) is the vertical thrust force input that is sampled as described in Section 3.2.1. For summary, the outputs of the Policy network are 3 pairs of (λi, µi), and the control inputs are then directly sampled from the Gaussian distributions Ni(λi, µi) and applied to the MAV.

State vector

In order to achieve the target rallying mission, we need to provide relevant data to the agent. Therefore, we defined the following vector ot as the observation at the timestep t of the environment: • and dt is the current Euclidean distance between the hexacopter and the target.

ot = [
All the variables involved in the observation vector Eq. (3.11) are assumed to be measured, their estimation is out of the scope of this work (as they are provided by the Gazebo simulator). The dimension of this observation vector Eq. (3.11) is 19 and it has been standardized to have zero mean and a variance of 1. In order to provide a higher time horizon to the agent, we constructed the state vector out of the current and past two observations vectors [ot; ot-1; ot -2]. For the purpose of providing the agent a sense of "velocity" in the evolution of the state, we consider the twobytwo difference of these vectors such as velt = (otot-1) and velt-1 = (ot-1ot-2). We went even further and tried to add a sense of "acceleration" in the evolution of the state by including the difference between the latter vectors acct = (veltvelt-1). The resulting state vector is therefore defined as:

st = [ ot ; ot-1 ; ot-2 ; velt ; velt-1 ; acct ] (3.12)
The dimension of the state vector Eq. (3.12) is thus 114. Here, we synchronized the sampling rate (defined in Section 2.3.2) with the embedded odometry sensor which led to a sampling rate of about 20Hz. This means that every time a new sensor message is received from the topic associated with the odometry sensor, a new state vector is captured and the next action is sampled.

Design of a learningbased adaptive controller

As stated in Section 2.1.2, learningbased methods consist in exploiting standard modelbased control ar chitectures that are either tuned or redesigned by a modelfree algorithm in order to compensate for the unknown part of the model. The MAV is subject to an additive but unknown wind perturbation which can be modeled as:

u adp = u + u wind (3.13)
In this context, despite u wind being unknown, the PID control law can again be considered as the integral term that will ensure convergence to the steady state despite the wind disturbance. In order to derive the PID control law, let's defined here the control objective in more detail as stabilizing the MAV at a given target in space xw = [xwywzw] with a velocity v = ẋw. Defining the error vector as e = xvxw, the wind disturbance is taken into consideration by considering an additional steadystate variable z = t 0 e(τ )dτ . The augmented model with state vector X = [z, e, v] becomes a double integrator:

ż ė v = 0 1 0 0 0 1 0 0 0 z e v + 0 0 1 u (3.14)
The corresponding PID control law is defined as:

u = -kiz -kpe -k d v (3.15)
The poles of the closedloop system are solutions to the following equation:

λ 3 + λ 2 k d + λkp + ki = 0 (3.16)
We propose to reparametrize the PID control law to make it adaptive using pole placement. This way, the action space for learning purposes is limited to desired solutions in the real part of the pole map, which prevents sampling unnecessary solutions in the space of the control gains. The desired constants τ1 > 0, τ2 > 0, τ3 > 0 are then defined as:

λ1 = -1 τ1 ; λ2 = -1 τ2 ; λ3 = - 1 τ3 (3.17)
Since each one is solution to (3.16), it follows that:

   1 -1 τ 1 1 τ 2 1 1 -1 τ 2 1 τ 2 2 1 -1 τ 3 1 τ 2 3    ki kp k d =    1 τ 3 1 1 τ 3 2 1 τ 3 3    ⇔ M K T = N (3.18)
Finally, the gains of the controller (3.15) are obtained as

K T = M -1 N : ki = 1 τ1τ2τ3 , kp = τ1 + τ2 + τ3 τ1τ2τ3 , k d = τ1τ2 + τ1τ3 + τ2τ3 τ1τ2τ3 , (3.19)
where τi ∈ R + .

We proposed to use the SAC algorithm to estimate at each time step the best values of poles Eq. (3.16) that can be mapped into gains values according to Eq. (3.19) to derive the resulting control input. For this first design of a learningbased adaptive controller, we proposed to use the incremental pole adjustment method which consists in adding a small value to the poles starting from a modelbased configuration of pole values that will be named in the following as nominal pole values. By doing so, we avoid jumping from a configuration of gains to a totally different one, which can give rise to undesired oscillations here as this first mapping Eq. (3. 19) is not bounded. The nominal set of pole values is obtained using the empirical ZieglerNichols method in an environment without wind disturbance. The resulting poles for each DoF are then equal to:

τ1 = 1, τ2 = 2.5, τ3 = 0.875. (3.20)
which is, when using our mapping proposed in Eq. (3.19), equivalent in the space of gains to:

kp = 2, ki = 0.457, k d = 2.542. (3.21)
This nominal set of pole values is not optimal because it was determined for one operating condition only which in addition did not incorporate wind disturbance (because it was too difficult to make the MAV converge under wind disturbance without a good enough set of gain values). Therefore, our approach consists in starting from this nominal set (3.20) and updating the pole values at each time step using τi(t+1) = τi(t)+∇i with ∇i ∈ [-0.01; +0.01]. Then, the updated gains are used to compute the associated PID control input that is ultimately applied. This is denoted as learningbased adaptive control because the resulting control input Eq. (3.15) is a function of modelbased information (i.e. the measured error values) and of the outputs of a deep neural network (i.e. the adjusted pole values). The objective of the learning agent is thus to build a predictive model that directly maps ∇i from the current state to adjust the poles starting from this initial configuration:

π θ : S ⊂ R 222 → A ⊂ R 18 x = [st] T → [λi, µi] . (3.22)
With the learningbased adaptive controller, the outputs of the Policy network are the 9 pairs (λi, µi) of mean and standard deviation. The ∇i added to the pole τi is modeled by Normal distributions defined as:

N (∇i) = (2πµi) -1/2 exp{- 1 2µi (x -λi) 2 }. (3.23)
In summary, with this controller the outputs of the Policy network are 9 pairs (λi, µi) of mean and standard deviation that are used to model normal distributions Ni(λi, µi). The value ∇i is then sampled from Ni(•).

The pole τi(t) is updated with τi(t + 1) = τi(t) + ∇i(t). Finally, the resulting poles are transformed back into the space of gains and the PID control law is computed and applied.

State vector

We used a slightly different observation vector from the modelfree scheme (3.11). Indeed, we propose to add the resulting pole values and the PID controller outputs (3.43) in the observation vector: where dim(τrpt) = 9, pidrpt = [ϕr ; θr ; T ] and dim(ot) = 37. We constructed the state vector out of this observation vector st as earlier Eq. (3.12), resulting in a state vector of dimension 222.

ot =[

Reward function

The following reward function rt has been designed in order to teach the agent how to complete the mission of target rallying:

rt =      r receded if drate ⩽ 0, r f orward if drate > 0, r reached if dt ⩽ d reached , r f ailed if zw / ∈ [0.25; 20], (3.25)
where rt is the reward of the agent at time step t; drate is the distance rate to the target performed between the last two time steps such as drate = dtdt-1; d reached is the limit value beneath which we consider the target to be reached ; zw ∈ RW is the MAV altitude; both r reached and r f ailed are terminal rewards determined at the end of the ongoing episode. Each of these terms represents the specific features of the desired behavior of the MAV:

• r receded is a constant negative reward equal to -20 that is sent to the agent whenever the robot is getting away from the target (or staying immobile).

• r f orward is a positive reward generated when the relative distance to the target is decreasing as:

r f orward = C1 × e -[( d t 1+d rate )× 1 C 2 ] 2
With this design, we encourage the robot to move toward the target as fast as possible. We chose the value of the constants C1 in order to scale the positive reward signal. This is particularly important because as mentioned in [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF], the SAC algorithm is particularly sensitive to the scaling of the reward signal which is the magnitude of the reward value. We followed the recommendation prescribed in [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] and chose to set C1 = 20 in order to obtain a positive reward scale of 20 (which we found in practice to be the reward scale that gave us the best performance for this problem). The constant C2 represents how sparse is r f orward based on the distance to the target. We chose the value C2 = 20 empirically. Therefore, the positive reward is equal to:

r f orward = 20 × e -[( d t 1+d rate )× 1 20 ] 2 (3.26)
• A constant positive reward is sent to the agent when it succeeded to complete the mission, meaning dt ≤ d reached . This generates r reached = +1000.

• If the MAV altitude zw exceed a threshold, the constant negative reward r f ailed = -550 is generated.

Training

A training episode is defined as follows: at the beginning of the episode, the MAV is set at the center of the environment at an altitude of 3 meters with roll, pitch and yaw angles equal to 0. The mission then begins and is considered a success if the relative distance to the target is inferior to a predefined threshold d reached and as a failure if an error signal is generated, both cases ending the episode.

Otherwise, the episode is ended if the number of iteration steps reaches the maximum value allowed per episode which is set at 300. The training for both controllers consisted in performing 1 000 000 iterations in an environment with a varying wind field (as described in Section B). To help the agent, The LB strategy presents a much higher convergence speed to a significant success rate than the MF strategy, which shows the great potential of combining modelbased classical controllers with learning procedures. This can be explained by the modelbased part of the LB controller which allows it to choose relatively good actions despite being at the early stage of the training session. Therefore, from the beginning of the training, the LB controller is able to explore a much higher part of the reward space than the one of the MF controller. We believe this significantly helps the DRL algorithm to find better overall strategies. We also evaluated a fixed control strategy which consisted of a PID controller with the fixed nominal poles configuration (3.20). In the following, this modelbased optimal but nonadaptive controller will be denoted as OFP for Optimal Fixed Poles controller.

d

Discussion

The outcomes of this evaluation are provided in Tables 3.5 and 3.6. In terms of control performance, the learningbased controller is on average completing the task in 281 control steps. The Modelfree and OFP controllers take on average 357 and 488 control steps respectively. In other words, the proposed learning based controller is between 27% and 73% faster than the other methods (respectively for the Modelfree and OFP controllers). In terms of reward, the learningbased controller is outperforming the other methods with a mean total reward between 1.80 and 5.68 times higher than the other methods (respectively for the Modelfree and OFP controllers). The highest reward per step is obtained with the learningbased controller where it is between 2.29 and 9.86 times higher than the other methods (respectively for the Modelfree and OFP controllers).

The learningbased controller is also dominating in terms of task performance. It displays a success rate of 91.6% against 74.2% and 50.6% respectively for the Modelfree and OFP controllers. The positive reward rate represents the rate of actions that resulted in the reduction of the distance to the target over an episode length. This variable tells us how stable the convergence to the steady state is. We can see that despite using neural networks, both the Modelfree and Learningbased controllers display a similar positive reward rate that is between 1.40 and 1.45 times higher than the OFP controller. This particular result shows the benefits of adapting the controller response using deep reinforcement learning as with the OFP controller, the poles are fixed and do not vary over the episode.

Furthermore, on average, fewer actions are required to achieve the task with the LB controller despite shar ing the same sampling rate (see Section 2.3.2). The mean reward per step of the LB controller is more than 2 times higher than the Modelfree one. On the other hand, the OFP controller is showing a much lower success rate, close to 50%. We observed that with this strategy, failures mostly consisted in cases where the MAV is close to the steady state but is being deviated by the wind gust. The vehicle is then not able to recover from this disturbance because the fixed poles are not conservative enough to provide satis fying performance over a large spectrum of wind characteristics. These first results validated the proposed learningbased adaptive control design. By keeping a modelbased control structure (i.e. PID control law), we can use deep reinforcement learning to design an adaptive poleplacement procedure where the controller parameters are a function of the process state. Despite not taking into account the current disturbance, it is still implicitly observed by the adjustment mechanism (i.e. neural network) as the value of the vehicle velocity and acceleration are included in the neural network input vector.

Nevertheless, this first design Eq. (3.16) holds the Routh-Hurwitz stability criterion that by considering only pole values of null imaginary component and negative real component, it ensures the stability of linear time invariant systems. Here, stability refers to the fact that the system output is bounded, which means that as time goes to infinity, the system will converge to a steady state. This stability component is not enough for realworld systems, where the great number of disturbances and uncertainties make the system time variant, and thus the Routh-Hurwitz stability criterion does not hold anymore. In this case, the stability of the control system can still be ensured using the classic Lyapunov stability theory [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF]. In this field, systems are modeled from an energy point of view, and the stability of the system is assessed in terms of energy dissipation. In other words, if we can have the mathematical proof (using Lyapunov stability theory) that the energy of a system is always reducing over time, we can guarantee that the system is stable and will reach a steady state. This approach has been studied and used for decades and there are various tools already existing to perform such stability analysis on AUVs. However, as soon as a nonlinear function approximator, namely neural networks, are present in the control system, it makes the classic Lyapunov stability analysis difficult or even impossible. For this reason, and in accordance with the internship of Hector Kohler in our laboratory at ENSTA Bretagne, we proposed a methodology to assess the Lyapunov stability of a learning based adaptive controller which we present in the next Section 3.2.3.

Lyapunov stability of learningbased adaptive control

In the case of modelbased adaptive control, the stability analysis of AUVs control systems is straightforward to conduct and successful study can be found in the literature [START_REF] Gong | Lyapunovbased model predictive control trajectory tracking for an autonomous underwater vehicle with external disturbances[END_REF][WSS21] [START_REF] Shen | Trajectory Tracking Control of an Autonomous Underwater Vehicle Using LyapunovBased Model Predictive Control[END_REF]. However, as previously discussed, in Section 3.2.2, the Lyapunov stability analysis [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF] is difficult to apply to control systems that use deep neural networks because we can not predict beforehand all the possible values that these nonlinear function approximators can output. If we want to design a stable AUV control system, one designer can restrict the space of control parameter possible values to a Lyapunovbased space using the numerous Lyapunov function candidates from the literature [START_REF] Fossen | Nonlinear Modelling And Control Of Underwater Vehicles[END_REF]. However, by restricting such a way the space of control parameters, we might lose all the benefits of using deep reinforcement learning to adjust their values. In other words, if there are only a few desired possible values, there might not have an advantage to adjusting them. Moreover, the resulting Lyapunovbased space of control parameters does not take into account the process variation and therefore is not optimal over the entire operating regime. For this reason, we propose to analyse empirically the stability component of an AUV learningbased adaptive control system. The choice of algorithms and methodology was made by Hector Kohler who is the first author of the resulting paper, by the application remains the one of interest of this thesis which is the adaptive control of AUV. The results were summarized and published [START_REF] Kohler | PID Tuning using CrossEntropy Deep Learning: a Lyapunov Stability Anal ysis[END_REF] and presented at the 14th IFAC Conference on Control Applications in Marine Systems, Robotics, and Vehicles (CAMS) 2022. In the following, we summarize the main results of this paper, and its preprint version can be found in Appendix C.

Task Description

In this study [START_REF] Kohler | PID Tuning using CrossEntropy Deep Learning: a Lyapunov Stability Anal ysis[END_REF], we address the same target rallying mission with again the RexRov2 platform using UUV Simulator (see Section 2.3.2). In the following, we will use a slightly different notation compared to the one from the paper [START_REF] Kohler | PID Tuning using CrossEntropy Deep Learning: a Lyapunov Stability Anal ysis[END_REF] in order to match the notations of this thesis. The tracking error e is defined as the error between, the current AUV's position and orientation, and a fixed target represented by a desired position and orientation. Thus, the control objective consists in minimizing the error between the vehicle state xv and the desired setpoint xw:

e = xv -xw. ( 3

Stability analysis methodology

Learningbased adaptive control methods, where neural networks are used, are dominating in modern robotic applications. However, we can observe different dynamics in maritime applications where such methods are yet to be successfully deployed on real platforms and for long operations. This can be explained by the fact that in underwater applications as studied in this thesis, we have limited observability of the process due to the limited embedded sensors. In addition, these vehicles have to face various disturbances acting on their body that we are yet able to model properly. In this context, learningbased methods can hardly be considered as little to no guarantee can be provided on the output of a neural network. This can be very dangerous for AUVs, which can lead to damage on the platform or worse to lose in the sea. For this reason, the community is particularly interested in certifying these types of methods by using classic stability tools from control theory. Nevertheless, whenever we are using a neural network, it is not straightforward to apply the standard Lyapunov stability methodology [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF]. We propose here a methodology to apply such analysis to a learningbased control system similar to the ones proposed so far in this thesis. In particular, we are interested in studying how much a solution obtained from a neural network can hold some stability components without having explicitly included them in the neural network optimization scheme. The objective is to assess how different the solutions obtained from Lyapunov theory are from a Deep Learning method.

To achieve this, we consider again the RexRov2 platform described in Section 2.3.2. Because we have access to the thruster allocation matrix Eq. (2.98), the vehicle can be controlled in all of the DoFs of interest (i.e. surge, sway, heave, roll, pitch and yaw). Therefore, the control problem 3.2.3 can be framed as a double integrator:

u = B -1 [ J T (xv) kpe + ki t 0 e(τ )dτ -k d ẋv + g(xv) ], (3.30)
where u ∈ R 6 are the control inputs; the PID gain matrices kp, ki, k d ∈ R 6×6 are the output of a neural network; xv ∈ R 6 is the current UUV's state, i.e. position and orientation (Eq. 3.28); ẋv ∈ R 6 is the temporal derivative of the state vector; g(xv) is the sum of external forces acting on the UUV's body (in our case, gravity); e ∈ R 6 is the tracking error (Eq. 3.27); B -1 is the fixed known thrusters allocation matrix mapping the control input u into a combination of thruster power inputs resulting to the desired movement and J T (η) is a transformation matrix.

The principal advantage of the considered PID regulator (Eq. 3.30) is that global stability and convergence analysis of the system has been well formalized by Fossen [START_REF] Fossen | Nonlinear Modelling And Control Of Underwater Vehicles[END_REF]. Lyapunov stability theory [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF] is straightforward to apply with such a control law. Following [START_REF] Fossen | Nonlinear Modelling And Control Of Underwater Vehicles[END_REF], there exists a Lyapunov function candidate V (x) for the considered UUV such that:

V (x) = 1 2 x T M -1 η αI 0 αI kp ki 0 Ki αki x, (3.31)
where α ∈ R is a small positive constant and the PID gains are kp, ki,

k d ∈ R 6×6 ; M -1
xv is related to the UUV's mass and can be computed from the current xv; we define st the control loop's state (not to be mistaken with the UUV's state): st = p, xv,

t 0 e(τ )dτ T ∈ R 18 and p = Mx v ẋT v ∈ R 6
is the generalized momentum depending on the UUV's mass and velocity. Lyapunov stability theory [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF] tells us that the convergence to the steady state of the feedback loop can be guaranteed by assessing the value of the Lyapunov function (3.31) which is a function of the vehicle state. Thus, the control loop is stable at state x if and only if:

V (x) > 0 and V (x) < 0. (3.32)
With this design, we can assess the stability of the AUV process at each time step by computing the Lyapunov function candidate Eq. (3.31).

We are also interested in the stability of the control parameters. In fact, we are often concerned by the values that a neural network can feed as output. There is little to no guarantee that these values will always remain small. Again, following Lyapunov stability theory [START_REF] Liberzon | Review of Liapunov functions and stability in control theory[END_REF], there exists theoretical constraints on the gain matrices kp, ki, k d and the small constant α such that local stability is guaranteed when the initial conditions of the systems are closed to x = 0. In other words, if these constraints on the controller parameters are respected, we can locally ensure further stability of the feedback loop. According to the proposed Lyapunov function (Eq. 3.31), the stability of the control parameters is guaranteed (Eq. 3.32) if the following constraints are satisfied:

         k d > Mη, ki > 0, kp > k d + 2 α ki, 1 2 (1 -α)k d -αMη + α 2 6 i=1 (ηi -η id ) ∂Mη ∂η i > 0, α > 0, (3.33)
We found that, when limiting the parameter space to a value satisfying (Eq. 3.33), the resulting space is so small that the benefits of adaptive control are merely preserved. Therefore, we propose not to take into account the stability constraints in the parameters optimization. Our objective is then to assess to what extent the resulting solution, obtained from an ANN, can still hold some stability components. In order to facilitate the stability analysis, we want to reduce the dimension of the space depicted in Eq. (3.33). First, we can transform the constraints (3.33) into equalities as follows:

             k d = Mη + M1, ki = 0 + M2, kp = k d + 2 α ki + M3, α = -k d (-k d -2Mη + 6 i=1 (ηi -η id ) ∂Mη ∂ηi ) max + ϵ, ( 3

Design of the learningbased adaptive controller

We propose to learn a stochastic predictive model π θ (parameterized by θ) presented by a neural network, that maps the system state vector x into the controller parameters of the PID law (as illustrated in Figure 3.4):

π : Ωx ⊂ R 18 → Θ ⊂ R 19 x = [st] T → [λi, µi] (3.36)
The neural network takes as input the control loop's state vector st Eq. (3.31) and returns as outputs the 19 pairs (λi, µi) of mean and standard deviation that are used to model the control parameters Eq. (3.35) where each variable is represented by a Gaussian distribution Ni defined as:

Ni(λi, µi) = (2πµi) -1/2 exp{- 1 2µi (x -λi) 2 }. (3.37)
The control parameters [Λ1, Λ2, Λ3, ϵ] ∈ R 19 are thus determined using the Gaussian distributions Eq. (3.37), which allow us to assess the stability of the control system. We have empirically set up an architecture composed of 2 hidden layers of 32 hidden nodes each, with the Sigmoid activation function applied to each layer. This results in a total of (18 + 1) × 32 + (32 + 1) × 32 + 2 × ((32 + 1) × 19) = 2918 parameters (ω) to learn from data. In a second stage of the mapping π, the PID parameters [Λ1, Λ2, Λ3, ϵ] are obtained by sampling from the resulting Gaussian distributions N (µ, σ). The Eq. 3.34 allows to the computation of the final PID parameters and using Eq. 3.30 the PID control inputs are derived. Accordingly, we can more easily assess the Lyapunov stability of the system. The overall control strategy is illustrated in Figure 3.4.

We propose to optimize the weights of the neural network with the CrossEntropy Method (CEM) which is a direct search optimization approach (i.e no gradient is computed). It is an Estimation of Distribution Algorithm (EDA) inspired by Natural Evolution Strategies [START_REF] Wierstra | Natural Evolution Strategies[END_REF]. With CEM, during one iteration k, N sets of weights S k i=1...N are sampled from a Normal distribution directly in the space of weights R 2918 . At each iteration k, N evaluations are made to determine the current best weights S k best=1...N ×ρ with respect to a given cost function. In our case, we used the following classic control performance (based on multi steps within an episode and connected to the tracking error (Eq.3.27) index as a cost function to minimize:

J = steps 1 6 6 i=1 (η d,i -ηi) 2 .
(3.38)

The mean and covariance of the Normal distribution are then updated as the mean and the covariance of the N × ρ best sets of weights obtained at iteration k -1. Noise σ 2 noise is added to the covariance of the best weights to avoid local optima. We randomly sample the next iteration weights as:

S k = N (mean(S k-1 best ), Cov(S k-1 best ) + σ 2 noise ) (3.39)

Analysis protocol Training settings

We used the following CEM hyperparameters that have been chosen through a grid search: population size The training dynamics are represented with the evolution of the MSE on the setpoint in Figure 3.5a and with the evolution of the covariance of the normal distribution in Figure 3.5b. We can see in Figure 3.5a that using the CEM approach, we rapidly converge to what seems to be a minimal MSE value. The target rallying mission is successfully performed with an error on the setpoint which converges to a minimal value. On the other hand, we can see in Figure 3.5b that the covariance of the normal distribution is not converging to a minimal value. This means that the CEM algorithm has yet not converged to optimal values. A high value of this covariance means that the sphere, that is used to explore the space of parameters of the neural network, is getting bigger. This means that there remain some parameters that can be optimized. However, as we did not observe a notable gain in MSE reduction by letting the CEM optimization algorithm run longer, we proposed to stop it after 200 epochs despite this high covariance value.

Evaluation settings

Our objective is to measure the stability ability of the AUV learningbased adaptive controller against process variation. For this objective, we propose three evaluation scenarios based on induced disturbance: none, Gaussian noise in sensor measurements, and Gaussian noise in control inputs with current disturbances denoted respectively as scenarios 1, 2, and 3. The length of the episode is now increased to 2000 time steps. The initial state of the UUV is changed at the beginning of each episode. For simplicity purposes, the proposed controller will be denoted in the following as LB PID. We compare the LB PID controller to its Lyapunovbased counterpart which consists in a PID controller whose parameters are set to M1, M2, M3 = Mi × J6×6 with:

Mi =        0.5 -10 -5 0 0 0 0 0 0 0.5 -10 -5 0 0 0 0 0 0 0.5 -10 -5 0 0 0 0 0 0 0.5 -10 -5 0 0 0 0 0 0 0.5 -10 -5 0 0 0 0 0 0 (0.5 -10 -5 ) + 10 -5        (3.40)
The values used for the Lyapunovbased PID controller Eq. (3.40) have been chosen such as to satisfy as much of the constraints on the control parameters stability Eq. (3.34). The resulting controller remains adaptive (as the gains are a function of the vehicle state x) and will be denoted as naive PID henceforth. The only difference between these controllers is how each of these controllers adjusts the value of the parameters used to derive the PID law (3.30), making the comparison fair. The objective of the evaluation is to compare both controllers in terms of control performance but also in terms of system and parameters stability thanks to the methodology described in Section 3.2.3. The evolution of the stability metrics is illustrated in Figures 3.8a to 3.8f. We can see in Figures 3.8a, 3.8b and 3.8c that despite not taking into account any stability constraints on the optimization procedure, the LB PID matches the system stability of the purely Lyapunovbased controller (i.e. the naive PID). This is due to the fact that the vehicle itself is fundamentally designed to be highly stable. This means that when facing no disturbance, the AUV will naturally stabilize itself to a steady state where no energy dissipation occurs. The fact that the control input is a function of a neural network has little impact on the stability of the state. Nonetheless, we can observe a divergence of the state stability in scenario 3 where we are facing sea current disturbance. This is due to the fact that it is not possible to design a vehicle that will remain stable against any possible process variations, which by definition is infinite, while the number of actuators is finite. When looking at the control parameters' stability, we observe a different trend. In fact, in the three scenarios, the control parameters of the LB PID controller are never satisfying the constraints Eq. (3.33) while the parameters of the purely Lyapunovbased controller satisfy them. With the latter, we can even notice that the control parameters' stability tends to increase over time, as the closer we are to the target, the smaller the Lyapunovbased control parameters gets and the less likely it is to diverge from it.

Discussion

From this study [START_REF] Kohler | PID Tuning using CrossEntropy Deep Learning: a Lyapunov Stability Anal ysis[END_REF], we were able to analyze and measure the stability of a learningbased adaptive controller. The main result is that despite the use of a neural network, the state stability of the AUV matches the one displayed by a Lyapunovbased controller. In the following, we will present a summary of the different suggestions rising from these preliminary studies in order to adapt the proposed learningbased adaptive control system to an AUV.

Suggestions

Based on the preliminary studies presented in Section 3.2, we identified three challenges that we wanted to tackle in this thesis in order to propose a satisfying learningbased adaptive control method for AUVs. These challenges guided the design of the proposed solution, and before presenting the novel learning based adaptive control system, we list below the thought processes that led to its design:

1. Interpretability of the control system: when designing a control system, we are interested in tuning our system so as to reach the desired control performance. In control theory, these performances are traditionally expressed in terms of desired settling time, overshoot, frequency of damped oscillation, or even slew rate. These requirements are usually provided by the operators, which are the experts that will be using the AUVs, and the task of the control designer is to provide them with a control system capable of answering these needs. In the case of the PID control law, as studied in this thesis, these desired control performances can not be analyzed directly in the space of gains as they are defined in a temporal plane. For this reason, we would be interested in transforming these gain values in the space of poles which is defined in a frequency plane (using the Laplace transform). There, the aforementioned performance criterion can be easily derived and the space of pole values can be bounded so as to ensure that we meet them. Therefore, we could be able to define a region in the space of poles where the control performance is at worst equal to the desired one, and which can be interpreted physically more easily.

2. Parameters adjustment against unobservable disturbance: the principal challenge of the AUV applica tion is that we have limited observability of the process. In the case of observable disturbances, the control parameters can be adjusted using modelbased adaptive control theory such as LQR optimiza tion [START_REF] Lucas | PID, LQR and LQRPID on a quadcopter platform[END_REF], Gain Scheduling [START_REF] Benoit | AEROSPACE LAUNCH VEHICLE CONTROL: A GAIN SCHEDULING APPROACH[END_REF], or H2/H∞ control [START_REF] Arzelier | Multiobjective H2 / H∞ / ImpulsetoPeak Control of a Space Launch Vehicle[END_REF]. However, in our application, the vehicle is not equipped with sensors allowing us to measure the current disturbance. Therefore, these modelbased methods can not be used. In this context, we need to use some modelfree optimization methods and deep reinforcement learning. It allows us to map the pole values to a state representation of the process (which is different from the state of the AUV) that can encompass a large amount of information on the process, and thus in a modelfree manner: we do not need any process model to found the optimal pole values (see Section 2.1.2).

3. Performance robustness of RLbased agent: deep reinforcement learning methods are associated to a great number of parameters that influence greatly the performance of the resulting agent. Because of their stochastic nature, these types of methods are associated with a high performance variance. A key element to control the robustness of the agent lies in the Experience Replay mechanism described in Section 2.2.6. Despite some works [ZS17][Sch+16] [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF] showing the effect and the limits of the original ER, it is mostly ignored by the community. Our objective is to propose a new algorithmic pro cedure to improve the ER technique by taking into account the characteristics of the replay mechanism taking place in biological systems. This way, the resulting methodology can be applied to any OffPolicy RL algorithm.

Based on the above suggestions, we present in the next Section 3.3 the contributions of this thesis. The objective of the theoretical and algorithmic elements is to address the aforementioned challenges in order to propose a satisfying adaptive control system for AUVs. Following the preliminary studies 3.2, we first present an augmented design of the learningbased adaptive control system that can be tuned according to to control performance requirements that can be physically interpreted. Then, we will present a novel ER technique that aims at reducing the performance variation of RLbased agents.

A novel learningbased adaptive controller

Design of the modelbased structure

Feedback controller

We introduce now the final version of the proposed learningbased adaptive controller. This work assumes that the controlled vehicle is fully observable and controllable. This means that each of the vehicle's DoF is measurable and the desired vehicle states (within the operating regimes) are supposed to be accessible. However, the model of the process B is not available and we are facing unobserved disturbances. In this context, a PID controller is a suitable modelbased structure to regulate the known part of the process. We can take into account the unknown current disturbance by considering the steadystate error variable σ = t 0 e(τ )dτ . We can rewrite the statespace equations with the augmented state vector X = [σ, e, ẋ] as:

d dt σ e ẋ = 0 1 0 0 0 1 0 0 0 A σ e ẋ + 0 0 1 B u. (3.41)
The PID statespace representation is given by:

Ẋ = (A -BK)X. (3.42)
The PID control law can then be derived as:

u = kpe + kiσ + k d ẋ, (3.43)
with kp, ki and k d ∈ R + , antiwindup added on the integral term such as max(σ) = max(u) and a lowpass filter is applied on the derivative term to reduce oscillations induced by process noise. By considering this control structure, the proposed method can be applied to any closedloop control process where essentially we have access to feedback from the system outputs.

Adaptive direct poleplacement strategy

Among the various procedures and rules that can be applied for PID tuning [START_REF] Wang | Handbook of PI and PID Controller Tuning Rules, Aidan O'Dwyer[END_REF], a fundamental technique consists of assigning a set of specific values, P = {λ1 λ2 . . . λn}, to the eigenvalues of the feedback loop A -BK. Given that these eigenvalues determine the poles of all the transmittances where the associated state matrices are involved, this procedure is denoted as PolePlacement. We can define a (normalized) control polynomial as:

C(s) = s n + c1s n-1 + • • • + cn-1s + cn, (3.44)
whose roots are the λi, which can be assigned the characteristic polynomial of A -BK with:

C(s) = det(sI -(A -BK)). ( 3
|A -BK -λI| = -λ(λ(k d + λ) + kp) -ki, = -λ 3 -λ 2 k d -λkp -ki, = 0. (3.46)
To ensure minimal stability of the feedback loop (in terms of output boundness), the poles of (3.46) must be placed in the complex left halfplane. For this purpose, we only consider as eigenvalues candidates the solutions of:

λ 3 + λ 2 k d + λkp + ki = 0. (3.47)
In order to maintain the dimension of the gain space, we propose as pole values candidates τi ∈ R + the following design:

λ1 = -1 τ1 ; λ2 = -1 τ2 ; λ3 = -1 τ3 . (3.48)
With this formulation (3.48), we essentially propose a symmetric mapping where each pole contributes equally to the closedloop eigenvalues. The PolePlacement design can be written as:

     -1 τ 3 1 + k d τ 2 1 - kp τ 1 + ki = 0 -1 τ 3 2 + k d τ 2 2 - kp τ 2 + ki = 0 -1 τ 3 3 + k d τ 2 3 - kp τ 3 + ki = 0 (3.49)
Since τ1, τ2 and τ3 are solutions to (3.47), it follows that:

   1 -1 τ 1 1 τ 2 1 1 -1 τ 2 1 τ 2 2 1 -1 τ 3 1 τ 2 3    ki kp k d =    1 τ 3 1 1 τ 3 2 1 τ 3 3    ⇔ M K T = N (3.50)
The gains of the control law (3.43) are obtained by transforming back the poles with K T = M -1 N as:

ki = 1 τ1τ2τ3 ; kp = τ1 + τ2 + τ3 τ1τ2τ3 ; k d = τ1τ2 + τ1τ3 + τ2τ3 τ1τ2τ3 (3.51)
This mapping Eq. (3.51) is totally novel and it is the first contribution of this thesis.

Using Eq. (3.51), the bounds for the controller parameters can be defined based on control constraints that are easier to derive in the poles domain. In the present case, with the design of Eq. (3.48), for any τi > 0, the poles of the feedback loop A -BK are placed on the xaxis of the complex left halfplane, reducing the settling time and overshoot. This leaves us with the settling time requirement to define in order to set the desired distance from the yaxis and thus bound the value of the space of poles. In accordance with the considered control objective, we can define the desired maximum settling time of the closedloop control ς as the maximum time (in seconds) after which we want the system outputs to stay around a percentage χ (e.g. χ = 0.05 for 5%) of its desired values. Accordingly, the upper bound of the space of poles is derived as λmax = ln (χ) ς . Therefore, as λi ≤ λmax from Eq. (3.48), we can derive the upper bound of the poles:

τmin < τi ≤ 1 -λmax . (3.52)
There exist a solution for all C(s) (3.44) if and only if the pair (A,B) is controllable, which is assumed in this thesis. In the case of a SingleInput system, the solution is unique. In the case of MultiInput systems, as studied here, the number of free components of the matrix K is greater than the n eigenvalue constraints. Accordingly, there exist an infinite number of solutions, among which it is not trivial to define an Optimal solution. In fact, depending on the process dynamics, we might favor one particular configuration of pole locations in the space (3.52) over another. The Linear Quadratic (LQ) optimization scheme then composes an alternative framework to define optimal values.

In contrast, we propose to use DRL to adapt these parameters for the control of an AUV, which consist in searching for the best values possible within (3.52) based on the process measurements. More precisely, our approach consists in using a Deep Policy Gradient method [START_REF] Richard | Reinforcement learning an introduction Second edition[END_REF] whose objective is to explore the space of poles (3.52) in order to find at each time step the best values possible (in this space) for each control input. The policy objective is to adjust them based on the process variation and with respect to a reward function that emphasizes the control objective. This is denoted as a Direct method because the controller parameters are adjusted directly without the need for estimation of any process parameters. With the mapping in Eq. (3.51), the learning action space is limited to desired solutions in the poles space as illustrated in Figure 3.9. This ensures that for any pole values chosen by the ANN in the parameter cube 3.9, the resulting control input will maintain the poles of the closed loop in the left halfplane.

Incremental VS direct pole adjustment

The continuous space of desired poles illustrated in Figure 3.9 can be explored in different ways. In this thesis we considered two approaches denoted as Incremental and Direct approaches:

• In the incremental case, we propose to first initialize the poles at some satisfying values within the resulting space of desired solutions (which is bounded according to the desired control performance as previously explained in Section 3.3.1). The initial pole values are chosen using the Ziegler-Nichols methodology, a modelbased heuristic approach. This way, we ensure that this nominal configuration provides satisfying minimal performance despite not being optimal for the complete operating regimes. Then, during operation, we adapt the pole values by adding or subtracting a small value from this nominal configuration.

• In the direct case, the poles switch from one value to another without a smooth transition. The agent can access any value of the space instantly.

The incremental procedure has the advantage of being particularly stable in terms of feedback oscillations.

In fact, despite changing the controller parameters at each time step, the change being small, the feedback loop has the time to converge to a steady state.

Nevertheless, the solution obtained from the Ziegler-Nichols method is not adaptive and can only compen sate for the process characteristics observed when applying the method. In particular, when the disturbances are large and/or when we have a limited measurement ability of the system state, the solution obtained from this heuristic method is not conservative enough to ensure satisfying performance over a wide spectrum of operating conditions. Therefore, with the incremental approach, it can take a notable amount of time before the control parameters move to proper values for the given operating conditions, which can often be too late. This motivated the direct pole adjustment where we can move from one pole configuration to another instantly between two time steps.

This learningbased adaptive control method can be seen as a modelfree variation of modelpredictive con trol (MPC) (see Figure 3.11). In comparison with MPC, with our method:

1. we do not need a system model, which is exactly the case study of this thesis. Because of the various disturbances and process uncertainties, we do not have a system model of our AUV process.

2. we do not rerun the entire optimization forward in time but rather infer instantly from a deep neural network. MPC relies on the fact that fast computational hardware is available to run the optimization online at every time step, which is most of the time not the case in our underwater applications. The vehicles of interest are equipped with smallsize CPU cards, such as Raspberry Pi, that are not powerful enough for realtime optimization.

3. the computed action is obtained from a nonlinear function of a state representation of the overall pro cess and not only of the system state. Therefore, the resulting solution takes into account much more information (the dimension of the process state is of the order of hundreds while the dimension of the vehicle state is of the order of tens). The resulting solution can thus more easily take into account uncertainties that are not included in the state variables of the controlled system.

Design of the modelfree optimization algorithm

Stochastic policy

When having access to little to no information on the disturbances and when facing a timevarying process, as considered here, modelfree adaptation can be exploited. In order to take into account the uncertainties in the poles selection, we propose to use DRL to build a stochastic predictive model π θ that maps a state vector st into the pole values:

π θ : S ⊂ R dim(S) → A ⊂ R 3×dim(u) x = [st] T → [λi, µi] . (3.53)
where N (τi) is the probability density function of τi that is modeled by a Normal distribution N (τi) as:

N (τi) = (2πµi) -1/2 exp{- 1 2µi (x -λi) 2 }, (3.54)
with λi ∈ R and µi ∈ R + are the mean and variance of p(τi) that are estimated by the Policy network. There fore, the outputs of the Policy network are the 3×dim(u) pairs of (λ, µ) representing the Normal distributions N (τi) used to sample the poles for each control input ui. In practice, the current Ti(t) is sampled from N (Ti) after applying an invertible squashing function (i.e. tanh) to N (Ti) (in order to bound the Gaussian distribu tion) and after using the change of variable to compute the likelihoods of the bounded action distribution (see Appendix C of [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] for the complete details of this process). Designing this stochastic function (3.53) is numerically expensive due to the dimensions of the underlying spaces, excluding realtime computation with modelbased methods only. The DRL framework allows us to iteratively build an estimate of this optimal mapping function.

The related methods cited later in Section 3.1 are mostly relying on the DDPG and TD3 algorithms. They are known to involve intensive tuning of hyperparameters to work properly. Additionally, in order to take into account the ocean current disturbances, one designer might favor a stochastic policy that is known to be more robust to uncertainties and to partially observable processes (at the cost of being less stable during training).

For these reasons, we chose to use the Soft ActorCritic algorithm described in Section 2.2.8. The SAC algorithm builds a stochastic policy where the action distributions are modeled by Gaussian distributions3

Stabilizing TDLearning

The Mean squared error (MSE) (squared L2 norm) is the most commonly used error criterion in the context of supervised learning. However, it is not appropriate for TDLearning defined in Section 2.2.8 where the target values (i.e. labels) move at each gradient update. Depending on the amplitude of the reward function, one can often encounter MSE in the order of tens to hundreds, and as the targets move, we are naturally more subject to sudden error increase. The associated MSE generates a high value of error derivative which can easily lead to exploding gradients and divergence. For this reason, we used the Smooth L1 loss function from Pytorch [START_REF] Paszke | PyTorch: An Imperative Style, HighPerformance Deep Learning Library[END_REF] for Critic's optimization as it is less sensitive to outliers compared to the standard MSE loss function. In our case, the Smooth L1 loss function acts as the L2 loss for errors of absolute value lower than 1 (i.e. fewer oscillations when the error is low) and as the L1 loss function otherwise (i.e. steady gradient for large error). It is less sensitive to outliers and reduces the chance of exploding gradient compare to the standard MSE loss. As outlined in Section 2.2.9, the standard ER technique that is commonly used to optimize the above neural networks is associated with a number of challenges. In the following section, we present our contribution to the ER mechanism in order to improve its sample efficiency and stability.

Design of the bioinspired experience replay Experience Replay

Offpolicy reinforcement learning tells us that we can learn the optimal policy from samples generated by any other policies. Following this heuristic, we can optimize the actor similarly to the critics, by using the agent's past experience. The stateoftheart method to manipulate the agent's past experience in this context is called Experience Replay (ER) and was presented in Section 2.2.6.

A deeper look: the CER method

Nevertheless, even with this first ER formulation, there is a great number of parameters that are usually ignored despite having an impact on the learning performance, including:

The replay buffer size: the total number of transitions that the replay buffer can store. When its maximum size is reached, the replay buffer is accessed in a firstinfirstout fashion. The bigger the replay buffer size, the more the data will look like it is IID, which in turn improves the gradient update quality. However, if the replay buffer is too big, an important transition will have much less chance of being used to update the policy, which could impair the learning process. In contrast, if the replay buffer is too small, the learned policy can be the result of an overfitting process on recent transitions, which precludes performance improvement.

The age of a transition: the number of gradient steps taken by the agent since the transition was generated. This value can be seen as a measure of the extent to which the transitions stored in the Replay Buffer are offpolicy, as it tells us how different the current policies are from those stored in the buffer. The age of the oldest policy stored increases with respect to the buffer size.

The replay ratio: the number of gradient updates per environment transition. It can be viewed as a mea sure of the frequency at which the agent is learning using existing data versus learning from collecting new experiences.

The size of the replay buffers, however, can impact negatively the learning performance [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF]. There are two competing methods that can be used to solve this issue: the Combined ER (CER) [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF] and the Prioritized ER (PER) [START_REF] Schaul | Prioritized Experience Replay[END_REF]. CER consists of adding the latest transition performed to the minibatch pooled over the replay buffer, whereas with PER important transitions, as measured by their associated TDerror (2.73)(2.74), are given a higher probability to be used in the gradient updates. Using CER, however, the last transition will undoubtedly be sampled and instantly affect the policy.

However, even with CER, a drop in performance was observed for certain sizes of replay buffer, at some point of the training (even when tuning the learning rate). This behavior was related to the process itself rather than to the aforementioned parameters [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF]. As written in [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF] "CER is a workaround ... and future effort should focus on developing a new principled algorithm to fully replace ER." In this paper, we propose a new ER mechanism with the ambition to decouple the performance of the agent from process complexity (as observed with CER).

Missing biological elements

A recent detailed analysis of ER was provided in [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF], where an analysis of the effects of the afore mentioned parameters was presented. Several conclusions on how the parameters can affect the learning dynamics were drawn, which motivated the ER design proposed in this work. These conclusions can be summarised as follows:

• Increasing the replay capacity while fixing the age of the oldest policy improves the performance be cause it lowers the chances of overfitting to a small subset of (state, actions).

• As the agent trains, it spends more time in higher quality regions of the environment (as measured by rewards), thus learning to better estimate the return in such regions leads to further gains in perfor mance.

• Increasing the buffer size with a fixed replay ratio has varying improvements. The replay ratio stays constant when the buffer size is increased because of both the replay capacity and the age of the oldest policy increase. If one of these two factors is independently modulated, the replay ratio will change.

A bioinspired experience replay

The design of the proposed approach for ER has been mostly motivated by these findings which we tried to incorporate into the ER scheme. A recent study [START_REF] Tyler | Replay in Deep Learning: Current Approaches and Missing Biological Elements[END_REF] undertook a comprehensive comparison between the replay mechanism that takes place in biological brains and those in artificial learning systems. We list below some findings related to DRL only.

Replay (in biological systems

) is temporally structured. Temporally correlated experience sequences are used for learning and memory combinations. This allows for more combinations of neurons which leads to faster emergence of temporal waking experiences. This feature is largely ignored by existing meth ods that only replay static and uncorrelated inputs.

Replay is modulated by reward and only a few selected experiences are replayed. It seems intuitive that not all experiences are useful for learning a new task. Some experiences are more important than others because they incorporate higherquality information about the process dynamics. The challenge here is twofold: how to model this information quality and how to measure it.

Replay is treated differently for novel versus nonnovel inputs. This allows for selective replay to be weighted by novelty. Biological systems tend to reduce drastically the attention given to old experiences versus that given to recent ones as the latter contains more information within them.

We propose a novel bioinspired experience replay (BIER) technique to take into account the abovementioned challenges while keeping in mind the requirements associated with Gradientbased optimization. In this method, the agent experience is divided into two distinct memory units (as illustrated in Figure 3.14) and samples are drawn from them differently as described next.

SequentialPartial Memory: this buffer is denoted as B1 and is similar to the one from the original ER scheme. Here its maximum size is set to 1, 000, 000 and contains old and new transitions. We believe that, especially in the robotic case, the optimal behavior is highly temporally correlated. With robots, early actions do have an impact on future states. Let's consider for example the case of a robotic arm whose objective is to grasp a cup of coffee and place it on the other side of the table. If the robot first grasps the cup in an unsteady manner (because grasping it fast may lead to a local high reward), even if it acts perfectly afterward, there is a high probability that it will spill the contents or drop the cup along the way. Learning this temporal relationship is thus essential in order to learn what truly good behavior is, in the physical sense, is. In addition, even with very different operating conditions, the robot's behavior remains quite similar. This means that the shape of the trajectories remains within a bounded space. Therefore, our hypothesis is that learning on sequences can lead to further gains (compared to temporally uncorrelated samples) because what we learn on a trajectory can directly be applied to future ones that the agent has yet to encounter. Following this heuristic, we propose to sample random sequences of transition from Buffer B1 (i.e. transitions that are successive within the buffer since we store each of them in a firstinfirstout fashion). The vector of experiences E sampled from B1 is composed of n samples as E = [ei ; ei+1 ; . . . ; ei+n], for 0 < i < m-n. While this replay procedure sounds very appropriate for a biological system, here we are optimizing MLPs with limited numbers of parameters and learning abilities. Using highly correlated samples often leads to repeatedly overfitting to those locally correlated samples and never really learning the true value of the functions (i.e. it ends up oscillating between different overfitting regimes). This is commonly observed in onpolicy methods where the ANNs are optimized using samples generated by the same policy. For this reason, we propose to consider sequences that are only partial by storing 1 out of 2 transitions in the buffer B1 which:

• adds a regularization effect by feeding incomplete sequences to the ANNs that encourages these net works to further learn the real value of the functions.

• reduces the age of the oldest policy contained in this buffer, which improved performance in [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF].

Optimistic Memory: both bad and good behaviors are important when learning a new task because they both contain information about the process. We have been able to observe a number of cases where "positive reinforcing" is much more efficient with biological systems, for example with animal training where good and bad behaviors are respectively rewarded with treats or no response (rather than punishment). It was shown in [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF] that trying to estimate values of highquality regions (as measured by the rewards) results in better performance. In addition, as the agent learns, its performance improves and better transitions are performed. Such transitions are important because they can further improve the data collection of the agent in the future. However, as shown in [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF], when using a large replay buffer, such transitions are likely to influence the policy later. Their probability to be sampled decreases as the replay buffer size increases, slowing down performance improvement. Following these heuristics, our objective with this second buffer B2 is to be optimistic about past experience, by increasing the probability of using transitions associated with such highquality regions.

We propose to store in B2 the upper outliers of the reward distribution that we consider to be the best transi tions. Outliers can be defined according to diverse metrics depending on the nature of the variable distribu tion. The challenge is that we can not predict the distribution shape beforehand. For instance, with our reward function, the closer the robot gets to the setpoint, the higher the maximum value of possible reward becomes (hence, the optimal policy should lead to a reward distribution of Pearson shape). In practice, however, the closer the vehicle is to the setpoint the more difficult it becomes for it to physically reduce the errors (which is more akin to a Gaussian distribution). Depending on the system, the operating conditions, and the reward function (among others), the reward distribution can switch between various shapes, potentially making the predefined metric not robust to different distribution assumptions. Thus, we propose to consider a transition as an outlier of interest and to store it in B2 if its associated reward r(st) is:

r(st) > E[r(st)], (3.55)
where the expected value E[r(st)] is computed over the last 50,000 rewards generated that are stored as an additional variable M . The size of M was chosen in order to compute the expected reward over a moving window of approximately 100 episodes in order to give more importance to novel inputs, similar to biolog ical systems [START_REF] Tyler | Replay in Deep Learning: Current Approaches and Missing Biological Elements[END_REF]. This choice of expected value as a metric is related to the subtracted baseline in Eq. ( 2.77) that is the Value function, resulting in the Advantage function A(s, a) = Q(s, a) -V (s). This func tion represents the benefits of changing the current policy as a positive value of A(s, a) indicates that the evaluated pair of stateaction is associated with a QValue higher than the expected one. Our assumption is that transitions that meet the criteria expressed in Eq. (3.55) are associated with positive values of A(s, a).

Using samples from the Optimistic Memory will therefore mostly improve the expected return of the policy, leading to faster discovery of successful trajectories.

The maximum size of B2 is set to 10, 000. It is drastically smaller than B1 because, as the agent's performance improves over the course of training, what was considered a good transition is most likely to be outdated. Therefore, the reduced buffer size ensures that we focus on the current best transitions. Contrary to the first buffer, we propose to sample uncorrelated items from B2 as single transitions are iteratively stored in this buffer. Finally, the minibatch is constructed of n samples from each memory unit.

This Experience Replay mechanism illustrated in Figure 3.14 is totally novel and it is the second contribution of this thesis. In terms of algorithm complexity, as measured by the number of operations, let's consider the original ER algorithm to be associated with a complexity of O( 1). As presented in Appendix A, the CER algorithm is accordingly associated with a complexity of O( 2) and the BIER algorithm with a complexity of O(6).

Summary

In this section, we presented the research approach that lead to the design of the proposed learningbased adaptive control system. First, in section 3.1 we presented related works in AUVs learningbased adaptive control. We were able to identify different challenges and opportunities for contributions that we initially tried to explore with some preliminary studies presented in Section 3.2. Based on these primary results, we design a novel learningbased adaptive control system which, as illustrated in Figure 3.15, is composed of two main components:

1. First, we propose a novel adaptive poleplacement strategy where the gains of a modelbased control structure are transformed in the space of poles. There, the space of possible pole values can be bounded according to desired control performance and stability requirements. As illustrated by the color code in Figure 3.15, the control system is divided into two modes: offline and online.

In the online control loop (in red), the policy is exploited and the estimated poles are used to compute the PID control inputs that regulate the real platform. In the offline learning loop (green), simulated data are used to improve the critics with offpolicy TD learning using the BIER method. The policy is updated using the policy gradient of the SAC algorithm Eq. (2.77) which incorporates entropy maximization.

In the next sections, we will present how to adapt the proposed learningbased adaptive control system to AUVs. This validation was performed under simulation but also on a real platform.

Simulated validation

The final objective of this thesis is to validate the proposed method on a real platform. However, as out lined earlier throughout this thesis, DRL algorithms are known to require a big tuning effort in order to reach satisfying performance. For this reason, we perform a series of validation under simulation before applying the method to a real vehicle. We present now the simulated validation of the method proposed in Section 3.3 to AUVs using the UUV Simulator [START_REF] Manhães | UUV Simulator: A Gazebobased package for underwater intervention and multirobot simulation[END_REF]. In the first study, we apply solely the learningbased adap tive controller to the RexRov2 platform for a maneuvering task. Then, in a second study, we propose an improvement of the controller design for the same vehicle but for another task, along with the BioInspired Experience Replay mechanism. The section ends with a summary of the findings with some suggestions for experimental validation.

Learningbased: application to AUV maneuvering under sea current disturbance

As the main focus of this thesis is AUVs, we present now how we extend the learningbased adaptive con troller, that we first used on a MAV in [Cha+22] (presented in Section 3.2.2), to the RexRov2 platform. The final objective of this thesis is to use the proposed method on a real platform. To that end, we perform differ ent validation of the method under simulation to ensure the safety and The results were published [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF] and presented at the 13th IFAC Conference on Control Applications in Marine Systems, Robotics, and Vehi cles. In the following, we summarize the main results of this paper, and its preprint version can be found in Appendix C.

Task description

In this study [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF], we address the control problem of AUV maneuvering, which can be summarized as the stabilization of an underwater vehicle at a fixed velocity and orientation (x ref , with ẋref = 0). The state vector is hence defined as x = [vx ; vy ; vz ; ϕ ; θ ; ψ] T . This is different from the previous target rallying task Eq. (3.2) because we are not regulating in terms of positions but in terms of velocities. The vehicle is fully actuated but subject to external disturbances (sea currents) that are here not observable (noted that relative current can be estimated with some types of DVL incorporating a bottom lock). Let the error between the present (xi) and the desired (x ref i ) state variable be defined as ei = x ref i -xi. The task of steering the AUV outputs in order to maintain the error signals within a specific threshold, over a predefined amount of time (guaranteeing the vehicle stabilization), can be achieved if the following control objective is met:

∀ t ′ ∈ [t -ς, t], ∄i ∈ R u such as | ei(t ′ ) | > χ, ( 4.1)
where R u is the space of control inputs, the current time step is denoted as t and ς is the length of the period of time over which we want all the errors ei to be less than the desired value χ. This class of control objective is used in various AUV missions, such as autonomous docking or underwater inspection, where a conservative regulation of the vehicle's outputs is required because collision can be very dangerous for both the vehicle and the environment. In this study, we consider the control task where the AUV has to maintain a constant forward velocity and orientation while being subject to sea current disturbance.

Design of the learningbased controller

Improved poleplacement strategy

The performance of a PolePlacement controller is directly related to the location of its poles which makes their tuning a critical task. There exist a set of fundamental design rules to guide this choice [START_REF] Chilali | H∞ Design with pole placement constraints: an LMI approach[END_REF] but in general, one should not choose closed loop poles that are highly negative, making the system fast respond ing (in the frequency domain) which leads to large bandwidth and thus amplification of noise. In our case, with the design (3.48), for any value of τi > 0, the poles of the controller are placed in the left halfplane which ensures its stability. This leaves us with the settling time required to define in order to bound the value of the poles. In accordance with the control objective (4.1), we define the desired settling time of the closeloop control ς = 10 seconds as the time after which we want the system outputs to stay within χ = 2.5% around its desired values. These values were chosen following empirical tests when performing the ZieglerNichols methodology. The upper bound of the poles is then derived as

λmax = ln (χ) ς = ln (0.025) 10 = -0.368 (4.2)
Therefore λi ⩽ λmax (3.48), from which we can derive the upper bound of the poles

τmin < τi ⩽ 1 -λmax = 2.710 = τmax (4.3)
We choose τmin = 0.1 because for lower values the control inputs are too expensive in terms of control efforts and too aggressive for our control objective. Thus, the bounds of the poles are defined as

0.1 ≤ τi ≤ 2.710 (4.4)
Again, the learning objective is to estimate a policy πµ that directly maps the pole locations from the AUV state in order to reach the control objective. The final actions are control inputs applied to control the AUV in terms of velocities and orientation:

π θ : S ⊂ R dim(S) → A ⊂ R 3×6 x = [st] T → [λi, µi] , ( 4.5)
where the dimension of the action space is 18 because we have 3 gains, thus 3 poles, for each degree of freedom. These poles candidates are then applied to compute the gains (3.51) that are used to derive the control PID law (3.43) for each control input as described in Section 3.3.2.

State vector

At each time step, the agent captures an observation vector of the process ot as described below:

ot = [ at-1 ; O ; V ; Ω ; ui ; ei ; eL 2 ] (4.6)
where

• at-1∈R 18 are the estimated pole values (4.5),

• the Euler orientation of the vehicle are O = [ϕ, θ, ψ],

• its linear and angular velocities are respectively V = [vx, vy, vz] and Ω = [ω ϕ , ω θ , ω ψ ],

• the vector ut ∈ R 6 is composed of the current PID controller outputs,

• ei ∈ R 6 are the errors on each setpoint,

• and eL 2 is the Euclidean distance to the steadystate defined as Eq. (4.9).

Due to latencies and uncertainties in the process, the dynamics can become nonMakorvian which signifi cantly degrades the learning performance. For this reason, we construct the state vector st out of the current and past 4 observation vectors (this choice of the number of frames is empirical and is motivated by related works), thus dim(st) = 200.

Reward function

In accordance with the considered control objective Eq. (4.1), we design the following reward function:

r(st) = r succeed if ∀t ∈ [t-100; t], | ei(t) | ≤ χ, r regulation , otherwise. (4.7)
The reward signal r regulation is a binary reward signal defined as:

r regulation = 20 × e -(e L 2 ) 2 × (1 + dt e i (t) ) if dt e i (t) > 0, -20, otherwise. (4.8)
where eL 2 is the Euclidean distance to the steadystate:

eL 2 = i=dim(u) i=1 e 2 i (t) (4.9)
and dt e i (t) is the derivative of the error ei computed over two time steps such as:

dt e i (t) = e 2 i (t -1) -e 2 i (t) (4.10)
This reward function highly recompenses error decrease and penalizes any deviation from the Euclidean path to the steady state. If the control objective is met, the reward r succeed = 500 is generated which ends the current episode.

Training

The training consisted in performing a total of 1 million time step iterations. Each episode has a maximum length of 300 time steps (equivalent to 20 seconds). This parameter also required tuning because depending on the problem, an overflow of steps can highly degrade and slow the learning, while too few steps often cause the policy to overfit. The complete list of training hyperparameters is provided in Table 4.1 with the details on the DRL framework. A training episode is defined as follows:

1. At the beginning of the episode the AUV is initialized at the position (x0, y0, z0) = (0, 0, -40) with null velocity and a random orientation (ψ0, θ0, ϕ0) ∈ [ -π 4 ; π 4 ]. 2. A random set of current variables is generated such as vc ∈ [0, 0.5] and [hc, jc] ∈ [ -π 4 ; π 4 ] which are then kept constant during the episode.

A random vector of setpoints is generated such that

Λ ref = [vx, 0, 0, 0, 0, 0] T with vx ∈ [0.5, 1.5].
4. Then, the offpolicy exploration strategy is used and the episode ends when the step number reaches 300 or r succeed is generated.

Evaluation

For comparison, we used the optimal PID controller provided in the UUV Simulator to show the benefits of our approach. It is a 6DoF PID controller whose parameters have been optimized using SMAC [START_REF] Hutter | Sequential ModelBased Optimization for General Algorithm Configuration[END_REF].

Both controllers are therefore based on the exact same structure (PID type control law) for a fair comparison. Evaluation outcomes are provided in Tables 4.2, 4.3, and 4.4. We use the following metrics: the success rate, the mean reward per step and the positive reward rate (as described in Section 3.2.2) computed over all episodes for each scenario. We defined three evaluation scenarios as follows:

• Scenario 1: the setpoints are fixed during the whole episode while the sea current variables vary.

• Scenario 2: the sea current variables are fixed during the whole episode while the setpoints vary.

• Scenario 3: both sea current variables and setpoints vary during the episode.

When varying, these variables have a sinusoidal shape defined as:

Discussion

In terms of success rate, the proposed learningbased adaptive controller is exhibiting the best performance. Compare to the OPF controller, the success rate of our learningbased adaptive controller is ∼ 4 times higher in the first scenario, and ∼ 11 times higher on the second and third scenarios. When the disturbance varies over time, the OPF controller is not able at all to compensate for it with its fixed parameters. The learning based controller on the other hand is able to adapt to variations in the current disturbance. We can see that the OPF controller is not able to complete the task despite its ability at keeping the error very low. The numerous variation in the process makes the task of maintaining the error within a threshold challenging for a controller with fixed parameters. In fact, if only one DoF exceeds the threshold value, the counter is reset to zero, and to objective is not completed.

In terms of reward per step, the learningbased controller is outperforming the OPF controller with a value that is 1.61, 10.87, and 7.06 times higher than the OPF controller (respectively for scenarios 1, 2, and 3).

The positive reward rate displayed by the learningbased controller is also the highest in every evaluation scenario. It is 1.21, 1.73, and 1.5 times higher than the OPF controller (respectively for scenarios 1, 2, and 3). When comparing the scenarios between each other, we can see that the performance difference is low between scenarios 1 and 2 and is large when comparing scenario 3 with the other scenarios.

As we have been able to observe, the drop in performance is larger in scenario 3 where there is process and disturbance variation. In order to identify which process variations have a big impact on the agent performance, we propose next to evaluate the agent on more various scenarios. We present in the following section improvements on the design of the modelbased and modelfree parts as well as on the reward function and the exploration ability of the agent. We will also evaluate the proposed BIER method against the standard CER method.

CER vs BIER: application to AUV regularization under sea current disturbance

In the previous study [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF], we were able to observe that the agent performance varies a lot against increasing process variation. In order to identify which process variation the performance is most sensitive to, we propose in the next study to improve the AUV learningbased adaptive control system and to evaluate it to a greater variety of scenarios. In addition to an improved controller design, we also evaluate the proposed BIER method against the standard CER technique. The first objective of this study is to identify which process variation causes the drop in performance in order to design a proper incremental environment complexity [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Reinforce ment Learning of DepthBased Robot Navigation[END_REF] technique which will allow an easier simtoreal transfer of the policy. Secondly, the objective is to evaluate the BIER method in terms of training dynamics and evaluation performance. The results were summarized in the paper [CHA+22] whose preprint version can be found in appendix C.

Task description

In this study [CHA+22], we addressed the exact same AUV maneuvering task described earlier in Section 4.1.1 that we previously studied in [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF].

Improvement of the learningbased controller

In accordance with the control objective shown in Eq. (4.1), and due to the lower performance observed against highly varying disturbances in [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF], we define the desired maximum settling time of the closed loop control ς = 10 seconds as the maximum time after which we want the system outputs to stay around χ = 5% (and not 2.5% as we did in [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF] because it was too small) of its desired values. The upper bound of the space of poles is derived as:

λmax = ln (χ) ς = ln (0.05) 10 = -0.2995 (4.12)
Therefore, as λi ≤ λmax from Eq. (3.48), we can derive the upper bound of the poles:

τmin < τi ≤ 1 -λmax = 3.338 = τmax (4.13)
Again, we set τmin = 0.025 because, for lower values, the control inputs are too expensive in terms of control efforts and too aggressive for our control objective. Thus, the bounds of the poles are defined as: 0.025 ≤ τi ≤ 3.338 (4.14)

State vector

At each timestep, the agent captures an observation vector ot representing the process dynamics that we defined as:

ot = [at-1 ; Θ ; V ; Ω ; ut ; et ; eL2 ; drate ; δχ], (

where

• at-1 ∈ R 18 are the last action estimated (i.e. poles value),

• Θ = [ϕ; θ; ψ] are the Euler orientation of the vehicle (roll, pitch, and yaw respectively),

• V = [vx; vy; vz] and Ω = [ω ϕ ; ω θ ; ω ψ ] are its linear and angular velocities,

• ut ∈ R 6 are the last control inputs applied,

• et ∈ R 6 are the error values on each setpoint,

• eL2 and dt e i (t) as described in Section 4.1.2,

• and δχ ∈ [0, 1] is a variable that keeps track of the number of successive steps where all the errors are within the threshold (i.e. if δχ = 1, the control objective is achieved).

The dimension of the observation vector ot is therefore equal to 42. Noted that with this observation vec tor (4.15), the current disturbance characteristics are not included. In order to improve the process observ ability and follow our previous results [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Reinforce ment Learning of DepthBased Robot Navigation[END_REF], we construct our state vector st out of the current and past observation vectors along with their twobytwo difference. This results in a 126dimensional state space defined as:

st = [ot ; ot-1 ; ot-1 -ot]. (4.16)

Reward function

The following terminal reward signal is defined to take into account the longterm stabilization requirement:

rsuccess = 1000 if ∀ t ∈ [t-100, t], | ei(t) | ≤ χ. (4.17)
Therefore, performing the desired control objective, depicted in Eq. (4.1), will generate this reward, Eq. (4.17), and its value was chosen in order to make sure that, for all trajectory lengths, the maximum sum of return is obtained only by stabilizing the vehicle. Otherwise, the reward r(st) is generated. Let's define the Euclidean norm of the errors vector as eL2(t) = i=dim(u) i=1 e 2 i (t) and its derivative is computed over the last two frames and denoted as drate(t). The reward r(st) is then defined as:

r(st) = C1 × exp -(eL2(t) × C2) 2 (4.18)
Here, we empirically chose the reward scale as C1 = 40, which gave us the best performance, by following advice from [START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]. The reward signal, Eq. (4.18), is equal to its maximum value possible per step (that is C1) only when all the current errors are equal to zero. As AUVs move slowly, successive states display error signals ei(t) of minor and similar amplitude. We find that this addition of C2 = 10 (compared to the previous work [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF]), makes it easier for the critics to differentiate the StateValue of successive states without altering the reward scale as limx→0 C × e -x = C. This reward function, Eq. (4.18), encourages the agent to reduce the errors as much and as fast as possible and the vehicle stabilization is further promoted by generating the maximum reward possible per step.

Exploration strategy

We used the adaptive parameter noise [START_REF] Plappert | Parameter Space Noise for Exploration[END_REF] technique (as described in Section 2.2.5) with the following values: the metric Es[•] is estimated over a batch of 1000 samples from the Replay Buffer, the initial standard deviation is 0.60, the threshold is set to 0.10 and the update rate is set to α = 1.01.

Training

The training consists in performing a total of 3000 episodes. Each episode has a maximum length of 500 time steps (equivalent to ∼35 seconds). The training episode characteristics are defined exactly as described in Section 4.1.3. We compare two versions of the learningbased adaptive controller, one that uses the CER method and one that uses the BIER method. Again, we compare the resulting policies with the OFP controller from the UUV Simulator defined in 3.2.1. The complete list of training hyperparameters is provided in Table 4.5 with the details on the DRL framework. The training curves are provided in Figure 4.1 with first the normalized return, the setpoint RMSE, and the normalized standard deviation of the return. Learningbased adaptive controllers are represented in blue and green with respectively the CER and BIER methods used for the offpolicy TD learning. The modelbased counterpart of the control structure, namely the OFP controller, is represented in yellow with a dashed line.

The performance of the OFP controller is the mean value obtained over 500 random training episodes. As we can see in the first plot of Figure 4.1, both methods are able to learn the task and converge toward what seems to be a maximum value of the reward. In the second plot of Figure 4.1, the control performance is displayed in terms of RMSE on the setpoint. We can see that both learningbased adaptive controllers exceed the control performance of the modelbased controller, which is represented by the vertical lines. The agent trained using the BIER method was able to exceed the performance of the OFP controller, in less than half the number of episodes compared to the standard CER method. In the third plot of Figure 4.1, the evolution of the return standard deviation is represented. We can see that the performance improvement is smoother with the BIER method which exhibits a lower reward standard deviation (which tends to reduce over time contrary to the CER agent). With the CER method, the variance is higher, with spikes that even drive the agent to lower performance than those obtained with the OFP controller. Noted that the OFP controller displays the lowest standard deviation (third plot of Figure 4.1), thanks to the model information incorporated in the SMAC [START_REF] Hutter | Sequential ModelBased Optimization for General Algorithm Configuration[END_REF] method. The results show that despite only manipulating the past experience differently, we can, with the BIER method, learn faster and with improved stability (i.e. lower standard deviation).

Evaluation

In order to visualize the effect of process variation, we present below different evaluation scenarios of in creasing complexity as measured by desired setpoint and current velocity.

Scenario 1: the setpoint range is the same as during training but no current disturbances are applied. This scenario is therefore in theory simpler than the training one.

Scenario 2: the process is the same as during training but with setpoint and current variables that were not seen during training (but are still in the same range). In Table 4.6 and 4.7, the metrics for each scenario were computed over 500 episodes (different from each other). The line "Baseline" denotes the performance obtained at the end of the training, which incorporates the exploration strategies described in Section 4.2.2. For the other scenarios, the exploration is removed. We propose in Figure 4.2 a bar chart of these results to simplify the analysis of the results.

Discussion

The CER and BIER agents are able to stabilize the vehicle over the first 3 scenarios where the performance is matching the training one. When increasing the desired setpoint value (i.e. scenario 3), we can see the associated RMSE slightly increasing but the performance remains satisfying. The performance drop is the largest when increasing the sea current disturbance (i.e. scenarios 4, 5, and 6). The sensibility to this disturbance is further depicted by the performance difference that is much smaller between scenarios 5 and 6 compare to between scenarios 3 and 4. We believe this is due to the current characteristics not being explicitly included in the state vector. Therefore, since the agent did not experience such disturbance during training, its performance drop drastically in these scenarios (i.e. scenarios 4, 5, and 6).

The BIER agent performs better in scenario 1 compared to scenario 2 which is simpler than the training scenario as it does not incorporate sea current disturbance. This suggests that overfitting is not happening, otherwise even in a simpler scenario, the performance would be lower than the training one (i.e. scenario 2). More interestingly, its performance compared to CER has also increased in scenarios 5 and 6, despite the distribution shift being particularly large there. This suggests that the policy obtained with the BIER method has better generalization abilities as its performance increased even on scenarios very different from the training one. We can see that the OFP controller is much more sensitive to sea current disturbance compare to setpoint variation (as the latter is included in the model). This tendency was already observed in the previous paper [START_REF] Chaffre | Direct Adaptive PolePlacement Controller using Deep Reinforcement Learning: Application to AUV Control[END_REF] (presented in Section 4.1.4) but we can clearly see here that disturbance variation can be compensated more with the learningbased adaptive controller. The learningbased controllers are able to exceed the OFP controller in scenario 6, despite not having experienced such disturbances during training. In Figures 4.3a to 4.3f we plot the evolution of the error on each DoF for an episode from scenario 5. In this scenario, despite not having seen such values of the desired setpoint and current velocity, the learningbased adaptive controller is still able to compensate for these uncertainties and to maintain the errors close to 0. We can see that the controller parameters vary differently depending on the DoF which shows that some DoF is more sensitive to current disturbance.

Summary

In this section, we presented the simulated validation of the proposed learningbased adaptive controller. First, we described how we adapt the control design for an AUV with an improved poleplacement strategy. The resulting control system was then evaluated under simulation for an AUV maneuvering task. From the results of this first validation, we are able to see that the proposed method is doing better at stabilizing the AUV compared to a purely modelbased optimal, but nonadaptive, version of the exact same control structure. We noticed that the performance of the learningbased adaptive controller drastically drops with increased process variation and proposed to study the cause of this incident.

Then, we proposed a second validation study to evaluate the performance of the proposed control system against more process variation. We also evaluated at that time the proposed experience replay method by training the exact same control system with the standard CER method or with the proposed BIER strategy. From these simulated results, we are able to see that the BIER method leads to faster training and more stable performance stability during training and evaluation. We are also able to identify that it is the change in setpoint and current disturbance that is hard to compensate for the learningbased adaptive control system.

After this simulated validation, the next step is to validate the proposed method on a real platform. In the next section, we will present our protocol for reallife validation with a presentation of an improved design of the control system and the experimental protocol.

Experimental validation

A major challenge in reinforcement learning is to transfer successfully the policy to the environment of interest.

There has been a lot of effort in the development of DRL as it becomes more and more efficient to train. The bottleneck of DRL remains the simtoreal transfer. As outlined in [START_REF] Dulacarnold | An empirical investigation of the challenges of realworld reinforce ment learning[END_REF], realworld reinforcement learning is associated to a number of challenges:

• Learning on the real platform limits the number of samples and is often not feasible. The real systems of interest are often slowmoving, fragile, or expensive enough to operate such that the data they generate is too costly to consider current methods.

• Most real robotic systems have delays in either sensing, actuation, or reward feedback. These can be caused by some safety checks or just because depending on the system, the effect of actions can take a long time to manifest (which is particularly the case with AUVs).

• Many practical realworld processes have large and continuous state and action spaces which can represent critical issues for RL algorithms [START_REF] Dulacarnold | Reinforcement Learning in Large Discrete Action Spaces[END_REF][Tes+19].

• Partial observability and nonstationary that in the case of AUVs appear as noise (e.g. uncalibrated/bro ken sensors) or as stochasticity (e.g. the vehicle behaves differently at each session).

In this section, we present the results of our campaign of experimental validation illustrated in Figure 5.1, where we transfer our policy on a real robot, emphasizing approximately 280 minutes (or ∼4h40) of oper ating time, which took us several months to complete and where we encounter most of the above challenges. Wheare, Andrew Lammas, and by all my supervisors, I was able to perform reallife validation of the proposed learning based adaptive controller showing the gains in performance compared to the standard method and its ease of deployment.

Task description

In accordance with the sensors available on the vehicle and the pool characteristics, we decided to address the control problem of multistations keeping that is to stabilize the vehicle, for a given amount of time, succes sively at different setpoints in space that is represented by a position and orientation. The state of the vehicle at the timestep t denoted as st is defined by its Cartesian position and Euler orientation st = [xt yt zt ψt θt ϕt] T (respectively roll, pitch and yaw for its orientation). A setpoint is defined as sw = [xw yw zw ϕw] T . The control objective is to minimize the Euclidean distance dt between the AUV and the setpoint:

dt = (xw -xt) 2 + (yw -yt) 2 + (zw -zt) 2 + (ϕw -ϕt).
(5.1)

The task of station keeping can be achieved if the following control objective is met:

∀ t ′ ∈ [t -ς, t], ∄i ∈ R u such as | ei(t ′ ) | > χ, ( 5.2)
where d reached is the desired threshold value that we want the Euclidean distance dt to be lower than for the setpoint to be considered reached. This class of control objective is used in various AUV missions, such as autonomous docking or underwater inspection. In our case, as described in the following, to test the performance of our control system, we defined multiple setpoints where we want the vehicle to stabilize. The experimental validation consists in performing station keeping at several setpoints in space. Starting from an initial position, the vehicle is required to perform station keep for an amount of 1000 timesteps at each setpoint (which is equivalent to 45 seconds). One session is, therefore, equivalent to ∼ 7 minutes and consists in performing the stationkeeping task illustrated in Figure 5.2. We evaluated our learningbased adaptive control against two environment configurations: with and without current disturbance. In order to provide significant value to our results, we performed this stationkeeping task 10 times for each control system and for each disturbance configuration. The results provided in Section 5.6 are therefore the mean values over this 10 trials. In order to reduce the bias of this evaluation, we propose to, before the beginning of each experiment protocol, stabilize the vehicle at the first setpoint. It consists in launching the control system when the vehicle is not too far from this setpoint and letting the control system regulate the vehicle over for the same amount of timesteps required during the experiments (i.e. 1000). Then, the session starts from this same setpoint. In practice, it means that there is an additional setpoint 1 in the list presented in Table 5.1, which is not taken into account when we assess the performance of the controllers. By doing that, we further minimize the bias of the evaluation as both controllers will start roughly exactly from the same state. We found this practice to be particularly relevant when we have current disturbances in the pool. In this case, without the double setpoint 1, the start of the sessions is very different due to the drift generated by the current.

The experiment task is illustrated in Figure 5.2 where the 9 setpoints are illustrated. The disturbance gen erator and the camera are fixed to a metallic arm that is fixed on the side of the pool at approximately 30 centimeters and 2 meters from the edge (respectively). The vehicle is asked to perform station keep at each setpoint in ascending order. The setpoints have been placed

Pose estimation without GPS

The Bluerov platform used for our experiments is equipped with an IMU and a frontal camera. No sensor allows us to directly estimate the position of the vehicle (such as a DVL). Therefore, we had to design a hybrid localization system composed of a camera and marker. The marker is placed on the top side of the vehicle while the camera is facing down toward the as illustrated in Figure 5.3. Then, a Kalman Filter uses data from the IMU and from the visual tracking system to estimate the position and orientation of the Bluerov. This solution is provided directly by the Centre for Maritime Engineering at FLINDERS University and therefore no further details will be given on its design as it is not the scope of the thesis.

Disturbance generator

In order to evaluate the robustness of the controllers against disturbance, we proposed to create an artificial current in the water tank. To that end, we fixed two thrusters of type T200 (the same as the ones on the Bluerov platform) on the aluminum arm where the camera is attached as illustrated in Figure 5.4. We chose a particular placement and orientation of the thruster such as to optimize the field of effect in the pool. The thrusters are controlled through ESC input that we set to 1625, which according to Blue Robotics documen tation gives around 8 Newtons of thrust per thruster. The total current draw for the pair is approximately 2.7A, providing a power draw of around 38 Watts.

Design of the learningbased adaptive controller Soft ActorCritic with Automatic Temperature Adjustment

Motivated by the fact the we are deploying our learningbased adaptive controller on a real vehicle, and because we have access to a suboptimal simulated model of the vehicle, we proposed to use this time the second version of the SAC algorithm (see Section 2.2.8). The main difference between these two version is that with the latter, the temperature alpha in the maximum entropy objective function Eq. (2.70). With this change, the reward scale does not needs to be tuned as the relative weight of the entropy term is adapted to satisfy a minimal entropy constraint. In fact, as studied in [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF], the reward scale influences greatly the explorationexploitation tradeoff of maximum entropy reinforcement learning methods. The optimal value of reward scale is difficult to determine beforehand and is essentially impossible to predict for a physical system; This was illustrated with the Minitaur platform, a smallscale quadruped with eight DoFs, considered in [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF], for which two orders of magnitude larger reward scale was required to work properly (compared to environments from the OpenAI gym benchmark suite [START_REF] Brockman | OpenAI Gym[END_REF]). On the other hand, the target entropy Eq.

(2.82) is easier to tune as it is a function of the dimension of the action space. It was proposed in [START_REF] Haarnoja | Learning to Walk via Deep Reinforcement Learning[END_REF] that the target entropy is equal to 1 per action dimension which can be interpreted as setting the weight of the entropy in the objective function as 1 percent for a given action space of dimension n. Therefore, the resulting dual constraint optimization for the policy is defined as:

J(α) = Es t ∼D,a t ∼πµ -α log πµ(at|st) + α × 18 . ( 5.3)
with H = -18 = -dim(u) is the target entropy that is equal to the dimension of the action space which is here 18 (since we have 3 pole values per DoF and we are controlling the vehicle in the 6DoFs 2.3.2).

State vector

At each timestep, the agent captures an observation vector ot representing the process dynamics that we defined as:

ot = [ at-1 ; Θ ; V ; V ; Ω ; et ; eL2 ], (5.4) where
• at-1 ∈ R 18 are the last action estimated (i.e. poles value),

• Θ = [ϕ; θ; ψ] are the Euler orientation of the vehicle (roll, pitch and yaw respectively),

• V = [vx; vy; vz] and Ω = [ω ϕ ; ω θ ; ω ψ ] are respectively its linear and angular velocities,

• V = [ vx; vy; vz] is its linear acceleration,

• et ∈ R 6 are the error values on each setpoint,

• and eL2 as described in Section 4.1.2,

The dimension of the observation vector ot is therefore equal to 40. Noted that with this observation vec tor (5.4), the current disturbance characteristics are not included. In order to improve the process observ ability and following our previous results [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Reinforce ment Learning of DepthBased Robot Navigation[END_REF], we construct our state vector st out of the current and past observation vectors along with their twobytwo difference. This results in a 120 dimensional state space defined as:

st = [ot ; ot-1 ; ot-1 -ot].
(5.5)

Reward function

Since we are using the second version of SAC 2.2.8, the reward scale does not required to be tuned. Thus, we proposed the following reward design:

r(st) = exp [-(eL2(t))] (5

Randomized environmental complexity as domain randomization

Despite the stability components of our learningbased adaptive controller, training directly on the real platform can not be considered due to the autonomy of the batteries. We, therefore, proposed to perform the training on a simulated version of the BlueRov platform. The resulting policy will then be transferred to the real platform. As discussed in Section 2.2.9, the distribution shift problem in reinforcement learning is mainly due to the difference in the spaces of states observed during training and evaluation. In our case here, the distribution shift arises from the fact that the policy will be trained in simulation, where the space of states is too perfect compared to what will be observed by the real robot (due to noise on sensorial data coming from imperfection, latency, power loss, and a great number of other natural phenomena). In addition, as we were able to observe in a previous work [CHA+22] (see Section 4.2), despite solving the task in training, the resulting policy can still has notably lower performance when the characteristics of the training environment greatly vary (see Section 4.2.5).

There exist many techniques to reduce the reality gap between simulation and the real world, and the most used one is denoted as Domain Randomization (DR). In DR, the environment that we have access to is denoted as source domain while the environment that we want to transfer to is denoted as target domain.

Training is often possible only in the source domain where we have N randomization parameters that we can module in order to vary the characteristics of the domain. Thus, we can define a configuration ξ as sampled from a randomization space ξ ∈ Ξ ⊂ R N . During training, DR consists in collecting data from the source domain with the parameters randomization applied. By doing that, the policy is exposed to a great variety of environments and learns to generalize better than when exposed to a single environment. The randomization parameters can control the appearance of the environment such as:

• position, shape, and colors of the objects,

• texture of material,

• lighting condition,

• random noise added to images,

• or position, orientation, and field of view of the simulated camera.

These parameters can also control the physical dynamics of the domain such as:

• mass and dimension of objects,

• mass and dimension of vehicles,

• damping and friction of the joints,

• observation noise,

• or action delay. We proposed what we called incremental environment complexity training in a previous paper [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Re inforcement Learning of DepthBased Robot Navigation[END_REF] which can be seen as a variation of domain randomization. It consisted in training the agent in different variations of the environment in terms of the complexity of the task (as measured by the amount and shape of obstacles in the environment). Then, the agent transits between these domains based on its current performance as measured by the success rate. The advantage of this approach is that if the policy is not able to solve the current complexity, it will be sent back to the previous one that was solved which ensures not getting stuck in a bad regime. With the right tuning, we can ensure that the agent transit smoothly between each configuration until the last one. However, with this approach, We can not control the amount of data collected in each complexity configuration. One configuration can be way more explored than another which can lead to overfitting.

In the paper [Cha+22] presented in Section 3.2.2, we proposed another incremental environment complexity methodology where we increased over time the complexity of the task until it reaches the complexity of the target domain. We made sure this time that each configuration is explored for the same amount of timesteps to avoid overfitting. However, we can not in advance guarantee that the agent will solve each configuration.

A highcomplexity configuration might require more time for the agent to solve compared to a simpler one.

Following these results, and based on the performance variation observed in our paper [CHA+22] (presented in Section 4.2) we proposed for the considered process here to divide the training environment in three complexity configuration as measured by the amount of disturbance as follows:

• Configuration 1: no disturbance at all.

• Configuration 2: sea current disturbance that does not vary within the episode.

• Configuration 3: see current disturbance that change at a random time within the episode between timestep 100 and 400, out of 500. The value 500 was chosen as the maximum value for the length of the episodes in accordance with the desired settling time 5.1.

Finally, before the beginning of each episode, the agent has a probability to experience each complexity configuration P that is equal for each of them. By doing so, we are sure that each configuration will be explored uniformly (to avoid overfitting) and the hardest configuration of the environment, that is the closest one to the target domain, will be experienced very early in the training phase (to force the agent to diversify its actions). This methodology is illustrated in Figure 5.7 where the choice of complexity configuration is performed after the end of each episode.

Exploration strategy

We used the adaptive parameter noise [START_REF] Plappert | Parameter Space Noise for Exploration[END_REF] technique (as described in Section 2.2.5) with the following values: the metric Es[•] is estimated over a batch of 1000 samples from the Replay Buffer, the initial standard deviation is this time equal to 1.0, the threshold is set to 0.10 and the update rate is reduced to α = 1.005. As recommended in [START_REF] Plappert | Parameter Space Noise for Exploration[END_REF], in order to not get stuck, which can still happen with a perturbed policy, we combine the parameter noise with an ϵgreedy policy 2.2.5 where each action holds an independent probability ϵ = 0.10 to be random. During the evaluation, both parameter noise and ϵgreedy are removed.

Training

The learning consists in performing a total of 3000 episodes of maximum timesteps set to 500 which takes approximately 4 hours (given that the training is performed in realtime factor which means that is it equivalent to 4 hours of reallife usage of the vehicle). Before an episode begins, the environment, the configuration of the environment characteristics is chosen as described in Section 5.4. The complete list of hyperparameters is provided in Table 5.2 with the details on the DRL framework. The results provided in Tables 5.35.6 are associated with the experimental scenario where no current dis turbance is applied to the vehicle. We can see that our learningbased (LB) controller displays better per formance compared to the modelbased (MB) controller. In terms of root mean squared error (RMSE) on the setpoint, the LB controller holds the smallest RMSE for every setpoint. On average, the gain in RMSE is about ∼ 241% with our LB controller. When we take a look at the standard deviation (Std) of the RMSE, which can be seen as a measure of robustness, we can also observe that the LB controller is doing better than the MB controller on every setpoint. The Std of the RMSE is again on average about 2 times smaller than our method. This tendency is furthermore perceptible is the violin plots provided in Figure 5.8 where we can see the median and quartile values computed over the 10 trials. We can see that the performance of the LB controller is notably better (i.e. lower error) and way more robust than the MB controller (i.e. less disseminated). On average, the gain in Std of RMSE is about ∼ 148% with our LB controller. When we take a look at the norm of the control inputs in Table 5.5, which can be seen as a measure of power consumption, we can observe a less apparent difference between the models. In fact, for the first half of the setpoint, the LB controller requires smaller control inputs to stabilize the vehicle, while it requires larger control inputs to stabilize the vehicle at the second half of setpoints. On average, the LB controller consumes 15% more energy than the MB controller. Finally, when we take a look at the return of the agent in Table 5.6, which can be used as a metric to assess if the policy is behaving as desired, we can that our LB controller is again doing better every setpoint. The normalized mean return of the LB controller is on average about 1.13 times higher than the MB controller. When facing current disturbance, the benefits of our proposed method are even more prominent. This is illustrated in the results provided in Tables 5.75.10 is associated with the experimental scenario where we apply a current disturbance to the vehicle. Again, we can see that our LB controller displays better perfor mance compared to the MB controller. In terms of RMSE on the setpoint, the LB controller holds the smallest RMSE for every setpoint. On average, the gain in performance in terms of setpoint regularization is about ∼ 207% with our LB controller. In terms of the Std of the RMSE, the LB controller is also doing better than the MB controller on every setpoint. As illustrated in Figure 5.9 where we can see the median and quartile values computed over the 10 trials. We can see that the performance of the LB controller is notably better (i.e. lower error) and way more robust than the MB controller (i.e. less disseminated). On average, the gain in Std of RMSE is about ∼ 314% with our LB controller. When we take a look at the norm of the control inputs in Table 5.5, which can be seen as a measure of power consumption, we can observe a less apparent differ ence between the models. In fact, for the first half of the setpoint, the LB controller requires smaller control inputs to stabilize the vehicle, while it requires larger control inputs to stabilize the vehicle at the second half of setpoints. On average, the LB controller consumes 9% more energy than the MB controller. Finally, when we take a look at the return of the agent in Table 5.6, which can be used as a metric to assess if the policy is behaving as desired, we can that our LB controller is again doing better on every setpoint. The normalized mean return of the LB controller is on average about 1.38 times higher than the MB controller.

Results

In Figures 5.10a to 5.10d, we provide violin plots of the results. On the top row, we plot the performance of both controller without disturbance (left) and against current disturbance (right). On the bottom row, we plot the performance displayed without disturbance and against current disturbance by the modelbased controller (left) and by the proposed learningbased adaptive controller (right). We can see that despite using a suboptimal simulated model of the AUV, the policy performed notably better when transfer on the real platform.

In Figures 5.11a to 5.16b, we plot the evolution of each DoF during a trial under current disturbance. We can see that the proposed learningbased adaptive controller displays lower overshoot and is better at tracking the desired trajectory.

Summary

In this section we presented the experimental validation of the proposed learningbased adaptive control system. In order to successfully transfer the policy from simulation to the real world, we proposed a domain randomization strategy where the agent faces different levels of environment complexity as measured by the amplitude and variation of the sea current disturbance. In addition, we proposed this time to use the second version of the SAC algorithm, with the automatic adjustment of the temperature parameter (see Section 2.2.8). This way, we do not need an intensive and empirical tuning of the reward scale parameter anymore. We compare our method to a purely modelbased version of the exact same control structure that was proposed in [START_REF] Wu | 6DoF Modelling and Control of a Remotely Operated Vehicle[END_REF]. The results show that the learningbased adaptive controller is doing better at stabilizing the AUV in both operating conditions: without and with sea current disturbance.

General conclusions and perspectives

Contributions summary

In this thesis we proposed a learningbased approach to the problem of adaptive control for autonomous underwater vehicles, with a focus on external disturbances. Following an indepth preliminary investigation (Section 3), we identified a set of key problems which make the problem challenging. We now summarized these topics and specify how we tackled them in the learningbased adaptive control system in Section 3.3 and how we evaluate it under simulation and in the real world as presented in Section 4 and 5 respectively.

Adjustment mechanism:

In the PID control framework, the control parameters are the gains of the control law. These variables can take numerous values and we have discussed why in our case, classic modelbased tuning methods [START_REF] Wang | Handbook of PI and PID Controller Tuning Rules, Aidan O'Dwyer[END_REF] can not be considered and why it is difficult to define the bounds of this space. In addition, control performance requirements are not straightforward to derive in the space of gains. In Section 3.3.1, this space of gains is transformed into the space of poles where only desired values are considered (i.e. poles of null imaginary part and negative real part) and where it is easier to impose bounds on the poles to meet predefined control performance requirements (e.g. desired settling time or overshoot). To achieve that, we introduced a novel mapping for the PID control law allowing us to switch from one domain to the other. This enabled us to bound the space of poles with respect to the desired control performance, which is not possible in the space of gains. Finally, we considered maximum entropy deep reinforcement learning to explore this space of desired values, allowing us to adapt the controller response to process variations (because the control parameters are a nonlinear function of a state representation of the process) and to guarantee some stability components on the system despite the use of deep neural networks. In fact, the proposed poleplacement design ensures that the poles hold the Routh-Hurwitz stability criterion and we have seen in Section 3.2.3 that despite not taking into account Lyapunov stability components in the learning procedure, the resulting policy is matching the stability of a purely Lyapunovbased adaptive controller. We have been able to study and show that this combination, compared to a completely DRLbased modelfree controller, is better at controlling the AUV and rejecting current disturbance. The proposed strategy can be used whenever a feedback controller can be considered, that is the case of most closedloop systems.

Unobservability of the disturbance:

The sea current disturbance that we want to compensate for is not observed by the control system. In other words, the value of the current velocity and orientation is not included in the state vector of the process. Nevertheless, we cope with this limitation by augmenting the state vector in such a manner that the effect of the disturbance can still be noticed. In practical terms, even without measuring the current directly, we can observe changes in current by observing the state of the vehicles. The state vector incorporates variables such as the AUV linear and angular velocities, on which the current disturbance has a direct impact. In Section 4.2.5, we show that despite not measuring it explicitly, with this procedure the variation in current disturbance is still detected and is compensated by fast adjustment of the pole values.

Sample efficiency:

Deep policy gradient methods require a great number of interactions with the environ ment to converge to a satisfying behavior. This can be explained by the fact that these methods do not learn what a good action is, but rather which action is better than the others. For this reason, these methods need to experience a large number of interactions to understand which set of actions is really the optimal one for the longterm return objective. We have been able to study how various variables of deep policy gradient methods can influence their sample efficiency which can be defined as: 1) the number of gradient steps before reaching performance convergence and 2) the standard deviation of the agent performance. Among the existing variables, we proposed to study the Experience Replay mechanism following numerous studies from the literature [ZS17][Sch+16][Fed+20] and [START_REF] Tyler | Replay in Deep Learning: Current Approaches and Missing Biological Elements[END_REF] which emphasize the impact of this tech nique. In Section 3.3.3 we proposed a novel BioInspired Experience Replay allowing us to train deep policy gradient methods faster and with improved stability. It consists in adding elements from the biological replay mechanism [START_REF] Tyler | Replay in Deep Learning: Current Approaches and Missing Biological Elements[END_REF] into our deep learning systems. BIER can be applied to any rewardbased MDP.

Simtoreal transfer:

The transfer of a policy trained under simulation into the real world is difficult due to the reality gap between simulators and the operating environment. There exist numerous techniques to fa cilitate this procedure, but in our case, since we are transferring control parameters, environment complexity randomization was proposed. We first propose this method in [START_REF] Chaffre | SimtoReal Transfer with Incremental Environment Complexity for Re inforcement Learning of DepthBased Robot Navigation[END_REF] which consists in dividing the train ing environment into subenvironments of the increasing complexity of the task. Then during training, the agent transit between the randomized environment based on its current performance. For the experimental validation of our method, we proposed an improved design of this procedure where the agent holds the same probability to experience each configuration of environment complexity. We showed that what is important is the variety of the samples and that the agent can directly face the most difficult configuration of the environ ment early in training without damaging its longterm performance. Moreover, we showed that transferring control parameters is easier than transferring control inputs because we do not learn the dynamics of a spe cific system and environment but rather we learn how to adapt to the variation of a process: from this point of view, the difference between two vehicles will be how the errors on the setpoint vary. This was highlighted in Section 5 where despite training the policy using a model of the real vehicle that is far from optimal (i.e. the physical parameters of the simulated model do not exactly match the behavior of the real vehicle), the result ing policy was able to control effectively the real vehicle, even better than a purely nonlinear modelbased controller.

Perspectives

Following the study, the design, and the simulated and experimental validation of the proposed learning based adaptive control system, I have been able to identify different perspectives to improve these steps.

In this section, I will present from a more personal point of view, and discuss what is blocking the usage of these types of control systems on real platforms and in industrial contexts.

Beyond PID control: In this thesis I have been focusing on the PID control law. This choice was motivated by the fact that the AUVs of interest are fully controllable and in this case, a PID control law can ensure con vergence to the steady state when time goes to infinity. Another reason, which might be the most important one, is that we do not have a reasonably good model of AUV processes to derive such optimal designs. Nevertheless, in theory, our methodology can be applied to any type of feedback control law, including: LQR

[Arg+13], H2/H∞ PID [ACP06], L2gain control [FE12], MPC [Ers+21], Gain Scheduling [Cle+02], …, etc.
The main difference with these control laws would be the number of control parameters and the mathemat ical framework used to compute the resulting control inputs (e.g. the control inputs are the results of an optimization problem with H∞ PID controller and not derived from a linear function as with a standard PID controller). The methodology would remain the same, that is to use a deep reinforcement learning algorithm to adjust these parameters to process variations. A first improvement perspective would be to consider in stead of the PID law, a nonlinear control law that would provide a better regularization ability (at the cost of being more expensive to compute compared to a PID controller) and with additional stability components. Lyapunov stability component can be directly incorporated in the design of the control structure when using for example L2gain control [START_REF] Mohsen | Robust 2 control for a class of nonlinear systems: A parameter varying Lyapunov function approach[END_REF] or MPC [START_REF] Shen | Trajectory Tracking Control of an Autonomous Underwater Vehicle Using LyapunovBased Model Predictive Control[END_REF]. By using such nonlinear control structures, the de ployment of learningbased adaptive control systems on real platforms could be more easily received by both academia and the industry as we could in addition to gaining in control performance, guarantee the safety of the systems, similarly to the classic modelbased methods but without a complete model of the process.

Learning how to learn: After these past years of experience in DRL, I have been able to feel how sensitive these algorithms are. There are so many parameters to tune that even for the simplest DRL algorithm, it can take several months for a system designer to build an agent that generalizes well [START_REF] Ghosh | Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability[END_REF]. Among the various variables that we can control, in this thesis I have been most interested in the Experience Replay (see Section 2.2.6). I decided to study its impact because I strongly believe that if we manipulate the agent's past experience in a more clever manner, we can influence the future data collection of the agent leading to higher quality interactions and thus faster learning. Despite having been pointed by the leaders of the field, like Richard S. Sutton [START_REF] Zhang | A Deeper Look at Experience Replay[END_REF], Sergey Levine [START_REF] Mendonca | MetaReinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling[END_REF], Yoshua Bengio [START_REF] Fedus | Revisiting Fundamentals of Experience Replay[END_REF], or David Silver [START_REF] Schaul | Prioritized Experience Replay[END_REF], this issue remains pretty much ignored by the community with replay mechanism in our deep learning systems that are still missing important biological elements [START_REF] Tyler | Replay in Deep Learning: Current Approaches and Missing Biological Elements[END_REF]. In this thesis, I tried to study this challenge by proposing the BIER method which is a first attempt at incorporating these biological elements in deep reinforcement learning. With BIER, we considered a good interaction as one associated with a reward value higher than the current mean reward value and then use these samples more often in the minibatch gradient descent procedure. This choice of criterion is of course far from optimal. The distribution of reward can take various shapes: depending on the reward function itself (e.g. if the reward function is symmetric or not, the mean value can hold a different nature), depending on the stage of training (i.e. at the beginning of training we can expect a reward distribution that is more uniformed compare to the one observed by the end of training), depending on the task itself (e.g. with the classic mountain car problem [START_REF] Patachi | A Comparative Performance Study of Reinforcement Learning Algorithms for a Continuous Space Problem[END_REF] where the only way for the agent to reach the goal is to take initial actions that prevent it from reaching the goal), or quite simply depending on the presence or not of rewards. A direct improvement would be to adapt the criterion to the shape of the reward distribution, for example: if the current reward distribution is more likely a Gaussian distribution, the mean or even the third quantile can be considered as a threshold to define outliers, on the other hand, if the current reward distribution is more likely a Pearson distribution, based on how skewed it is we might consider different criterion to define outliers. The value of the reward is not the only reason why we would be more selective in the past experience selection. The policy gradient formulation Eq. (2.45) tells us that these type of solution methods does not learn what a good action is, but rather learns which action is better compared to the others. Accordingly, we could fairly ask ourselves: if only considering a sample of experience associated with a QValue higher than the mean one will induce a positive change (with respect to the RL objective that is to maximize future return) in the policy, why should we be concerned by any other samples? In fact, since we are only interested in the maximum value of the QValue function (as discussed in Section 2.2.9), why would we be interested in the other parts of the function? Another perspective, or domain of interest, would be to study if it is still possible to learn the task by considering for example storing in the Replay Buffer, only the samples that meet this value criterion. This way, we would ensure that each gradient update will induce a policy improvement in terms of return, leading to faster learning. However, from an optimization point of view, using such few and particular samples to optimize a nonlinear function approximator can be very dangerous and lead to QValue overestimation or overfitting.

Real life reinforcement learning:

In theory, the optimal procedure would consist in training deep policy gradient methods directly in the real world, removing the distribution shit problem of reinforcement learning. In practice, using today's DRL methods in the real world can not be considered for numerous reasons [START_REF] Dulacarnold | An empirical investigation of the challenges of realworld reinforce ment learning[END_REF].

The principal problem is that they need a large number of interactions to reach a satisfying performance, which is also computationally expensive, and before reaching that stage the agent behavior can be dangerous for the platforms and operators. An improvement perspective would be to perform what is denoted as offline reinforcement learning and then adjust (to a smaller extent) the solution obtained from this first phase by online reinforcement learning during operation. Offline reinforcement learning consists in training the agent on a finite set of experiences until the convergence of the policy. This set can be obtained for instance in our case, by running a modelbased controller, i.e. OFP controller, on the AUV for a short amount of time and recording the resulting st, at, rt, st+1. Offline reinforcement learning can be seen as a variation of imitation learning [START_REF] Rashidinejad | Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism[END_REF] in the sense that we will not be able to learn a policy better than the one used to generate the samples. However, this will be a way better starting point compared to a random policy, because the OFP controller despite not being optimal, ensures some minimal control performance and stability components.

The proposed learningbased adaptive control system would then be used during operation with a limited update ability, e.g. with a lower learning rate and a lower gradient update rate. In the long run, we will need to build machines that can directly learn from real data, because that is the only way that we will get them to improve perpetually. If not, if they have to rely on simulated data, eventually the simulator becomes the bottleneck.

Simulation is not a substitute for being able to not utilize real experience: Another strategy to leverage deep policy gradient methods would be to close the reality gap between the simulator and the physical world.

In this thesis, we used solely Gazebo as the simulation framework. It allowed us to simulate underwater processes and several disturbances, but it remains far from the complexity of real sea dynamics. On the other hand, trying to have an overly realistic simulation does not sound feasible as the number of particles and their interactions to model grows exponentially in the underwater context. Running such simulations will not allow us to use effectively the methods presented in this thesis. Another point of view is to augment the simulation so as to make the set of simulated states closer to the one encountered during the operation. One possible way to do that is to extract from reallife data what is truly representative of these interactions, to then use this knowledge as a model to adapt the simulated interactions to look more like real interactions, and one technology that can be used to achieve this are Autoencoders [Vin+10] [Vin+08]. Autoencoders are a very specific type of deep neural network used for unsupervised learning. It learns codings of unlabeled data by attempting to regenerate the input from the encoding. This procedure can be seen as a learningbased version of Principal Component Analysis [START_REF] Wold | Principal Component Analysis[END_REF] which aims at reducing the dimension of the representation space. This strategy is already used on dynamical systems such as the Lorenz, Rossler, and LotkaVolterra systems [START_REF] Bakarji | Discovering Governing Equations from Partial Measurements with Deep Delay Autoencoders[END_REF] where a deep autoencoder network is used to learn a coordinate transformation from the delay embedded space into a new space where it is possible to represent the dynamics in a sparse, closed form. In our case, we could use Autoencoders to learn a function that maps the state st that incorporates the past action performed (input of the encoder) to the next state st+1 of the same size (output of the decoder). By doing that, we essentially force the model to learn a lowdimensionality representation of the dynamics of the real world. Then, we could use the resulting optimized Autoencoders to adapt the simulation by feeding as input the simulated st and observing as output the augmented state zt that will be transformed such as to be more realistic. Then, the vector zt will be used as input to the policy network instead of st. The objective is to train on samples that are close to the ones observed in real life despite using a simulator. This would allow, in theory, to simulate an infinite amount of realistic data to train the deep policy gradient methods and thus reduce the distribution shift of reinforcement learning. Nevertheless, reallife experiments in more complex conditions, compared to the pool experiments presented in Section 5, will have to be conducted to further validate the relevance of this approach.

External and internal disturbance rejection:

In this thesis, I initially made the assumption that the AUVs of interest are fully controllable. In other words, this means that we consider the vehicle to always operate as requested and as planned. The scope of the thesis was thus defined as designing methods to detect and compensate for the external disturbance taking the form of seainduced disturbing forces. However, in practice, the vehicle behavior will not always be as expected, and sometimes even far from it because of power loss of the thruster or body wrench disturbance. These types of disturbance are in contrary internal to the AUV itself. They directly come from the component of the vehicles. An improvement of the method proposed in this thesis would be to take into account both external and internal disturbances. Contrary to the external disturbance, the internal ones can not be compensated without some a priori knowledge. We can not detect a deviation from normal behavior if we do not have a model of its expected behavior. This model can then either be known beforehand, or estimated using some parametric system identification techniques, including but not restricted to: Extended Kalman Filter, Least Square, Best Linear Unbiased Estimate, Maximum Likelihood Estimate, or neural networks. An intuitive methodology would be to use deep learning to perform this task [20]. The resulting estimation would then be incorporated into the proposed learningbased adaptive control system to compensate for both external and internal disturbances.

Learning to adapt to process variation, and not to a particular task or system: Across the different studies realized in this thesis, I was able to apply the learningbased adaptive control system to different types of vehicles (AUV and MAV), under different nature of disturbances (sea current disturbance and wind gust disturbance), and for different class of task (target rallying, outputs regularization, and station keeping). In all cases, the exact same control system was used and was able after learning to reach satisfying per

formance. An open question that remains is whether or not it would be possible for such a control system trained for example on a target rallying mission, to be used on an outputs regularization task and maintain its performance. Similarly, could we train on an AUV and transfer successfully the resulting policy to a MAV? My opinion is that it could be possible. In fact, in our learningbased design, the neural network takes as input a state representation of the process and estimate the associated poles of the PID control law. We are not learning to map one particular system as the variables in the state vector are essentially the same (i.e. IMU feedback). Whether we are trying to perform target rallying, outputs regularization, or station keeping, from a control objective point of view, the only difference is that in some cases the setpoint is a constant or a function of time. Therefore, the inputs of the control structure, that is the PID controller, remain the same: the error on the setpoint.

B The RotorS Simulator package

The Gazebobased package called RotorS [START_REF] Furrer | RotorS-A Modular Gazebo MAV Simulator Framework[END_REF] is a highfidelity simulation framework for MAVs that does not require any additional components to simulate highlevel tasks (e.g. pathplanning, collision avoidance, or visionbased problems). A complete model of the Firefly MAV is directly included in the package along with various world models and a plugin to simulate wind fields. We relied on the wind plugin provided by RotorS to generate wind fields in the environment. This plugin allows to define the wind as a 3D field sampled over a regular grid and each point specifies a wind velocity vector (in m.s -1 ) which varies here between -5m.s -1 and +10m.s The main forces acting on the vehicle come from gravity and the thrust of the rotors. Rotors drag and air friction are neglected here to simplify the model. It is classically assumed that the hexacopter is a rigid body with a symmetrical structure, and tensions in all directions are proportional to the square of the propeller speed. The equations of motion follow as:

ζ = v v = -ge3 + R b m Ω 2 i Ṙ = Rω I ω = -ω × Iω - Jr(ω × e3)Ωiτ (B.2)
where R the rotation matrix from RB to RW ; ω is the skew symmetric matrix ; Ωi is the ith rotor speed ; I the body inertia ; Jr the rotor inertia ; b is the thrust factor and τ is the torque applied to the body frame due to the rotors. A classical cascade control structure is adopted [START_REF] Bertrand | A hierarchical controller for miniature VTOL UAVs: Design and sta bility analysis using singular perturbation theory[END_REF], where the builtin lowlevel controller of the RotorS package is used to track a reference in roll ϕr, pitch θr and thrust T . The yaw angle ψ is kept constant without loss of generality. Under the smallangle assumption on ϕ and θ, the guidance model reduces to the doubleintegrator model:

ζ = v, v = u = [ux, uy, uz] T , (B.3)
where the computed accelerations are converted into lowlevel control inputs as:

T = m(uz + g) ; θr = m T (cψux + sψuy) ; ϕr = m T (sψux -cψuy). (B.4)
Evaluation of a Deep-Reinforcement-Learning-based Controller for the Control of an Autonomous Underwater Vehicle Yoann Sola * , Thomas Chaffre * † , Gilles Le Chenadec * , Karl Sammut † , Benoit Clement * † * Lab-STICC UMR CNRS 6285 at ENSTA Bretagne, Brest, France, benoit.clement@ensta-bretagne.fr † Centre for Maritime Engineering, College of Science and Engineering, Flinders University, Adelaide, Australia, SA 5042, karl.sammut@flinders.edu.au

Abstract-The development of efficient controllers in underwater environments has long been a challenging topic, hindered mostly by the lack of understanding in process variations under these conditions. It is without a doubt difficult for an autonomous underwater vehicle to behave as instructed while being constantly trying to compensate for the disturbing forces that act on its body. Recently, noteworthy improvements have been made in the model-free control theory, allowing the use of Reinforcement-Learning-based controllers in terrestrial and aerial robotic contexts. In contrast, the underwater control field has been largely dominated by controllers based on the Proportional-Integral-Derivative structure. In this paper we compare a PID controller with a leading-edge Deep Reinforcement Learning algorithm, the Soft Actor-Critic. These controllers have been tested within marine robotics simulations, using a realistic simulated environment including external disturbances for a waypoint tracking mission. The results obtained reveal the superiority of PID controllers in terms of success rate and precision, but not in terms of thruster usage. Future works will include the test of these controllers with an underactuated AUV, to demonstrate the guidance abilities of the learning approach.

Index Terms-AUV, control theory, deep reinforcement learning, simulation, precision, thrusters usage

I. INTRODUCTION

The control of Autonomous Underwater Vehicles (AUVs) is a particularly challenging task. In the strongly unstructured and uncertain marine environment, AUVs must constantly try to compensate for disturbing forces such as wind, wave and current action that act on their body. The wave frequency changes with weather condition variations and will fluctuate even more since it is also influenced by the vehicle relative velocity and heading. An AUV operating in comparable environment needs to perform well on specific tasks such as path following or dynamic positioning, while being robust to unexpected events (propeller failures, seafloor rocks, other vehicles, etc.) and any type of disturbances (sea currents, sensor noize, etc.). The decision-making process of an AUV is provided by what is called a Guidance, Navigation and Control (GNC) system. The navigation function involves estimating the state of the AUV, typically its pose, its linear and angular velocities and their respective derivatives. The estimated state is then used by the guidance and the control parts. One of the most commonly used state observers is the Extended Kalman filter [1]. In this work we make the assumption that the kinematic state of the AUV is known, and we will only focus on the guidance and control aspects of the GNC system. The guidance is a high-level control subsystem, whose task is to define setpoint values for the low-level control algorithm. The type of guidance algorithm to be used is defined according to the mission we want the AUV to perform (waypoint tracking, obstacle avoidance, etc.) and may come from different areas: for example potential fields methods [2] or line-of-sight [3]. The control part is provided by low-level controllers which compute the inputs needed by the AUV's actuators (propellers and fins) in order to follow the setpoint values provided by the guidance system. Several approaches can be used in control theory. Depending on the particularities of the task, the robotic platform and the environment, one designer can use different methods: adaptive control, robust control, optimal control, sliding mode control, etc. The paper is organized as follows. In section II we provide a presentation on recent work concerning the control of AUVs and then follow this up with a discussion on the use on Reinforcement Learning in robotics. In section III we present details on our deep-reinforcement-learningbased model-free controller. Finally, simulation results and the comparison with the PID controller are assessed in section IV.

II. RELATED WORK

A. Classic Control Approaches

The Proportional-Integral-Derivative (PID) controller is the most commonly used low-level controller for many applications. It is composed of three terms: the proportional term which is proportional to the tracking error (the difference between the controlled variable and the setpoint to reach), the derivative of the tracking error and its integral. The command law is then structured according to different strategies: one PID controller for each degree of freedom (DOF) [6] ; pairs of PID controllers, with the output of the first one becoming a setpoint value for the second one [7].

The PID controllers relies on a mathematical model of the controlled system which captures relatively well its dynamics around some operating conditions. Difficulties appear with variations of internal parameters of the system model and in the character of the disturbances. Vulnerabilities increase with variations in the process dynamics. In real-life problems, it is difficult to describe these mechanisms analytically because there are many different sources for the variations and in most cases, the underlying reasons for them are yet not fully understood.

In this study we have been focusing on approaches based on controllers which do not rely on any physical model of the system. These approaches are classified as model-free algorithms. The recent surge in data collection and analysis has greatly supported the use of Reinforcement Learning, a machine learning technique, as an optimization method for model-free controllers. We introduce in the next subsection the main topics of this approach.

B. Deep Reinforcement Learning in Robotics

Traditional methods for the control of an autonomous vehicle usually rely on multiple model-based blocks that perform waypoint navigation algorithms, inertial and/or visual odometry and low-level control. These methods required expert knowledge and usually do not possess sufficient robustness to process variations when working in unstructured and uncertain conditions like under water. On the other hand, model-free control methods relying on Reinforcement Learning (RL), have recently shown superior performances in the control of terrestrial [8], aerial [9] and underwater robots [10] [11]. The objective of RL methods is for a learning agent to find an optimal strategy behaviour (called a policy and denoted by π) from trials and errors. Those repeated interactions are in the form of the execution of an action a t ∈ A from state s t ∈ S which make the agent transit to a new state s t+1 ∈ S. This transition produce a feedback signal noted r ∈ R known as the reward, which quantitatively transcribes how good was this transition in regard to a given state s and with respect to a reward function R(.). This type of procedure can thus be framed as a Markov Decision Process where the objective is to maximize the expected discounted total reward:

min θ ∑ t∈1,...,T ∑ τ∈1,...,T r τ γ τ-t log(π θ (a t |s t )) (1)
The policy π(a t |s t ) is generally modelled with a parameterized function with respect to θ . Two functions are exploited to improve the policy. The first one is called the Value function, denoted by V (s) and represents the expected long-term reward achieved starting from a state s and then by following the policy π θ . If this function is known, it is possible to perceive how "good" it is for the agent to be in the state s and then follow the current policy. The second function commonly used is called the Q-Value function, denoted by Q(s, a) and similarly to the Value function, it measures the expected longterm reward, but this time is achieved from a state and action pair following the same policy. This quantity illustrate how good for the agent it is to take action a from state s and then follow the policy π θ . If these functions are fully known, the given policy is optimal since, in that case, at each timestep, the agent would be aware which action to execute to transit to the next state that would maximize its total future rewards. With this purpose, many RL approaches seek to estimate these two functions in order to improve the policy. We propose now to present classical RL approaches and their application in the robotics domain. Reinforcement-learning-based control methods can be divided in two categories: model-based and model-free approaches * . Model-based methods are strongly influenced by control theory. In this instance, a total or partial knowledge of the environment "model" x is provided which mathematically tells how it will respond to the agent's transition:

x t = f (x t-1 , u t-1 ) (2)
This model is either given or learned. In the first case, during training or acting, the agent can request the model to provide a prediction of the next reward and the next state or the expected future reward or even the full distribution of future states and future rewards. If it has to be learned, a random policy is initially used to observe the trajectories. The model is then fitted using the sampled data. Standard model-based RL methods include Dynamical Programming optimization algorithms [12]. This family of methods have the strong advantage of being sample efficient. They only need a few samples to learn models since they behave linearly, at least in the local proximity. Once the model and cost function are known, we can design the optimal control without further sampling. However, even though model-based algorithms have better physical assumptions and approximations on a given task, they might succeed only for the specific task. In contrast, model-free methods uses much more samples but are able to learn directly one or several simpler quantities without any model. Typical model-free methods are Monte Carlo Control [13], Q-Learning [14] or Actor-Critic [15] algorithms where the agent rely only on samples from the environment to change its behavior. In this study we focus on a particular type of model-free RL approach called Policy Gradient. They aim at directly modifying the parameters θ of the agent's policy to maximize its expected future discounted reward (1). In the following, we provide recent improvements made in this family of techniques.

A turning point in Policy Gradient methods arose from the paper [16] where the mechanism of Experience Replay (ER) and the use of a neural network in the Q-Learning architecture to approximate the Q-Value function were introduced. With this technique (ER), at each timestep the agent's experience e t = {s t , a t , r t , s t+1 , a t+1 } is stored in a dataset D = {e t , e t+1 . . . e T }, pooled over every episodes into a "replay buffer". Q-Learning updates are then applied to sample experience e ∼ D randomly drawn from D. This arbitrary selection of samples has the benefit of reducing correlations between consecutive samples and therefore reduce the variance of the updates. When using experience replay, it is necessary to learn Off-Policy because the updated parameters are different from the ones used for sample collection. This means that a policy π θ ′ has to be used to generate samples in order to optimize the policy π θ . Thereafter, the community has started to use the term "Deep" Reinforcement Learning to refer to approaches where the latter functions are approximate using neural networks. With this procedure, V. Mnih el al. [16] demonstrated that their agent was able to outperform the best human players on numerous games of the Atari 2600 console. The principal drawback of this approach is that it can only tackle discrete action space which seems hardly applicable in a robotic context. In fact, a designer would prefer a continuous controller to allow its robots to achieve smooth and precise movements. Significant research efforts have since been made to address this issue.

In [17] in particular, a new theorem called Deterministic Policy Gradient (DPG) has been proposed to achieve Reinforcement Learning with continuous action space. The policy is now deterministic which means that, at a given state, a given agent will always choose the same action to perform, until it is updated by the Deep RL algorithm. There are no more probabilities for each action to be selected. Moreover, its parameters are updated with gradient ascent in order to maximize the total amount of future discounted rewards. In [17], the gradient of each policy parameter with respect to the objective function to maximize is ensured by the policy gradient theorem. The DPG algorithm was improved in [18], so as to define the Deep Deterministic Policy Gradient (DDPG) algorithm. As previously explained in Deep RL, the term "Deep" refers to the use of Neural Networks (NN) to replace some features of the classical DPG. The DDPG algorithm proposes an Actor-Critic architecture, where one NN is used to model the deterministic policy, called the actor, while another one is modeling the Q-Value function, called the critic. Moreover, the target values needed by the loss function of the two NNs are themselves generated by two others NN, called the target actor and the target critic with regard again to the reward function. Finally the parameters of the target actor and the target critic are updated using exponentially moving averages of the parameters of the actor and the critic NNs. This algorithm has been used widely and notably for the Endto-End navigation of robots such as in [19] for a TurtleBot platform or in [20] for a UAV.

Lastly, in [21] a policy gradient approach with remarkable sampling efficiency and exploration capabilities was proposed. By expanding the latter improvements made in policy gradients methods combined with a brand new objective function formulation, T. Haarnoja et al. [21] demonstrated that their algorithm called "Soft Actor-Critic" (SAC) outperforms the RL methods cited so far (included DDPG and PPO [22]). In this study we proposed a model-free controller based entirely on the Soft Actor-Critic for the control and the guidance of an AUV which, to our knowledge, has never been proposed. We go further by evaluating the robustness to external disturbances of our deep-reinforcement-learning-based controller and of a classical PID controller in a ROS-based realistic simulated environment (UUV Simulator package [4]) where, in addition to sensor noise and controller output noise, we simulate various high amplitude variations in sea currents to more realistically disturb the vehicle.

III. MATERIALS

In this section, the mathematical background of the algorithm used is provided. To facilitate reader comprehension, we define the Value function as:

V π (s) = E a∼π θ [G t |S t = s t ] (3)
with

G t = ∞ ∑ k=0 γ k R t+k+1 the discounted future reward
The parameter γ ∈ [0; 1] is a discount factor that penalizes rewards in the far future. It is used because we consider that the future state of our agent might have higher uncertainty and that distant rewards have minor immediate benefits. The Q-Value function is defined in the same manner as:

Q π (s, a) = E a∼π θ [G t |S t = s t , A t = a t ] (4)
A. Soft-Actor-Critic

The Policy Gradient algorithms cited so far demand a large sampling work and possess vulnerable convergence properties that require fastidious hyperparameter tuning. On the other hand, in [21] the SAC algorithm, an Off-Policy Actor-Criticbased Policy Gradient method, has been proposed to cope with these issues by, among others things, incorporating a measure of the entropy of the policy in the reward. This approach is based on the following components:

• An Actor-Critic architecture with separate policy and value function estimators. • An Off-Policy technique that allow the reuse of previously collected samples (II-B). • The maximization of the entropy to ensure stability and exploration. Models with high entropy explore more and have better generalization capabilities because they are able to choose near optimal strategies during their decision making process. This means that at any timestep, for multiple actions that seem to be equally good, a policy with high entropy should assign each an equal probability to be chosen. The policy π θ is therefore trained to maximize the expected return and the entropy at the same time:

J(π θ ) = T ∑ t=0 E (s t ,a t )∼ρ π θ [r(s t , a t ) + αH(π θ (.|s t ))] (5)
where ρ π θ (s t ) denotes the state marginal of the trajectory distribution induced by a policy π θ (a t |s t ), the term H(.) is the entropy measure and α is what we call the "temperature" parameter which determine the relative importance of the entropy against the reward. This approach aims at learning three functions with the first one being a soft Value-function V (s t ) that is trained to minimize the squared residual error:

J V = E s t ∼D [ 1 2 (V (s t ) -E a t ∼π θ [Q(s t , a t ) -log π θ (a t |s t )]) 2 ] (6)
with D the distribution of previously sampled experiences Secondly, a soft Q-function Q(s t , a t ) is trained to minimize the soft Bellman residual:

J Q = E s t ∼D 1 2 Q(s t , a t ) -Q(s t , a t ) 2 (7) with Q(s t , a t ) = r(s t , a t ) + γ E s t +1∼p [V (s t+1 )]
Finally, the policy parameters are learned by minimizing the expected Kullback-Leibler divergence at each step:

J π (θ ) = E s t ∼D D KL π θ (.|s t )|| exp (Q(s t , .)) Z(s t ) (8)
where Z(.) is the partition function used to normalize the distribution. Three NNs are used to approximate these functions thanks to the use of target values generated by others NNs. These functions are referred to as "soft" functions because an exponentially moving average, with a smoothing constant τ = 5e -3 , is used to update the target value network weights (i.e. soft updates). This procedure which has already been used in other works [18][23] has proved that using a separate target value network that slowly tracks with soft updates the actual value function, improves greatly the stability of the training process. In addition, two Q-Value functions are trained independently to optimize their associate J Q i (7). The minimum of these Q-Value functions are then used in the gradients of the Value function (6) and Policy (8). As mentioned in [21], this method alternates between sampling experience from the environment with the current policy and updating the function approximators using the stochastic gradients from batches sampled from a replay buffer.

B. Training setup

Policy-Gradient-based models are particularly sensitive to the shape of the reward function. The smallest variations can lead to the convergence or not of the model, and the magnitude and the evolution of each term of the computed function must be chosen carefully. The reward function is often crafted by trial and error, since no global rules exist in the reinforcement learning theory, and the expression of the function can drastically change from one task to another. Moreover, the transfer from simulation to real world is a major concern in reinforcement learning for robotics. In this perspective, the simulated environment used during the training must be realistic enough, while not being too complex for the agent. Indeed, if the task requested is too complex, the agent might not be able to converge: for example if the state returned by the environment is not descriptive enough, or if the reward function is too constraining. A trade-off between the reward function and the complexity of the environment must therefore be found.

Since the proposed approach here only focuses on the highlevel guidance and low-level control parts of a GNC system for an AUV, the navigation considerations are not tackled. Therefore we consider that the AUV has access to good estimates of its pose, its linear and angular velocities, and its tracking errors with respect to the waypoint. The environment state X t observed by the SAC algorithm at time t is given as follows :

X t = [ x, Θ, v, Ω, φ e , x e , u t-1 ] T (9)
where x = [ x, y, z ] is the position vector of the AUV in Cartesian coordinates, Θ = [ ψ, θ , φ ] is its orientation vector expressed with Euler angles (respectively roll, pitch and yaw), v is its linear velocity vector (the derivative of x), Ω is the angular velocity vector (the derivative of Θ), φ e is the error between its current yaw and the desired yaw leading directly towards the waypoint, x e is the error between its current position and the desired position, corresponding to the waypoint position, and u t-1 is the vector of the inputs of the actuators computed at the previous step, at time t -1.

In order to perform a waypoint tracking mission, we designed the reward function r t defined in (10). It was inspired partly by [11], where the reward function takes into account low-level variables such as linear and angular velocities and their respective references. We adapted this work for higher level of control, taking directly into account the position of the AUV and its reference. Therefore the guidance and control parts of the GNC system of the AUV are both provided by the SAC algorithm.

r t =        r toward if d t -d t-1 > 0 r backward else r waypoint if d t < ε r limit if z / ∈ [z min , z max ] (10)
where r t is the reward received by the agent at time t, d t is the current relative distance between the AUV and the waypoint to reach, z min and z max are the authorized limits for the vertical movement z of the AUV, and ε is a strictly positive real number.

Each term appearing in (10) represents a specific feature of the global desired behavior of the AUV:

• r toward is a variable reward given when the distance d t is decreasing, which means that the AUV moves toward the waypoint. It is defined as follows :

r toward = λ 1 (d t -d t-1 ) -λ 2 Ω (11)
where λ 1 and λ 2 are positive weighting terms. The term weighted by λ 1 rewards large movements toward the waypoint, while the term weighted by λ 2 penalizes strong angular speeds, and promotes indirectly a softer use of the actuators. • r backward is a constant negative reward given when the distance d t is increasing, which means that the AUV moves backward the waypoint. • r waypoint is a constant positive reward given to the agent when the AUV reaches the waypoint, which leads to a terminal state, ending the current episode. • r limit is a constant negative reward given to the agent when the vertical movement of the AUV exceeds the limits defined by [z min , z max ], which leads to a terminal state, ending the current episode. We chose the value of all the parameters in order to give to the signals r toward and r backward a magnitude of around 10, as recommended in [21]. To make the training more realistic, we randomize several parameters of the environment. Details on this procedure are given in the following section.

IV. SIMULATIONS AND RESULTS

A. Simulation setup

The method proposed in this paper has been evaluated under simulation only, by using a model of the RexROV 2 platform (see Figure 1a) included in the UUV Simulator package [4]. It is a ROS-Gazebo-based simulator, which includes packages needed for underwater robotics simulation. The RexROV 2 is a 6 DOFs cube-shaped Remotely Operated underwater Vehicle (ROV), propelled by 6 thrusters. As shown in Figure 1b, the layout of thrusters can easily change the orientation of the RexROV 2, since some of them are able to act on more than one DOF. Consequently, the DOFs of this ROV are strongly correlated, unlike other cube-shaped ROVs which generally can independently move horizontally or vertically. The control task is therefore harder in our setup than with a classical thrusters configuration. The RexROV 2 simulation model incorporates an Inertial Measurement Unit (IMU), a Doppler Velocity Log (DVL) and a pressure sensor. We will designate it as an AUV in the following.

We compared DRL-based controller to a 6 DOFs fixed-gains PID controller tuned for a precise control of the RexROV 2 and provided by the UUV Simulator. More specifically, the PID controller regulates the entire pose of the AUV: the three coordinates of the position of the AUV and the three angles of its orientation. At the beginning of each episode, we initialize the vehicle at a position x = [0, 0, -20] (in meters and relative to the frame attached to the Gazebo world center) with a random orientation Θ, where ψ = θ = 0 and φ ∈ [0; 360] (in degrees and with respect to the center of mass of the vehicle body). The waypoint is placed at the beginning of each episode at a random position located 50 meters from the AUV. The sensors measurements included in our state vector incorporate added noise such as: for each x i ∈ X, x i = x i + σ i with σ i a sample uniformly distributed over the interval [0.05; 0.1] except for the past actions x t-1 where σ i is a sample uniformly distributed over the interval [0.01; 0.05] this time. We also added fluctuating sea currents to our underwater simulated environment. The current velocity c v ∈ [0; 1] (in m.s -1 ) and angles, (c ha ; c va ) ∈ [-0.5; 0.5] (for horizontal and vertical angles respectively in radians), are randomly modified every 100 timesteps during training and testing.

B. Results analysis

Simulation results are presented here for the proposed DRLbased controller, that have been trained over 1200 episodes and a standard PID controller. The training has been stopped after approximately 4 hours because convergence in terms of performance were reached (i.e. the variance of rewards was small enough, for us, to consider the policy satisfying). The evaluation consist of using both controllers for a series of 500 episodes each with the same environment parameters: AUV initial position, waypoint positions and sea currents variations. We measure their performance in term of success rate (percentage of episodes where the AUV reaches the waypoint), temporal mean and standard deviation (SD) of the distance error dδ to the ideal trajectory (the direct path from the initial position of the AUV to the waypoint location), but also in terms of thrusters usage under the form of the temporal mean of the Euclidean norm of the input vector u at each step (with each element of u being a signal between -240 and +240).

We can observe that the PID controller showed better performances in terms of success rate and distance error while the DRL-based controller is better in terms of thruster usage, and thus indirectly in power consumption. The difference in power consumption can be explained by the characteristics of the controllers. In fact, the PID controller incorporates the correlation between the DOFs of the RexROV 2 platform, whereas the DRL-based controller does not. This means that to move in a specific direction, the PID controller knows exactly which combination of thrusters to use. The DRL-based controller, on the other hand, estimates directly the inputs to the actuators and tends to use fewer thrusters than the PID controller to perform the same movement. This leads to smaller power consumption but poorer control performances than the PID controller.

V. CONCLUSIONS

In this paper we have proposed a model-free controller trained end-to-end with Deep Reinforcement Learning and compared it to a standard PID controller for the control and guidance of a 6 DOF AUV. The results provided in this study show that a PID controller is better than a DRL-based controller in terms of global success of the mission, but not in terms of actuators usage, given that the thrusters were 11.14% less solicited by the latter controller. Moreover the reward function implemented here gives satisfying results, even without taking into account the whole complexity of the task (e.g. penalizing directly strong actuators inputs, or deviations from the ideal trajectories). Proposed future directions for this research will include designing more sophisticated reward functions and comparing the two controllers in an underactuated context where the robot is not strong enough to fight the currents. In this case, the PID controller would (ineffectively) keep trying to go straight to the waypoint while the DRL-based controller should be able to discover new strategies such as letting itself be carried by the advantageous currents in order to optimize its motion toward the waypoint.

Learning-based vs Model-free Adaptive Control of a MAV under Wind Gust ⋆ Thomas Chaffre 1,2 , Julien Moras 3 , Adrien Chan-Hon-Tong 3 , Julien Marzat 3 , Karl Sammut 2 , Gilles Le Chenadec 1 , and Benoit Clement 1,2

1 Lab-STICC UMR CNRS 6285, ENSTA Bretagne, Brest, France {thomas.chaffre,gilles.le chenadec}@ensta-bretagne.org 2 Centre for Maritime Engineering, Flinders University, Adelaide, SA, Australia {karl.sammut,benoit.clement}@flinders.edu.au 3 DTIS, ONERA, Université Paris-Saclay, 91123 Palaiseau, France {julien.moras,adrien.chan hon tong,julien.marzat}@onera.fr Abstract. Navigation problems under unknown varying conditions are among the most important and well-studied problems in the control field. Classic model-based adaptive control methods can be applied only when a convenient model of the plant or environment is provided. Recent model-free adaptive control methods aim at removing this dependency by learning the physical characteristics of the plant and/or process directly from sensor feedback. Although there have been prior attempts at improving these techniques, it remains an open question as to whether it is possible to cope with real-world uncertainties in a control system that is fully based on either paradigm. We propose a conceptually simple learning-based approach composed of a full state feedback controller, tuned robustly by a deep reinforcement learning framework based on the Soft Actor-Critic algorithm. We compare it, in realistic simulations, to a model-free controller that uses the same deep reinforcement learning framework for the control of a micro aerial vehicle under wind gust. The results indicate the great potential of learning-based adaptive control methods in modern dynamical systems.

Introduction

Aerial vehicles are exposed to a mixture of perturbations that fluctuate vigorously because of the hazardous environment they are evolving in. They operate over a wide range of speeds and altitudes. Their dynamics are nonlinear and can also be of time-varying nature (as they fly, their mass slowly decreases due to fuel consumption and their center of gravity can greatly vary). Micro Aerial Vehicles (MAVs) are challenging in some other ways. They have to constantly compensate for the distribution of forces that act on their small airframes because of wind and turbulence. These forces can significantly increase when weather conditions change and are also influenced by the velocity and heading of the vehicle. Because they are small, rigid, light, and move slowly, MAVs are highly sensitive to wind gusts even if mild. In addition to the latter unpredictable (external) forces, we face the difficulty of accurately modeling their (internal) dynamics under these conditions. With the development of flight controllers for hypersonic aircraft by NASA in the early 1950's [1], it was found that conventional linear feedback control strategies (e.g. fixed-gain control [2], robust control [3], sliding mode control [4] or fuzzy logic control [5]) are too limited to handle such entire regimes. It is also true for the MAVs context where for a variety of reasons it is almost impossible to manually re-tune the control parameters, these include:

-The uncertainty level of the wind speed is high and can vary more quickly than can be compensated by these types of controllers.

-The response at some operating points has to be overly conservative in order to satisfy specifications at other operating points. -The controlled process itself varies significantly during operations (the motors' and propellers' efficiency for example).

The field of adaptive control broadly addresses these challenges by granting the control law some flexibility to modify its action based on the process variations. Despite the great advancements conducted in the theoretical area, the expansion of autonomous agents to the real-world is still limited. This is mostly due to the high dimensional problems that often appear when working under real conditions where it usually requires many degrees of freedom to describe the robot state. The lack of basic knowledge of the various natural phenomena taking place in the robot's environment, e.g. wind gust for MAVs or sea current for autonomous underwater vehicles (AUVs) makes it even more difficult to actively control it [6] with model-based adaptive methods.

For the past few years, model-free adaptive control methods have attracted significant attention. They exploit strong statistical tools that provide control systems the ability to automatically learn and improve from experience without being explicitly told how to. In particular, model-free adaptive methods based on deep reinforcement learning (DRL) have shown promising achievements. This was possible especially thanks to the use of deep neural networks to extract physical insights through sensory data (empowered by progress in data collection with high fidelity simulations, cheaper storage drives, faster computers, etc). However, as stated in [7], the extension of DRL techniques for robotic tasks raises serious questions. Compared to other application fields, robots have to interact with a dynamic environment where relevant information about the system is not always accessible or tractable over time.

In their up-to-date thorough investigations [8], G. Dulac-Arnold et al. presented real-world reinforcement learning challenges that are still not resolved, these include Satisfying Environmental Constraints, High-Dimensional Continuous State and Action Spaces or Multi-Objective Reward Functions. In addition, the notions of stability (in terms of Lyapunov stability) in the DRL framework is more deeply investigated [9] but the lack of formalism is still often highlighted by the control community as a significant concern. The purpose of this study is to explore to what extent model-free control techniques combined with the classical theory of dynamical systems can be, at least, part of the solution to these great challenges. This is the objective of the learning-based control area [10] which aims to combine the advantages of the aforementioned paradigms (modelbased and model-free) into one hybrid control scheme. Therefore, we propose a learning-based adaptive control system composed of a state-feedback control law whose parameters (i.e. gains) are tuned by the Soft Actor-Critic DRL algorithm [11] using the pole placement method. It is fairly compared, in a realistic simulation setting, with a model-free adaptive method trained end-to-end with the same DRL framework (as initially applied to robot navigation in [12]). The control objective treated is a hexacopter performing a waypoint rallying mission under unknown wind fields.

The paper is organized as follows. Section 2 gives details on the theoretical bases of the proposed control strategies and related work. In Section 3, we present background material on the simulation and control of MAVs. Section 4 is dedicated to the description of our contribution, with the application of DRL to the design of model-free and learning-based controllers. A description of the simulated training and evaluation sessions is provided in Section 5 with the obtained results. Finally, a thorough analysis of the outcomes and some suggestions to extend the work further are presented in Section 6.

Related work

An adaptive controller is essentially a controller that can adjust its own behavior in response to changes in the dynamics of the process and the character of the disturbances [13]. Since ordinary feedback control also seeks to reduce these undesirable effects, one needs to clearly distinguish between the two. In this paper, we will designate as adaptive, a control system that provides the desired system performance by changing its parameters and/or structure in order to reduce the uncertainties effect and to improve the knowledge of the desired system. As mentioned by Nikolai M. Filatov [14], the goal of adaptive systems is to switch off the adaptation as soon as the system uncertainty is reduced sufficiently and to then use the adjusted controller with a fixed structure and fixed parameters. Although it is more common in the control community to classify methods based on the nature of the model (e.g. linear vs nonlinear, continuous vs discrete), we classify here adaptive controllers based on their dependence on the mathematical model of the controlled system. This algebraic representation of the dynamics of the system can take several forms such as differential equations, state-space representations, or frequency domain representations. We can thus identify three types of adaptive control methods: totally model-based, totally model-free, and learning-based. Since the approach presented here consists in merging together some model-based controllers and model-free algorithms, in this chapter we will present the basis of the aforesaid techniques and recent applications to the problem of MAVs under unknown wind fields. Then, we will see how reinforcement learning can be used to strengthen control laws, which will lead to the introduction of the proposed learning-based adaptive controller.

Formulation of the adaptive control problem:

Consider the system described by the following continuous-time state equations:

ẋ(t) = f (x(t), u(t), p(t), t), y(t) = h(x(t), u(t), t), (1)
where x ∈ R n is the system state2 ; u ∈ R nu is the control vector ; p(t) ∈ R p is the constant or time-varying vector composed of the unknown or uncertain parameters of the model ; y ∈ R m is the output vector of interest. The probability density of the initial values P[x(0), p(0)] is assumed to be known. We can define the set of outputs and control inputs available at time t as:

Γ t = {y(t), . . . , y(0), u(t -1), . . . , u(0)}, t = 0, 1, . . . , T -1, Γ 0 = {y(0)} (2)
The control performance index can have the following form:

J(t) = 1 t t 0 ē2 (τ )dτ, ē(t) = w(t) -y(t) (3)
where w(t) is the targeted set point. The general problem of adaptive control consists in finding the control policy u(t) = u t (Γ t ) ∈ Ωt that minimizes the performance index (3) for the system described by (1) with Ωt the domain in the space R nu where the admissible control values are defined. Backward recursion of the following stochastic continuous-time dynamic programming equations can give the optimal control of the above problem:

J t-1 (Γ t-1 ) = min u(t-1)∈Ωt-1 1 t t 0 ē2 (τ )dτ |Γ t-1 , (4)
J t (Γ t ) = min u(t)∈Ωt 1 t t 0 ē2 (τ )dτ + J t+1 (Γ t+1 )|Γ t ,
for t = t -2, t -3, . . . , 0.

(

This optimal solution is usually impractical to derive analytically from ( 4) and ( 5) because of the dimension of the underlying spaces, even for simple cases. Near-optimal solutions can be obtained with model-based adaptive methods.

Model-based adaptive control

The design of a controller usually begins with a mathematical model of the system to be controlled. For mechanical systems of moderate dimension, it is possible to write down such a model (e.g. based on the Newtonian, Lagrangian, or Hamiltonian formalism) and eventually linearize its dynamics around a fixed point or periodic orbit. Model-based adaptive control refers to adaptive control methods that are completely reliant on this type of model. This means that the feedback control law will adjust the controller parameters online to compensate for model uncertainty. This adaptation can be performed in different manners, which gives the classification of model-based methods in two groups: direct and indirect. In direct schemes, designers attempt to estimate the control parameters. The adjustment rule tells directly how the controller's parameters should be updated. It is done without intermediate calculations involving plant parameter estimates. This is possible because in many cases, there exist measurable variables that correlate well with changes in the process dynamics. The Gain Scheduling [15] and Model-Reference Adaptive Systems [16] algorithms are examples of direct schemes. On the other hand, if the estimates of the process parameters are updated and the controller parameters are obtained from the solution of a design problem using the estimated parameters, we obtain what is called an indirect approach. This kind of controller can be seen as automation of process modeling and design, in which the process model and the control design are updated at each sampling period. The Self-Tuning Regulators [17] and Dual Controllers [18] are well-known indirect methods. Instead of further extending the theory basis of model-based control (since it is not the main focus of this study), we describe here recent applications to the control of quadrotor MAVs under wind disturbances. An output controller was proposed in [19] to cancel wind disturbances where the nonlinear dynamical mathematical model of a quadcopter was decomposed into two parts: a static MIMO transformation and few SISO channels. It was assumed that the unknown wind force acts on each channel of the quadcopter as a constant signal that has to be canceled. They proposed to first design a virtual control law for each SISO channel and then to inverse the MIMO transformation to get the control laws for the initial considered system. Their approach showed, under simulation, that it was able to efficiently reach steady-state under unknown wind perturbations. Later on, to reduce the undesirable effects of wind on a similar quadcopter, A. Razinkova et al. [20] chose another strategy. They assumed the wind to be an unknown but uniform and time-varying force acting on the vehicle body in the X and Y -axes. Therefore, they proposed to augment a PD controller with additional terms adapting to the aforesaid forces. The resulting PD control law according to an axis i takes the form:

u i = K p × e(t) + K d × e(t)/d t -γ ĩ
, where e(t) is the position error and γ ĩ the adaption term. Such an adaptation converges [20] if and only if the adaptation rate γ remains positive. Later on, in [21] a L 1 adaptive controller [22,23] (which is an advanced version of MRAC [16]) was proposed to control a quadcopter that is performing wind turbine inspection. They proved the robustness of their controller with respect to wind under simulated and real flights by comparing it to a basic Linear Quadratic Regulator (LQR). More recently, in [24] the gains of a PID controller have been tuned especially in order to handle wind gusts. They tuned this classical feedback controller in the H 2 optimal control framework and compared it with the existing LQR-based tuning method [25]. Simulated experiments proved that the classical PID controller combined with a dedicated parameter adjustment mechanism is better at rejecting wind disturbances than the classical LQR method.

These approaches have in common the use of the Certainty Equivalence (CE) approach, which means that the estimation uncertainty is not taken into consideration. The wind estimates are used in the control law as if they were the real values of the unknown wind field. The CE paradigm has been used for a long time and makes it possible to generate a wide class of model-based adaptive controllers by combining different on-line parameter estimation algorithms (e.g. sensitivity methods, positivity and Lyapunov design, gradient and leastsquares methods, etc) with different control laws (e.g. PID, LQG, LQR, Pole placement, etc). However, it was proved in [14] that control systems based on the CE approach are not always optimal and can be far from so. A straightforward improvement would be to apply reinforcement on the uncertainty estimates over time, in a sort of learning process, as a mean to optimize control performance. This goal, combined with the development of complex systems, where precise modelization is extremely difficult, has motivated the development of the model-free adaptive control field.

Model-free adaptive control

The model-free wording refers here to the fact that these families of controllers do not rely on any mathematical model of the controlled system. They aim at describing complex systems from observational data collected directly by embedded sensors rather than first-principles modeling. As shown in Figure 2, modelfree algorithms can be seen as an optimization problem where the goal is to minimize a cost function without closed-form knowledge of the function or its gradient. A classic approach widely used in the model-free control framework is the Extremum-Seeking (ES) methods [26,27]. We present now a simple ES algorithm by considering the same equations of state (1). The goal of the control is again to optimize the performance of the system described by the cost function (3). We model it as a smooth function J(x, u) : R n × R → R. We will denote it simply by J(u) because the state vector x is driven by u. In order to write convergence results we need the following assumptions:

1. There exists a smooth function l : R → R n such that:

f (x(t), u(t), p(t), t) = 0, if and only if x(t) = l(u(t)) (6)
2. For each u ∈ R, the equilibrium x = l(u) is locally exponentially stable.

3. There exists (a maximum) u * ∈ R such that:

(J • l) (1) (u * ) = 0 (J • l) (2) (u * ) < 0 (7)
Based on these assumptions, one can easily design a simple extremum seeker with proven convergence bounds. One of the simplest way to maximize J is to use a gradient-based ES control law as:

u = k dJ du , k > 0 (8)
The convergence of (8) can be analysed with the Lyapunov function:

V = J(u * ) -J(u) > 0, for u = u * (9)
The derivative of V gives:

V = dJ du u = -k dJ du 2 ≤ 0 (10)
This proves that the algorithm drives u to the invariant set dJ du = 0, which is equivalent to u = u * . However, as simple as it seems, this approach requires the knowledge of the gradient of J. An ES-based model-free approach was proposed in [28] for the optimal control of a quadcopter carrying different payloads. They proved, with experimental flights, that their model-free controller is able to find the speed that maximizes the flight time (endurance) or flight distance (range) of the vehicle when transporting an unknown payload. They designed specific cost functions for each of these goals. Then, the ES algorithm was applied to estimate an optimal velocity which is used to transform the desired path into a trajectory. They proved that the ES scheme is able to find unknown, time-varying, operating points that minimize these cost functions directly from sensorial measurements and without any model of the MAV power consumption. is to minimize a well-defined cost function J within the space of possible control laws. The off-line learning loop provides experiential data to train the controller. The vector z is composed of all the information that may factor into the cost.

Model-free control is a blossoming field where a handful of strong techniques are currently being developed and applied to minimize or maximize specific cost functions by using the system outputs as shown in Figure 2. Reinforcement learning (RL) is currently the leading technique used in this research area. It is a class of machine learning methods in which an autonomous agent in an environment has to learn how to take actions in order to maximize the notion of cumulative reward [29]. More formally, the objective of RL is for a learning agent to find an optimal strategy behavior (called a policy and denoted by π) from experimental trials and errors. Those repeated interactions are under the form of the execution of an action a t ∈ A from state s t ∈ S which makes the agent transit to a new state s t+1 ∈ S. This transition produces a reward signal denoted by r(s t , a t ) ∈ R, which quantitatively transcribes how well the agent is doing in the environment. The policy is a rule used by the agent to decide what actions to take and is mathematically defined as a function:

π : S → A (11)
This type of procedure may be framed as a Markov Decision Process [30] along a trajectory (a sequence of T actions and T + 1 states in the environment) τ = (s 0 , a 0 , s 1 , a 1 , . . . , a T -1 , s T ). Hence, the probability of a trajectory for a π policy is:

P (τ |π) = ρ 0 (s 0 ) T -1 t=0 P (s t+1 |s t , a t )π(a t |s t ) (12)
where ρ 0 (s 0 ) means that the starting state s 0 is randomly sampled from the distribution ρ 0 and π(a t |s t ) means that the action a t is randomly sampled by the policy π in the state s t . The overall optimization problem considered in RL is to determine the policy π ⋆ which maximizes the expected return when the agent decides to take actions according to this policy:

π ⋆ = arg max J RL (π) (13)
where J RL (π) denotes the expected return

J RL (π) = τ T -1 t=0 r t • P (τ |π) = E τ ∼π T -1 t=0 r t (14)
Both RL and model-free paradigms are based on the same optimization formalism. We can easily see the similarity between this cost function ( 14) and the one from the ES algorithm (8). Note that the terms model-based and model-free can also be found in the RL theory but do not correspond to the ones used in the control field. In RL, they refer to whether or not a model of the environment (a function which predicts state transitions and rewards) is given beforehand to the RL algorithm to improve the policy whereas, in control, it is the model of the controlled dynamical system that might be provided.

Various methods exist to tackle the RL optimization problem. Among them, Policy Gradient techniques have enabled to use RL in real-world robotic contexts (see [31] for an extensive definition of these approaches and [32,[START_REF] Kahn | Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation[END_REF]34] for successful real-world robotic applications). These techniques are based on the use of a stochastic policy formally denoted by π θ , where θ is a vector of parameters. The vector θ is estimated iteratively by gradient ascent. The expression of the gradient of J(π θ ) with respect to θ is given by:

∇ θ J RL (π θ ) = E τ ∼π θ T t=0 ∇ θ log π θ (a t |s t )(Q π θ (s t , a t ) -V π θ (s t )) (15)
where V π θ (s t ) is V -value function quantifying the expected return if the trajectory starts in a state s t taking actions upon the policy π θ :

V π θ (s t ) = E τ ∼π θ T t=0 r t |s 0 = s t (16)
and Q π θ (s t , a t ) is the Q-value function quantifying the expected return if the trajectory starts in a state s t , takes an action a t and then takes actions upon the policy π θ :

Q π θ (s t , a t ) = E τ ∼π θ T t=0 r t |(s 0 = s t , a 0 = a t ) (17)

Learning-based adaptive control

Learning-based controllers can be seen as hybrid control schemes. They are used when we only have access to an imperfect physics-based model of the system. The idea is to strengthen it with some model-free algorithm to compensate for the uncertain or the missing parts of the model. This compensation can either be done directly by learning the uncertain parts or indirectly by tuning the controller parameters to cope with the uncertainty. To illustrate this idea, we consider the system (1) with a specific structure that can be written as:

ẋ(t) = f 1 (x(t), u(t), t) + f 2 (x(t), p(t), t), y(t) = h(x(t), u(t), t) (18)
In this model, f 1 represents the known part of the plant that can be efficiently driven by a classical model-based control architecture, while f 2 represents the unknown part of the model that can be compensated by some model-free learning algorithm. The goal of learning-based controllers is to use a model-free step to optimize an unknown performance function and then use a model-based control law to guide the system's dynamics toward the optimal performance. The idea of combining both techniques is indeed attractive. One designer could take advantage of the model-based design, with its stability characteristics, and add to it the advantages of model-free learning, with its fast convergence and robustness to uncertainties. Overall, in Neural-Network (NN) learning-based control design, the idea is to write the model of the plant as a combination of a known and unknown part (i.e. the disturbances). The NNs are then used to estimate the unknown part of the model. As a result, a controller based on the known part (model-based) and the NN estimates of the uncertainties (model-free) is determined to realize some desired regulation or tracking performance. To illustrate this concept we formalize it below, in a similar spirit as the formalism from [10] where a state-space model of (1) under the Brunovsky form is considered as:

ẋ1 = x 2 , ẋ2 = f (•) + u, (19)
where f (•) = f 1 (x(t), u(t), t)+f 2 (x(t), p(t), t), with f 1 known and f 2 an unknown smooth function of the state variables x = (x 1 , x 2 ) T and u the control signal.

The unknown part of the model, namely f 2 , is estimated by a NN as:

f2 = Ŵ T S(x(t)), (20)
where Ŵ = ( ŵ1 , . . . , ŵN ) T ∈ R N is the estimated vector of synapse weights of the neural network node and S(x) = (s 1 (x), . . . , s n (x)) T is the regressor vector, with s i , i = 1, . . . , N . Consider the reference model:

ẋref1 = x ref2 , ẋref2 = f ref (x), (21)
where f ref is a known nonlinear smooth function of the desired trajectories

x ref = (x ref 1 , x ref 2 ) T .
A basic learning-based controller can be defined as:

u = -e 1 -c 1 e 2 -Ŵ T S(e) + v, (22)
with:

e 1 = x 1 -x ref 1 , e 2 = x 2 -v, v = -c 2 e 1 + x ref 2 v = -c 2 (-c 2 e 1 + e 2 ) + f ref (x ref ), (c 1 , c 2 > 0) Ẇ = Γ (S(e)e 2 -σ Ŵ ), (σ > 0 and Γ T > 0) (23)
It can be observed that u is now a function of the Brunovsky form (19) (i.e., model-based information), and the remaining part is based on the NN estimates of f 2 (model-free estimation). The unknown part is here estimated by a neural network but other strategies can be used in the learning-based scheme. This concept of using NNs to estimate the unknown parameters (i.e. the wind field for our use case) seems appealing. In fact, as stated by the universal approximation theorem [35] for any function f (x), regardless of its complexity, there exists a neural network such that for every possible input x, the value f (x) is a feasible output from this network. If the external disturbances are well estimated, efficient control laws can be designed accordingly. This idea was investigated in [36,37] where a Geometric Adaptive Controller was proposed based on NNs for a quadcopter in wind fields. Two control laws were defined with adaptive control terms denoted as ∇i to mitigate the effects of the unknown disturbance. These adaptive control terms were computed with neural networks, exactly like in (22), as: ∇i = W T i ς(z i ) with zi = V T i x nni ( Vi being the current estimate of the ideal weighting parameters and x nni the inputs of the neural network). They proved with real experiments that the learning-based controller succeeded to complete the considered backflip maneuver followed by a stable hovering flight whereas it was not possible without the disturbance rejection terms. Another strategy was proposed in [38], where a learning-based safety-preserving cascaded QP controller (SPQC) using Gaussian Processes (GP) [39] has been proposed for safe trajectory tracking by a quadcopter in a cluttered environment. More precisely, the cascade controller is composed of two QP controllers: a position and an attitude level QP controller. The first one generates the desired thrust while the second makes use of it, together with the high confidence uncertainty interval obtained via GPs, to compute the desired body rotational rates. They evaluated their approach under numerical simulation whose results proved that the proposed learning-based controller is able to perform a trajectory tracking task with obstacle avoidance capacity under changing wind fields.

Compared to the approaches presented in this related work Section, we propose to only treat the parameters adjustment task. We design a learning-based controller that uses a DRL-based model-free algorithm to perform online-tuning of the parameters of a state-feedback controller. We compare this method to the model-free strategy initially applied in [12] but this time for the application of a waypoint rallying mission by a MAV under unknown wind gusts.

MAV simulation, modeling and control

This Section presents the ROS package used to simulate the aerial vehicle and wind perturbations and the corresponding model of the Firefly platform, which is the hexacopter considered in this study. The principal elements necessary to understand the control designs proposed in Section 4 are also derived.

RotorS package

Testing algorithms on physical platforms can be very time-consuming and dangerous for the robot. It can be even riskier when one designer wants to perform the training of machine learning algorithms directly on the real robot (because of their initial hazardous behavior). There exist many robotic simulation tools to reduce field testing time and make debugging easier. In this work, we relied on the ubiquitous Gazebo simulator [40] which is connected to the Robot Operating System framework (ROS [41]). ROS is an open-source meta-operating system for robots that facilitates the reuse of code.

The Gazebo-based package called RotorS [42] is a high-fidelity simulation framework for MAVs (developed by the Autonomous Systems Lab team from ETH Zurich) that does not require any additional components to simulate highlevel tasks (e.g. path-planning, collision avoidance, or vision-based problems). A complete model of the Firefly MAV is directly included in the package along with various world models and a plugin to simulate wind fields. We relied on the wind plugin provided by RotorS to generate wind fields in the environment. This plugin allows to define the wind as a 3D field sampled over a regular grid and each point specifies a wind velocity vector (in m.s -1 ). More specifically, we employed the environment named hemicyl (see Figure 3) and its pre-configured wind field, the field used is composed of 6282 vertices and the wind velocity vary here between -5m.s -1 and +10m.s -1 .

  x B y B z B   =   cθcψ cθsψ -sθ sφsθcψ -cφsψ sφsθsψ + cφcψ sφcθ cφsθcψ + sφsψ cθsθsψ -sφcψ cφcθ     x W y W z W   (24)
The main forces acting on the vehicle come from gravity and the thrust of the rotors. Rotors drag and air friction are neglected here to simplify the model. It is classically assumed that the hexacopter is a rigid body with a symmetrical structure, and tensions in all directions are proportional to the square of the propeller speed. The equations of motion follow as:

ζ = v v = -ge 3 + R b m Ω 2 i Ṙ = Rω I ω = -ω × Iω - J r (ω × e 3 )Ω i τ (25)
where R the rotation matrix from R B to R W ; ω is the skew symmetric matrix ; Ω i is the i-th rotor speed ; I the body inertia ; J r the rotor inertia ; b is the thrust factor and τ is the torque applied to the body frame due to the rotors. A classical cascade control structure is adopted [43], where the built-in low-level controller of the RotorS package is used to track a reference in roll φ r , pitch θ r and thrust T. The yaw angle ψ is kept constant without loss of generality. Under the small-angle assumption on φ and θ, the guidance model reduces to the doubleintegrator model:

ζ = v v = u = [u x , u y , u z ] T (26)
where the computed accelerations are converted into low-level control inputs as:

T = m(u z + g) θ r = m T (cψu x + sψu y ) φ r = m T (sψu x -cψu y ) (27)

Controller parametrization

For future use by the learning-based control approach (Section 4.4), a specific parametrization is adopted to control the system (26).

  ż ė v   =   0 1 0 0 0 1 0 0 0     z e v   +   0 0 1   u (28)
The corresponding state-feedback controller, equivalent to a PID, is such that:

u = -k i z -k p e -k d v (29)
The poles of the closed-loop system are solution to the following equation:

λ 3 + λ 2 k d + λk p + k i = 0 (30)
The controller has been re-parametrized using pole placement so as to guarantee convergence to steady-state without oscillations. This way, the action space for learning purposes is limited to desired solutions in the real part of the pole map, which prevents from sampling unnecessary solutions in the space of the control gains. The desired constants τ 1 > 0, τ 2 > 0, τ 3 > 0 are then defined as:

λ 1 = -1 τ 1 ; λ 2 = -1 τ 2 ; λ 3 = - 1 τ 3 (31)
Since each one is solution to (30), it follows that:

   1 -1 τ1 1 τ 2 1 1 -1 τ2 1 τ 2 2 1 -1 τ3 1 τ 2 3      k i k p k d   =    1 τ 3 1 1 τ 3 2 1 τ 3 3    ⇔ M K T = N (32)
Finally, the gains of the controller (29) are obtained as K T = M -1 N :

k i = 1 τ 1 τ 2 τ 3 k p = τ 1 + τ 2 + τ 3 τ 1 τ 2 τ 3 k d = τ 1 τ 2 + τ 1 τ 3 + τ 2 τ 3 τ 1 τ 2 τ 3 (33)
J V (Ψ ) = E st∼D 1 2 (V Ψ (s t ) -E[Q π θ w (s t , a t ) -log π θ (a t , s t )]) 2 (37)
with Ψ the target value function. The NN parameters θ of the policy are updated in order to minimize the expected Kullback-Leibler divergence:

J π (θ) = E st∼D D KL π θ (•|s t ) exp(Q π θ w (s t , •)) Z π θ w (s t ) (38)
with Z π θ w a partition function used to normalize the distribution. The (•) ∼ D means that the expected values are computed using the pair (s t , a t ) and (s t ) sampled from a replay buffer D (see Section 4.2) in which the agent experience e t = (s t , a t , r t , s t+1 ) is stored at each time-step. Moreover, the authors use an exponentially moving average, with a smoothing constant τ = 5e -3 , to update the target value network weights w (the parameters Ψ are also update in the same way since they are directly related to w). Thus these weights are constrained to change slowly from one iteration to another. The name Soft Actor-Critic is based on this soft update procedure. Secondly, the SAC algorithm optimizes a stochastic policy in an off-policy manner [45]. This means that the policy used to explore the environment is different from the one that is being evaluated and improved. Finally, the policy is trained to maximize simultaneously the expected return and its entropy, which is an amount of informative randomness:

J SAC (π) = T t=1 E (st,at)∼ρπ [r(s t , a t ) + αH(π θ (.|s t ))] (39)
where:

H(π θ (.|s)) = - a∈A π θ (a) log π θ (a|s) (40)
The term H(π θ ) is the entropy measure of policy π θ and α is a fixed temperature parameter that determines the relative importance of the entropy term (an automatic adjustment mechanism for α was proposed in [44] by considering the constrained optimization problem of maximizing the expected return while satisfying a minimum entropy constraint but was not used in our implementation). The authors state that entropy maximization leads to policies that have better exploration capabilities. Note that this RL problem (39) is different from the initial RL problem (13). The resulting Q-value and V-value functions will therefore also include this entropy term. Two additional Q-functions are used to reduce positive bias in the policy improvement step, which is known to degrade performance of value based methods [46,47]. They are trained independently and the minimum of the two Q-functions is used as proposed in [47].

The SAC algorithm is part of the family of deep reinforcement learning approaches which try to exploit the strong representation capabilities offered by NNs to represent the Q-value, State-value and policy functions. The NNs structure of our SAC implementation is shown in Figure 4. ) and learning rate l r = 3e -4 . Other parameters are a discount factor γ = 0.99 for (34), a number of 256 hidden units per hidden layer and one gradient update is performed at each time-step.

Experience Replay

To improve its performance, one needs to learn from its past actions. In [52], L. Lin showed that past experience of the learning agent should and could be used in an effective way by using what he called Experience Replay (ER). An experience is the result of a transition to a new state which can be framed as a quadruplet e t = (s t , a t , r t , s t+1 ). The ER technique consists in storing at each time-step the experience e t of the agent in a data-set (a replay buffer) D = e 1 , . . . , e n of fixed and substantial size. During training, the RL algorithm updates are then applied to mini-batches of experiences randomly pooled over the stored samples to reinforce the functions estimates.

This intuitive concept can be refined by distinct means. It seems natural that when learning a new task, some experience might be more valuable than the others and should be used more often in the update process. This relevance is in general not directly accessible, therefore a specific criterion has to be chosen to quantify it. In [53], T. Schaul proposed the Prioritized Experience Replay (PER) technique which consists in sampling more often from the replay buffer the transitions with high expected learning progress, as measured by the magnitude of their temporal-difference (TD) error. By doing so, they were able to obtain state-of-the-art performance on the Atari 2600 benchmark suite.

Nevertheless, depending on the task, PER can be very computationally expensive. Another parameter that has been considered is the replay capacity that is the total number of transitions possibly stored in the replay buffer. In [54], S. Zhang showed that both large and small size replay buffers can significantly damage the performance. They proposed Combined Experience Replay (CER) to cope with this problem. It consists of adding the latest transition to the minibatch pooled over the replay buffer which only requires O(1) extra computation compares to PER to reduce the negative effect of the buffer size. The difference with PER is that by using CER, the latest transitions will undoubtedly be sampled. We decided to use the CER technique and to keep a mini-batch composed of randomly sample experiences e rd n and the last one as: mini-batch = [e rd 1 ; e rd 2 ; . . . ; e rd n-1 ; e

-1 ] (41)
with n = 256, the size of our mini-batch. We kept a replay capacity of 10 6 as recommended in the original paper of the SAC [11].

Application to the model-free controller

We will now present the use of the previous deep reinforcement learning framework to build a model-free controller. The methodology follows the same line of thought as our previous work [12]. The objective of the learning agent is to build a predictive model, using the SAC algorithm, that directly maps the actions (27) that control the MAV from the current state:

π θ : s t → [ φ r ; θ r ; T ] (42)
where (φ r ,θ r )∈ [-π 6 ; + π 6 ] and T ∈ [m × g; m × (g + 3.0)] (with m = 1.544Kg, g = 9.81m.s -1 ). In order to achieve the target rallying mission, we need to provide relevant data to the agent. Therefore, we defined the following vector o t as the observation at the time-step t of the environment: ). We went even further and tried to add a sense of "acceleration" in the evolution of the state by including the difference between the latter vectors acc t = (vel t -vel t-1 ). The resulting state vector is therefore defined as:

o t = [ a t
s t = [ o t ; o t-1 ; o t-2 ; vel t ; vel t-1 ; acc t ] (44)
The dimension of this state vector is 114. The reward function r t has been designed in order to teach the agent how to complete the mission of target rallying:

r t =        r receded if d rate 0 r f orward if d rate > 0 r reached if d t d reached r f ailed if z w / ∈ [0.25; 20] (45)
where r t is the reward of the agent at time t ; d rate is the distance rate to the target performed between the last two steps such as d rate = d t -d t-1 ; d reached is the limit value beneath which we consider the target to be reached ; z w ∈ R W is the MAV altitude; both r reached and r f ailed are terminal rewards determined at the end of the ongoing episode. Each of these terms represent the specific features of the desired behavior of the MAV:

r receded is a constant negative reward equal to -20 that is sent to the agent whenever the robot is getting away from the target (or staying immobile). -r f orward is a positive reward sent when the relative distance to the target is decreasing. It is defined as: With this design, we encourage the robot to move toward the target as fast as possible. We chose the value of the constants C 1 in order to scale the positive reward signal. This is particularly important because as mentioned in [11], the SAC algorithm is particularly sensitive to the scaling of the reward signal which is the magnitude of the reward value. We followed the recommendation prescribed in [11] and chose to set C 1 = 20 in order to obtain a positive reward scale of 20 (which we found in practice to be the reward scale that gave us the best performance for this problem). The constant C 2 represents how sparse is r f orward based on the distance to the target. We chose the value C 2 = 20 empirically. Therefore, the positive reward is equal to: We define as sampling rate, the rate at which a state is sampled from the environment after the execution of the actions. A good practice consists in synchronizing it with the slowest sensor of the system which ensure no potential loss of information in the state vector and the fastest sampling rate. We synchronized it with the embedded odometry sensor which led to a sampling rate of about 20Hz.

r f orward = C 1 × e -( d t 1+d rate )× 1
r f orward = 20 × e -( d t 1+d rate )× 1

Application to the learning-based controller

As stated in Section 2.3, learning-based methods consist in exploiting standard model-based control architectures that are either tuned or redesigned by a modelfree algorithm in order to compensate for the unknown part of the model. The MAV is subject to an additive but unknown wind perturbation. Therefore, we combined the PID controller (29) under parametrization [START_REF] Kahn | Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation[END_REF] within the DRL framework (Section 4.1) to iteratively auto-tune the feedback gains parameters (i.e. the constants τ 1 , τ 2 and τ 3 ), with the ambition of optimizing online a desired performance cost function. For behavior stability purpose, instead of directly trying to estimate the values of τ i , the SAC algorithm estimates at each timestep small increments ∇ i that are added to the τ i in order to cope with the wind disturbances. By doing so, we avoid jumping from a configuration of gains to a totally different one, which usually give rise to undesired oscillations. The gains (33) are then computed using τ i = τ i + ∇ i with ∇ i ∈ [-0.01; +0.01] (while making sure that 3 ≥ τ 1 ≥ τ 2 ≥ τ 3 ≥ 5×10 -3 ) and the resulting control law ( 29) is finally applied. The τ i parameters are initialized to the nominal configuration τ 1 = 1, τ 2 = 2.5, τ 3 = 0.875. The objective of the learning agent is now to build a predictive model that directly maps ∇ i from the current state to adjust the gains starting from this inital configuration:

π θ : s t → [ ∇ τ1 roll ; ∇ τ2 roll ; ∇ τ3 roll ; ∇ τ1 pitch ; ∇ τ2 pitch ; ∇ τ3 pitch ; ∇ τ1 thrust ; ∇ τ2 thrust ; ∇ τ3 thrust ] (47)
We used a slightly different observation vector from the model-free scheme (43). Indeed, we propose to add the resulting parameters and the PID controller outputs (29) in the observation vector: where dim(τ rpt ) = 9 and pid rpt = [φ r ; θ r ; T] . The dimension of this observation vector is 37. We constructed the state vector out of this observation vector as earlier (44), resulting to a state vector of dimension 222. We used the exact same reward function and parameters as described earlier (45) to train this controller.

o t =[ a t

Results

A training episode is defined as follows: at the beginning of the episode, the MAV is set at the center of the environment at an altitude of 3 meters with roll, pitch and yaw angles equal to 0. A target is then initialized at a fixed and uniformly random position Λ = [Λ x ; Λ y ; Λ z ] T with [Λ x ; Λ y ] T ∈ [-20; -5[∪]5; 20] and Λ z ∈ [2; 20]. The mission then begins and is considered as a success if the relative distance to the target is inferior to a predefined threshold d reached and as a failure if an error signal is generated, both cases ending the episode. Otherwise, the episode is ended if the number of iteration steps reach the maximum value allowed per episode that is set at 300. The training for both controllers consisted in performing 1 000 000 iterations in a environment with a varying wind field (as described in Section 3.1). To help the agent, d reached is reduced during training as follows: at first d reached = 3m, from iteration 250 000th we set d reached = 2m and from iteration 500 000th we set d reached = 1m. The Py-Torch framework [50] was used to carry out the numerical experiments, along with the CUDA toolkit [51] and an RTX 2060 GPU card, allowing us to perform the training of one controller in approximately 10 hours. It can be seen in Figure 5 that during training, both the learning-based (LB) and model-free (MF) controllers were able to reach a high success rate under unknown wind gust disturbances, which shows the applicability of the SAC DRL procedure for this type of aerial navigation problems. The LB strategy presents a much higher convergence speed to a significant success rate than the MF strategy, which shows the great potential of combining model-based classical controllers with learning procedures. This can be explained by the model-based part of the LB controller which allows it to choose relatively good actions despite being at the early stage of the training session. Therefore, from the beginning of the training the LB controller is able to explore a much higher part of the reward space than the one of the MF controller. We believe this significantly helps the DRL algorithm to find better overall strategies. The evaluation consisted in performing the target rallying mission in areas of the same environment that had never been explored by neither controllers during training (i.e. the wind field in these areas were totally unknown to the neural networks) with d reached = 1m. The evaluation is composed of a total of 500 episodes, different from each other in terms of target position and with a max step size per episode of 1000. The targets during evaluation were uniformly distributed in the space defined by Λ = [Λ x ; Λ y ; Λ z ] T with [Λ x ; Λ y ] T ∈ [-50; -20[∪]20; 50] and Λ z ∈ [2; 20]. The same set of evaluation episodes was used for each controller. We also evaluated a fixed control strategy which consisted in a PID controller with the fixed nominal poles configuration (4.4). The outcomes of this evaluation are provided in Table 1. We can see that the LB controller achieves a better performance for each metric. Noted that it requires, on average, less actions to achieve the task with the LB controller despite sharing the same sampling rate. The mean reward per step of the LB controller is more than 2 times higher than the MF one. On the other hand, the fixed control strategy is showing a much lower success rate, close to 50%. We observed that with this strategy, failures mostly consisted in cases where the MAV is close to the steady-state but is being deviated by the wind gust. The vehicle is then not able to recover with fixed-pole parametrization. It shows the benefits of adaptive control methods when facing unknown disturbances. Note that with SAC, the policy is trained to maximize also the entropy, therefore the mean action does not always correspond to the optimal action for the maximum return objective.

Conclusions

We present a novel application of the SAC procedure for learning both model-free and learning-based adaptive controllers applied to the autonomous navigation of a MAV under wind gust conditions. These two strategies have been trained and compared in the same ROS-Gazebo reference simulation. Both strategies were able to reach high levels of performances under unknown uncertainty conditions, but the learning-based scheme, with a judicious parametrization of its modelbased control part, exhibited a much faster convergence rate. Our results suggest that learning-based adaptive methods can be much more efficient than modelfree ones and allow a better stability analysis in the DRL scheme.

INTRODUCTION

Operating in a constantly perturbed marine environment, autonomous underwater vehicles (AUVs) must compensate for wave and current induced forces acting on their body. In this context, a common practice is to employ a PID type control law with fixed parameters that are typically obtained using model-based optimization theory.

Although PID controllers can be made to work reasonably well under static known environment conditions, the performance of the fixed parameter controller may diminish under adverse conditions when exposed to large dynamic variations in the process. Performance of PID regulators can however, be improved considerably by accommodating such changes using online PID regulator tuning techniques, such as adaptive control theory Nevertheless, the stability analysis remain difficult to derive in the ANN framework. There is no theoretical guarantee of performance and ANNs tends to be over parameterized which results in poor performance under real conditions. The contribution of this work can be summarized in two parts: 1.) we propose a learning-based tuning procedure for the adaptive Pole-Placement of a PID controller which ensures stable poles selection despite using ANNs, and 2.) we modified and extended the method that we originally designed for an aerial vehicle [Chaffre et al. (2021)] to the underwater environment and show the ease of applying it to a new class of process.

This paper is organized as follows: Section 2 reviews related work in DRL-based adaptive control for AUVs, Section 3 presents the simulated robotic platform and environment used, and in Section 4 a complete description of our approach is provided. The DRL-based tuning strategy is detailed in Section 5. Results obtained from testing the controller within a realistic simulation environment are provided in Section 6 to demonstrate the effectiveness of our proposed learning-based adaptive controller. In Section 7, the paper concludes with a discussion of the performance of the proposed approach.

RELATED WORK

In the underwater domain, process variation can occur in many forms and qualities [Fossen (2011)]. Among others, currents are often the dominants forces acting on an AUV. They are difficult to counter because they are caused by multiple phenomena mixing together such as: tidal movement, surface wind, heat exchange at the sea surface, salinity changes or density variation in higher depths. It is commonly assumed that sea currents vary slowly with time and therefore they are usually represented as a static and uniform force. This way, they can theoretically be compensated for by including an integral action.

Nevertheless, in practice the frequency of the current induced forces vary much more than assumed because they are also influenced by the velocity and heading of the vehicle. In addition, the motors and thrusters efficiency can also vary significantly during operation. It is therefore almost impossible to re-tune the control parameters using linear feedback control methods. The field of adaptive control theory has emerged as a solution to tackle this type of process. This broad family of techniques aims at allowing the control law to hold some flexibility by means of online parameters refinement. It has been used widely in the underwater context where the principal sources of disturbances (i.e. wind, waves and current) are mostly not observable. Recently, more concerted effort has been put in the design of adaptive controllers using DRL in order to adapt to a wide range of operating conditions.

In [START_REF] Carlucho | Auv position tracking control using end-to-end deep reinforcement learning[END_REF]], the Deep Deterministic Policy Gradient (DDPG) algorithm developed by [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF]] was used to teach an AUV how to perform a waypoint rallying mission. They took as inputs the position of the vehicle, its positioning error, its velocity and the last actions performed. The DDPG algorithm was then trained to directly map the thruster commands to these input variables in order to reach a waypoint. They designed a reward signal that is a function of the error of the distance to the target point to train the agent. Simulation results proved that the controller is able to complete the task even when facing unexpected fault scenarios (i.e. not seen during training). Indeed, the agent was able to complete the task even when facing a thrust reduction of 90% (despite undershooting the goal before reaching it in this case). These results showed the adaptive abilities of ANNbased adaptive control strategies.

The DDPG algorithm was later used in [Wang et al. (2018)] to perform adaptive trajectory planning for multiple AUVs with the added objective of satisfying constraints related to kinematics, communication range and sensing area. To do so, they designed a specific cost function and modified their Experience Replay approach. The first included terms that transcribe the field uncertainty, the cost of the trajectory and the constraints induced by the trajectory. In addition, they decided to store past experience of the agent (using the Experience Replay technique) that specifically satisfy the communication range and sensing area constraints in distinct replay buffers. The gradient updates are then applied to those transition that satisfy both requirements resulting in a policy that chooses actions that comply with the constraints. Simulation results showed that their approach is able to achieve a performance matching that of a benchmark method that assumes perfect knowledge of the field hyper-parameters.

From another design point of view, the DDPG algorithm was more recently combined with a Proportional-Derivative (PD) controller in [Knudsen et al. (2019)] for the station keeping of a Remotely Operated underwater Vehicle (ROV). Such controllers are denoted as learningbased adaptive controllers and can be seen as a mix of model-based and model-free control schemes. The hybrid controller was evaluated through simulations and real life experiments. The results proved that the hybrid controller is able to compensate for unknown external disturbances thanks to the DRL tuning while ensuring cautious control of the known part of the process with the PD controller.

As mentioned above, DRL techniques are increasingly being used in control and efforts are being made to combine them with classical control scheme, taking advantage of both model-based and model-free theory. Following this idea, we present a hybrid adaptive control strategy for the control of an AUV in the presence of unknown sea currents. In contrast to [Knudsen et al. (2019)], it is based on a PID controller. The gains of the PID controller can take a wide variety of values and their bounds are not trivial to choose for tuning. One could need many disparate values of gains in order to be overly conservative to satisfy many operating points. Some terms might also need to be removed at some point for safety or power conservation needs. Instead of using a DRL algorithm to directly explore the space of gains, our objective is to express them in a different space, as large as possible, where bounds and control performance criterion are easy to derive. For this purpose, the PID controller has been re-parametrized using a Pole-Placement strategy. It allows us to convert the gains in the poles domain. The poles are then tuned in this space by using the maximum entropy DRL algorithm called Soft Actor-Critic [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]]. The resulting poles are then transformed back into the space of gains and the PID control law is finally applied as illustrated in Figure 1.

AUV SIMULATION AND MODELING

ROS packages

Because DRL algorithms are sample inefficient and their initial behavior can be unpredictable and possibly risky for robotic platforms, it is mandatory to perform their training in realistic simulators. If realistic enough, the simulator can theoretically provide an infinite amount of training data and the reality gap can be reduced with Domain Randomization [START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF], [START_REF] Sadeghi | CAD 2 RL: Real singleimage flight without a single real image[END_REF]]. For this purpose, we used the ROS [Quigley (2009)] package called UUV Simulator [Manhães et al. (2016)], to train our controller. This package provides a Gazebo-based simulation of underwater environments and the possibility to use the existing RexROV2 platform [Berg (2012)] as a test vehicle. In addition, it is also possible to simulate several disturbances including sea currents. They are simulated as an uniform force in the Gazebo world, represented by a linear velocity v c (in m.s -1 ), an horizontal h c and vertical angle j c (in radians).

AUV model

A ROV platform can be modelled using the general equations of motion for a marine craft, which can be written in the vectorial form according to [Fossen (1994)] as:

η = J Θ (η)ν M ν + C(ν)ν + D(ν)ν + g(η) = δ + δ cable (1)
where η and ν are the position and velocity vectors respectively, δ is the control force vector and δ cable is the vector describing the umbilical forces from the cable attached to the ROV. The RexROV2 is a ROV-type platform provided in UUV Simulator. It is propelled by 6 thrusters (complete details on its equation of motions are provided in [Berg (2012), McCue (2016), Yang et al.

(2015)]). The control vector u is obtained by transforming the actuator force vector:

δ = T(α)Ku (2)
where T(α) ∈ R n×r is the thrust allocation matrix ; K is the thrust coefficient matrix ; δ is the control force vector in nDOF and u ∈ R r is the actuator input vector. The package [Manhães et al. (2016)] allows us to setup a vector of thruster contribution for every DOFs (for clarity, sin(•) and cos(•) are denoted as s• and c• below):

T i =        Surge Sway Heave Roll Pitch Yaw        =        cθcφ sθcφ sφ -Zsθ + Y sφ -Zcθ + Xsφ -Y cθ + Xsθ        (3)
These vectors are then assembled into a thrust allocation matrix T = [T 1 , . . . , T 6 ] which decribes the relationship between propeller thrust and the vehicle speed (its calculation is available in [Carlton (2018)]). Using this allocation matrix, the control inputs u are transformed as u = [v x ; v y ; v z ; ω ψ ; ω θ ; ω φ ] T where [v x ; v y ; v z ] T are linear velocity inputs (in m.s -1 ) and [ω ψ ; ω θ ; ω φ ] T are torque inputs (in radians) expressed in the reference frame attached to the center of mass of the simulated RexROV2 platform.

A PID CONTROLLER WITH DIRECT POLE-PLACEMENT

The control objective treated here is to control the AUV at a fixed velocity and orientation Λ ref with Λref = 0. We define the error as e = Λ ref -Λ. We take into account the wave-generated disturbance by considering the steady-state error variable σ = t 0 e(τ )dτ . The proposed control strategy is based on a classical PID applied on each axis. Unfortunately, the gain tuning is not easy and a reinforcement learning method is used to do it. In order to accelerate and make it easy to interpret, the PID controller is seen as a double integrator with a state-feedback. This gain mapping is then easier to tune as shown in Section 5.

State space system

The augmented model with the state vector X = [σ, e, Λ] becomes

d dt σ e Λ = 0 1 0 0 0 1 0 0 0 A σ e Λ + 0 0 1 B u (4)
The PID control law is defined as

u = k p e + k i σ + k d Λ (5)
where max(k i σ) = 2000 for anti-windup compensation and u(t) ∈ [-2000; +2000]. We consider the control objective to be achieved if the value of each errors e i on the setpoint Λ refi stays around a predefined percentage χ of the desired value over a predefined time period of ς steps.

Pole-Placement strategy

The PID state-space representation is given by

Ẋ = (A -BK)X (6)
Among the various possible procedures that can be used to determine the gain K, a fundamental technique consists of assigning a set of specific values, P = {λ 1 λ 2 . . . λ n }, to the eigenvalues of the feedback loop A -BK. Given that these eigenvalues determine the poles of all the transmittance where the associated state matrices are involved, this procedure is denoted as Pole-Placement. We can define a (normalized) Control Polynomial as

C(s) = s n + c 1 s n-1 + • • • + c n-1 s + c n (7)
whose roots are the λ i and assign it as the characteristic polynomial of A -BK with

C(s) = det(sI -(A -BK)) (8)
Equations ( 6) and ( 8) yield

|A -BK -λI| = -λ 1 0 0 -λ 0 -k i -k p -kdλ = -λ -λ 1 -k p -kd -λ - 0 1 -k i -kd -λ (9)
This can be rewritten as

|A -BK -λI| = -λ(λ(k d + λ) + k p ) -k i = -λ 3 -λ 2 k d -λk p -k i = 0 (10)
This is true if and only if

λ 3 + λ 2 k d + λk p + k i = 0 (11)
In order to maintain controller stability, the poles of ( 11) must be placed in the left half-plane (i.e. A -BK is Hurwitz) which ensure convergence of the controller to steady-state without oscillations. For this purpose, the poles of the controller ( 5) must be solutions of the equation (11). The desired poles τ 1 > 0, τ 2 > 0, τ 3 > 0 are then defined as

λ 1 = -1 τ 1 ; λ 2 = -1 τ 2 ; λ 3 = -1 τ 3 (12)
The Pole-Placement design can be written as

       -1 τ 3 1 + k d τ 2 1 - kp τ1 + k i = 0 -1 τ 3 2 + k d τ 2 2 - kp τ2 + k i = 0 -1 τ 3 3 + k d τ 2 3 - kp τ3 + k i = 0 (13)
Since τ 1 , τ 2 and τ 3 are solutions to (11), it follows that

   1 -1 τ1 1 τ 2 1 1 -1 τ2 1 τ 2 2 1 -1 τ3 1 τ 2 3    k i k p k d =    1 τ 3 1 1 τ 3 2 1 τ 3 3    ⇔ M K T = N (14)
Finally, the gains of the control law (5) are obtained with

K T = M -1 N as k i = 1 τ 1 τ 2 τ 3 k p = τ 1 + τ 2 + τ 3 τ 1 τ 2 τ 3 k d = τ 1 τ 2 + τ 1 τ 3 + τ 2 τ 3 τ 1 τ 2 τ 3 (15)
This mapping allows optimisation of the exploration space.

Choice of closed-loop poles

The performance of a Pole-Placement controller is directly related to its poles location which make their tuning a critical task. There exist a set of fundamental design rules to guide this choice [START_REF] Chilali | H ∞ design with pole placement constraints: an lmi approach[END_REF]] but in general one should not choose closed loop poles that are highly negative, making the system fast responding (in the frequency domain) which leads to large bandwidth and thus amplification of noise.

In our case, with the design (12), for any value of τ i > 0, the poles of the controller are placed in the left half-plane which ensure its stability. This leaves us with the settling time requirement to define in order to bound the value of the poles. In accordance with the control objective (4.1), we define the desired settling time of the close-loop control ς = 10 seconds as the time after which we want the system outputs to stay within χ = 2.5% around its desired values. The upper bound of the poles is derived as

λ max = ln (χ) ς = ln (0.025) 10 = -0.368 (16)
Therefore λ i λ max (12), from which we can derive the upper bound of the poles

τ min < τ i 1 -λ max = 2.710 = τ max (17)
We choose τ min = 0.1 because for lower values the control inputs are too expensive in terms of control efforts and too aggressive for our control objective. Thus, the bounds of the poles are defined as

0.1 ≤ τ i ≤ 2.710 (18)
There exist a solution for all C(s) (7) if and only if the pair (A,B) is controllable, which is assumed here. In the case of a Single-Input system, the solution is unique. In the case of Multi-Input systems as studied here, the number of free component of the matrix K is greater than the n eigenvalue constraints. Accordingly, there exist an infinite number of solutions, among which it is not trivial to define an Optimal solution. In fact, depending on the process dynamics, we might favor one particular configuration of pole locations in the space (18) over another. The Linear Quadratic (LQ) optimization scheme then composes an alternative framework to define optimal values.

In contrast, we propose to use DRL to adapt these parameters for the control of an AUV, which consist in searching for the best values possible within (18) based on the process measurements. More precisely, our approach consist in using a Deep Policy Gradient method [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]] whose objective is to explore the space of poles (18) in order to find at each time step the best values possible (in this space) for each control input. The policy objective is to adjust them based on the process variation and with respect to a reward function which emphasizes the control objective. This is denoted as a Direct method because the controller parameters are adjusted directly without the need for an estimation of any process parameters. With this mapping (15), the learning action space is limited to desired solutions in the poles space. This ensure that for any pole values estimated by the ANN, the resulting control law will maintain the poles of the controller in the left half-plane.

This adaption is performed in closed loop and therefore exploits an extensive state representation of the process. We believe that this is highly beneficial to the parameters adaptation which is hence able to capture a large variety of appropriate poles. In the well known original Direct Adaptive Pole-Placement (DAPP) suggested by H. Elliott [START_REF] Elliott | Direct adaptive pole placement with application to nonminimum phase systems[END_REF], [START_REF] Elliott | Arbitrary adaptive pole placement for linear multivariable systems[END_REF]], there is neither an explicit reference model nor an error beteen the reference model output and system. The desired dynamical behavior is used implicitly in the process of indirect identification. In our version of DAPP, there is no parameters identification and the desired behavior is solely characterize by a reward signal, making its design a critical task.

REINFORCEMENT LEARNING FRAMEWORK

Reinforcement Learning (RL) is a subclass of Machine Learning methods where a learning agent that is evolving in an environment has to learn how to take optimal actions. The agent interacts with its environment by executing at each time step action a t ∈ A from state s t ∈ S which makes it transit to a new state s t+1 ∈ S. This transition produces a scalar value r t ∈ R known as the reward. The sequence of T actions and T + 1 states in the environment may be framed as a Markov Decision Process [Howard (1960)] along a trajectory ζ = (s 0 , a 0 , s 1 , a 1 , . . . , a T -1 , s T ). This stochastic process is completely defined by p(ζ) involving the initial probability s 0 ∼ p(s 0 ), the policy π µ : S → A parameterized by µ and the state transition probability p(s t+1 |s t , a t ). The optimization problem historically considered in RL is to estimate the policy π ⋆ which maximizes the expected return while following this policy:

π ⋆ = arg max J RL (π µ ) (19)
where J RL (π µ ) denotes the expected return along the trajectory ζ as:

J RL (π µ ) = E ζ∼p(ζ) T -1 t=0 r t = T t=1 E (st,at)∼ρπ µ r(s t , a t ) (20

Soft Actor-Critic

The learning objective is here to estimate a policy π µ that directly maps the pole locations from the AUV state in order to reach the control objective:

π µ : s t → [ τ vx ; τ vy ; τ vz ; τ roll ; τ pitch ; τ yaw ] (21)
where dim(τ i ) = 3, thus the dimension of the action space is 18. These poles candidates are then applied to compute the gains (15) that are used to derive the control law (5) for each control inputs. For this purpose, we used the algorithm named Soft Actor-Critic. It is denoted as a DRL method because the parameter µ is estimated by an ANN. Contrary to the classic DRL optimization objective (19), the Soft Actor-Critic aims at maximizing the expected return as well as the entropy of the policy:

J SAC (π µ )= T t=1 E (st,at)∼ρπ µ [r(s t , a t ) + αH(π µ (.|s t ))] (22)
where

H(π µ (.|s)) = - a∈A π µ (a) log π µ (a|s) (23)
The term H(π µ ) is the entropy measure of policy π µ and α is a fixed temperature parameter that determines the relative importance of the entropy term (i.e. the trade-off between reward maximization and entropy maximization). By doing this, the policy is forced to explore sub-optimal solutions (i.e. actions associated to similar high Q-Values) until it identifies which ones are really better for the long term maximization objective. This has been proven to greatly improve the sampling efficiency of off-policy DRL techniques [START_REF] Haarnoja | Reinforcement learning with deep energy-based policies[END_REF]]. The SAC method includes estimation of the Q-Value and State-Value functions by ANNs whose parameters w and Ψ are respectively updated to minimize the errors [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]]:

J Q (w) =E (st,at)∼D 1 2 Q πµ w (s t , a t ) -r(s t , a t ) + γE st+1∼ρπ(s) [V πµ Ψ (s t+1 )] 2 (24)
J V (Ψ) = E st∼D 1 2 (V Ψ (s t ) -E[Q πµ w (s t , a t ) -log π µ (a t , s t )]) 2 (25)
where w and Ψ respectively, denote the parameters of the Q-Value and State-Value networks. The ANN estimated parameters µ of the policy function are updated in order to minimize the expected Kullback-Leibler divergence:

J π (µ) = E st∼D D KL π µ (•|s t ) exp(Q πµ w (s t , •)) Z πµ w (s t ) (26)

State vector

At each time step, the agent captures an observation vector of the process o t as described below:

o t = [ a t-1 ; O ; V ; Ω ; u i ; e i ; e L2 ] (27)
where a t-1 ∈R 18 are the estimated pole values (21); the Euler orientation of the vehicle are O = [φ, θ, ψ]; its linear and angular velocities are respectively V = [v x , v y , v z ] and Ω = [ω φ , ω θ , ω ψ ]; the vector u t ∈ R 6 is composed of the current PID controller outputs, e i ∈ R 6 are the errors on each setpoint and e L2 is the Euclidean distance to the steady-state as in (30). Due to latencies and uncertainties in the process, the dynamics can become non-Makorvian which significantly degrade the learning performance. For this reason, we construct the state vector out of the current and past 4 observation vectors, thus dim(s t ) = 200.

Artificial neural network architecture

Five multilayer perceptron networks are trained in order to minimize the above loss functions ( 24)-(26

) To remove dependency on the initial weights, we use a random policy for the first 50, 000 time steps. Then, we add to each action a Gaussian noise N(0, 01).

Mini-batch gradient descent is applied at each time step with a mini-batch size of 256 past transitions, randomly sampled from a replay buffer of maximum size 10 6 and with the CER technique [START_REF] Zhang | A deeper look at experience replay[END_REF]].

Reward function

In accordance with the control objective, the following reward function is introduced:

r(s t ) = r succeed if ∀t ∈ [t-100; t], |e i (t)| ≤ χ, r regulation , otherwise. (28)
The reward signal r regulation is a binary reward signal defined as:

r regulation = 20 × e -(e L 2 ) 2 × (1 + dt ei(t) ) if dt ei(t) > 0, -20, otherwise. (29)
where e L2 is the Euclidean distance to the steady-state:

e L2 = i=dim(u) i=1 e 2 i (t) (30)
and dt ei(t) is the derivative of the error e i computed over two time steps such as:

dt ei(t) = e 2 i (t -1) -e 2 i (t) (31)
This reward function highly recompenses error decrease and penalizes any deviation from the Euclidean path to the steady-state. If the control objective is met, the reward r succeed = 500 is generated which ends the current episode.

Training scenario

The training consisted in performing a total of 1 million time step iterations. Each episode has a maximum length of 300 time steps (equivalent to 20 seconds). This parameter also required tuning because depending on the problem, an overflow of steps can highly degrade and slow the learning, while too few steps often cause the policy to over-fit. A training episode is defined as follow:

(1) At the beginning of the episode the AUV is initialized at the position (x 0 , y 0 , z 0 ) = (0, 0, -40) with null velocity and a random orientation

(ψ 0 , θ 0 , φ 0 ) ∈ [ -π 4 ; π 4 ]. (

Evaluation scenarios

We defined three evaluation scenarios based on the characteristics of the process:

• Scenario 1: the setpoints are fixed during the whole episode while the sea current variables vary. • Scenario 2: the sea current variables are fixed during the whole episode while the setpoints vary. • Scenario 3: both sea current variables and setpoints vary during the episode.

When varying, these variables have the form: for [h c , j c , v c ] randomly chosen. The evaluation consists in performing 500 episodes for each scenario with a maximum step size per episode of 500 and different from each other in terms the above mentioned variables.

[v x , h c , j c , v c ] = D 1 × sin(D 2 × t) + D 3 (32)

SIMULATION RESULTS

Evaluation outcomes

As a benchmark, we used the optimal PID controller provided in the UUV Simulator to show the benefits of our approach. It is a 6-DoF PID controller whose parameters have been optimized using SMAC [Hutter et 2019). These methods are used to iteratively build an estimate of an optimal policy function, mapping the process states to the control parameters. The DRL-based methods oblige to frame the control process as a Markov Decision Process (MDP). The associated Bellman's equations are then estimated using ANNs, usually involving Gradient-based learning. In contrast, we proposed in this paper to use CEM to optimize the ANN, removing the need for the MDP formulation.

AUV MODELING AND SIMULATION

AUV model

A mathematical model of a ROV platform can be derived using the general equations of motion for a marine craft, which can be written in the vectorial form according to Fossen (2011) as:

η = J Θ (η)ν M ν + C(ν)ν + D(ν)ν + g(η) = δ + δ cable (1)
where η and ν are the position and velocity vectors respectively, δ is the control force vector and δ cable is the vector describing the umbilical forces from the cable attached to the ROV. The control vector u is obtained by transforming the actuator force vector: δ = T (α)Ku, where T (α) ∈ R n×r is the thrust allocation matrix; K is the thrust coefficient matrix; δ is the control force vector in n degrees of freedom (DoF) and u ∈ R r is the actuator input vector. The package Manhães et al. (2016) allows us to design a vector of thruster contribution for every DoFs.

ROS packages

We used the ROS-based package called UUV Simulator proposed in Manhães et al. (2016), to train our controller. This package provides a Gazebo-based simulation of underwater environments and the possibility to use the existing RexROV2 platform described in Berg (2012) as a test vehicle. In addition, it is also possible to simulate several disturbances including sea currents. They are simulated as a uniform force in the Gazebo world and represented by a linear velocity v c (in m.s -1 ), a horizontal h c , and a vertical angle j c (in radians).

OUR LEARNING-BASED ADAPTIVE CONTROL FRAMEWORK

In this paper, we address the problem of setpoint regulation where the UUV's objective is to minimize its tracking error in all 6 DoFs. The tracking error e is deőned as the error between, the current UUV's position and orientation, and a őxed desired position and orientation as:

e = η d -η. (2)
The position and orientation of the UUV is:

η = (x, y, z, roll, pitch, yaw), (3)
and the desired setpoint:

η d = (0, 0, 0, 0, 0, 0). (4)
In our LB adaptive control framework, a parametrization such as an artiőcial neural network, of the PID's gains is learned (in simulation or in the real world). In the LB control framework's testing phase, at every state measurement, a control step is performed. In our case, during a control step, the state of the control process is used as input of a trained neural network. The outputs of the network parametrize the PID gains (see section 5) used to compute the control input: the UUV's thrusters. The PID control law is deőned as the following:

u = B -1 [ J T (η) k p e + k i t 0 e(τ )dτ -k d η + g(η) ], (5)
where u ∈ R 6 are the control inputs; the PID gain matrices k p , k i , k d ∈ R 6×6 are the output of a neural network; η ∈ R 6 is the current UUV's state, i.e. position and orientation (Eq. 3); η ∈ R 6 is the temporal derivative of the state vector; g(η) is the sum of external forces acting on the UUV's body (in our case, gravity); e ∈ R 6 is the tracking error (Eq. 2); B -1 is the őxed known thrusters allocation matrix mapping the control input u into a combination of thruster power inputs resulting to the desired movement and J T (η) is a transformation matrix.

The principal advantage of the considered PID regulator (Eq. 5) is that global stability and convergence analysis of the system has been well formalized by Fossen Fossen (1994). Lyapunov stability theory Liberzon (2005) is straightforward to apply with such a control law. Following Fossen (1994), there exists a Lyapunov function for the considered UUV such that:

V (x) = 1 2 x T   M -1 η αI 0 αI k p k i 0 K i αk i   x, (6)
where α ∈ R is a small positive constant and the PID gains are k p , k i , k d ∈ R 6×6 ; M -1 η is related to the UUV's mass and can be computed from the current η; x is the control loop's state (not to be mistaken with the UUV's state):

x = p, η, t 0 e(τ )dτ T ∈ R 18 and p = M η ηT ∈ R 6 is the generalized momentum depending on the UUV's mass and velocity. The control loop is stable at state x if:

V (x) > 0 and V (x) < 0. (7)
Following Lyapunov stability theory Liberzon (2005), there exists theoretical constraints on the gain matrices k p , k i , k d and the small constant α such that local stability is guaranteed when the initial conditions of the systems are closed to x = 0. According to the proposed Lyapunov function (Eq. 6), the vehicle stability and convergence to steady-state are guaranteed (Eq. 7) if the following constraints are satisőed:

           k d > M η , k i > 0, k p > k d + 2 α k i , 1 2 (1 -α)k d -αM η + α 2 6 i=1 (η i -η id ) ∂Mη ∂ηi > 0, α > 0, (8)
Preprints, IFAC CAMS 2022 Kgs. Lyngby, Denmark. September [14][15][16]2022 We found that, when limiting the parameter space to value satisfying (Eq. 8), the resulting space is so small that the beneőts of adaptive control are merely preserved. Therefore, we propose not to take into account the stability constraints in the parameters optimization. Our objective is then to assess to what extent the resulting solution, obtained from an ANN, can still hold some stability components. In order to facilitate the stability analysis, we aim at reducing the dimension of the space depicted in Eq. ( 8). First, we can transform (8) into equalities as follows:

       k d = M η + M 1 , k i = 0 + M 2 , k p = k d + 2 α k i + M 3 , α = ∥ -k d (-k d -2Mη+ 6 i=1 (ηi-η id ) ∂Mη ∂η i ) ∥ max + ϵ, (9)
where M 1 , M 2 and M 3 are three 6 × 6 positive matrices and ϵ is a small positive constant. With this transformation ( 9), the fulőllment of the Lyapunov stability constraints (8) can now be veriőed by only assessing the value of [M 1 , M 2 , M 3 , ϵ] ∈ R 3×6×6+1,>0 . In order to further reduce this dimension space, we apply a diagonal transformation on the matrices M i : M i = P Λ i P -1 , where Λ i ∈ R 6 are positive vectors and P ∈ R 6×6 is a positive invertible matrix chosen randomly beforehand. Thanks to this transformation, we can now assess the value of [M 1 , M 2 , M 3 , ϵ] ∈ R 3×6×6+1,>0 (and K p , K i , K d with Eq.9) by only accessing:

[Λ 1 , Λ 2 , Λ 3 , ϵ] ∈ R 19,>0 . (10)

PID TUNING USING DEEP LEARNING

In order to take into account the process uncertainties, we propose to learn a stochastic predictive model π ω (parameterized by ω) that maps the system state vector x into the controller parameters of the PID law:

     π : Ω x ⊂ R 18 → Θ ⊂ R 109 x = p, η, t 0 e(τ )dτ T → [k p , k i , k d , α] (11)
This mapping is composed of two stages. The őrst stage is a trainable neural network whose input is the state vector x, aiming at predicting the vector [Λ 1 , Λ 2 , Λ 3 , ϵ] ∈ R 19 . To this end, we model each variable n ∈ {1, . . . , 19} of this vector as a random variable with an independent Normal distribution as:

N n (µ n , σ 2 n ) = (2πσ 2 n ) -1/2 exp{- 1 2σ 2 n (x -µ n ) 2 }, (12)
The goal of this network is to output the 19 pairs of (µ, σ) representing the action distributions. We have empirically set up an architecture composed of 2 hidden layers of 32 hidden nodes each, and with the Sigmoid activation function applied to each layer. This results in a total of (18 + 1) × 32 + (32 + 1) × 32 + 2 × ((32 + 1) × 19) = 2918 parameters (ω) to learn from data. In a second stage of the mapping π, the PID parameters [Λ 1 , Λ 2 , Λ 3 , ϵ] are obtained by sampling from the resulting Gaussian distributions N(µ, σ). The Eq. 9 allows to compute the őnal PID parameters and using Eq. 5 the PID control inputs are derived. With this setting, we can more easily assess the Lyapunov stability of the system. The overall control strategy is illustrated in Figure 1.

LEARNING WITH THE CROSS ENTROPY METHOD

We propose to learn/optimize the weights of the neural network with the CEM. The CEM is a direct search method (i.e no gradient is computed). It is an Estimation of Distribution Algorithm (EDA) inspired by Natural Evolution Strategies [START_REF] Wierstra | Natural evolution strategies[END_REF]. With CEM, during one iteration k, N sets of weights S k i=1...N are sampled from a Normal distribution directly in the space of weights R 2918 . At each iteration k, N evaluations are made to determine the current best weights S k best=1...N ×ρ with respect to a given cost function. In our case, we used the following classic control performance (based on multi steps within an episode and connected to the tracking error (Eq.2) index as a cost function to minimize:

J = steps 1 6 6 i=1 (η d,i -η i ) 2 . ( 13
)
The mean and covariance of the Normal distribution are then updated as the mean and the covariance of the N × ρ best sets of weights obtained at iteration k-1. Noise σ 2 noise is added to the covariance of the best weights to avoid local optima. We randomly sample the next iteration weights S k = N(mean(S k-1 best ), Cov(S k-1 best ) + σ 2 noise ).

EXPERIMENTS AND ANALYSIS

Experimental settings

Simulation: We control a simulated UUV in Gazebo as described in Section 3. The simulation is running in realtime and the frequency of the control loop is 20 Hz. The outputs of the ANNs, are used to compute the PID control law that is ultimately applied to the 6 thrusters. The complete training loop of the ANN is given in Figure 1. Preprints, IFAC CAMS 2022 Kgs. Lyngby, Denmark. September [14][15][16]2022 Stability metrics:

Thanks to the proposed Lyapunov function (6) and mapping (10), we can directly assess the state and parameters stability. We propose to measure the controller's stability as the respective percentage of steps satisfying the state and parameters constraints.

Evaluation setup: We propose three evaluation scenarios based on induced disturbance: none, Gaussian noise in sensor measurements, and Gaussian noise in control inputs with current disturbances. The length of the episode is now increased to 2000 timesteps. The initial state of the UUV is changed at the beginning of each episode. We compare our LB adaptive controller to its Lyapunov-based counterpart which consist in a PID controller whose parameters are set to: M 1 , M 2 , M 3 = diag(0.5 -10 -5 , 0.5 -10 -5 , 0.5 -10 -5 , 0.5-10 -5 , 0.5-10 -5 , 0.5-10 -5 )+10 -5 ×J 6×6 (which satisőes the parameters stability constraints in Eq.( 9)). The resulting controller remains adaptive (as the gains are a function of the vehicle state x) and will be denoted as naive PID henceforth. The only difference between these controllers is the value of the parameters used to derive the PID law (5), making the comparison fair.

Control performance

The control performance is measured as the MSE on the setpoint. The evolution of this performance is illustrated in the following őgures: when facing no disturbance in őgure 2; when facing Gaussian noise on the vehicle position and orientation feedback in őgure 3 and when facing Gaussian noise on the control inputs and sea current disturbance in őgure 4. . Control performance with noisy control inputs and sea current disturbance. We can see that the naive PID (in blue) results overall in better performances in terms of setpoint tracking compared to the LB PID (in orange). Nevertheless, we can see that the relative performance of the LB PID with respect to the naive PID's performance improves with increased uncertainty. In Figures 3 and4, we can see that the performance of the LB PID matches the naive PID for approximately 500 timesteps, while without disturbance, its performance drops notably earlier as shown in őgure 2. The evolution of the vehicle's state is represented in Figures 5678910. We can observe a difference in regulation dynamics depending on the DoF. For instance, we can see that the depth of the UUV is not successfully regulated by none of the controllers. Due to the short latency between episodes and the position of the UUV's CoG, the UUV slightly sinks at the beginning of the episode, altering its depth and yaw angle (z, yaw). This explains the vertical drift in their associated errors observed in the őgures. In addition, as seen in Figures 5678910, the LB PID tends to regulate effectively more DoF compare to the naive PID (which fails at regulating (z, yaw), see Figures 5,7,9).

Stability performance

The evolution of the stability metrics is illustrated in Figures 11-16. We can see in Figure 11 that both controllers reach a similar level of state stability. The naive PID exceeds the stability performance of the LB PID in both state and parameters stability. We believe that this is thanks to the stochastic nature of the adaptation (see Eq.( 12)). Note that similar state stability does not guarantee similar control performances. Indeed, as mentioned before, both controllers reach the same level of state stability, however, we have seen that the MSE of the LB PID increases over time, so its control performance decreases but the control loop remains stable. Finally, we can see a mismatch between state and parameters stability: despite reaching a percentage of state stability as good as the naive PID, the gains obtained from the LB PID are not satisfying the constraints ( 8). This suggests that the Lyapunov function holds a limited conservatism. In other words, the Lyapunov-based space of parameters seems extremely small compared to the space of parameters obtained from the ANN. Future work could focus on incorporating the parameter constraints in the learning optimization scheme to analyze the difference in the resulting spaces. We proposed the use of CEM for PID tuning to adapt to process variation. Despite not considering stability components in the CEM procedure, the proposed LB PID is able to match the state stability obtained with the Lyapunovbased PID controller. Our result also suggests that the LB PID is better at compensating process variation. Learningbased adaptive control might be a key ingredient toward the safe deployment of fully autonomous UUVs. AbstractÐDeep Reinforcement Learning (DRL) methods are dominating the field of adaptive control where they are used to adapt the controller response to disturbances. Nevertheless, the usage of these methods on physical platforms is still limited due to their data inefficiency and the performance drop when facing unseen process variations. This is particularly perceived in the Autonomous Underwater Vehicles (AUVs) context as studied here, where the process observability is limited. To be effective, DRL-based AUV control systems require the use of methods that are data-efficient (in order to reach a satisfactory behavior with a sufficiently fast response time) and are resilient (to ensure robustness to severe changes in operating conditions). With this ambition, we study in this paper the effect of the Experience Replay (ER) mechanism on the performance variation of a DRL-based stochastic adaptive controller. We propose a new ER method (denoted as BIER) that takes inspiration from the biological Replay Mechanism and compare it to the standard method denoted as CER. We apply it to the Soft Actor-Critic, a maximum entropy DRL algorithm, for use with an AUV maneuvering task that consists in stabilizing the vehicle at a given velocity and pose. The training results show that our BIER method exceeds the performance of the nonadaptive optimal model-based counterpart of the controller in less than half the number of episodes compared to CER. We proposed different evaluation scenarios of increasing complexity as measured by desired velocity value and amplitude of current disturbance. Our results suggest that the BIER method achieves improved learning stability and better generalization abilities.

Index TermsÐDeep reinforcement learning (DRL), neural networks, machine learning (ML), adaptive control.

I. INTRODUCTION A. Adaptive Control

Autopilots for unmanned systems are usually designed based on the feedback provided from velocity and orientation sensors. In the case of autopilot systems for autonomous underwater vehicles (AUVs), the main objective in the design is to compensate for waves and current-induced disturbing forces acting on their body. Existing AUV autopilots are however only able to compensate for low-frequency components of seainduced disturbances. It seems natural to assume that the AUV performance could be improved by taking the nature of the disturbances into account in the design of the autopilot.

Adaptive control [1] provides what seems to be an ideal framework to this end. The objective of this technique is This work was supported by ISblue project, Interdisciplinary graduate school for the blue planet (ANR-17-EURE-0015) and co-funded by a grant from the French government under the program ºInvestissements d'Avenirº.

The first author's scholarship is jointly supported by the doctoral school of Bretagne-Loire University with a grant from the Brittany Region under the program ºMobilit e Internationaleº and by Flinders University with a grant provided by the South Australian Government.

to adjust automatically the control parameters when facing unknown or time-varying processes such that the desired performance threshold is met. Developed in the late 1950s, adaptive control frameworks have been considerably expanded and used in various fields, their application has been facilitated by the rapid progress in microelectronics and the increasing interaction between laboratories and companies, from aerospace to maritime industries. As a result, adaptive controllers started to be widely adopted in the industry in the early 1980s. It was established at that time that robust designs with fixed parameters are too limited to handle complex regimes. The study of adaptive controllers for AUV manoeuvring is associated with various challenges, including:

Unknown dynamics: the uncertainty associated with describing precisely the states of waves or currents is high. This, together with its dynamic nature, prevents linear feedback control methods from achieving optimal performance of the plant. This becomes more critical in the presence of changes in weather conditions that impose a multiplicative factor in the component of the induced forces. The disturbance period will also vary with the speed of the vehicle and its orientation relative to the waves.

Nonlinearity: the controller response at some operating points must be overly conservative in order to satisfy the specification at other operating points. This is difficult to achieve for fixed parameters obtained through local linearization, that do not encompass the entire regime envelope.

Thruster efficiency: a fully-actuated vehicle can often become underactuated as its forward speed increases. This is especially true for hovering AUVs which are designed to move at low speeds and steer using their thrusters only. As the forward speed increases, the efficiency of lateral motions is drastically reduced, making it impossible for the platform to account for pure lateral motions.

System reliability: if the performances of one or more thrusters become increasingly less effective, the control system should be able to detect this and engage a new control algorithm specially designed to accommodate the failures and, if possible, to complete the mission.

A class of adaptive control methods, known as learningbased adaptive controllers, have been developed to tackle some of these limitations. This family of solutions uses machine learning algorithms capable of compensating for the unknown part of a process while also maintaining optimal control of its known part using traditional methods.

B. Learning-based adaptive control

Real-world systems are in general nonlinear and their motion equations, parameters and system measurements are affected by uncertainty. A realistic scheme is to consider that the process model is partially available. In learningbased adaptive controllers, model-free algorithms are used to mitigate this lack of a complete description of the process by finding (learning) an approximate representation of the unknown parts, or by fitting (tuning) the best control parameters for a target behavior. The dynamics of such a system can be represented as the sum of known (f 1 ) and unknown (f 2 ) parts:

ẋ(t) = f 1 (t, x, u) + f 2 (t, x, p), y(t) = h(t, x, u), (1)
where classical model-based control methods can be used to efficiently control f 1 , and f 2 which can be approximated by model-free learning algorithms. In other words, learningbased control methods take advantage of the fast convergence and robustness to uncertainty of learning algorithms to approximate an unknown performance function, while applying model-based control laws to maintain some stability requirements of the system. More recently, in Neural-Network (NN) learning-based control design, the unknown part of the model can be estimated by a NN, whose weights are obtained using some model-free optimization procedure. Among the various techniques, a prominent candidate to that end is Deep Reinforcement Learning (DRL).

In DRL, the learning process happens through the interaction between an agent and the environment [2]. DRL is traditionally defined as a Markov Decision Process (MDP) expressed as the tuple ⟨S, A, T, R⟩, in which: S is the set of possible states; A is the set of actions that can be executed by the agent; T is the transition function that defines the probability of reaching a successor state s ′ ∈ S, from the application of an action a ∈ A in a state s ∈ S; R is the reward function.

The DRL general framework can be summarised as follows:

1) At an instant t, an action a ∈ A is chosen by the agent in a state s ∈ S; 2) The execution of this action leads the agent to a state s t+1 ∈ S and the agent receives a scalar value r t , the reward signal. This signal is a numerical representation of the action outcomes with respect to a reward function R(s). The goal of DRL is to maximize this reward function; 3) Finally, the agent updates the value of executing action a based on the received reward, according to the specific DRL algorithm applied.

In the domain of optimal control, the agent is identified with the controller, environment is the controlled system (or plant) and action is the control signal [2]. Among the various existing DRL algorithms [3], Deep Policy Gradient methods (that use gradient descent for the purpose of optimizing a decision making function, denoted as policy, with respect to the expected return) are deemed the most suitable method for handling robotic domains for the following reasons:

1) Domain dimensionality: the large size of the state-space in a real robot environment makes the application of value iteration based methods unfeasible;

2) Non-observable disturbances: any real environment, beyond toy scenarios, is partially observable. In this context, environment modelling and trajectory planning are extremely difficult tasks for any model-based or value-based DRL method, since unexpected states can corrupt the remaining part of the estimated trajectory, resulting in failures.

Deep Policy Gradient methods are essentially relying on the Actor-Critic architecture [4], where a value and a policy function are estimated simultaneously in order to improve the agent performance. This formulation has led to the development of what is known as off-policy methods where the estimates of these functions can be improved using the Experience Replay (ER) mechanism. ER is the state-of-the-art method for the automatic selection of past experience to improve an agent's future behavior, allowing the use of samples from various distribution. In contrast, with the on-policy formulation, only samples generated by the same policy can be considered for the optimization process.

The off-policy Deep Policy Gradient methods are increasingly being applied to AUVs adaptive control [5]± [8] where the standard ER method is used to improve DRL-based agents for the purpose of adapting to process variations. These works rely on deterministic policies that are highly stable during training but display lower adaptation abilities compared to their stochastic counterparts. Nevertheless, the performance of Deep Policy Gradient methods is highly sensitive to the distribution shift problem that is in the context of DRL, the difference between the training and evaluation set of states. A key element toward the reduction of this sensibility is the ER mechanism, and its effect on the resulting policy is the main focus of this paper.

This paper is organized as follows: in Section II the ER mechanism is introduced with its standard formulation in DRL and its biological counterpart. In Section III we present an application of learning-based adaptive control theory for the control of an AUV. The detailed architecture of the learningbased adaptation is provided in Section III-B. We present our proposed ER approach in Section IV. The training settings are provided in Section V along with the training results. The evaluation scenarios are described in Section VI with an analysis of the results, leading to some open questions and perspectives for future work.

II. EXPERIENCE REPLAY

The concept of ER [9] employs the agent's past experience to improve its current behavior. It aims to artificially make the agent's experience look Independent and Identically Distributed (IID). This is highly desirable in order to not concentrate the updates to a limited area of the desired functions.

Given that an agent's experience at the time step t is defined as the quadruplet e t = (s t , a t , r t , s t+1 ), the ER method consists of storing (at each time step) the experience e t in a memory unit D = {e 1 , . . . , e t } of fixed size, also known as the replay buffer. Then, the neural networks are trained by performing mini-batch gradient descent of past experiences randomly pooled over the replay buffer. The estimators are hence trained on IID samples that are generated by various trajectories and policies and they are, therefore, more representative of the true function. However, even with this first ER formulation, there is a great number of parameters that are usually ignored despite having an impact on the learning performance, including:

The replay buffer size: the total number of transitions that the replay buffer can store. When its maximum size is reached, the replay buffer is accessed in a first-in-first-out fashion. The bigger the replay buffer size, the more the data will look like it is IID, which in turn improves the gradient update quality. However, if the replay buffer is too big, an important transition will have much less chance of being used to update the policy, which could impair the learning process. In contrast, if the replay buffer is too small, the learned policy can be the result of an overfitting process on recent transitions, which precludes performance improvement.

The age of a transition: the number of gradient steps taken by the agent since the transition was generated. This value can be seen as a measure of the extent to which the transitions stored in the Replay Buffer are off-policy, as it tells us how different the current policies are from those stored in the buffer. The age of the oldest policy stored increases with respect to the buffer size.

The replay ratio: the number of gradient updates per environment transition. It can be viewed as a measure of the frequency at which the agent is learning using existing data versus learning from collecting new experiences.

The size of the replay buffers, however, can impact negatively on the learning performance [10]. There are two competing methods that can be used to solve this issue: the Combined ER (CER) [10] and the Prioritized ER (PER) [11]. CER consists of adding the latest transition performed to the mini-batch pooled over the replay buffer, whereas with PER important transitions, as measured by their associated TDerror (22) (23), are given a higher probability to be used in the gradient updates. Using CER, however, the last transition will undoubtedly be sampled and instantly affect the policy.

Nevertheless, even with CER, a drop in performance was observed for certain sizes of replay buffer, at some point of the training (even when tuning the learning rate). This behavior was related to the process itself rather than to the aforementioned parameters [10]. As written in [10] ªCER is a workaround ... and future effort should focus on developing a new principled algorithm to fully replace ER.º In this paper, we propose a new ER mechanism with the ambition to decouple the performance of the agent from process complexity (as observed with CER).

A recent detailed analysis of ER was provided in [12], where an analysis of the effects of the aforementioned parameters was presented. Several conclusions on how the parameters can affect the learning dynamics were drawn, which motivated the ER design proposed in this work. These conclusions can be summarised as follows:

• Increasing the replay capacity while fixing the age of the oldest policy improves the performance because it lowers the chances of overfitting to a small subset of (state,actions). • As the agent trains, it spends more time in higher quality regions of the environment (as measured by rewards), thus learning to better estimate the return in such regions leads to further gains in performance. • Increasing the buffer size with a fixed replay ratio has varying improvements. The replay ratio stays constant when the buffer size is increased because of both the replay capacity and the age of the oldest policy increase. If one of these two factors is independently modulated, the replay ratio will change. The design of the proposed approach for ER has been mostly motivated by these findings which we tried to incorporate in the ER scheme. A recent study [13] undertook a comprehensive comparison between the replay mechanism that takes place in biological brains and those in artificial learning systems. We list below some findings related to DRL only.

Replay (in biological systems) is temporally structured. Temporally correlated experience sequences are used for learning and memory combination. This allows for more combinations of neurons which leads to faster emergence of temporal waking experiences. This feature is largely ignored by existing methods that only replay static, uncorrelated inputs.

Replay is modulated by reward and only a few selected experiences are replayed. It seems intuitive that not all experiences are useful for learning a new task. Some experiences are more important than others because they incorporate higher quality information about the process dynamics. The challenge here is twofold: how to model this information quality and how to measure it.

Replay is treated differently for novel versus non-novel inputs. This allows for selective replay to be weighted by novelty. Biological systems tend to reduce drastically the attention given to old experiences versus that given to recent ones as the latter contain more information within them.

In this paper, we propose a new ER mechanism to include these biological insights while keeping in mind the constraints related to the regression problem.

III. A LEARNING-BASED ADAPTIVE CONTROLLER

In this paper, we address the control problem of AUV manoeuvring, which can be summarized as the stabilization of an underwater vehicle at a fixed velocity and orientation (x ref , with ẋref = 0). The state vector is hence defined as x = [x y z ϕ θ ψ] T . The vehicle is fully actuated but subject to external disturbances (sea currents) that are here not observable (noted that relative current can be estimated with some types of DVL incorporating a bottom lock). As defined in Section I-B, the process dynamics can be framed as the combination of its known f 1 and unknown f 2 parts. Our approach consists in using the knowledge of f 1 to design a model-based control structure, which is then combined with a model-free learning algorithm to compensate for f 2 . The resulting learning-based controller guides the controller parameters toward the optimal ones that best compensate for both f 1 and f 2 without prior estimation of some process parameters, leading to a Direct Adaptive Control scheme. Let the error between the present (x i ) and the desired (x refi ) state variable be defined as e i = x refixi .

The task of steering the AUV outputs in order to maintain the error signals within a specific threshold, over a predefined amount of time (guaranteeing the vehicle stabilization), can be achieved if the following control objective is met:

∀i ∈ R u , ∀ t ′ ∈ [t -ς, t], ∃! | e i (t ′ ) | > χ, (2)
where R u is the space of control inputs, the current time step is denoted as t and ς is the length of the period of time over which we want all the errors e i to be less than the desired value χ. This class of control objective is used in various AUV missions, such as autonomous docking or underwater inspection, where a conservative regulation of the vehicle's outputs is required.

We use the ROS-based UUV Simulator [14] which contains a small library of different vehicles of cubic and torpedo shapes. For this study, we used the RexROV2 platform, a cubic shape AUV 1 . The UUV Simulator can simulate several current and wave disturbances, thruster dynamics, and body wrench disturbances. When incorporated in simulations, the induced forces have a realistic physical impact on the robot and fluid dynamics. The sea current disturbance (which is the main focus of this study) is modeled as a uniform force acting over the Gazebo environment. This force is represented by a linear velocity, v c (in m.s -1 ), a horizontal h c and a vertical angle j c (measured in radians). The simulated RexROV2 platform is equipped with an IMU which feedbacks its linear velocities and orientation (Euler angles). These variables are accessible through ROS topics, which are essentially data pipelines to access the state of the simulated objects. Our software architecture consists in using the simulation metadata to train the learning algorithms considered in this work.

A. Design of the model-based part of the controller

We introduce now a simple learning-based adaptive controller formulation. We first proposed it in [15] where we demonstrated its superiority to its nonadaptive optimal modelbased counterpart (denoted in this manuscript as OFP for Optimal Fixed Poles controller) that we used here to compare our controllers. In particular, we show in [15] that only the learning-based adaptive controller (that we introduce now) was able to regulate the vehicle against time-varying disturbances. Moreover, we provide in this paper an enhanced model-based design that leads to better performance. This work assumes that the controlled vehicle is fully observable and controllable. This means that each of the vehicle's DoF is measurable and the desired vehicle states (within the operating regimes) are supposed to be accessible. In this context, a PID controller is a suitable method to regulate the process. We can take into 1 See our last study [15] for further information on the simulated vehicle.

The PID state-space representation is given by:

Ẋ = (A -BK)X. (4)
The PID control law can then be derived as:

u = k p e + k i σ + k d ẋ, (5)
with k p , k i and k d ∈ R + . Among the various procedures and rules that can be applied to tune the PID gains [16], a fundamental technique consists of assigning a set of specific values, P = {λ 1 λ 2 . . . λ n }, to the eigenvalues of the feedback loop A-BK. Given that these eigenvalues determine the poles of all the transmittances where the associated state matrices are involved, this procedure is denoted as Pole-Placement. We can define a (normalized) control polynomial as

C(s) = s n + c 1 s n-1 + • • • + c n-1 s + c n , (6)
whose roots are the λ i , which can be assigned the characteristic polynomial of A -BK with:

C(s) = det(sI -(A -BK)). (7)
Equations ( 4) and ( 7

The desired λ i are solutions of

λ 3 + λ 2 k d + λk p + k i = 0. (9)
To ensure the control loop stability, the poles of (9) must be placed in the complex left half-plane (i.e. A-BK is Hurwitz). For this purpose, the poles of the controller (5) must be solutions to (9). In order to derive such poles (τ 1 , τ 2 , τ 3 ) ∈ R + , we the following eigenvalues design is considered:

λ 1 = -1 τ 1 ; λ 2 = -1 τ 2 ; λ 3 = -1 τ 3 (10)
The Pole-Placement design can be written as:

       -1 τ 3 1 + k d τ 2 1 - kp τ1 + k i = 0 -1 τ 3 2 + k d τ 2 2 - kp τ2 + k i = 0 -1 τ 3 3 + k d τ 2 3 - kp τ3 + k i = 0 (11)
Since τ 1 , τ 2 and τ 3 are solutions to (9), it follows that:   

1 -1 τ1 1 τ 2 1 1 -1 τ2 1 τ 2 2 1 -1 τ3 1 τ 2 3      k i k p k d   =    1 τ 3 1 1 τ 3 2 1 τ 3 3    ⇔ M K T = N (12)
The resulting gains of the control law (5) are obtained by transforming back the poles with K T = M -1 N as:

k i = 1 τ 1 τ 2 τ 3
; k p = τ 1 + τ 2 + τ 3 τ 1 τ 2 τ 3 ; k d = τ 1 τ 2 + τ 1 τ 3 + τ 2 τ 3 τ 1 τ 2 τ 3 (13) With the mapping in Eq. ( 13), the bounds for the controller parameters can be defined based on control constraints that are easier to derive in the poles domain. In the present case, with the design of Eq. (10), for any τ i > 0, the poles of the feedback loop are placed on the x-axis of the complex left halfplane. By doing that, we set the desired oscillation frequency and percentage overshoot to 0. This leaves us with the settling time requirement to define in order to set the desired distance from the y-axis and thus bound the value of the space of poles. In accordance with the control objective shown in Eq. ( 2), we define the desired maximum settling time of the closed loop control ς = 10 seconds as the maximum time after which we want the system outputs to stay around χ = 5% of its desired values. The upper bound of the space of poles is derived as:

Therefore λ i ≤ λ max (Eq. ( 10)), from which we can derive the upper bound of the poles

τ min < τ i ≤ 1 -λ max = 3.333 = τ max (15)
We set τ min = 0.025 because, for lower values, the control inputs are too expensive in terms of control efforts and too aggressive for our control objective. Thus, the bounds of the poles are defined as: 0.025 ≤ τ i ≤ 3.333 (16) There is a solution for all C(s) in Eq. ( 6) if and only if the pair (A,B) is controllable, that is assumed here. In the case of a Single-Input system, the solution is unique. In the case of Multi-Input systems, as studied here, the number of free components of the matrix K is greater than the n eigenvalue constraints. Accordingly, there exist multiple solutions, among which it is not trivial to define an optimal one.

When no model of the uncertainties is provided, as investigated here, model-free adaptation can be exploited. In order to take into account the uncertainties in the poles selection, we apply DRL to build a stochastic predictive model π µ that maps a state vector s t into the pole values: π µ : s t → [τ vx ; τ vy ; τ vz ; τ φ ; τ θ ; τ ψ ], π µ : s t → N (τ i ), (17) where dim(τ ) = 18 and N (τ i ) is the probability density function of τ i that is modeled by a Normal distribution as:

N (τ i ) = (2πµ i ) -1/2 exp - 1 2µ i (x -λ i ) 2 , (18)
where λ i ∈ R and µ i ∈ R + are the mean and variance of p(τ i ) that are estimated by the Policy network. Therefore, the outputs of the Policy network are the 18 pairs of (λ, µ) representing the Normal distributions N (τ i ) used to sample the poles for each degree of freedom. Designing this stochastic function (17) is numerically expensive due to the dimensions of the underlying spaces, excluding real-time computation with model-based methods only. The DRL framework allows us to iteratively build an estimate of this optimal mapping function. We compare the resulting DRL-based controllers to the OFP controller (provided by the UUV Simulator) whose pole values have been optimized using SMAC [17], a modelbased method, thus without learning. The OFP controller is not adaptive as the poles are fixed. Details on the resulting poles can be found on the UUV Simulator website2 . We present in the next section our use of DRL to adapt the pole values.

B. Design of the DRL-based model-free learning procedure

The related methods cited earlier in Section I are mostly relying on the DDPG and TD3 algorithms. They are known to involve intensive tuning of hyperparameters to work properly. Additionally, in order to take into account the ocean current disturbances, one designer might favor a stochastic policy that is known to be more robust to uncertainties and to partially observable processes (at the cost of being less stable during training). For these reasons, we chose to use another Deep Policy Gradient method, named Soft Actor-Critic. It has been exploited to solve many DRL benchmark environment using the exact same NN architecture and hyperparameters. A complete description of this algorithm is detailed next. Our learning-based architecture is therefore composed of 3 main items that we present now.

1) Soft Actor-Critic: The Soft Actor-Critic (SAC) [18]± [20] is a current state-of-the-art Policy Gradient algorithm integrating three key components:

• An improved exploration and stability in performance thanks to entropy maximization [21]. • An Actor-Critic architecture [4] with separate Values and Policy networks. • An off-policy formulation enabling the use of past collected data with Experience Replay [9].

Instead of optimizing only the expected sum of rewards, the objective function of SAC also maximizes the entropy of the behavior policy, that is weighted by a constant, α, as follows:

J SAC (π µ ) = T t=1
E (st,at)∼ρπ µ [r(s t , a t ) + αH(π µ (.|s t ))],

is known as the Shannon entropy measure of the policy π that is represented by an ANN parameterized by µ. By trying to maximize the entropy of the policy and the reward at the same time, the policy is driven to take the best actions while remaining as random as possible. This results in better exploration and improved robustness to uncertainty thanks to entropy maximization [21], [22].

For this purpose, the entropy term is explicitly incorporated in the State-Value function V (s t ) as:

V (s t ) = E[Q(s t , a t ) + αH(π µ (.|s t ))],

= E[Q(s t , a t )α log π µ (a t |s t )],

where α is controlled indirectly by the reward scale (see Section V-B). In order to reduce Actor-Critic value overestimation [23], [24], the State-Value function is estimated by an ANN parameterized by Ψ using the minimum of two different Q-Value estimates represented by two ANNs parameterized by Υ 1 and Υ 2 , respectively. We used TD-Learning [25] to iteratively build an estimate of the State-Value function (22) and Q-Value function (23). For the State-Value function, the parameters Ψ are thus optimized to minimize the TD-error:

J V (Ψ) = V πµ Ψ (s t ) -min Q πµ Υ1 (s t , a t ), Q πµ Υ2 (s t , a t )
log π µ (.|s t ) . (22) Similarly, the parameters Υ i of the i-th Q-Value function estimator are optimized to minimize the TD-error: (23) where γ = 0.99 is the discount actor and Ψ ′ is defined in Section III-B2. The parameters µ of the Policy network are then optimized [18] in order to minimize the expected Kullback-Leibler (KL) divergence between the current policy and the exponential of the Q-Value function that is normalized by a function Z Υ as:

J Q (Υ i ) = Q πµ Υi (s t , a t ) -r(s t , a t ) + γ × V πµ Ψ ′ (s t+1 ) ,
J π (µ) = E st∼D D KL π µ (•|s t ) Q * (s t , •) Z Υ (s t ) , (24)
where, Q * (s t , a t ) = exp min Q πµ Υ1 (s t , a t ), Q πµ Υ2 (s t , a t ) , ∀s t , a t . (25) When using the distribution expressed in Eq. ( 25) as a target for the policy shown in Eq. ( 24), the agent is forced to explore actions proportionally to their associated exponential Q-Values. This positive transformation allows a smarter exploration-exploitation tradeoff as negative Q-Values will be transformed into small but positive ones, forcing the policy to make progress along sub-optimal strategies until the algorithm finds which value is better for the future gains.

The gradient of ∇ µ J π (µ) is approximated in [18] by:

∇µ J π (µ) = ∇ µ log π µ (
The derivative in Eq. ( 26) is easier to compute compared to that in Eq. ( 24) and allow the use of Gradient Descent to optimize the parameters µ of the Policy ANN. Considering Eq. ( 21), the parameters µ are consequently optimized exactly for the desired maximum entropy objective (19). The soft Qupdate (23) guarantees that Q πnew (s t , a t ) ≥ Q π old (s t , a t ) and the repeated policy updates (26) ensure convergence toward the optimal policy π * 3 .

3 See Appendix B.2 and B.3 of [18] for the mathematical proof).

2) Stabilizing TD-Learning:

The TD learning algorithm, summarized in Eq. ( 22) and (23), is different from Gradient Descent in the sense that the target value changes at each update, transforming the loss landscape. This generates training behaviors that are unstable and divergent. To tackle this problem, it is common practice to have a separate copy of the considered value network, denoted as ªtargetº network, whose parameters are moving slowly or are fixed over a predefined amount of iteration steps. Here, we define a Target State-Value network V Ψ ′ (s) that is then used to compute the Q-Value TD error (Eq. ( 23)) and thus with Ψ ′ slowly tracking the value of Ψ. In our case, we used the soft updates procedure which, for the target network parameters, consists in slowly tracking the parameter of the State-Value function using an exponential moving average ∆ = 0.005. We used the Smooth L1 loss function from Pytorch [26] for the Critics optimization as it is less sensitive to outliers compared to standard MSE.

3) Artificial neural networks: Our implementation of the SAC algorithm is composed of 5 fully-connected Multilayer Perceptron (MLP): two Q-Value networks (with shared architectures), a Value and a Target-Value networks (with shared architectures) and a Policy network. We used the same ANN architecture as proposed in the original SAC paper [18] where each network is composed of 2 hidden layers of 256 hidden units each. The Pytorch framework [26] and CUDA toolkit [27] were used to implement this architecture along with an Nvidia RTX 2070 GPU card for the gradient and simulation processing. The SAC algorithm builds a stochastic policy where the action distributions are modeled by Gaussian distributions 4 . There are several advantages of considering a stochastic policy: it prevents early convergence of the policy variance, it encourages exploration in the value function by increasing the value of regions of state space that lead to highentropy policy, and the resulting policy tends to perform much more consistently compared to its deterministic counterpart with improved robustness to uncertainties [18]. We used Adam [28] as optimizer for all networks with the learning rate l r = 3e -4 . The Leaky ReLU activation function is applied to all hidden layers and gradient descent is applied using a mini-batch of size 256. Layer Normalization [29] (LN) is added before the activation function of all the hidden layers and L2 weight penalty of 1e -3 is incorporated to the Critics only. The weights and biases are initialized from the Gaussian distribution N (0, 2/f ), where f is the fan-in of the layer.

IV. A BIO-INSPIRED EXPERIENCE REPLAY

This section presents the ER method proposed in this paper, which is called BIER for Bio-Inspired ER. In this method, the agent experience is divided into two distinct memory units (as illustrated in Figure 1) and samples are drawn from them differently as described next.

Sequential-Partial Memory: this buffer is denoted as B 1 and is similar to the one from the original ER scheme. Here its maximum size is set to 1, 000, 000, and contains old and new transitions. We believe that, especially in the robotic case, the optimal behavior is highly temporally correlated. With robots, early actions do have an impact on the future states. Learning this temporal relationship is thus essential in order to learn what a truly good behavior, in the physical sense, is. In addition, even with very different operating conditions, the robot's behavior remains quite similar. This means that the shape of the trajectories remains within a bounded space. Therefore, our hypothesis is that learning on sequences can lead to further gains (compared to temporally uncorrelated samples) because what we learn on a trajectory can directly be applied to future ones that the agent has yet to encounter. Following this intuition, we propose to sample random sequences of transition from Buffer B1 (i.e. transitions that are successive within the buffer since we store each of them in a first-in-firstout fashion). The vector of experiences E sampled from B 1 is composed of n samples as E = [e i ; e i+1 ; . . . ; e i+n ], for 0 < i < mn.

While this replay procedure sounds very appropriate for a biological system, here we are optimizing MLPs with limited numbers of parameters and learning abilities. Using highly correlated samples when performing gradient descent often leads to overfitting or local minima. In fact, sequential states are strongly correlated. This is often observed in onpolicy methods where the ANNs are optimized using samples generated by the current policy. This usually leads to repeatedly overfitting to those local correlated samples and never really learning the true value of the functions (i.e. it ends up oscillating between different overfitting regimes). For this reason, we propose to consider sequences that are only partial by storing 1 out of 2 transitions in the buffer B 1 which:

• adds a regularization effect by feeding incomplete sequences to the ANNs that encourages these networks to further learn the real value of the functions. • reduces the age of the oldest policy contained in this buffer, which improved performance in [12].

Optimistic Memory: both bad and good behaviors are important when learning a new task because they both contain information about the process. We have been able to observe a number of cases where ªpositive reinforcingº is much more efficient with biological systems, for example with animal training where good and bad behaviors are respectively rewarded with treats or no response (rather than punishment). It was shown in [12] that trying to estimate values of highquality regions (as measured by the rewards) results in better performance. In addition, as the agent learns, its performance improves and better transitions are performed. Such transitions are important because they can further improve the data collection of the agent in the future. However, as shown in [10], when using a large replay buffer, such transitions are likely to influence the policy later. Their probability to be sampled decreases as the replay buffer size increases, slowing down performance improvement. Following these heuristics, our objective with this second buffer B 2 is to be optimistic about past experience, by increasing the probability of using transitions associated with such high-quality regions.

We propose to store in B 2 the upper outliers of the reward distribution that we consider to be the best transitions. Outliers can be defined according to diverse metrics depending on the nature of the variable distribution. The challenge is that we can not predict the distribution shape beforehand. For instance, with our reward function, the closer the robot gets to the setpoint, the higher the maximum value of possible reward becomes (hence, the optimal policy should lead to a reward distribution of Pearson shape). In practice, however, the closer the vehicle is to the setpoint the more difficult it becomes for it to physically reduce the errors (which is more akin to a Gaussian distribution). Depending on the system, the operating conditions, and the reward function (among others), the reward distribution can switch between various shapes, potentially making the predefined metric not robust to different distribution assumptions. Thus, we propose to consider a transition as an outlier of interest and to store it in B 2 if its associated reward r(s t ) is:

r(s t ) > E[r(s t )], (27)
where the expected value E[r(s t )] is computed over the last 50,000 rewards generated that are stored as an additional variable M . The size of M was chosen in order to compute the expected reward over a moving window of approximately 100 episodes in order to give more importance to novel inputs, similarly to biological systems [13]. This choice of expected value as a metric is related to the subtracted baseline in Eq. ( 26) that is the Advantage function A(s, a). This function represents the benefits of changing the current policy. If the value of A(s, a) is positive, then the probability to take the evaluated action (with respect to. the given state) will be increased because its Q-Value is higher than the expected one (with respect to. the current policy). Our assumption is that transitions that meet the criteria expressed in Eq. ( 27) are associated with positive values of A(s). Gradient updates using samples from the Optimistic Memory will therefore mostly improve the expected return of the policy, leading to faster discovery of successful trajectories.

The maximum size of B 2 is set to 10, 000. It is drastically smaller than B1 because, as the agent's performance improves over the course of training, what was considered as a good transition is most likely to be outdated. Therefore, the reduced buffer size ensures that we focus on the current best transitions. Finally, contrary to the first buffer, we propose to sample n uncorrelated items from B 2 as single transitions are iteratively stored in this buffer.

V. TRAINING

This section describes the training settings (including training scenario, reward function, state vector, and exploration strategy) with the hyperparameters choice. It ends with an analysis of the training results.

A. Scenarios definition

The training consists of performing a total of 3000 episodes (as no further notable improvement in setpoint regulation is observed after approximately 2500 episodes). The maximum length of a training episode was set at 500 timesteps (equivalent to 25 seconds). We define a training episode as follows: Fig. 1. Illustration of our bio-inspired procedure. The agent's past experience is divided in two memory units: 1 out of 2 transitions are stored in the first buffer and we sample from it temporal sequences of experiences; we store in the second buffer the best transitions as measured by reward and with respect to the current policy. This procedure takes advantage of the resilience from on-policy sampling while keeping the data efficiency from the off-policy formulation.

1) The robot is initialized at a depth of 40 meters with a random orientation (ψ, θ, ϕ) ∈ [ -π 4 ; π 4 ] and at null velocity. 2) The current variables are randomly chosen such that v c ∈ ]0, 0.5] and [h c , j c ] ∈ [ -π 4 ; π 4 ] (which remain fixed during the episode) and a random vector of setpoints is generated such that x ref = [v x , 0, 0, 0, 0, 0] T with v x ∈ [0.1, 0.5] (m.s -1 ). 3) Then, the off-policy π µ (a|s) behavior is used.

4) The episode ends when the control objective ( 2) is met or when the episodic step number exceed 500.

B. Reward shaping

Compared to our previous work [30], the control objective treated here is fundamentally more complex since we are not trying to reach one singular state but rather a sequence of desired successive states. We propose the following terminal reward signal to take this into account: r success = 1000 if ∀ t ∈ [t-100, t], |e i (t)| ≤ χ. (28) Therefore, performing the desired control objective, as shown in Eq. (2), will generate this reward, Eq. ( 28), and its value was chosen in order to make sure that, for all trajectory lengths, the maximum sum of return is obtained only by stabilizing the vehicle. Otherwise, the reward r(s t ) is generated. Let's define the Euclidean distance to the desired setpoint as e L2 (t) = i=dim(u) i=1 e 2 i (t) and its derivative is computed over the last two frames and denoted as d rate (t). The reward r(s t ) is then defined as:

r(s t ) = C 1 × exp -e 2 L2 (t) × C 2 (29)
The performance of SAC is highly dependent on the choice of reward scale (or amplitude) which, in the case of our reward function Eq. ( 29), is controlled by the constant C 1 . The reward scale can be interpreted as the inverse of the temperature parameter α from (19) which controls the stochasticity of the resulting policy. Here, we empirically chose C 1 = 40, which gave us the best performance, by following advice from [20]. The reward signal, Eq. ( 29), is equal to its maximum value possible per step (that is C 1 ) only when all the current errors are equal to zero. As AUVs move slowly, successive states display error signals e i (t) of minor and similar amplitude. We find that this addition of C 2 = 10 (compared to our previous work [15]), makes it easier for the critics to differentiate the State-Value of successive states without altering the reward scale as lim x→0 C × e x = C. This reward function, Eq. ( 29), encourages the agent to reduce the errors as much and as fast as possible and the vehicle stabilization is further promoted by generating the maximum reward possible per step, as shown in Eq. (28). In order to improve the exploration ability provided by the maximum entropy framework, we used adaptive parameter noise [31] which consists in adding noise to the parameters of the policy network.

C. Process Observability

At each timestep, the agent captures an observation vector o t representing the process dynamics that we defined as: o t = [a t-1 ; Θ ; V ; Ω ; u t ; e t ; e L2 ; d rate ; δ χ ],

where a t-1 ∈ R 18 are the last action estimated (i.e. poles value); Θ = [ϕ; θ; ψ] are the Euler orientation of the vehicle (roll, pitch and yaw respectively); V = [v x ; v y ; v z ] and Ω = [ω φ ; ω θ ; ω ψ ] are its linear and angular velocities; u t ∈ R 6 are the last control inputs applied; e t ∈ R 6 are the error values on each setpoint; e L2 and d rate as described in Section V-B; and δ χ ∈ [0, 1] is a variable which keeps track of the number of successive steps where all the errors are within the threshold (i.e. if δ χ = 1, the control objective is achieved). The dimension of the observation vector o t is therefore equal to 42. Noted that with this observation vector (30), the current disturbance characteristics are not included. In order to improve the process observability and following our previous results [32], we construct our state vector s t out of the current and past observation vectors along with their two-by-two difference. This results in a 126 dimensional state space defined as s t = [o t ; o t-1 ; o t-1 -o t ].

D. Training results

Figure 2 shows the training dynamics with the normalized mean return per episode, the RMSE on the setpoint per episode, and the normalized mean return standard deviation per episode. The yellow dashed lines represent the performance of the OFP controller. It is fair to compare these controllers, as they are based on the exact same control structure as our learning-based controller, that is the PID structure. Thus, the only difference between the three controllers is the value of the poles used to compute the PID control inputs.

As we can see in Figure 2, both methods are able to learn the task and converge toward what seems to be a maximum value of the reward. In the third plot of Figure 2, we can see that the performance improvement is smoother with the BIER method which exhibits a lower reward standard deviation (which tends to reduce over time contrary to the CER agent). With the CER method, the variance is higher, with pikes that even drive the agent to lower performance than those obtained with the OFP controller. Noted that the OFP controller displays the lowest standard deviation (third plot of Figure 2), thanks to the model information incorporated in SMAC [17].

The agent trained using the BIER method was able to exceed the performance of the OFP controller, in less than half the number of episodes compared to the standard CER method. It is represented in Figure 2 by the vertical lines. Therefore, when DRL can be considered on the physical platform, a designer might favor the use of our BIER method for improved data efficiency and learning stability.

VI. EXPERIMENTS AND ANALYSIS

The challenge of conducting DRL in real-life lies in the drop in performance caused by the distribution shift that is here the difference between the training and evaluation set of states. In order to visualize this behavior in our use case, we present below different evaluation scenarios of increasing complexity as measured by desired setpoint and current velocity.

Scenario 1: the setpoint range is the same as during training but no current disturbances is applied. This scenario is therefore in theory simpler than the training one.

Scenario 2: the process is the same as during training but with setpoint and current variables that were not seen during training (but are still in the same range). Scenario 5: both the setpoint and current variables are increased to the range defined in scenarios 3 and 4.

Scenario 6: we keep the increased range from scenario 5, and at a random timestep during the episodes (between the 100th and 400th timestep), we vary the current variables (velocity and orientation) within the same training range.

In Table I, the metrics were computed over 500 episodes (different from each other) for each evaluation scenario. The line ªBaselineº denotes the performance obtained at the end of the training, which incorporates the parameter noise described in Section V-B. For the other scenarios, this noise is removed. In order to visualize more easily the performance variation, we propose in Figure 3 an illustration of these results.

The CER and BIER agents are able to stabilize the vehicle over the first 3 scenarios where the performance is matching the training one. When increasing the desired setpoint value, we can see the associated RMSE slightly increasing but the performance remains satisfying. The performance drop is the largest when increasing the sea current disturbance (i.e. scenarios 4, 5, and 6). The sensibility to this disturbance is further depicted by the performance difference that is much smaller between scenarios 5 and 6 compare to between scenarios 3 and 4. We believe this is due to the current characteristics not being explicitly included in the state vector.

The BIER agent performs better on scenario 1 compared to scenario 2 which is simpler than the training scenario as it does not incorporate sea current disturbance. More interestingly, its performance compared to CER has also increased on scenarios 5 and 6, despite the distribution shift being particularly large there. This suggests that the policy obtained with the BIER method has better generalization abilities. We can see that the OFP controller is much sensitive to sea current disturbance compare to setpoint variation (as it is included in the model). Noted that the learning-based controllers are able to exceed the OFP controller on scenario 6, despite not having experience such disturbance during training.

These results show that ER does have an impact on the quality of the resulting policy. By only manipulating differently the agent's past experience, we were able to make learning faster, and with improved robustness to increased process variations. Reducing the distribution shift problem in DRL remains an active field of research and future work could focus i) on improving process observability and ii) studying how conservative can a DRL-based policy be.

VII. CONCLUSION

In this article, we proposed a new Bio-Inspired Experience Replay which aims at incorporating concepts from the biological Replay Mechanism in the context of DRL. We found that by taking inspiration from nature, while keeping in mind the requirements from Gradient-based optimization, the adaptive controller can learn faster and with improved stability. Our results also suggest that learning-based adaptive control might be a key ingredient toward real-life autonomous robotic applications, by taking advantage of the available information on the process, with a model-based structure, and using modelfree learning to adjust its parameters to compensate for unmodeled process variation, via Deep Reinforcement Learning. Abstract: Autopilots for unmanned systems are usually designed based on the feedback provided by velocity and orientation sensors. In the case of autopilot systems for autonomous underwater vehicles (AUVs), the main objective in the design is to compensate for waves and currentinduced disturbing forces acting on their body. Existing AUV autopilots are however only able to compensate for low-frequency components of sea-induced disturbances. It seems natural to assume that the AUV performance could be improved by taking the nature of the disturbances into account in the design of the autopilot.

Adaptive control provides what seems to be an ideal framework for this end. The objective of this technique is to adjust automatically the control parameters when facing unknown or timevarying processes such that the desired performance threshold is met. Developed in the late 1950s, adaptive control frameworks have been considerably expanded and used in various fields, their application has been facilitated by the rapid progress in microelectronics and the increasing interaction between laboratories and companies, from aerospace to maritime industries. As a result, adaptive controllers started to be widely adopted in the industry in the early 1980s. It was established at that time that robust designs with fixed parameters are too limited to handle complex regimes. The study of adaptive controllers for AUV maneuvering is associated with various challenges, and the focus of this thesis was the external disturbances including: Unknown dynamics: the uncertainty associated with describing precisely the states of waves or currents is high. This, together with its dynamic nature, prevents linear feedback control methods from achieving optimal performance of the plant. This becomes more critical in the presence of changes in weather conditions that impose a multiplicative factor in the component of the induced forces. The disturbance period will also vary with the speed of the vehicle and its orientation relative to the waves.

Nonlinearity : the controller response at some operating points must be overly conservative to satisfy the specification at other operating points. This is difficult to achieve for fixed parameters obtained through local linearization, that do not encompass the entire regime envelope.

In this thesis, we considered the case where the AUVs have limited observability of the process and therefore the aforementioned uncertainties are not measured by the system. A class of adaptive control methods, known as learningbased adaptive controllers, have been developed to tackle some of these limitations. This family of solutions uses model-free optimization methods capable of compensating for the unknown part of a process while also maintaining optimal control of its known part using traditional model-based control structures. Among the various modelfree methods, deep reinforcement learning is currently leading the field. They exploit strong statistical tools that provide control systems the ability to automatically learn and improve from experience without being explicitly told how to.

The objective of this thesis was to formalize a novel learning-based adaptive control using deep reinforcement learning and adaptive poleplacement control. In addition, we proposed a novel experience replay mechanism that takes into account the characteristic of the biological replay mechanism. The methods were validated in simulation and in real life, demonstrating the benefits of combining both theories against using them separately.

Figure 1 . 1 :

 11 Figure 1.1: Procedure to decide what type of controller to use.

  cl = P u + d = P Kr -P ky cl + d (1 + P K)y cl = P Kr + d y cl = P

Figure 1 . 3 : 2 Figure 1 . 4 :

 13214 Figure 1.3: Block diagram of an AUV control loop with a PI controller.

Figure 1 . 5 :Figure 1 . 6 :

 1516 Figure 1.5: Block diagram of an AUV control loop with a PI controller.

Figure 2 . 1 :

 21 Figure 2.1: Block diagram of a Gain Scheduling control system. It can be framed as a feedback control system in which the feedback gains are tuned on the basis of the operating conditions.

Figure 2 . 3 :

 23 Figure 2.3: Block diagram of a modelreference adaptive system (MRAS).

Figure 2 . 4 :

 24 Figure 2.4: Illustration of the objective function J(•) considered for this example (a) and the associated PBESC system (b).

  ) the function f ref is a known nonlinear smooth function of the desired trajectories x ref = (x ref 1 , x ref 2 ) T . A basic learningbased controller can be defined as: u = -e1 -c1e2 -Ŵ T S(e) + v, with

Figure 2 . 5 :

 25 Figure 2.5: Illustration of an RL process where the agent is interacting with its environment.

Figure 2 . 6 :

 26 Figure 2.6: Illustration of the impact of the discount factor on future rewards.

Figure 2 . 7 :

 27 Figure 2.7: Illustration of the agentenvironment interaction in a Markov Decision Process.

Figure 2 . 8 :

 28 Figure 2.8: Illustration of RL methods based on the nature of the decisionmaking process: in modelbased methods, the actions are the result of deterministic planning, in valuebased methods the actions are pooled over the entire set of possible actions, and in policy gradient methods the actions are sampled from a probability density function.

Figure 2 . 9 :

 29 Figure 2.9: Backup diagram of DPbased method for the statevalue function [SB18].

Figure 2 . 10 :

 210 Figure 2.10: Backup diagram of MCbased methods for the statevalue function [SB18].

2 .

 2 for t = 0, . . . , T : (a) From St choose At = arg max a∈A Q(St, a) (using ϵgreedy exploration strategy in general). (b) After applying action At, we observed the reward Rt+1 and transit to the next state St+1. (c) From St+1 pick action similarly to step (a): At+1 = arg max a∈A Q(St+1, a).

Figure 2 . 11 :

 211 Figure 2.11: Backup diagram of TD learning methods for the statevalue function [SB18].

Figure 2 . 12 :

 212 Figure 2.12: Illustration of the update behavior obtained with the vanilla gradient of the reward function. The values depicted here were chosen only to illustrate that the resulting gradient is proportional to the QValue and does not hold any mathematical accuracy.

Figure 2 . 13 :

 213 Figure 2.13: Illustration of the update behavior obtained when using the statevalue function as a baseline to reduce

1 . 3 .

 13 Initialize s, w, θ at random 2. Sample a ∼ π(a|s; θ) For t = 1, . . . , T : (a) Sample reward rt ∼ R(s, a) and the next state s ′ ∼ P (s ′ |s, a). (b) Sample the next action performed a ′ ∼ π(s ′ , a ′ ; θ). (c) Update the policy parameters as: θ ← θ + α θ Q(s, a; w)∇ θ log π(a|s; θ). (d) Compute the TD error at timestep t as: Gt:t+1 = rt + γQ(s ′ , a ′ ; w) -Q(s, a; w).(e) Update the parameters w according to this TD error: w ← w + αwGt:t+1∇wQ(s, a; w).

  (f) Move to the next transition a ← a ′ and s ← s ′ .

Figure 2 . 14 :

 214 Figure 2.14: In the ActorCritic architecture, the value function is learned in addition to the policy. This greatly reduces the variance in the policy gradient update

Figure 2 . 15 :

 215 Figure 2.15: Illustration of an offpolicy TDLearning procedure where the policy used for exploration is different from the one that is optimized.

1. Initialize s, w and θ 2 .

 2 For t = 1, . . . , T : (a) Sample reward rt ∼ R(s, a) and the next state s ′ ∼ P (s ′ |s, a). (b) Sample the next action performed a ′ ∼ π(s ′ , a ′ ; θ).

Figure 2 . 18 :

 218 Figure 2.18: Illustration from [Haa+18a] of the performance variation observed depending on the version of the SAC.

Figure 2 . 19 :

 219 Figure 2.19: Illustration of the moving target value problem from reinforcement learning.

Figure 2 . 20 :

 220 Figure 2.20: Spatial distribution of rewards obtained with the reward defined in Eq. (2.88).

Figure 2 . 21 :

 221 Figure 2.21: Spatial distribution of rewards in the (x, y) plane obtained with the reward defined in Eq. (2.89). The points that are at the same distance to the waypoint (i.e. yellow line) can now be associated with a different reward value based on the derivative of the distance.

  Figure 2.22.

Figure 2 . 22 :

 222 Figure 2.22: Illustration of the distribution shift problem in reinforcement learning. There are no guarantees that the solution obtained from training, often using samples from a simulator, will remain valid on new state samples (e.g. reallife samples).

Figure 2 . 23 :

 223 Figure 2.23: Imagine the green curve is the true function and the red curve is our fit. While the red curve is good in most

Figure 2 . 24 :

 224 Figure 2.24: The RexRov2 platform in a simulated Gazebo environment from UUV Simulator.

  .98) These vectors are then assembled into a thrust allocation matrix T = [T1, . . . , T6] which describes the re lationship between propeller thrust and the vehicle speed, [Car18]. Using this allocation matrix, the control inputs u are transformed as u = [vx; vy; vz; ω ψ ; ω θ ; ω ϕ ] T where [vx; vy; vz] T are linear velocity inputs (in m.s -1 ) and [ω ψ ; ω θ ; ω ϕ ] T are torque inputs (in radians) expressed in the reference frame attached to the center of mass of the simulated RexROV2 platform.

Figure 2 . 25 :

 225 Figure 2.25: RexROV2 thruster positions, side view. Source: [Ber12].

Figure 2 . 26 :

 226 Figure 2.26: RexROV2 thruster positions, front view. Source: [Ber12].

Figure 2 . 27 :

 227 Figure 2.27: Illustration of the Bluerov configuration kit (left) and The Bluerov thruster configuration from topdown view.

Figure 2 . 28 :

 228 Figure 2.28: General block diagram of a DRLbased learningbased adaptive control system.

Figure 3 . 1 :

 31 Figure 3.1: In (a) we plot the distance error dδ from the ideal trajectory over time steps. and in (b) we display the Euclidean norm of the input vector u over time steps for the associated episode.

Figure 3 . 2 :

 32 Figure 3.2: Trajectory in the XY plan performed during an episode by both controllers with Cv = +0.096 m.s -1, C ha = -0.1324 rad and Cva = 0.083 rad.

  A target is then initialized at a fixed and uniformly random position Λ = [Λx; Λy; Λz] T with [Λx; Λy] T ∈ [-20; -5[∪]5; 20] and Λz ∈ [2; 20].

  reached is reduced during training as follows: at first d reached = 3m, from iteration 250 000th we set d reached = 2m and from iteration 500 000th we set d reached = 1m. The PyTorch framework [Pas+19] was used to carry out the numerical experiments, along with the CUDA toolkit [Nic+08] and an RTX 2060 GPU card, allowing us to perform the training of one controller in approximately 10 hours. It can be seen in Figure 3.3 that during training, both the learningbased (LB) and modelfree (MF) controllers were able to reach a high success rate under unknown wind gust disturbances, which shows the applicability of the SAC DRL procedure for this type of aerial navigation problems.

Figure 3 . 3 :

 33 Figure 3.3: Training curves showing the mean reward and success rate computed per 100 episodes over a moving window of 100 episodes.

  .27) where the state of the vehicle η d is its position and orientation: xv = [xv ; yv ; zv ; ψv ; θv ; ϕv], (3.28) and the desired setpoint is defined as: xw = [0, 0, 0, 0, 0, 0]. (3.29)

  .34) where M1, M2 and M3 are three 6 × 6 positive matrices and ϵ is a small positive constant. With this transfor mation (3.34), the fulfillment of the Lyapunov stability constraints (3.33) can now be verified by only assessing the value of [M1, M2, M3, ϵ] ∈ R 3×6×6+1 . In order to further reduce this dimension space, we apply a diago nal transformation on the matrices Mi: Mi = P ΛiP -1 , where Λi ∈ R 6 are positive vectors and P ∈ R 6×6 is a positive invertible matrix chosen randomly beforehand. Thanks to this transformation, we can now assess the value of [M1, M2, M3, ϵ] ∈ R 3×6×6+1 (and kp, ki, k d with Eq.3.34) by only accessing:[Λ1, Λ2, Λ3, ϵ] ∈ R 19 .(3.35)

Figure 3 . 4 :

 34 Figure 3.4: Block diagram of the proposed method for PID tuning using CrossEntropy Deep Learning. The control parameters are adapted during operation and are represented in the form of a function of the state of the vehicle (i.e. neural network).

N = 25 ,

 25 proportion to keep ρ = 0.2 and added noise σ 2 noise = 0.1. Each training episode is composed of 200 time steps. One epoch is defined as performing one episode using each of the N sets of parameters candidate. The training consists in performing 200 epochs, thus a total of 200 × 200 × 25 = 10 6 time steps.

( a )

 a Mean squared error on the setpoint. (b) Covariance of the normal distribution.

Figure 3 . 5 :

 35 Figure 3.5: Training curves showing the evolution of the MSE and the covariance during training.

( a )

 a Evolution of the MSE for scenario 1. (b) Evolution of the MSE for scenario 2. (c) Evolution of the MSE for scenario 3.

Figure 3 . 6 :

 36 Figure 3.6: Illustration of the control performance by both controllers over the different scenarios.

Figure 3 . 7 :

 37 Figure 3.7: Illustration of the errors on each DoF by both controllers over the different scenarios.

( a )

 a Evolution of the state stability during an episode from sce nario 1. (b) Evolution of the state stability during an episode from sce nario 2. (c) Evolution of the state stability during an episode from sce nario 3. (d) Evolution of the stability of the parameters during an episode from scenario 1. (e) Evolution of the stability of the parameters during an episode from scenario 2. (f) Evolution of the stability of the parameters during an episode from scenario 3.

Figure 3 . 8 :

 38 Figure 3.8: Illustration of the state and parameters stability of each controller over the different scenarios.

  .45) Equations (3.42) and (3.45) yield:

Figure 3 . 9 :

 39 Figure 3.9: Parameter cube representation of the proposed learningbased controller. The continuous space of possibles poles is restricted to stable locations. It is explored by a deep reinforcement learning algorithm (yellow lines) that uses these variables to compute the control input.

Figure 3 . 10 :

 310 Figure 3.10: Illustration of the resulting parameter values for a different combination of pole values. For τ i ≥ 1, the resulting gains grow linearly while when τ i ≤ 1, the resulting gains grow exponentially.

Figure 3 . 11 :

 311 Figure 3.11: Illustration of the difference between MPC and our proposed method. The learningbased adaptive con troller (b) samples the optimal actions directly from the neural network.

Figure 3 . 12 :

 312 Figure 3.12: Plots of the L1, L2 and Smooth L1 loss functions from Pytorch [Pas+19].

Figure 3 . 13 :

 313 Figure 3.13: The deep neural network structure for our implementation of the SAC algorithm. The networks are com posed of dense layers only and of two hidden layers of 256 hidden nodes for each network. Each layer is a fullyconnected layer represented by its type, output size, and activation function. The networks uses the same optimizer (Adam [KB15]), activation function (Leaky Relu [Xu+15]) and learning rate lr = 3e -4 .

Figure 3 . 14 :

 314 Figure 3.14: Illustration of the proposed bioinspired procedure. The agent's past experience is divided into two memory units: 1 out of 2 transitions are stored in the first buffer and we sample from it temporal sequences of experiences; we store in the second buffer the best transitions as measured by reward and with respect to the current policy. BIER takes advantage of the resilience from onpolicy sampling while keeping the data efficiency from the offpolicy formulation.

Figure 3 . 15 :

 315 Figure 3.15: Illustration of the overall proposed learningbased adaptive control system.

[

  vx, hc, jc, vc] = D1 × sin(D2 × t) + D3, (4.11) with D1 ∈ [0.1; 0.25], D2 ∈ [0.5; 1] and D3 ∈]1; 1.25] for vx and D1 ∈ [0.25; 1], D2 ∈ [0.5; 1] and D3 ∈]0; 0.5] for [hc, jc, vc] randomly chosen. The evaluation consists in performing 500 episodes for each scenario with a maximum step size per episode of 500 and different from each other in terms of the abovementioned variables. The outcomes of this evaluation are provided in Tables 4.2, 4.3, and 4.4 with the success rate, the mean reward per step and the positive reward rate.

Figure 4 . 1 :

 41 Figure 4.1: Training curves for both Experience Replay methods. The BIER agent outperforms the CER agent in both learning speed and variance despite having access to a reduced variety of samples. Both learningbased controllers outperform the OFP controller).

Scenario 3 :Scenario 4 :Scenario 6 :

 346 the setpoint vx is increased to the range [0.5,1.0], and the current variables are the same as during training. the setpoint characteristic is the same as during training but the current velocity vc is increased to [0.5,1.0] (m.s -1 ) and (hc, jc) ∈ [-π, -π 2 [∪] π 2 , π]. Scenario 5: both the setpoint and current variables are increased to the range defined in scenarios 3 and 4. we keep the increased range from scenario 5, and at a random timestep during the episodes (between the 100th and 400th timestep), we vary the current variables (velocity and orientation) within the same training range.

Figure 4 . 2 :

 42 Figure 4.2: Illustration of the evaluation performance. The only difference between CER and BIER is how, during training, the agent's past experience was used. Noted that, despite having been trained only under process variation from scenario 2, the learningbased adaptive controllers outperform the nonadaptive optimal modelbased controller on scenario 6 where the distribution shift is particularly large.

Figure 4 . 3 :

 43 Figure 4.3: Evolution of the error on each DoF for an episode from scenario 5.

( a )

 a The BlueRov2 platform resting on the side of the water tank at FLINDERS University. (b) A Kalman filter fuses IMU and camera data to estimate the marker's position. (c) Jonathan Wheare and myself monitoring the robot from the computer station. (d) Andrew Lammas adjusting the disturbance system before an experimentation session.

Figure 5 . 1 :

 51 Figure 5.1: The experimental validation took several months and required the efforts of a team. Helped by Jonathan

Table 5 . 1 :

 51 List of the setpoints and their coordinates (in meters).

Figure 5 . 2 :

 52 Figure 5.2: Illustration of the stationkeeping task performed for our experimental validation. The vehicle has to stabilize at 9 setpoints in space while being perturbed by an external current source.

( a )

 a Front view of the vehicle. (b) Side view of the vehicle. (c) Illustration of the camera feedback (left half of the screen) and the resulting pose estimate represented by the position vectors (right half of the screen).

Figure 5 . 3 :

 53 Figure 5.3: The data from sensors and the marker tracking system are fused using a Kalman filter to compute an estimation of the vehicle position.

Figure 5 . 4 :

 54 Figure 5.4: Two thrusters generate current disturbance (a) while the camera is fixed at the end of the arm and is facing down (b).

Figure 5 . 5 :

 55 Figure 5.5: Illustration of the setup for the experiments.

. 6 )

 6 Contrary to the previous reward function designs Eqs. (3.7), (3.25), (4.7), and (4.18), the reward scale is not controlled here. This means that the reward signal (5.6) is defined in r(st) = [0, 1].

Figure 5 . 6 :

 56 Figure 5.6: Conceptual illustration of domain randomization. The ambition is to make the policy experience so many variations of the source domain that the target domain will only be another variation among the others.

Figure 5 . 7 :

 57 Figure 5.7: Illustration of the proposed domain randomization method. We proposed to divide the training environ ment into three configurations in terms of environment complexity. At the beginning of each episode, the environment characteristics are set to one of these configurations with an equal probability P for each case.

Figure 5 . 8 :

 58 Figure 5.8: Illustration of the experimental performance of the controllers without disturbance.

Figure 5 . 9 :

 59 Figure 5.9: Illustration of the experimental performance of the controllers with disturbance.

Figure 5 . 10 :Figure 5 . 11 :Figure 5 . 12 :Figure 5 . 13 :

 510511512513 Figure 5.10: Illustration of the experimental performance (the complete results are provided in Tables 5.35.10). In the first row, we present the performance of both controllers without disturbance in Figure 5.10a, and against current disturbance in Figure 5.10b. On the second row, we display the performance of each controller against both operating conditions with the performance of the modelbased controller in Figure 5.10c and the learningbased controller in Figure 5.10d.

- 1 .Figure B. 1 :

 11 Figure B.1: Visualization of the simulated environment in Gazebo (a) and the wind field in Paraview (b). Characteristics of the wind field can be find on the RotorS plugin page. The parametrization of the Firefly platform considers two reference frames: the worldfixed inertial frame RW and the body reference frame RB which is attached to the center of mass of the hexacopter. Coordinates in the world frame are denoted as [xW , yW , zW ] T while they are denoted as [xB, yB, zB] T in the body frame. The pose of the hexacopter is given by its position ζ = [xW , yW , zW ] T and orientation η = [ϕ, θ, ψ] T in the three Euler angles (respectively roll, pitch and yaw). For the sake of clarity, sin(•) and cos(•) are abbreviated as s• and c• in the next equation. The transformation from RW to RB is given by: xB yB zB = cθcψ cθsψ -sθ sϕsθcψcϕsψ sϕsθsψ + cϕcψ sϕcθ cϕsθcψ + sϕsψ cθsθsψsϕcψ cϕcθ xW yW zW

  (a) Rexrov2 in the ROS-Gazebo environment [4]. (b) Thrusters layout of the Rexrov2 (top view) [5].

Figure 1 :

 1 Figure 1: Illustration of the simulated robotic platform with its thrusters configuration and the environment used for the training and testing of both controllers.

  Evaluation results under simulation for both controllers over the same set of 500 test episodes.

Figure 2 :

 2 Figure 2: Trajectory followed during an episode for both controllers with C v = +0.096m.s -1, C ha = -0.1324rad and C va = 0.083rad.

Figure 3 :

 3 Figure 3: Performances of both controllers for the same episode characteristics as in Figure 2.

Fig. 1 :

 1 Fig. 1: General block diagram of an adaptive control system. It is composed of two loops: a normal feedback loop with the plant and the controller and a second loop with the parameter adjustment mechanism (which is often slower than the first one).

Fig. 2 :

 2 Fig. 2: General block diagram of a classical model-free controller. The control objective

Fig. 3 :

 3 Fig. 3: Visualization of the simulated environment in Gazebo (a) and the wind field in Paraview (b). The complete details of the wind field we used can be find on the RotorS-based extended wind plugin page.

  The control objective is to stabilize the robot at a waypoint ζ ref with a velocity v = 0. Defining e = ζ -ζ ref and taking into account unmodeled disturbances (e.g. wind) by considering an additional steady-state error variable z = t 0 e(τ )dτ , the augmented model with state vector X = [z, e, v] becomes:

Fig. 4 :

 4 Fig. 4: The neural networks structure for our implementation of the SAC. The networks are composed of dense layers only and of two hidden layers for each network. Each layer is a fully-connected layer represented by its type, output size and activation function. The networks uses the same optimizer (Adam [48]), activation function (Leaky Relu [49]) and learning rate l r = 3e -4 . Other parameters are a discount factor γ = 0.99 for (34), a number of 256 hidden units per hidden layer and one gradient update is performed at each time-step.

-

  A constant positive reward is sent to the agent when it succeeded to complete the mission, meaning d t ≤ d reached . This generates r reached = +1000.-If the MAV altitude z w exceed a predefined threshold, the constant negative reward r f ailed = -550 is generated.

Fig. 5 :

 5 Fig.5: Training curves showing the mean reward and success rate computed per 100 episodes over a moving window of 100 episodes. Note that with SAC, the policy is trained to maximize also the entropy, therefore the mean action does not always correspond to the optimal action for the maximum return objective.

  [O'Dwyer (2006)]. Adaptive control methods are widely used in the context of dynamic processes and provide, what seems to be, an ideal framework for automatic tuning of regulators. New tuning techniques are emerging from the development of datadriven theory [Brunton and Kutz (2019)]. The well-known model-free adaptive method, Extremum Seeking [Ariyur and Krstić (2003)], has been proven to compensate for uncharacterised and unmodelled process variations. Machine ⋆ This work was supported by ISblue project, Interdisciplinary graduate school for the blue planet (ANR-17-EURE-0015) and cofunded by a grant from the French government under the program "Investissements d'Avenir".

Fig. 1 .

 1 Fig. 1. Parameter cube representation for adaptive Pole-Placement control. The continuous space of possibles poles is restricted to stable locations. It is explored by a deep reinforcement learning algorithm (yellow lines) that feed these parameters to a PID controller to derive the gains and thus the control law. This guarantees adaptability while converging to the steady-state. Learning based model-free adaptive methods have been suggested to tune PID regulators [Jayachitra and Vinodha (2014)]. Deep Reinforcement Learning techniques (DRL) in particular, have shown great performance as optimization methods for model-free adaptive scheme. They exploit the strong abilities of artificial neural networks (ANNs) to perform nonlinear mapping between sensor feedback and control inputs directly in closed-loop systems.

( 1 )

 1 )where ρ πµ is the state-action joint probability. In this article, we propose to use the Policy Gradient algorithm [Sutton and Barto (2005)] called Soft Actor-Critic which aims at modeling and optimizing the policy directly.It is composed of three key components: An Actor-Critic architecture[START_REF] Konda | Actor-critic algorithms[END_REF]] with separate values and policy networks. (2) An off-policy formulation that enable the use of past collected data with Experience Replay[START_REF] Lin | Reinforcement learning for robots using neural networks[END_REF]]. (3) Entropy maximization for improved stability and exploration[START_REF] Haarnoja | Reinforcement learning with deep energy-based policies[END_REF]].

; π 4 ]

 4 2) A random set of current variables is generated such asv c ∈ [0, 0.5] and [h c , j c ] ∈ [ -π4 which are then kept constant during the episode.(3) A random vector of setpoints is generated such thatΛ ref = [v x , 0, 0, 0, 0, 0] T with v x ∈ [0.5, 1.5].(4) Then, the off-policy exploration strategy is used and the episode ends when the step number reaches 300 or r succeed is generated.The PyTorch framework[START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]] was used along with the CUDA toolkit[START_REF] Nickolls | Scalable parallel programming with CUDA[END_REF]] and an RTX 2070 GPU card, allowing us to perform the training in approximately 18 hours. Note that the Gazebo simulation is run here at a real time factor, therefore this training time could be further reduced.

with D 1 ∈

 1 [0.1; 0.25], D 2 ∈ [0.5; 1] and D 3 ∈]1; 1.25] for [v x ] and D 1 ∈ [0.25; 1], D 2 ∈ [0.5; 1] and D 3 ∈]0; 0.5]

Fig. 1 .

 1 Fig. 1. Block diagram of the proposed method for PID tuning loop using Cross-Entropy Deep Learning.

Fig. 2 .

 2 Fig. 2. Control performance without disturbance.

Fig. 3 .

 3 Fig. 3. Control performance with noisy position and orientation measurements.

Fig. 4

 4 Fig.4. Control performance with noisy control inputs and sea current disturbance. We can see that the naive PID (in blue) results overall in better performances in terms of setpoint tracking compared to the LB PID (in orange). Nevertheless, we can see that the relative performance of the LB PID with respect to the naive PID's performance improves with increased uncertainty. In Figures3 and 4, we can see that the performance of the LB PID matches the naive PID for approximately 500 timesteps, while without disturbance, its performance drops notably earlier as shown in őgure 2. The evolution of the vehicle's state is represented in Figures5-10. We can observe a difference in regulation dynamics depending on the DoF. For instance, we can see that the depth of the UUV is not successfully regulated by none of the controllers. Due to the short latency between episodes and the position of the UUV's CoG, the UUV slightly sinks at the beginning of the episode, altering its depth and yaw angle (z, yaw). This explains the vertical drift in their associated errors observed in the őgures. In addition, as seen in Figures5-10, the LB PID tends to regulate effectively more DoF compare to the naive PID (which fails at regulating (z, yaw), see Figures5, 7, 9).

Fig. 5 .Fig. 6 .

 56 Fig. 5. The position and orientation errors of the naive PID during a non-disturbed episode.

Fig. 7 .

 7 Fig. 7. The position and orientation errors of the naive PID during an episode with measurements noise.

Fig. 8 .

 8 Fig. 8. The position and orientation errors of the LB PID during an episode with measurements noise.

Fig. 9 .Fig. 10 .

 910 Fig. 9. Position and orientation errors of the naive PID with noisy control inputs and sea current disturbance.

Fig. 11 .Fig. 12 .

 1112 Fig. 11. Evolution of the state stability during a nondisturbed simulation

Fig. 13 .Fig. 14 .Fig. 15 .

 131415 Fig. 13. Evolution of the state stability during a simulation with measurements noise

  account the current disturbance by considering the steady-state error variable σ = t 0 e(τ )dτ . We can rewrite the state-space equations with the augmented state vector X = [σ, e, ẋ] as:

  ) yield |A -BK -λI| = -λ(λ(k d + λ) + k p )k i , = -λ 3λ 2 k dλk pk i , = 0.

H

  (π µ (.|s)) = -a∈A π µ (a|s) log π µ (a|s)

Fig. 2 .

 2 Fig. 2. Training curves for both Experience Replay methods. The BIER agent outperforms the CER agent in both learning speed and variance. Both learning-based controllers outperform the model-based controller (i.e. OFP).

Scenario 3 :

 3 the setpoint v x is increased to the range [0.5,1.0], the current variables are the same as during training. Scenario 4: the setpoint characteristic is the same as during training but the current velocity v c is increased to [0.5,1.0] (m.s -1 ) and (h c , j c ) ∈ [-π, -π 2 [∪] π 2 , π].

Fig. 3 .

 3 Fig.3. Illustration of the evaluation performance. The BIER agent performs consistently better than the CER one despite having been trained using a reduced variety of samples.

  1. Initialize t = 0 and set initial state S0 2. At timestep t, we pick action according to At = arg max a∈A Q(St, a), with epsilongreedy strategy commonly used. 3. Execute At, then receive reward Rt+1 and transit to next state St+1.

4. Update the QValue function as:Q

  |ht) is the probability of taking action a given the history ht. The Thompson sampling strat egy is based on the Probability Matching decisionmaking strategy in the stochastic context where the probability of actions matches the probability of reward. It can be computationally intractable to estimate the posterior distributions, but Thompson sampling can still work using methods like Gibbs sampling, Laplace approximate, and the bootstraps for the approximation process.

	arg max	Qt(a))],	(2.56)
	a∈A
	where π(a;

The derivative expressed in Eq. (2.77) allows the use of Gradient Descent to optimize the parameters µ of the Policy ANN. Considering Eq. (2.72), the parameters µ are consequently optimized exactly for the desired maximum entropy objective (2.70). The soft Qupdate (2.74) guarantees that Q πnew (st, at) ≥ Q π old (st, at) and the repeated policy updates (2.77) ensure convergence toward the optimal policy π

  ∇µ log πµ(at|st) + ∇a t log πµ(at|st) -∇a t min(Q

	πµ Υ 1 (st, at), Q	πµ Υ 2 (st, at))∇µfµ(ϵt, st) . (2.77)

* (see Appendix B.2 and B.3 of [

  The state of the vehicle at the time step t denoted as xv(t) is defined by its Cartesian position and Euler orientation xv(t) = [xv yv zv ψv θv ϕv]

T (respectively roll, pitch and yaw for its orientation). The target is defined as xw(t) = [xw yw zw ϕw] T , and the values of these variables are provided by the Gazebo simulator as described in Section 2.24. We defined ow the bearing angle between the vehicle and the target as:

Table 3 . 1 :

 31 .1. The gains obtained from the modelbased optimization scheme.

	Gains	Surge	Sway	Heave	Roll	Pitch	Yaw
	Kp Ki Kd	11993.888 11993.888 11993.888 19460.069 19460.069 19460.069 321.417 321.417 321.417 2096.951 2096.951 2096.951 9077.459 9077.459 9077.459 18880.925 18880.925 18880.925

Table 3 . 2 :

 32 List of hyperparameters and their values.

	Training hyperparameter	Value
	SAC version Activation function Optimizer (all networks) Learning rate (all networks) Discount factor (γ) Minibatch size Target network smoothing coefficient (∆) 2.2.8 Update frequency (all networks) Layer Normalization [BKH16] (all networks) Replay buffer max size Replay start size Experience Replay method	1 (see Section 2.2.8) Leaky ReLU Adam [KB15] 3 × 10 -4 0.99 256 0.005 1 True 1e6 1e4 Original ER [Lin04]

Table 3 . 3 :

 33 Task performance.

	Performance criterion	MB Controller DRL Controller
	Success rate Mean dδ (m) SD of dδ (m) Mean of u	96% 3.81 3.53 541.42	86% 8.67 5.45 481.06

  2.1 but this time using a Micro Aerial Vehicle (MAV). The full description of the MAV and its simulation using Gazebo is provided in the paper in Appendix C. The state of the vehicle at the time step t denoted as xv is defined by its Cartesian position and Euler orientation xv = [xv yv zv ψv θv ϕv] T (respectively roll, pitch and yaw for its orientation). The target is defined as xw = [xw yw zw] T , and the values of these variables are provided by the simulator Gazebo. Similarly, the control objective is to minimize the Euclidean distance dt between the MAV and the target (see Section 3.2.1 for all the details on the error signals).

Table 3 . 4 :

 34 List of hyperparameters and their values.

	Training hyperparameter	Value
	SAC version Activation function Optimizer (all networks) Learning rate (all networks) Discount factor (γ) Minibatch size Target network smoothing coefficient (∆) Update frequency (all networks) Layer Normalization [BKH16] (all networks) Reward scale Automatic temperature adjustment Replay buffer max size Replay start size Experience Replay method	1 (see Section 2.2.8) Leaky ReLU Adam [KB15] 3 × 10 -4 0.99 256 0.005 (see Section 2.2.8) 1 True 40 False 1e6 1e4 CER (see Section 3.3.3)

Evaluation

The evaluation consisted in performing the target rallying mission in areas of the same environment that had never been explored by either controller during training (i.e. the wind field in these areas was totally unknown to the neural networks) with d reached = 1m. The evaluation is composed of a total of 500 episodes, different from each other in terms of target position and with a max step size per episode of 1000. The targets during the evaluation were uniformly distributed in the space defined by Λ = [Λx; Λy; Λz] T with [Λx; Λy] T ∈ [-50; -20[∪]20; 50] and Λz ∈ [2; 20]. The same set of evaluation episodes was used for each controller.

Table 3 . 5 :

 35 Control performance.

	Controller type	Mean step number Mean total reward Mean reward per step
	OFP Model-free Learning-based	488 357 281	710.797 2238.970 4034.434	1.454 6.256 14.346

Table 3 . 6 :

 36 Task performance.

	Controller type	Success rate Positive reward rate
	OFP Model-free Learning-based	50.6% 74.2% 91.6%	61.197% 86.056% 89.2%

  2. Secondly, the maximum entropy reinforcement algorithm called SAC (see Sections 2.2.8 and 2.2.8) is trained using the novel BioInspired Experience Replay (BIER) mechanism. It consists in being more selective in the choice of past interactions to use for the update of the Critic and Actor.

Table 4 . 1 :

 41 List of hyperparameters and their values.

	Training hyperparameter	Value
	SAC version Activation function Optimizer (all networks) Learning rate (all networks) Discount factor (γ) Minibatch size Target network smoothing coefficient (∆) Update frequency (all networks) Critics L2 regularization Layer Normalization [BKH16] (all networks) Reward scale Automatic temperature adjustment Replay buffer max size Replay start size Experience Replay method	1 (see Section 2.2.8) Leaky ReLU Adam [KB15] 3 × 10 -4 0.99 256 0.005 (see Section 2.2.8) 1 0.001 True 40 False 1e6 1e4 CER (see Section 3.3.3)

Table 4 . 2 :

 42 Success rate.

	Scenario	OFP DR-DAPP
	1 2 3	22% 7% 5%	88% 79% 56%

Table 4 . 3 :

 43 Mean reward per step.

	Scenario	OFP	DRDAPP
	1 2 3	9.398 1.226 0.878	15.139 12.110 6.200

Table 4 . 4 :

 44 Positive reward rate.

	Scenario	OFP	DR-DAPP
	1 2 3	75.3% 50.3% 54.0%	91.7% 87.2% 81.0%

Table 4 . 5 :

 45 List of hyperparameters and their values.

	Training hyperparameter	Value
	SAC version Activation function Optimizer (all networks) Learning rate (all networks) Discount factor (γ) Minibatch size Target network smoothing coefficient (∆) Update frequency (all networks) Critics L2 regularization Layer Normalization [BKH16] (all networks) Reward scale Automatic temperature adjustment Replay buffer max size Replay start size Experience Replay method	1 (see Section 2.2.8) Leaky ReLU Adam [KB15] 3 × 10 -4 0.99 256 0.005 (see Section 2.2.8) 1 0.001 True 40 False 1e6 1e4 CER/BIER (see Section 3.3.3)

Table 4 . 6 :

 46 Mean RMSE per step.

	Scenario	CER agent BIER agent
	Baseline 1 2 3 4 5 6	0.0364 0.0370 0.0350 0.0448 0.1483 0.1656 0.1802	0.0330 0.0366 0.0320 0.0418 0.1214 0.1508 0.1637

Table 4 . 7 :

 47 Normalized mean return.

	Scenario	CER agent	BIER agent
	Baseline 1 2 3 4 5 6	0.9104 ± 0.0461 0.9219 ± 0.0250 0.9072 ± 0.0309 0.9347 ± 0.0262 0.9108 ± 0.0456 0.9244 ± 0.0240 0.8774 ± 0.0416 0.9124 ± 0.0266 0.4078 ± 0.2965 0.5071 ± 0.2530 0.3556 ± 0.2846 0.4289 ± 0.2573 0.3238 ± 0.2219 0.3966 ± 0.2167

Table 5 . 2 :

 52 List of hyperparameters and their values for experimental validation.

	Training hyperparameter	Value
	SAC version Activation function Optimizer (all networks) Learning rate (all networks) Discount factor (γ) Minibatch size Target network smoothing coefficient (∆) Delayed update trick [FHM18] Critics L2 regularization Layer Normalization [BKH16] (all networks) Automatic temperature adjustment Replay buffer max size Replay start size Experience Replay method	2 (see Section 2.2.8) Leaky ReLU Adam [KB15] 3 × 10 -4 0.99 256 0.005 (see Section 2.2.8) True 0.001 True True 1e6 1e4 BIER (see Section 3.3.3)

Table 5 . 3 :

 53 RMSE without disturbance.

	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.1189 0.1366 0.1673 0.1433 0.1085 0.0914 0.1380 0.1311 0.1106	0.0414 0.0509 0.0546 0.0544 0.0740 0.0498 0.0534 0.0463 0.0622

Table 5 . 4 :

 54 Std RMSE without disturbance.

	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.0241 0.0325 0.0335 0.0332 0.0376 0.0247 0.0355 0.0471 0.0520	0.0147 0.0229 0.0242 0.0254 0.0295 0.0216 0.0237 0.0256 0.0285

Table 5 .

 5

		6: Normalized mean Return without disturbance.
	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.7698 0.7517 0.6934 0.7185 0.7871 0.8336 0.7353 0.7617 0.7785	0.8992 0.8815 0.8740 0.8738 0.8390 0.8863 0.8769 0.8929 0.8595

Table 5 . 7 :

 57 RMSE with disturbance.

	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.3723 0.3587 0.3420 0.3645 0.3686 0.3647 0.3648 0.3606 0.3826	0.0882 0.0844 0.0731 0.0783 0.1120 0.0797 0.1149 0.1385 0.1100

Table 5 . 8 :

 58 Std RMSE with disturbance.

	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.1573 0.0947 0.0888 0.1068 0.0893 0.1005 0.0969 0.0987 0.1206	0.0289 0.0330 0.0285 0.0294 0.0451 0.0277 0.0319 0.0530 0.0444

Table 5 .

 5

		10: Normalized mean Return with disturbance.
	Setpoint	Model-based Learning-based
	1 2 3 4 5 6 7 8 9	0.5260 0.6642 0.6648 0.6309 0.5326 0.5646 0.5461 0.6478 0.4831	0.7972 0.8008 0.8384 0.8348 0.7753 0.8309 0.7752 0.7486 0.7840

  -1 ; φ ; θ ; ψ ; v x ; v y ; v z ; ω φ ; ω θ ; ω ψ ; ζ t ; e t ; d t ] (43) where a t-1 are the last actions performed ; [φ; θ; ψ] are the Euler angles of the robot (roll, pitch and yaw) ; [v x ; v y ; v z ] and [ω φ ; ω θ ; ω ψ ] are its linear and angular velocities respectively in R W and R B ; ζ t represents its position in R W (obtained from its embedded odometry sensor), e t = [e x ; e y ; e z ] are the current errors on the set-points in terms of Euclidean distance and d t is the current Euclidean distance between the hexacopter and the target in R W . All the variables involved are assumed to be measured, their estimation is out of the scope of this work. We denote the target position by Λ = [Λ x ; Λ y ; Λ z ] T ∈ R W . The dimension of this observation vector is19 and it has been standardized to have zero mean and a variance of 1. In order to provide a higher time horizon to the agent, we constructed the state vector out of the current and past two observations vectors [o t ; o t-1 ; o t-2 ]. For the purpose of providing the agent a sense of "velocity" in the evolution of the state, we consider the two by two difference of these vectors such as vel t = (o t -o t-1 ) and vel t-1 = (o t-1 -o t-2

  -1 ; τ rpt ; pid rpt ; k p roll ; k i roll ; k d roll k p pitch ; k i pitch ; k d pitch ; k p thrust ; k i thrust ; k d thrust ; φ ; θ ; ψ ; v x ; v y ; v z ; ω φ ; ω θ ; ω ψ ; ζ t ; e t ; d t ]

	(48)

Table 1 :

 1 Evaluation results.

	Controller type	Mean step number	Mean total reward	Mean reward per step	Success rate	Positive reward rate
	Fixed Poles PID	488	710.797	1.454	50.6%	61.197%
	Model-free	357	2238.970	6.256	74.2%	86.056%
	Learning-based	281	4034.434	14.346	91.6%	89.2%

  26) networks. A fifth network, a separate target State-Value network that slowly tracks the actual State-Value network is employed and updated using an exponentially moving average with a smoothing constant Υ = 5 × 10 -3 as in [Lillicrap et al. (2016), Mnih et al. (2015)]. Because value estimates tend to overestimate when the policy is poor, and the policy tends to be poor if the value estimate itself is inexact, we delay their updates by updating the target State-Value and Policy networks after a fixed number of updates d = 2 to the critics as proposed in [Fujimoto et al. (2018)]. (2) Batch normalization [Ioffe and Szegedy (2015)] is used before the ReLU6 layers of each networks as suggested in [Lillicrap et al. (2016)].

	). Four of these include a Policy network, two Q-value networks and a State-Value network. They respectively take as input, the state (s The networks are trained simultaneously using Gradient descent and Adam [Kingma and Ba (2015)] as optimizer. Each network is composed of two hidden layers of 256 hidden units. The ReLU6 [Krizhevsky (2010)] activation function is applied to all the hidden layers, Tanh is applied to the output layer of the Policy network while no activation functions are applied to the final layers of the critics. The learning rate is fixed and equal to 3 × 10 -4 for all networks [Haarnoja et al. (2018)] and with γ = 0.99 for (24). The following addition to the first version of the SAC algorithm has been made: (1)

t ), the pair of state and actions (s t , a t ), and the state (s t ) vectors. The two Q-Value networks are used to reduce the overestimation bias of Actor-Critic methods [Hasselt et al. (2016), Fujimoto et al. (2018)]. The minimum between the two Q-Value estimates is used to compute the losses of the State-Value (25) and Policy

(

  Learning Stochastic Adaptive Control using a Bio-Inspired Learning Stochastic Adaptive Control using a Bio-Inspired

	Learning Stochastic Adaptive Control using
	a Bio-Inspired Experience Replay
	Experience Replay Thomas Chaffre, Paulo E. Santos, Gilles Le Chenadec, Experience Replay Estelle Chauveau, Karl Sammut, Senior Member, IEEE, Benoit Clement
	This paper was downloaded from TechRxiv (https://www.techrxiv.org).
	LICENSE
	CC BY 4.0
	SUBMISSION DATE / POSTED DATE
	03-03-2022 / 08-03-2022
	CITATION
	CHAFFRE, Thomas; SANTOS, paulo; LE CHENADEC, Gilles; CHAUVEAU, Estelle; CLEMENT, Benoit;
	Sammut, Karl (2022): Learning Stochastic Adaptive Control using a Bio-Inspired Experience Replay.
	TechRxiv. Preprint. https://doi.org/10.36227/techrxiv.19297577.v1
	DOI
	10.36227/techrxiv.19297577.v1

  a t |s t ) + ∇ at log π µ (a t |s t ) -∇ at min Q πµ Υ1 (s t , a t ), Q πµ Υ2 (s t , a t ) ∇ µ f µ (ϵ t , s t ) .

TABLE I EVALUATION

 I RESULTS FOR THE CER METHOD (LEFT) AND THE PROPOSED BIER METHOD (RIGHT).

	Scenario	mean RMSE per step	normalized mean return	mean RMSE per step	normalized mean return
	Baseline	0.0364	0.9104 ± 0.0461	0.0330	0.9219 ± 0.0250
	1	0.0370	0.9072 ± 0.0309	0.0366	0.9347 ± 0.0262
	2	0.0350	0.9108 ± 0.0456	0.0320	0.9244 ± 0.0240
	3	0.0448	0.8774 ± 0.0416	0.0418	0.9124 ± 0.0266
	4	0.1483	0.4078 ± 0.2965	0.1214	0.5071 ± 0.2530
	5	0.1656	0.3556 ± 0.2846	0.1508	0.4289 ± 0.2573
	6	0.1802	0.3238 ± 0.2219	0.1637	0.3966 ± 0.2167

See appendix C of[START_REF] Haarnoja | Soft ActorCritic: OffPolicy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] for more details on the change of variable applied to bound the Gaussian distributions.

* Here the terms model-based and model-free do not correspond to the ones used in II-A but refer to whether or not a model of the environment is given to the RL algorithm to improve the policy whereas in automation, it is the model of the controlled dynamical system that is provided.

⋆ This work was supported by ISblue project, Interdisciplinary graduate school for the blue planet (ANR-17-EURE-0015) and co-funded by a grant from the French government under the program "Investissements d'Avenir".

In this paper, x refers to state in terms of state space representation and s in regard to the context of machine learning because although sharing the same title, these entities do not hold the same nature.

Preprints, IFAC CAMS 2022 Kgs. Lyngby, Denmark. September 14-16, 2022 © 2022 the authors. Accepted by IFAC for publication under a Creative Commons License CC-BY-NC-ND

https://uuvsimulator.github.io

See appendix C of[18] for more details on the change of variable applied to bound the Gaussian distributions

Remerciements

Acknowledgements The authors thank Dr. Estelle Chauveau from Naval Group for the help provided. This work was supported by SENI, the research laboratory between Naval Group and ENSTA Bretagne.

ACKNOWLEDGEMENTS

This study was supported by SENI, the joint research laboratory between NAVAL GROUP and ENSTA Bretagne.

Methodology

In this Section, we present how to build a model-free and learning-based controller with a DRL framework composed of two components: the Soft Actor-Critic algorithm and the Experience Replay technique. Later, we present how to apply it to the model-free and learning-based control schemes.

Soft Actor-Critic

In this study, we used the same policy gradient algorithm as in our previous paper [12] to either maximize the RL-based cost function of a model-free controller (see Section 2.2) or to estimate the controller parameters of a learning-based controller (as described in Section 2.3). The Soft Actor-Critic (SAC) algorithm proposed by T. Haarnoja et al. in [11] is based on three concepts which we formally define thereafter.

First, the SAC algorithm uses an actor-critic architecture which concurrently learns both State value and Q-value functions and the policy π θ (a t |s t ) as well. During the training stage, the critic part updates parameters of the State or Q-value functions while the actor network updates the policy parameters in the direction suggested by the critic. Note that a recent version of the SAC algorithm [44] allows to only estimate the Q-value function, while we used in this work the original version of the algorithm [11] where the State value function is also estimated to help stabilizing the overall training process. Precisely, our implementation of the SAC algorithm aims at iteratively learning three functions modelized by three NNs:

a policy function π θ (a t |s t ) parameterized by the NN weights θ, -a soft Q-value function Q w (s t , a t ) parameterized by the NN weights w, -and a soft State value function V Ψ (s t ) parameterized by the NN weights Ψ .

Beyond their modelization, these soft Q-value and State value functions are induced by the policy π θ and defined as follows:

The terms ρ π (s) and ρ π (s, a) denote the state and the state-action marginals of the state distribution induced by the policy π θ (a|s). The NNs parameters w and Ψ of the Q-value and State value functions respectively, are the solutions minimizing the soft Bellman residual J Q (w) and the mean squared error J V (Ψ ): the positive reward rate (i.e. the rate of actions that made the error decreased) computed over all episodes for each scenario. We can see that the optimal PID is not able to complete the task despite its ability at keeping the error very low. The numerous variation in the process makes the task of maintaining the error within a threshold extremely hard for a controller with fixed parameters. Our method, on the other hand demonstrates better results with a higher success rate and mean reward per step especially for scenario 1 and 2. However, the performance of our controller is limited on scenario 3 with a success rate almost equal to 50%. This could be explained by the fact that during training, the robot was facing constant currents and setpoints only and therefore failed to build a policy that can complete the task when they both vary.

CONCLUSIONS

In this paper we have presented a direct tuning scheme for the adaptive control of an AUV under unobservable disturbances. We proposed a new application of the SAC algorithm to perform adaptive Pole-Placement. Although the stability analysis is limited to the controller, our mapping provides some guarantees on the outputs of the neural network, which is often highlighted by the control community as a major concern toward the use of such methods in the real world. Finally, we demonstrated the benefits of our approach with simulated comparison to the model-based optimal counterpart of our control structure.