Prof. I Alejandro Serrano-Maldonado

Dr Trung Dang-Bao

Dr Isabelle Favier

Itzel Guerrero-Ríos
email: itzelgr@unam.mx

Prof Daniel M Pla
email: pla@lhfa.fr

Montserrat Gómez
email: gomez@chimie.ups-tlse.fr

Prof Erika Martin

Guerrero-Ríos

Gómez

Glycerol boosted Rh-catalyzed hydroaminomethylation reaction, a mechanistic insight

Keywords: Hydroaminomethylation, Rhodium, Glycerol, Hydrogen transfer, Rh-catalyzed condensation

of selected organic compounds ( 1 H and 13 C NMR, MS).

Introduction

Among nitrogen compounds, amines play an important role as feedstock chemicals with a production on million-ton scale annually in the synthesis of agrochemicals, pharmaceuticals, solvents, dyes, monomers for polymerizations and functional materials, among the most important applications. [1] Despite the production of amines often requires expensive and sometimes toxic substances through conventional methods, hydroaminomethylation (HAM) of alkenes offers an alternative synthesis from an environmental and economic point of view. [2] HAM is a multi-component tandem transformation that consists of three consecutive steps: the metal-catalyzed hydroformylation of an alkene, followed by condensation of the resulting aldehyde with an amine leading to an enamine (or imine/iminium) intermediate that undergoes metal-catalyzed hydrogenation, giving water as the only concomitant product. [2a-g, 3] HAM is very attractive from an atom economy viewpoint, however it often requires harsh conditions, in particular using Rh-based systems, [2a-g] despite its well-known ability to catalyze hydroformylation processes. [4] In order to overcome this drawback, it is essential to design catalytic systems able to promote both hydroformylation and hydrogenation steps under relative mild conditions. In the literature, this approach has been considered and systems involving two catalysts have been described. Actually, Xiao [5] and Han [6] have employed a Rhbased catalyst together with an optically pure phosphoric acid as organocatalyst promoting imine hydrogenation towards the synthesis of chiral secondary amines, from styrene and aniline derivatives under mild conditions (total pressure : 2 -11 bar; 25 -50 °C; 16 -72 h). For the synthesis of linear amines, Beller and coworkers reported a catalytic system constituted by Rh(I) and Ir(I) precursors (Ir/Rh ratio ca. 10/1) in the presence of an excess of TPPTS, taking advantage of the known performance of Ir-based systems in the hydrogenation of imines and enamines, [7] leading to primary amines from terminal alkenes and ammonia (total pressure 78 bar; 110 -130 °C; 5 -10 h) with high regioselectivity (linear amines) and good chemoselectivity (primary amines) (Scheme 1a). [8] More recently, Hartwig's group has also reported a Rh/Ir catalytic system constituted by [Rh(acac)(CO)2]/BISBI (BISBI = 2,2'bis(diphenylphosphinomethylene)-1,1'-biphenyl) and Xiao's catalyst, an Ir(III) organometallic complex stable in the presence of carbon monoxide; [9] this catalytic system permits to work under milder conditions (total pressure 3.4 bar; 80 °C; 20 h), producing secondary linear amines from good to high yields, using a mixture of solvents (toluene/methanol) and an aqueous buffer solution of NaHCO2 (Scheme 1b). Notably, the role of bidentate ligands, such as the diphosphine BISBI, has proven to be key for such high regioselectivities. [10] Based on these precedents and taking into account the properties of glycerol as solvent in a wide range of transformations, [11] mainly involving metal-catalyzed homogeneous processes, [12] but also metal nanoparticles [13], [14] and catalyst-free reactions, [15] we envisaged to explore the role of glycerol as hydrogen donor in the HAM reaction of terminal olefins catalyzed by [Rh(acac)(CO)2]/TPPTS system through enhancing the reductive amination, which often is the rate limiting step of the HAM reaction (Scheme 1c). [16] This contribution aims to provide a mechanistic insight, in particular related to the role of rhodium in the amine/aldehyde condensation step and the hydrogenation of the resulting enamine, in glycerol under mild reaction conditions. Scheme 1. Strategies to conduct the multi-step HAM reaction under tailored conditions for the synthesis of primary amines.

Results and Discussion

Based on the established HAM mechanism, terminal alkenes are initially transformed into the corresponding linear and branched aldehydes by hydroformylation, which then react with an amine to produce the corresponding enamine by a condensation reaction. Hydrogenation of these latter intermediates affords the expected amines, provided that sidereactions such as alkene isomerization or hydrogenation are precluded (Scheme 2, illustrating the case of oct-1-ene and morpholine). We chose the HAM reaction of oct-1-ene (1) and morpholine (a) catalyzed by [Rh(acac)(CO)2]/TPPTS in glycerol as benchmark reaction (Scheme 2). Under optimized conditions, high chemoselectivity was obtained towards the formation of amines 1a at 120 °C and 10 bar (total pressure) of an equimolar CO:H2 mixture, only observing minor alkene isomerization (less than 10%; see entry 1 in Table 1). Notably, the use of the diphosphine BISBI instead of TPPTS exhibited lower reactivity (50% conversion) and no selectivity enhancement (l-1a / b-1a, 70/30). The reaction optimization survey in terms of temperature, pressure, time, Rh/phosphine ratio and solvent (for further details, see Tables S1 andS2 in the Supporting Information) raised concerns on crucial solvent effects that merit a special discussion.

Scheme 2. Hydroaminomethylation benchmark reaction of Rh-catalyzed oct-1-ene (1) and morpholine (a). By-products from 1 are also indicated (1H and 1l).

When toluene or dioxane were used, full conversion was observed (entries 2 and 3, Table 1), showing moderate chemoselectivity towards the HAM product due to competing olefin isomerization (mixture of internal alkenes) and substrate hydrogenation, albeit regioselectivity was similar for both solvents, leading to ca. 3:2 linear to branched ratio of 1a. Despite the high conversion and high chemoselectivity towards the formation of amines obtained in cyclopentyl methyl ether (CPME), the reaction was not regioselective (entry 4, Table 1). Given the reported efficiency of alcohols in promoting HAM reactions, [17] we sought to evaluate the effect of other protic solvents than glycerol. Whereas the use of water (entry 5, Table 1) led to an inactive catalytic system, both MeOH and butan-1-ol presented higher olefin isomerization (entries 6 and 7, Table 1) than glycerol (entry 1, Table 1). Thus, glycerol provided the best results in terms of chemo-and regioselectivity towards the formation of 1a, outperforming both non-protic and other protic solvents in terms of deleterious olefin isomerization/hydrogenation side-reactions.

Table 1. Rh-catalyzed HAM of oct-1-ene and morpholine in different solvents.

[a]

Entry Solvent

Conv.

(%) [b] Selectivity [b] 1I (%) 1H (%) 1a''(%) 1a (%) l-1a/b-1a Taking into account the remarkable effect of glycerol and given the fact that hydrogenation of the imine-, iminium-or enaminetype intermediates is generally accepted as the rate determining step for Rh-catalyzed HAM, [2a, 18] we considered that a hydrogen transfer (glycerol as reducing agent) could be operative in addition to the hydrogenation reaction, [11e] enabling low pressure conditions while overriding the need for a metal-based cocatalyst. [9] Thus, the nature of the hydrogen donor was further assessed with glycerol and butan-1-ol at 80 ºC, revealing that enamine reduction was faster in the latter albeit with moderate 1a selectivity (entries 1 and 2, Table 2). The effect of glycerol in boosting enamine reduction was observed when a solvent mixture of glycerol and butan-1-ol (0.25:0.75) was used, yielding a quantitative conversion and full chemoselectivity towards 1a, showing a slight decrease in terms of regioselectivity (entries 1 and 2 vs 3, Table 2).

Table 2. Rh-catalyzed enamine reduction studies of oct-1-ene and morpholine benchmark reaction in glycerol, butan-1-ol and a mixture of both.

[a]

Entry Solvent (glycerol:butan-1-ol)

Conv. (%) b

Selectivity [b] 1a''(%) 1a (%) l-1a/b-1a With the aim of evidencing the plausible role of glycerol as reducing agent, the reaction of 4-[(E)-non-1-en-1-yl]morpholine intermediate (l-1a'') [19] in glycerol under inert atmosphere, i.e. in the absence of molecular H2, was carried out (Scheme 3a). Interestingly, up to 67% conversion of l-1a'' to l-1a was obtained. This reactivity was confirmed by the hydrogen transfer of a tertiary enamine such as 4-(1-cyclohexen-1-yl)morpholine, yielding full conversion towards 4-cyclohexylmorpholine (Scheme 3b). Despite the ability of glycerol to reduce enamines, [START_REF]See the following reference for the non-catalyzed[END_REF] l-1a'' was only slightly reduced in the absence of catalyst (less than 5%), proving that enamine hydrogen transfer is catalyzed by rhodium. We next investigated whether the in-situ generated aldehydes in the initial hydroformylation step of the HAM process could also undergo hydrogen transfer processes under the optimized reaction conditions. Thus, a control test with nonanal (l-1') was carried out but no formation of nonan-1-ol was observed; alternatively, acetals from the reaction of the aldehyde and glycerol were obtained, because acetalization is probably faster than hydrogen transfer reaction. [21] We then focused on the reactivity of aldehydes towards enamine formation and subsequent reduction to the desired products, while avoiding aldol condensation pathways (Table Accordingly, the control tests were carried out. First, the condensation between nonanal (l-1') and morpholine (a) using a 1/1.5 molar ratio of reagents under inert atmosphere at 120 °C in the presence of catalyst, afforded full conversion but mainly yielding the aldol condensation product (62%) together with only 38% of the desired l-1a product (coming from the reduction of enamine l-1a'' by glycerol as discussed above; entry 1, Table 3). Under H2/CO pressure, the formation of l-1a was slightly improved (entry 2, Table 3). However, in the absence of catalyst, the aldol product was favored (up to 88%; entry 3, Table 3), indicating the benefic role of the Rh/TPPTS system in terms of chemoselectivity. The selectivity towards the desired amine substantially enhanced when a large excess of morpholine was used (l-1'/a ratio of 0.1/3), leading up to 71% of l-1a (entry 4, Table 3). These latter conditions, but under H2/CO pressure, gave 98% of l-1a (entry 5, Table 3); in the absence of catalyst, aldol condensation products were obtained up to 49% yield (entry 6, Table 3), in agreement with the trend observed using a lower excess of morpholine (entry 3, Table 3). These results proved that both a low concentration of aldehyde (probably shifting the equilibrium towards the formation of enamine l-1a'' by the amine excess) and the presence of the rhodium catalyst are necessary in order to prevent the formation of aldol condensation products, as well as the reducing effect of glycerol facilitating the enamine reduction. Consequently, under catalytic conditions, the condensation step must be fast enough to avoid the accumulation of aldehydes throughout reaction; in this step, rhodium plays a crucial role in the chemoselectivity, hampering the formation of aldol condensation products. This reactivity behavior was corroborated by the GC-MS monitoring of the benchmark reaction under catalytic conditions (see below). Selectivity [b] l-1a (%) aldol byproduct (%) [a] General conditions: nonanal (l-1'), morpholine (a), 1 mol% of [Rh(acac)(CO)2] and 2 mol% of TPPTS unless otherwise noted, 2 mL of glycerol, CO (5 bar) : H2 (5 bar) unless otherwise noted, 120 °C, 6 h; see Table S3 in the Supporting Information for full details. [b] Conversion (full conversion observed for the experiments indicated in the table), chemo-and regio-selectivity were determined by 1 H NMR and GC using n-dodecane as internal standard; n.d., not determined. [c] 5% of the enamine l-1a" was observed. [d] 71% conversion. [d] The aldol condensation product (35%) further condenses with morpholine to yield a by-product in 14% (m/z 335).

Despite high pressures are usually required for hydroaminomethylation, [4b, 4c] the methodology described herein permitted to work under relative low pressure. Notably, in the absence of morpholine, the hydroformylation of oct-1-ene towards l-1' and b-1' in glycerol only worked under high pressure (25 bar H2 / 25 bar CO), favoring the formation of the branched regioisomer (l-1'/b-1' ratio of 36/64, see Table S4 in the Supporting Information). Thus, this behavior suggests that the amine is involved in the hydroformylation step, [4b, 22] probably by coordination to the rhodium center, modifying the energy of intermediates and transition states [START_REF] Petrucci | For Rh(I) and Rh(III) complexes containing morpholine, see: a)[END_REF] and in consequence influencing the rate and selectivity of the hydroformylation reaction. [24] In order to assess the organic compounds present throughout the HAM reaction, reaction monitoring studies were performed by GC-MS and FTIR (Figure 1 and Figure S3 in the Supporting Information, respectively). The GC-MS study evidenced that the aldehyde intermediates (l-1' and b-1') arising from olefin hydroformylation, were present at rather low concentrations throughout the reaction in agreement with the behavior observed for the condensation step (see above). Thus, after 30 min of reaction, the molar fraction of enamine intermediates 1a'' is consistently higher than the corresponding aldehyde precursors 1', indicating that the reduction of the former represents the ratedetermining step for the overall HAM reaction. Despite the operando FTIR method did not permit the detection of organometallic intermediates or aldehydes under catalytic conditions due to their low concentration (see Figure S3 in the Supporting Information), the reaction profiles of oct-1-ene ( 1) conversion (disappearance of the C=C stretching band at 1641 cm -1 ), together with the presence of a transient enamine intermediate 1a'' (967 cm -1 band formation and decay) and the formation of the amine 1a (C-N stretching band at 1031 cm -1 increase) are consistent with GC monitoring data (Figure 1) and 1 H NMR analysis of the reaction mixture after the 24 h operando FTIR monitoring (see Figure S4 in the Supporting Information; for a control FTIR monitoring, see Figure S5). For detailed data, see Table S5 in the Supporting Information.

Alternative mechanistic HAM approaches consisting on the direct C-C coupling of oct-1-ene with 4,4'methylenedimorpholine aminal, N-formylmorpholine or N-(methylene)morpholinium chloride did not work under the optimized reaction conditions. Despite Rh nanoparticles were detected after the reaction by TEM (see Figure S6 in the Supporting Information), preformed Rh nanoparticles were totally inactive in the HAM benchmark reaction (see Figure S7 in the Supporting Information), indicating that hydroaminomethylation operates under a molecular regime. Entry Alkene Amine Conv.

(%) [b] Selectivity [b] 1x (%) [c] l-1x/b-1x 8) [d] 60/40 [g] 6 1 80 71 ( 10) [e] 85/15 ) [h] 80/20 9 4 >99 97 (n.d.) [i][j] 80/20

[a] General conditions: 1 mmol of alkene (1 -4), 1.5 mmol of amine (ag), 1 mol% of [Rh(acac)(CO)2], 2 mol% of TPPTS, 2 mL of glycerol, 5 bar CO / 5 bar H2 (pressure at room temperature), 120 °C, 6 h. [b] Conversion, chemoand regio-selectivity were determined by GC using n-dodecane as internal standard. [c] In brackets, percentage of internal alkenes from oct-1-ene. [d] 14% of tertiary amine was formed. [e] 19% of tertiary amine was formed. [f] Linear/branched ratio determined by 1 H and 13 C NMR. [g] Linear/branched for the tertiary amine was not determined. [h] 4% 4-chloroethylbenzene.

[i] 3% 4ethylanisole.

[j] Reaction conditions: 1 mmol of 4-vinylanisole, 1.5 mmol of morpholine, 1 mol% [Rh(acac)(CO)2], 1 mol% of TPPTS in 2 mL of glycerol, 5 bar CO / 15 bar H2 (pressure at room temperature), 120 ºC, 10 h.

With the optimized experimental conditions for the benchmark reaction, the HAM reaction scope was evaluated using primary and secondary amines, including aniline derivatives (ag), as well as different terminal alkenes (1 -4) giving high conversions towards the expected products (75 -94% conversion, Table 4). Concerning chemoselectivity, for secondary amines, the expected amine product was mainly obtained (entries 1-4, Table 4). However, with cyclohexylamine and aniline (entries 5-6, Table 4), ca. 15 -20% of tertiary amine was formed together with the expected secondary amine products, due to the highest reactivity of secondary amines, thus competing with the primary amine substrates. Modifying the ratio oct-1-ene/aniline (1/0.5, 1/1.5, 1/2), the selectivity was not improved; as expected, a higher amount of aniline favored the formation of bis-(octyl)phenyl amine (up to 33%, see Table S6 in the Supporting Information).

Concerning regioselectivity, the linear isomer was the major product in all cases, with a maximum l/b ratio of 85/15 (entries 5 and 7, Table 4). Ammonia-based reagents (NH3(aq), NH4OAc) and sulfonamide derivatives (methylsulfonamide, ptolylsulfonamide) did not react under these reaction conditions. Unfortunately, di-and trisubstituted alkenes (cyclooctene, ethyloleate, -pinene, -methylstyrene, 4-octene, limonene) did not undergo HAM (conversions lower than 20%, see Table S7 in the Supporting Information). Furthermore, aryl-substituted alkenes, such as 4-allylanisole (2), 4-chlorostyrene (3) and 4-methoxystyrene (4) showed good to excellent conversions towards HAM, together with high linear-tobranched outcomes (entries 7-9, Table 4). These results are particularly interesting for styrene derivatives, because the formation of the branched regioisomer in the Rh-catalyzed hydroformylation step is generally favored; [2a] however, the nature of ligand and solvent triggers a non-innocent effect on the regioselectivity, reversing this trend as proven by Zhang's group; they obtained the best linear/branched ratio using a tetraphosphorus-based ligand and tert-amyl alcohol as solvent. [25]

Conclusion

This contribution sheds light on the mechanism of multi-step Rhcatalyzed hydroaminomethylation in glycerol. For the benchmark reaction (oct-1-ene and morpholine as reagents), we evidenced paramount solvent effects in the reactivity of Rh-based catalytic systems, revealing the role of glycerol as reducing agent. Actually, we demonstrated that both Rh-catalyzed hydrogenation and hydrogen transfer processes cooperate to favor enamine reduction under relative low pressure (5 bar H2 / 5 bar CO). Moreover, we have shown that the enamine formation proceeds via a chemoselective condensation promoted by rhodium; under these conditions aldol reactions are precluded in contrast to the non-catalyzed organic condensation. The results described herein highlight both the fast rhodium-promoted condensation of the aldehyde with the amine to selectively form the enamine intermediate, as well as the role of glycerol as a hydrogen transfer agent, boosting the Rh-catalyzed reduction of this transient enamines and thus permitting to work at relative low pressure. The reaction scope of terminal alkenes and amines showed the versatility of the process exhibiting excellent chemoselectivity towards the formation of secondary and tertiary amines, with high linear-to-branched regioselectivity, including styrene derivatives (up to 85:15). Overall, the results reported herein stress the fact that glycerol enables a Rh-based catalytic system working under molecular regime, to perform one-pot HAM cascade reaction to reach a reactivity behavior that was thus far limited to multi-catalytic systems, [5][6][8][9] in particular those based on Rh/Ir bimetallic catalysts for the synthesis of linear amines. [9] Experimental Section General procedure for catalytic hydroaminomethylation. In a stainless steel reactor vessel (100 mL) were placed the desire amount of [Rh(acac)(CO)2] and tris(3-sulfophenyl)phosphine trisodium salt (TPPTS) together with glycerol (2 mL), the selected olefin (1 mmol), the selected secondary or primary amine (1.5 mmol) and n-dodecane as internal standard (1 mmol). The reactor was pressurized with the selected proportion of CO/H2 at the desire pressure, temperature and time. After the reaction was completed, the reactor vessel was cooled to room temperature and consecutively depressurized. Products were extracted with cyclohexane (5 x 3 mL), filtered through a Celite column and analyzed by gas chromatography to determine conversion and selectivity. Products were isolated and characterized by 1 H and 13 C NMR, and MS. The Rh content of extracted organic products was in the range of 0.01-0.08 ppm as determined by ICP-AES analyses.

Synthesis of Rh nanoparticles. In a Fisher-Porter bottle, [Rh(acac)(CO)2] (13.0 mg, 0.05 mmol) and TPPTS (28.5 mg, 0.05 mmol) were dissolved in 10 mL of glycerol at room temperature under Ar. The system was then pressurized with 3 bar of H2 and stirred at 80 °C during 18 h. The Fisher-Porter bottle was cooled down to room temperature and depressurized. The resulting black homogeneous solution of dispersed nanoparticles was dried under vacuum (0.05 mmHg) at 80 °C for 18 h to remove volatile organic compounds. The as-prepared Rh nanoparticles were characterized by TEM (see Figure S7 in the Supporting Information).

Scheme 3 .

 3 Scheme 3. Rh-catalyzed hydrogen transfer reaction of 4-[(E)-non-1-en-1yl]morpholine (a) and 4-(1-cyclohexen-1-yl)morpholine (b) in glycerol (for 1 H NMR crude analysis, see Figure S1 in the Supporting Information).

Figure 1 .

 1 Figure 1. Monitoring by GC-MS of the hydroaminomethylation of oct-1-ene (1) and morpholine (a) catalyzed by Rh(I)/TPPTS catalyst in glycerol. Reaction conditions: 1 mmol of 1, 1.5 mmol of a, 1 mol% of [Rh(acac)(CO)2], 2 mol% of TPPTS, 2 mL of glycerol, CO (5 bar)/H2 (5 bar), 120 °C. Molar fraction of products (xi) was determined by GC using n-dodecane as internal standard.For detailed data, see TableS5in the Supporting Information.

Table 3 .

 3 Reductive amination controls in glycerol.

	[a]

Table 4 .

 4 Reaction scope of Rh-catalyzed HAM in glycerol.

	[a]

Acknowledgements

The Centre National de la Recherche Scientifique (CNRS) and the Université Toulouse 3 -Paul Sabatier are gratefully acknowledged for their financial support. The French-Mexican International Associated Laboratory "Laboratoire de Chimie Moléculaire avec applications dans les Matériaux et la Catalyse" (LCMMC) is acknowledged for financial support (project number 69-2018). The work was also supported by PhD fellowships to A. S. M. (CONACyT) and T. D.-B. (Ministry of Education and Training of the Socialist Republic of Vietnam and CNRS). The authors thank Christian Pradel for TEM analyses and Corinne Routaboul (ICT, Toulouse) for helpful assistance with the infrared experiments with the ReactIR.

Entry for the Table of Contents

A multitalented monometallic Rh-based catalytic system in glycerol enables tandem hydroaminomethylation reaction under a molecular regime, via low pressure hydroformylation, Lewis acid-assisted condensation and enamine reduction boosted by hydrogen transfer.

Twitter: https://twitter.com/cat_tlse


domain: info.info-rb info.info-ai spi phys math info

text:
Thomas Chaffre

Julien Moras
email: julien.moras@onera.fr

Adrien Chan-Hon-Tong
email: adrien.chanhontong@onera.fr

Julien Marzat
email: julien.marzat@onera.fr

Sim-to-Real Transfer with Incremental Environment Complexity for Reinforcement Learning of Depth-based Robot Navigation

Keywords: Reinforcement Learning, Sim-to-Real Transfer, Autonomous Robot Navigation

Transferring learning-based models to the real world remains one of the hardest problems in model-free control theory. Due to the cost of data collection on a real robot and the limited sample efficiency of Deep Reinforcement Learning algorithms, models are usually trained in a simulator which theoretically provides an infinite amount of data. Despite offering unbounded trial and error runs, the reality gap between simulation and the physical world brings little guarantee about the policy behavior in real operation. Depending on the problem, expensive real fine-tuning and/or a complex domain randomization strategy may be required to produce a relevant policy. In this paper, a Soft-Actor Critic (SAC) training strategy using incremental environment complexity is proposed to drastically reduce the need for additional training in the real world. The application addressed is depth-based mapless navigation, where a mobile robot should reach a given waypoint in a cluttered environment with no prior mapping information. Experimental results in simulated and real environments are presented to assess quantitatively the efficiency of the proposed approach, which demonstrated a success rate twice higher than a naive strategy.

INTRODUCTION

State-of-the-art algorithms are nowadays able to provide solutions to most elementary robotic problems like exploration, mapless navigation or Simultaneous Localization And Mapping (SLAM), under reasonable assumptions [START_REF] Cadena | Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age[END_REF]. However, robotic pipelines are usually an assembly of several modules, each one dealing with an elementary function (e.g. control, planning, localization, mapping) dedicated to one technical aspect of the task. Each of these modules usually requires expert knowledge to be integrated, calibrated, and tuned. Combining several elementary functions into a single grey box module is a challenge but is an extremely interesting alternative in order to reduce calibration needs or expertise dependency. Some of the elementary functions can raise issues in hard cases (e.g., computer vision in weakly textured environment or varying illumination conditions). Splitting the system into a nearly optimal control module processing a coarse computer vision mapping output may result in a poorer pipeline than directly using a map-and-command module which could achieve a better performance trade-off.

For this reason, there is a large academic effort to try to combine several robotic functions into learningbased modules, in particular using a deep reinforcement strategy as in [START_REF] Zamora | Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo[END_REF]. A limitation of this approach is that the resulting module is task-dependent, thus usually not reusable for other purposes even if this could be moderated by multitask learning. A more serious limit is that learning such a function requires a large amount of trial and error. Training entirely with real robots is consequently unrealistic in practice considering the required time (even omitting physical safety of the platform during such learning process where starting behavior is almost random). On the other hand, due to the reality gap between the simulator and the real world, a policy trained exclusively in simulation is most likely to fail in real conditions [START_REF] Dulac-Arnold | Challenges of real-world reinforcement learning[END_REF]. Hence, depending on the problem, expensive real fine-tuning, and/or a complex domain randomization strategy may be required to produce a relevant policy. The explainability and evaluation of safety guarantees provided by such learning approaches compared to conventional methods also remain challenging issues [START_REF] Juozapaitis | Explainable reinforcement learning via reward decomposition[END_REF].

In this work, we address the problem of robot navigation in an uncharted cluttered area with Deep Reinforcement Learning. In this context we consider as a benchmark the task where a robotic agent (here a Wifibot Lab v4 mobile robot, see Figure 1) has to reach a given target position in a cluttered room. At the beginning of each episode, the robot starts at the same location but with a random orientation and gets the target position coordinates. The robot is equipped with a perception sensor providing a dense depth map (here an Intel RealSense D435), it has access to its current position in the environment and has to control the speed of its wheels (which are the continuous outputs of the proposed learning algorithm). The proposed training method is based on the Soft Actor-Critic method [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] coupled with incremental environment complexity. The latter refers to a technique which consists in splitting the desired mission requirements into several environments, each one representing a different degree of complexity of the global mission. Furthermore, an experimental setup is also proposed in this paper for testing or learning continuation in the real environment without human supervision. The paper is organized as follows. Section 2 presents related work in robotic navigation and reinforcement learning. Section 3 details the proposed method, particularly the agent structure, the reward generation and the learning procedure. Finally, experiments and results in both simulated and real environments are presented in Section 4.

RELATED WORK

Classical methods for robot autonomous navigation are usually based on a set of several model-based blocks that perform SLAM [START_REF] Mur-Artal | ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras[END_REF][START_REF] Engel | Large-scale direct SLAM with stereo cameras[END_REF][START_REF] Wurm | Octomap: A probabilistic, flexible, and compact 3d map representation for robotic systems[END_REF] and waypoint navigation algorithms [START_REF] S ¸ucan | The Open Motion Planning Library[END_REF][START_REF] Kamel | Model predictive control for trajectory tracking of unmanned aerial vehicles using robot operating system[END_REF][START_REF] Bak | Path following mobile robot in the presence of velocity constraints[END_REF]. The latter use either reactive methods, which are limited in horizon and sometimes trapped in local minima, or a combination of a trajectory planner using an additional mapping algorithm and a tracking controller. Trajectory planning is usually highly time-consuming and has difficulties to adapt to real-time computational constraints. It could be hinted that learning-based strategies will be able to achieve an implicit computational and informational trade-off between these techniques. In [START_REF] Koltun | Benchmarking classic and learned navigation in complex 3d environments[END_REF], classic and learning-based navigation systems have been compared. The modular navigation pipeline they proposed divides the navigation process into 4 sub-tasks: mapping, localization, planning and locomotion. They demonstrated that the classical system outperforms the learned agent when taking as input RBG-D values. In contrast, the classical method is very sensitive to the global set of modalities (i.e. without depth information, it fails catastrophically). Although model-based1 methods usually work well in practice, they demand expert knowledge and do not allow to tackle missions requiring great interaction with the environment. On the other hand, model-free approaches (in particular reinforcement learning) have shown impressive progress in gaming [START_REF] Silver | Mastering the game of go without human knowledge[END_REF], and are beginning to be widely applied for robotic tasks involving computer vision (see [START_REF] Carrio | A review of deep learning methods and applications for unmanned aerial vehicles[END_REF] for a review). This paradigm has also been successfully applied to video game environments [START_REF] Mnih | Playing Atari with deep reinforcement learning[END_REF][START_REF] Kempka | ViZDoom: A Doom-based AI research platform for visual reinforcement learning[END_REF][START_REF] Lample | Playing FPS games with deep reinforcement learning[END_REF], where the input space is similar to the control space of a robot. However these results cannot be readily applied in robotics, because these strategies have been learned and tested in the same virtual environment and moreover the reward (game score) is explicit. Dedicated Deep Reinforcement Learning strategies have already been applied to different robotic platforms and functions. In [START_REF] Chiang | Learning navigation behaviors end-to-end with autoRL[END_REF], pointto-point and path-following navigation behaviors that avoid moving obstacles were learned using a deep reinforcement learning approach, with validation in a real office environment. In [START_REF] Xie | Towards monocular vision based obstacle avoidance through deep reinforcement learning[END_REF], a dueling architecture based on a deep double-Q network (D3QN) was proposed for obstacle avoidance, using only monocular RGB vision as input signal. A convolutional neural network was constructed to predict depth from raw RGB images, followed by a Deep Q-Network consisting of a convolutional network and a dueling network to predict the Q-Value of angular and linear actions in parallel. They demonstrated the feasibility of transferring visual knowledge from virtual to real and the high performance of obstacle avoidance using monocular vision only. In [START_REF] Zamora | Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo[END_REF], a robot patrolling task was successfully learned in simulation. However, this strategy puts the emphasis on not hitting any obstacle more than anything else, therefore the system is not strongly forced to take risks (which is required when heading to a designated destination). Visual object recovery [START_REF] Sampedro | A fullyautonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques[END_REF] has also been considered, the task being defined as reaching an object described by an appearance where all objectives are described in a common vocabulary: proximity to obstacle and proximity of target are embedded in the visual space directly. These two approaches have only been validated in a simulation environment, therefore an undetermined amount of work remains to transfer them to a real robot and environment. In [START_REF] Kulhanek | Vision-based navigation using deep reinforcement learning[END_REF], a navigation task to an image-defined target was achieved using an advantage actor-critic strategy, and a learning process with increased environment complexity was described. The method we propose is similar in spirit to this one, but our contribution addresses mapless navigation as in [START_REF] Tai | Virtual-to-real deep reinforcement learning: Continuous control of mobile robots for mapless navigation[END_REF], which forces the system to take more risks: the system fails if it does not reach the target sufficiently fast, so going closer to obstacles (without hitting them) should be considered. Also, in this task, the system has to process both metric and visual information: distance to obstacles should be perceived from sensor measurements (image, depth), while the target is given as a coordinate. We propose a new learning strategy to tackle this problem, similar to Curriculum Learning (CL) [START_REF] Elman | Learning and development in neural networks: the importance of starting small[END_REF][START_REF] Bengio | Curriculum learning[END_REF] but easier to implement. CL aims at training a neural network more efficiently by using the concept of curriculum, a powerful tool inspired by how humans progressively learn from simple concepts to harder problems. Recent studies [START_REF] Zaremba | Learning to execute[END_REF][START_REF] Rusu | Sim-to-real robot learning from pixels with progressive nets[END_REF][START_REF] Openai | Solving rubik's cube with a robot hand[END_REF] on the application of this method in the robotic field have shown promising outcomes. A drawback of these approaches is the need for heavy simulation, however there is little alternative: fine-tuning in real life seems to be a candidate, but as the fine-tuning database may be quite limited, it is hard (with a deep model) to avoid overfitting and/or catastrophic forgetting. In this paper, we study the behavior of the policy transferred from a simulated to a real environment, with a dedicated hardware setup for unsupervised real testing with a mobile robot. It turns out that depth-based mapless navigation does not seem to require a too heavy domain randomization or fine-tuning procedure with the proposed framework based on incremental complexity.

SAC-BASED NAVIGATION FRAMEWORK

Preliminaries

For completeness, we recall here the Policy Gradient [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] point of view in which we aim at modeling and optimizing the policy directly. More formally, a policy function π is defined as follows:

π θ : S → A
Where θ is a vector of parameters, S is the state space and A is the action space. The vector θ is optimized and thus modified in training mode, while it is fixed in testing mode. The performance of the learned behavior is commonly measured in terms of success rate (number of successful runs over total number of runs). Typically for mapless navigation, a successful run happens if the robot reaches the targeted point without hitting any obstacle in some allowed duration.

In training mode, the objective is to optimize θ such that the success rate during testing is high. However, trying to directly optimize θ with respect to the testing success rate is usually sample inefficient (it could be achieved using e.g. CMA-ES [START_REF] Salimans | Evolution strategies as a scalable alternative to reinforcement learning[END_REF]). Thus, the problem is instead modelled as a Markov process with state transitions associated to a reward. The objective of the training is therefore to maximize the expected (discounted) total reward:

min θ ∑ t∈1,...,T ∑ τ∈t,...,T r τ γ τ-t log (π θ (a t |s t ))
Direct maximization of the expected reward is a possible approach, another one consists in estimating the expected reward for each state, and they can be combined to improve performance. A turning point in the expansion of RL algorithms to continuous action spaces appeared in [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] where Deep Deterministic Policy Gradient (DDPG) was introduced, an actor-critic model-free algorithm that expanded Deep Q-Learning to the continuous domain. This approach has then be improved in [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] where the Soft Actor Critic (SAC) algorithm was proposed: it corresponds to an actor-critic strategy which adds a measure of the policy entropy into the reward to encourage exploration. Therefore, the policy consists in maximizing simultaneously the expected return and the entropy:

J(θ) = T ∑ t=1 E (s t ,a t )∼ρ π θ [r(s t , a t ) + αH(π θ (.|s t ))] (1)
where

H(π θ (.|s)) = -∑ a∈A π θ (a) log π θ (a|s) (2)
The term H(π θ ) is the entropy measure of policy π θ and α is a temperature parameter that determines the relative importance of the entropy term. Entropy maximization leads to policies that have better exploration capabilities, with an equal probability to select nearoptimal strategies. SAC aims to learn three functions, a policy π θ (a t |s t ), a soft Q-value function Q w (s t , a t ) parameterized by w and a soft state-value function V Ψ (s t ) parameterized by Ψ. The Q-value and soft state-value functions are defined as follows:

Q w (s t , a t ) = r(s t , a t ) + γE s t+1 ∼ρ π (s) [V Ψ (s t+1 )] (3) V Ψ (s t ) = E a t ∼π [Q w (s t , a t ) -α log π θ (a t |s t )] (4)
Theoretically, we can derive V Ψ by knowing Q w and π θ but in practice, trying to also estimate the statevalue function helps stabilizing the training process.

The terms ρ π (s) and ρ π (s, a) denote the state and the state-action marginals of the state distribution induced by the policy π(a|s). The Q-value function is trained to minimize the soft Bellman residual:

J Q (w) = E (s t ,a t )∼R [ 1 2 (Q w (s t , a t ) -(r(s t , a t ) + γE s t+1 ρ π (s) [V Ψ(s t+1 )])) 2 ]
(5)

The state-value function is trained to minimize the mean squared error:

J V (Ψ) = E s t ∼R [ 1 2 (V Ψ (s t ) -E[Q w (s t , a t ) -log π θ (a t , s t )]) 2 ] (6)
The policy is updated to minimize the Kullback-Leibler divergence:

π new = arg min π ∈∏ D KL (π (.|s t ), exp(Q π old (s t , .) -log Z π old (s t ))) (7)
We use the partition function Z π old (s t ) to normalize the distribution and while it is intractable in general, it does not contribute to the gradient with respect to the new policy and can thus be neglected. This update guarantees that Q π new (s t , a t ) ≥ Q π old (s t , a t ), the proof of this lemma can be found in the Appendix B.2 of [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. Despite performing well in simulation, the transfer of the obtained policy to a real platform is often problematic due to the reality gap between the simulator and the physical world (which is triggered by an inconsistency between physical parameters and incorrect physical modeling). Recently proposed approaches have tried to either strengthen the mathematical model (simulator) or increase the generalization capacities of the model [START_REF] Ruiz | Learning to simulate[END_REF][START_REF] Kar | Meta-sim: Learning to generate synthetic datasets[END_REF]. Among the existing techniques that facilitate model transfer, domain randomization (DR) is an unsupervised approach which requires little or no real data. It aims at training a policy across many virtual environments, as diverse as possible. By monitoring a set of N environment parameters with a configuration Σ (sampled from a randomization space, Σ ∈ Ξ ∈ R N ), the policy π θ can then use episode samples collected among a variety of configurations and as a result learn to better generalize. The policy parameter θ is trained to maximize the expected reward R (of a finite trajectory) averaged across a distribution of configurations:

θ * = arg max θ E Σ∼Ξ E π θ ,τ∼e Σ [R(τ)] (8)
where τ is a trajectory collected in the environment randomized by the configuration Σ. In [START_REF] Vuong | How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?[END_REF], domain randomization has been coupled with a simple iterative gradient-free stochastic optimization method (Cross Entropy) to solve (8). Assuming the randomization configuration is sampled from a distribution parameterized by φ, Σ ∼ P φ (Σ), the optimization process consists in learning a distribution on which a policy can achieve maximal performance in the real environment e real :

φ * = arg min φ L(π θ * (φ) ; e real ), (9)
where

θ * (φ) = arg min φ E Σ∼P φ (Σ) [L(π θ ; e Σ )] (10)
The term L(π,e) refers to the loss function of policy π evaluated in environment e. Since the ranges for the parameters are hand-picked in this uniform DR, it can be seen as a manual optimization process to tune φ for the optimal L(π θ ;e real ). The effectiveness of DR lies in the choice of the randomization parameters. In its original version [START_REF] Sadeghi | Cad2rl: Real singleimage flight without a single real image[END_REF][START_REF] Tobin | Domain randomization for transferring deep neural networks from simulation to the real world[END_REF], each randomization parameter Φ i was restricted to an interval

Φ i ∈ [Φ low i ; Φ high i
], i = 1, . . . , N. The randomization parameters can control appearance or dynamics of the training environment.

Proposed Learning Architecture

State and Observation Vectors

The problem considered is to learn a policy to drive a mobile robot (with linear and angular velocities as continuous outputs) in a cluttered environment, using the knowledge of its current position and destination (external inputs) and the measurements acquired by its embedded depth sensor. The considered state is defined as:

s t = (o t , p t , h t , a t-1 ) (11)
where o t is the observation of the environment from the depth sensor, p t and h t are respectively the relative position and heading of the robot toward the target, a t-1 are the last actions achieved (linear and angular velocities). The elementary observation vector o t is composed of depth values from the embedded Intel RealSense D435 sensor. The depth output resolution is 640 × 480 pixels with a maximum frame rate of 60 fps. Since the depth field of view of this sensor is limited to 87°± 3°× 58°± 1°× 95°± 3°, the environment is only partially observable. To limit the amount of values kept from this sensor, we decided to sample 10 values from a specific row of the depth map (denoted as δ). By doing this, we sample the environment along the ( X; Y ) orthonormal plane, similarly to a LIDAR sensor (but within an angle of 58°).

In the following, the vector containing these 10 depth values captured from a frame at timestep t is denoted by F t . To be able to avoid obstacles, it seems natural to consider a period of observation longer than the current frame. For this reason, three different observation vectors have been evaluated:

• The current frame only,

o 1 t = [F t ]. • The three last frames, o 2 t = [F t ; F t-1 ; F t-2 ]
• The last two frames and their difference,

o 3 t = [F t ; F t-1 ; (F t -F t-1 )]
The sampling rate for the training and prediction of a policy is a critical parameter. It refers to the average number of s t obtained in one second. If it is too high, the long-term effects of the actions on the agent state cannot be captured, whereas a too low value would most likely lead to a sub-optimal policy. A good practice is at the very least to synchronize the sampling process with the robot slowest sensor (by doing this, every state s t contains new information). This was the depth sensor in our case, which is also the main contributor to the observation vector.

Policy Structure

The architecture of networks encoding the policy seemed to have little impact, therefore we did not put a lot of emphasis on this part and a single one has been selected. In order to optimize the functions introduced in Section 3.1, three fully-connected neural networks are used as shown in Figure 2. The ndimensional depth range findings, the relative target position and the last action achieved are merged together as a (n + 4)-dimensional state vector s t . The sparse depth range findings are sampled from the raw depth findings and are normalized between 0 and 1. The 2-dimensional relative target position is represented in polar coordinates with respect to the robot coordinate frame. The last action performed takes the form of the last linear and angular velocities that are respectively expressed in m.s -1 and rad.s -1 .

Reward Shaping

Reinforcement learning algorithms are very sensitive to the reward function, which seems to be the most critical component before model transfer strategy. A straightforward sparse reward function (positive on success, negative on failure) would most likely lead to failure when working with a physical agent. On the other hand, a too specific reward function seems too hard to be learned. However, we describe below how the reward shaping approach [START_REF] Laud | Theory and application of reward shaping in reinforcement learning[END_REF]) could lead to an interesting success rate in simulation. At the beginning of each episode, the robot is placed at an initial position P with a randomized orientation θ. The goal for the robot is to reach a target position T whose coordinates (x T , y T ) change at each episode (the target relative position from the robot is an input of the models). Reaching the target (considered achieved when the robot is below some distance threshold d min from the target) produces a positive reward r reached , while touching an element of the environment is considered as failing and for this reason produces a negative reward r collision . The episode is stopped if one of these events occurs. Otherwise, the reward is based on the difference dR t between d t (the Euclidean distance from the target at timestep t) and d t-1 . If dR t is positive, the reward is equal to this quantity multiplied by a hyper-parameter C and reduced by a velocity factor V r (function of the current velocity v t and d t ). On the other hand, if dR t is negative (which means the robot moved away from the target during the last time step), the instant reward is equal to r recede . The corresponding reward function is thus:

r(s t , a t ) =          C × dR t ×V r if dR t > 0 r recede if dR t ≤ 0 r reached if d t < d min r collision if collision detected (12)
where

V r = (1max(v t , 0.1)) 1/max(d t ,0.1) , r reached = 500, r collision = -550 and r recede = -10.

Without this velocity reduction factor V r , we observed during training that the agent was heading toward the target even though an object was in its field of view (which led to a collision). The reward signal based only on the distance rate dR t was too strong compared to the collision signal. With this proposed re-Figure 2: The network structure for our implementation of the SAC algorithm. Each layer is represented by its type, output size and activation function. The dense layer represents a fully-connected neural network. The models use the same learning rate l r = 3e -4 , optimizer (Adam) [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] and activation function (Leaky Relu, [START_REF] Maas | Rectifier nonlinearities improve neural network acoustic models[END_REF]). The target smoothing coefficient τ is set to 5e -2 for the soft update and 1 for the hard update.

ward function, we encourage the robot to get closer to the target and to decrease its velocity while it gets to the goal. In addition, it is important to relate the nonterminal reward to the distance of the current state to the target. This way, it is linked to a state function (a potential function) which is known to keep the optimal policy unchanged. More precisely, if the reward was simply defined as γd t+1d t , then the optimal policy would be the same with or without the shaping (which just fastens the convergence). Here, the shaping is a little more complicated and may change the optimal policy but it is still based on d t (see [START_REF] Badnava | A new potentialbased reward shaping for reinforcement learning agent[END_REF] for more details on reward shaping and its benefits).

Incremental Complexity Vs Naive

Sim-to-Real Transfer

The mission was divided into three distinct environments as shown in Figure 3. The first one (Env1) is an empty room. By starting training in this context, we try to force the agent to learn how to simply move toward the target. The second environment (Env2) incorporates eight identical static obstacles uniformly spread in the room. Training in these conditions should allow the agent to learn how to avoid obstacles on its way to the target. The last environment (Env3) includes both static and mobile obstacles. Two identical large static obstacles are placed near the initial position of the robot while four other identical mobile obstacles are randomly distributed in the room at the beginning of each episode. Transition from an environment to another is based on the success rate S rate for the last 100 episodes. If this value exceeds a specific threshold, the agent will move to the next environment or will be sent back to the previous one.

Transition from one environment to another is related to the local performance of the policy and is done during the current training session, ensuring the use of samples collected from various conditions to improve generalization. As illustrated in Figure 3, α 1 and β 1 rule transitions between Env1 and Env2 while α 2 and β 2 rule transition between Env2 and Env3. For this study, these parameters were set to α 1 = 90%, α 2 = 80%, and β 1 = β 2 = 50%. In the following, the "naive" strategy refers to training using only either Env2 or Env3. The training of all the models consisted of 5000 episodes with a maximum step size of 500 each. It was observed that learning with incremental complexity does not improve performance in simulation but has a critical impact in real life. It is relevant, since this domain randomization technique can be easily implemented for many other problems.

EXPERIMENTS

Training or evaluating a robotic agent interacting with a real environment is not straightforward. Indeed, both the training (or at least the fine-tuning) and the evaluation require a lot of task runs. So in this work, we used both simulation and real-world experiments and particularly studied the behavior of the transferred policy from the former to the latter. To do so, a simulation environment representative of the realworld conditions was built, and the real world environment was also instrumented to carry out unsupervised intensive experiments.

Simulation Experiments

The proposed approach has been implemented using the Robot Operating System (ROS) middleware and the Gazebo simulator [START_REF] Koenig | Design and use paradigms for Gazebo, an open-source multi-robot simulator[END_REF]. Some previous works already used Gazebo for reinforcement learning like Gym-Gazebo [START_REF] Zamora | Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo[END_REF][START_REF] Kumar | Offworld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research[END_REF]). An URDF model representative of the true robot dynamics has been generated. The R200 sensor model from the Intel RealSense ROS package was used to emulate the depth sensor (at 10 fps), and the Gazebo collision bumper plugin served to detect collisions. We created several environments that shared a common base, a room containing multiple obstacles (some fixed, others with their positions randomised at each episode). The training process was implemented with Pytorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] as a ROS node communicating with the Gazebo node using topics and services. Both the simulator and the training code ran on the same desktop computer equipped with an Intel Xeon E5-1620 (4C-8T, 3.5Ghz), 16GB of memory and a GPU Nvidia GTX 1080, allowing us to perform the training of one model in approximately 6 hours in the Cuda framework [START_REF] Ghorpade | Gpgpu processing in cuda architecture[END_REF]. The communication between the learning agent and the environment was done using a set of ROS topics and services, which facilitated transposition to the real robot.

Real-world Experiments

The real world experiment took place into a closed room measuring 7 by 7 meters. The room was equipped with a motion capture system (Optitrack) used by the robot and by a supervision stack (described in what follows). Four obstacles (boxes) were placed into the room at the front or the side of the robot starting point. The same desktop computer processed the supervisor and the agent. The robot used was a Wifibot Lab V4 robotic platform which communicated with the ground station using WiFi. It carried an Intel RealSense D435 depth sensor and an onboard computer (Intel NUC 7), on which the prediction was computed using the learned policy. Since the number of runs needed for training and validation is large, this raises some practical issues:

• A long operation time is not possible with usual mobile robots due to their battery autonomy.

• Different risks of damaging the robotic platform can occur on its way to the target with obstacle avoidance. To tackle these issues, we instrumented the environment with two components. First, the room setup allowed the robot to be constantly plugged into a power outlet without disturbing its movements. Secondly, we developed a supervisor node to detect collisions, stop the current episode, and replace autonomously the robot to its starting location at the beginning of a new episode. As detailed in Figure 5, the supervisor multiplexes the command to the robot embed-ded low-level controller (angular and linear speeds).

During the learning phase, it uses the command coming from the SAC node and during the resetting phase it uses the command coming from a motion planner node. The motion planner node defines a safe return trajectory using a PRM* path planner (S ¸ucan et al., 2012) and a trajectory tracking controller [START_REF] Bak | Path following mobile robot in the presence of velocity constraints[END_REF]. No data was collected for learning during this return phase. During the episode, the supervisor node (Figure 5) takes as input the linear and angular velocities estimated by our SAC model to send them to the robot. Whenever the episode is stopped, the supervisor takes as input the commands generated by the motion planner based on the mapping stack to make the robot move to a new starting position, without colliding with any element of the environment. Since the map is fixed, we built a ground truth 3D map (Figure 4) of the test environment before starting the experiment by manually moving the robot and integrating the depth sensor into an Octomap [START_REF] Wurm | Octomap: A probabilistic, flexible, and compact 3d map representation for robotic systems[END_REF] model (any other ground truth mapping technique would be suitable). Thanks to this infrastructure, we were able to run a large number of runtimes with a minimal need for human supervision.

Obviously, the duration of each real-life run is large (vs simulation), but the unsupervised evaluation of 100 runs can be performed in roughly ∼ 30 minutes, which is practical for evaluating the Sim-to-Real policy transfer.

Results

Experimental results for the approach proposed in the previous section are provided for the different observation vector configurations considered 2 . This evaluation consisted in a total of 5 sessions of 100 episodes each, conducted with the real robot thanks to the supervision stack described in Section 4.2. Let us stress that these performances are conservative due to safety margins included in the supervision stack but comparable for all models. It took us roughly 45 minutes to test one model under these conditions. Performances of the trained policies were finally assessed and compared in terms of mean success rate and mean reward over the 5 sessions. These results are provided in Tables 1 and 2 for the distinct cases outlined in Section 3.2. In these tables, we designate by F n the 10 depth values kept in the frame captured at time step n. This means that the first column indicates which observation vector o i t is used in the state s t . The second column specifies which environment has been used to train the models as shown in Section 3.3. It can ob-2 A video can be found at https://tinyurl.com/sim2realdrl-robotnav served that the models trained by using the incremental method (i.e. Env1-2-3 in the tables) obtain the best performances in terms of mean success rate as well as in mean reward over the 5 sessions. The best one among the models trained incrementally is the model whose observation vector consisted of the last two frames and their difference (o 3 t ) with a success rate of 47% and a mean reward of 38.751. The performance can thus be scaled twice using the incremental complexity sim-to-real strategy coupled with the SAC reinforcement learning strategy. This result is not trivial as depth-based mapless navigation is harder than mapless patrolling [START_REF] Zamora | Extending the OpenAI Gym for robotics: a toolkit for reinforcement learning using ROS and Gazebo[END_REF] or visual object recovery [START_REF] Sampedro | A fullyautonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques[END_REF], which do not need to go close to obstacles (and these methods were only tested in simulated environments). It could be noted that even the naive learned-in-sim policy achieves a non-trivial success rate. The success rate could most probably be improved by carrying out a fine-tuning training session in the real-world experiment, however this is beyond the scope of this paper.

CONCLUSIONS

In this paper, we have proposed a mapless navigation planner trained end-to-end with Deep Reinforcement Learning. A domain randomization method was ap- plied in order to increase the generalization capacities of the policy without additional training or finetuning in the real world. By taking as inputs only two successive frames of 10 depth values and the target position relative to the mobile robot coordinate frame combined with a new incremental complexity training method, the given policy is able to accomplish depthbased navigation with a mobile robot in the real world even though it has only been trained in a ROS-Gazebo simulator. When compared to a naive training setup, this approach proved to be more robust to the transfer on the real platform. The models trained in this study were able to achieve the mission in an open environment containing box-size obstacles and should be able to perform well in similar indoor contexts with obstacles of different shapes. However, they would most likely fail in environments such as labyrinths because the observation inputs o t will be too different.

A direct improvement could be to include a final reference heading which can be easily considered since the Wifibot Lab V4 robotic platform is able to spin around. Future work will focus on the fair comparison between model-based methods and such learning algorithms for autonomous robot navigation, as well as addressing more complex robotics tasks.

  (a) Robot learning in the ROS-Gazebo simulated environment. (b) Cable-Powered Wifibot with depth sensor. (c) Wifibot navigating in the real environment.

Figure 1 :

 1 Figure 1: Illustration of simulated and real environments for reinforcement learning of depth-based robot navigation.

Figure 3 :

 3 Figure 3: Illustration of the incremental complexity strategy. The policy is trained on multiple environments, each one representing an increment of subtasks (more complex obstacles) contributing to the global mission.

Figure 4 :

 4 Figure 4: Octomap ground truth of the environment. The frame denotes the robot position and the red ball the target position.

Figure 5 :

 5 Figure 5: Supervision stack for learning and testing in the real world.

Table 1 :

 1 Success rate (in %). F t designates depth measurements taken at time t. ;F t-1 ; F t -F t-1 ] Env2 24% [F t ; F t-1 ; F t -F t-1 ] Env3 33% [F t ; F t-1 ; F t -F t-1 ]

	Observation	Training	Success
	vector (o t )	environments	Rate
	[F t ]	Env2	21%
	[F t ]	Env3	29%
	[F t ]	Env1-2-3	32%
	[F t ; F t-1 ; F t-2 ]	Env2	38%
	[F t ; F t-1 ; F t-2 ]	Env3	17%
	[F t ; F t-1 ; F t-2 ]	Env1-2-3	42%
	[F t Env1-2-3	47%

Table 2 :

 2 Mean reward values. F t designates depth measurements taken at time t.

	Observation	Training	Mean
	vector (o t )	environments	reward
	[F t ]	Env2	-248.892
	[F t ]	Env3	-189.68
	[F t ]	Env1-2-3	-95.623
	[F t ; F t-1 ; F t-2 ]	Env2	-100.662
	[F t ; F t-1 ; F t-2 ]	Env3	-300.124
	[F t ; F t-1 ; F t-2 ]	Env1-2-3	22.412
	[F t ; F t-1 ; F t -F t-1 ]	Env2	-217.843
	[F t ; F t-1 ; F t -F t-1 ]	Env3	-56.288
	[F t ; F t-1 ; F t -F t-1 ]	Env1-2-3	38.751

Here the term model does not correspond to the one used in Section 3 but refers to whether or not a dynamical model of the controlled system is used in the control strategy whereas in RL, it is the model of the environment that would be provided to the RL algorithm.