N Bello

C Larignon

J Douin

Long-term thermal ageing of the 2219-T851 and the 2050-T84 Al-Cu alloys

Keywords: Aluminium alloys, Transmission Electron Microscopy, Plasticity, ageing treatment, Mechanical Properties

In this study, the influence of high temperature exposure on mechanical properties as well as on the microstructures of two Al-Cu alloys, has been investigated. Driven by an industrial will to use structural aluminium at higher temperature than usually stated, this work is focused on the characterization of the evolution of two commercial alloys, the 2219-T851 and the 2050-T84, induced by an ageing treatment of 1000 hours at 200°C. Careful TEM observations confirm the importance of the nanoscale precipitation and that the precipitate size and distribution strongly influence the mechanical properties of the studied alloys. Results unexpectedly show that the precipitates are more stable in time than the precipitates at this temperature. Regarding the mechanical properties related to ageing, it follows that the 2219-T851 alloy presents a reduced drop with time of its tensile test curves compared to the 2050-T84, thus a limited, and at least less pronounced, temperature dependence. For an industrial application on structural part, 2219-T851 alloy then appears to be a good candidate for long-term structural applications at intermediate temperatures.

Introduction

Aerostructures manufacturers are continually seeking solutions to reduce aircraft weight, and aluminium alloys, with their low-density giving rise to excellent specific mechanical properties, often appear as a pertinent choice when metallic parts are needed. Historically developed for lowtemperature applications, aluminium alloys are still being used with this specification, i.e. at temperature generally below 80°C. Some applications of aluminium alloys can be found at medium temperatures (in the range 80 -110ºC) but mainly for non-structural applications. Usually aerodynamic parts, as the wings leading edges for example, are made of aluminium alloys and the temperature can be higher than 110°C [START_REF] Chen | The effects of heat treatment on the microstructure and mechanical properties of the AA2618 DC cast plate[END_REF]. For the supersonic flight, the Concorde highest temperature was 127°C on the nose, an aerodynamic part non-essential for the aircraft structural integrity [START_REF]Accelerated Aging of Materials and Structures[END_REF]. In the case of heated structural parts, titanium alloys are preferred, essentially due to their better mechanical resistance at elevated temperature. Consequently, on aircraft structures, titanium alloys are being used for in-service temperatures above 80°C and up to 400°C (with ⍺, near-⍺, β, and/or near-β structures [START_REF] Peters | Titanium alloys for aerospace applications[END_REF][START_REF] Boyer | An overview on the use of titanium in the aerospace industry[END_REF]), while aluminium alloys are only used below 80°C-110°C for structural parts, as it is generally considered that their mechanical properties decrease excessively above 110°C. For structural applications, the replacement possibility of titanium alloys by aluminium ones for long-term use at an intermediate temperature, that is when the mechanical properties of aluminium are considered to be dropping down too rapidly, would, however, be of great interest, as such a replacement would allow weight reduction, but also cost savings through the low raw material and machinability costs. Aluminium alloys are indeed less expensive and easier machined than titanium alloys [START_REF] Niknam | Machinability and Machining of Titanium Alloys[END_REF].

Within the framework of this project, the aim was to find an aluminium alloy with a service temperature of, at least, 150°C. To be able to use aluminium alloys at this temperature on structural aircraft parts, ageing stability must be studied. In relation with industrial purposes, the maximum service temperature has been fixed to 200°C. Hence the selection of the ageing temperature to fit this targeted in-service temperature. Following these requirements, the alloy choice has been made: two qualified aluminium alloys of the 2xxx series have been selected as the aluminium-copper alloys of this series are supposed to have a good stability after thousands of hours in temperature when compared to the other heat treatable aluminium alloys [START_REF] Develay | Aluminium et alliages d'aluminium corroyés : propriétés métalliques Aluminium et alliages d'aluminium corroyés : propriétés métalliques[END_REF]. The 2219-T851 alloy was selected due to the good ageing performance [START_REF] Develay | Aluminium et alliages d'aluminium corroyés : propriétés métalliques Aluminium et alliages d'aluminium corroyés : propriétés métalliques[END_REF] as well as the 2050-T84 for the results on the temperature tensile properties. As-received heat treatment was also of interest and selected to correspond to the one already used for aircraft parts and to an overaged microstructure. Some results on the ageing of the 2xxx series aluminium alloys can be found in the literature but most of them are focused on elaboration process optimizations. Indeed, the applications of higher temperature during short periods of time, typically less than 200h [START_REF] Ceschini | Effect of thermal exposure on the residual hardness and tensile properties of the EN AW-2618A piston alloy[END_REF], can enhance mechanical properties but increase the production cost. Some recent research investigated over-ageing behaviour on Al-Cu-Li alloys between 200°C and 305°C for less than 200 hours [START_REF] Balducci | Effects of overaging on microstructure and tensile properties of the 2055 Al-Cu-Li-Ag alloy[END_REF][START_REF] Balducci | Thermal stability of the lightweight 2099 Al-Cu-Li alloy : Tensile tests and microstructural investigations after overaging[END_REF]. However, almost no information can be found on aluminium ageing for a period as long as 500h or more. In this work, the influence of a 1000 hours thermal ageing at 200°C at room temperature tensile properties and microstructure evolution of two aluminium alloys of the 2xxx series is investigated.

To follow the evolution of the studied alloys, multiscale characterization and mechanical testing have been conducted.

Materials

The two aluminium alloys selected for this study are the 2050-T84 and the 2219-T851 alloys, with composition ranges given in Table 1. Both alloys were delivered as plate-like bulk pieces, 130 mm thick for the 2050 and 80 mm thick for the 2219. For this study, the 2050 and 2219 alloys are respectively associated with the specific thermomechanical T84 and T851 treatments. The T84 heat treatment for the AA2050 corresponds to a first solution heat treatment at 525 ± 5 °C followed by a water quenching, then a cold work at approximately 4% and an artificial ageing at 155 ± 5 °C for 18 hours [START_REF] Kaufman | Understanding the Aluminum Temper Designation System[END_REF][START_REF] Danielou | Aluminium-copper-lithium alloy having improved mechanical strength and improved toughness[END_REF]. The T851 heat treatment on the 2219 alloy is similar but with a solution heat treatment at 535 ± 5 °C, followed by the cold working and an ageing treatment at 175 ± 3 °C for 18 hours. On this last alloy, another treatment was applied: a stress relief by stretching between 1 and 3% [START_REF] Kaufman | Understanding the Aluminum Temper Designation System[END_REF]. In the following, these thermomechanical treatments are called "as-received states".

The 2219 alloy has been developed in 1954 (year of standard designation). Some authors [START_REF] Gupta | Study of Aluminum Alloy AA2219 After Heat Treatment[END_REF][START_REF] Naga Raju | Microstructure and high temperature stability of age hardenable AA2219 aluminium alloy modified by Sc, Mg and Zr additions[END_REF] have noted the high temperature stability of this alloy. They conducted several mechanical tests on a 2219-T87 alloy heat-treated between 165 and 185°C and reported stable mechanical properties of the alloy after being exposed between 12 and 30 hours at the given temperatures.

The tensile yield stress value was slightly above 300 MPa and, ultimate tensile strength was around 400 MPa. Notice that these authors only studied these experimental conditions in order to improve the thermomechanical process optimization, as mentioned previously [START_REF] Gupta | Study of Aluminum Alloy AA2219 After Heat Treatment[END_REF][START_REF] Naga Raju | Microstructure and high temperature stability of age hardenable AA2219 aluminium alloy modified by Sc, Mg and Zr additions[END_REF]. The 2219 alloy mostly contains precipitates, which have been widely studied in the literature [START_REF] Meetsma | Refinement of the crystal structure of tetragonal Al2Cu[END_REF][START_REF] Costa Teixeira | The effect of shearresistant, plate-shaped precipitates on the work hardening of Al alloys: Towards a prediction of the strength-elongation correlation[END_REF][START_REF] Zhao | A facile electron microscopy method for measuring precipitate volume fractions in AlCuMg alloys[END_REF]. The precipitation sequence of the precipitates is now well accepted [START_REF] Lin | Precipitation hardening of 2024-T3 aluminum alloy during creep aging[END_REF][START_REF] Shen | Atomic-scale mechanism of the θ″ → θ′ phase transformation in Al-Cu alloys[END_REF]: starting from a solid solution alloy, precipitates grow from Guinier-Preston zones to coherent precipitates to semi-coherent precipitates to incoherent precipitates with the known Al 2 Cu stoichiometry. The precipitation sequence can be summarized as:

Where GP stands for Guinier-Preston zones [START_REF] Guinier | Structure of Age-Hardened Aluminium-Copper Alloys[END_REF][START_REF] Preston | Structure of Age-Hardened Aluminium-Copper Alloys[END_REF]. -precipitates are platelets-like precipitates lying in the {001} Al planes with various orientation relationships with the aluminium matrix [START_REF] Noble | Theta-prime precipitation in aluminium-copper-cadmium alloys[END_REF][START_REF] Bourgeois | Structure and energetics of the coherent interface between the θ′ precipitate phase and aluminium in Al-Cu[END_REF].

It is worth noting that coarser intermetallic precipitates such as T-phase Al 20 Cu 2 Mn 3 are also present in the alloy microstructure [START_REF] Costa Teixeira | The effect of shearresistant, plate-shaped precipitates on the work hardening of Al alloys: Towards a prediction of the strength-elongation correlation[END_REF]. These precipitates have an important influence on the corrosion sensitivity of the alloys [START_REF] Grilli | Localized corrosion of a 2219 aluminium alloy exposed to a 3.5% NaCl solution[END_REF], mechanical properties [START_REF] Chen | The effects of heat treatment on the microstructure and mechanical properties of the AA2618 DC cast plate[END_REF][START_REF] Liu | Effect of T-phase on microstructure of the hot rolled Al-Cu-Mg alloy[END_REF] and also grain size control [START_REF] Liu | Effect of T-phase on microstructure of the hot rolled Al-Cu-Mg alloy[END_REF][START_REF] Rioja | The evolution of Al-Li base products for aerospace and space applications[END_REF]. They have been studied by several research teams but won't be studied here as we will focus our investigation on mechanical properties.

The 2050 alloy has been more recently developed and is part of the AIRWARE® brand developed by Alcan Rhenalu SAS (now known as Constellium®) for space and aeronautical use [START_REF] Lequeu | Aluminum-copper-lithium alloy 2050 developed for medium to thick plate[END_REF]. This alloy is the third generation of Al-Cu-Li alloys developed by Alcan Rhenalu SAS, with a reduced lithium content. This alloy's macroscopic properties and microstructure have been characterized since the early 2010s in order to understand the microstructural changes characteristic of this generation compared with the previous generation of Al-Cu-Li alloys and even with other aluminium alloy series [START_REF] Rioja | The evolution of Al-Li base products for aerospace and space applications[END_REF][START_REF] Lequeu | Aluminum-copper-lithium alloy 2050 developed for medium to thick plate[END_REF]. Apart from precipitates, other nanometric precipitates can be found in Al-Cu-Li alloys [START_REF] Han | Microstructural evolution, mechanical property and thermal stability of Al-Li 2198-T8 alloy processed by high pressure torsion[END_REF]. Two types of precipitates can coexist: T 1 -Al 2 CuLi and . T 1 -Al 2 CuLi platelet-like precipitates are lying in the {111} Al planes. The T 1 -Al 2 CuLi crystallographic structure was first determined by Meetsma et al. [START_REF] Meetsma | Refinement of the crystal structure of tetragonal Al2Cu[END_REF] and recently confirmed by two different research teams [START_REF] Dwyer | Combined electron beam imaging and ab initio modeling of T1 precipitates in Al-Li-Cu alloys[END_REF][START_REF] Donnadieu | Atomic structure of T1 precipitates in Al-Li-Cu alloys revisited with HAADF-STEM imaging and small-angle X-ray scattering[END_REF]. Despite their observation on previous Al-Cu-Li generations [START_REF] Lequeu | Aluminum-copper-lithium alloy 2050 developed for medium to thick plate[END_REF][START_REF] Williams | Microstructural Analysis of Aluminium Alloy 2096 as a Function of Heat Treatment By[END_REF], precipitates have not been found in the 2050 alloy due to the low Li content. In addition to these precipitates, and due to the alloy chemical composition, another kind of precipitate can be expected. Indeed with the presence of magnesium and silver in this alloy, precipitates can be observed [START_REF] Muddle | The precipitate Ω phase in Al-Cu-Mg-Ag alloys[END_REF]. These precipitates are also lying in the {111} Al planes. Finally, as in the 2219, coarser precipitates related to heavier elements are also present in the microstructure, also affecting corrosion sensibility [START_REF] Proton | The influence of artificial ageing on the corrosion behaviour of a 2050 aluminiumcopper-lithium alloy[END_REF]. In this alloy, it has been found that Al 3 Zr and T-phase Al 20 Cu 2 Mn 3 precipitates are present [START_REF] Rioja | The evolution of Al-Li base products for aerospace and space applications[END_REF].

Experiments

The thermal ageing was conducted on bulk pieces placed at 200°C for 1000h under air pressure (noted 'aged' in the following). Ageing parameters have been chosen to correspond to a maximum peak temperature seen by the alloys during foreseen in-service solicitations.

To understand the alloys' behaviour before and after a long thermal ageing, several characterization methods were used: mechanical testing, optical microscopy and Scanning and

Transmitting Electron Microscopy (SEM and TEM).

-Mechanical testing was performed on cylindrical specimens with an effective diameter of 6 mm in accordance with the NF EN 2002-001 standard modified using a test speed fixed at 0.005 min -1 (approximately 0.8 .10 -4 s -1 ). The tests have been realized on a RKM250 Zwick equipment.

-Optical observations were made after etching the samples for 30 seconds with Keller's reagent. Widely used to reveal aluminium alloys microstructure, the Keller reagent is composed of 1 ml of 40 wt.% HF, 1.5 ml of 35 wt.% HCl, 2.5 ml of 68 wt.% HNO 3 and 95 ml of H 2 O [START_REF] Dix | Keller's reagent[END_REF].

-Scanning Electron Microscopy observations were performed using a dual-beam FEI Helios All observations were made on the as-received state alloys and after the long thermal ageing.

Mechanical properties

Tensile tests were realized at room temperature with the parameters given above on the asreceived and aged alloys (Figure 1). The values of yield and ultimate tensile strengths (referred in the following as YS and US) are reported in table 2.

First of all, for alloys in the as-received state, the measured stress-strain curves are in a good agreement with what was expected from the patent presented by Alcan in 2010 [START_REF] Warner | Aluminum-Copper-Lithium Alloy With Improved Mechanical Strength and Toughness[END_REF] or by Chemin et al. [START_REF] Chemin | Characterization of phases, tensile properties, and fracture toughness in aircraft-grade aluminum alloys[END_REF] for 2050-T84 and reported in previous works for 2219 alloy [START_REF] Kaufman | Understanding the Aluminum Temper Designation System[END_REF]. The 2050-T84 alloy has a high yield tensile strength, comparable to the alloys of the 7xxx series, and significantly higher than that of 2219, with equivalent ductility. As expected, after being submitted to a long-term ageing process, yield and ultimate tensile stresses of both alloys are significantly reduced. The most affected material by long-term ageing is the 2050 alloy for which the decrease of static yield tensile strength (YS) is estimated to 208 MPa in the Rolling Direction (RD), that is a reduction of about 40% from the value before ageing. The impact of ageing on the 2219 alloy is less pronounced with a reduction limited to 79 MPa in the RD, that is about 17%. The evolution of YS with thermal ageing is also more significant for the 2050 alloys with a 271 MPa (54%) drop than for the 2219 alloy (106 MPa, i.e. a reduction of 30% in the RD). The 2219 alloy then appears with slightly better capabilities than the 2050 after ageing. Clearly speaking, while the 2050 alloy shows a better tensile resistance than the 2219 alloy in the as-received condition, the 2219 alloy is better than the 2050 after ageing treatment. As this heat treatment could be abruptly correlated to real conditions of use, this may have profound industrial implications.

Another important point to emphasize is the differences in the stress-strain curve shapes, which can be clearly evidenced by plotting the Kocks-Mecking curves (work hardening rates as a function of reduced stress (Figure 1b)) [START_REF] Kocks | Physics and phenomenology of strain hardening: the FCC case[END_REF][START_REF] Kocks | Laws for Work-Hardening and Low-Temperature Creep[END_REF]. Heat treatment does not have a significant impact on the hardening rate of the 2219 alloy. On the contrary, there is a great change for the 2050 alloy before and after heat treatment. Between 1% and 3% of deformation, that is when the deformation process has reached a steady-state behaviour, the hardening rate of the as-received 2050 alloy is significantly lower than the ones for the aged 2050, whose behaviour looks like the one of the 2219 alloys, thus indicating a change in the involved deformation mechanisms.

Two hypotheses can be put forward to explain this change. Firstly, it could result from the elaboration process. Indeed, the 2219 alloy underwent a stress relief treatment by stretching, and not the 2050 alloy. On a general point of view, a higher initial dislocation density would result in an improved initial yield stress, essentially related to forest hardening, and consequently a lower hardening rate. However, it is not believed that a different initial dislocation density would modify the shape of the work-hardening rate curve.

Secondly, it is well known, and it has been observed in aluminium alloys [START_REF] Deschamps | The influence of precipitation on plastic deformation of Al-Cu-Li alloys[END_REF][START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF], that the hardening rate is very dependent upon the fine microstructure of the alloys. Indeed, the study of the influence of the size and shape of the precipitates on the hardening response indicates that smaller and more shareable precipitates are generally ascribed to a lower hardening rate than larger ones, which are bypassed during deformation through the Orowan process. A change of the strain-stress curve has already been observed for overaged Al-Cu-Li alloys [START_REF] Balducci | Effects of overaging on microstructure and tensile properties of the 2055 Al-Cu-Li-Ag alloy[END_REF][START_REF] Deschamps | The influence of precipitation on plastic deformation of Al-Cu-Li alloys[END_REF][START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF]. In an Al-Cu-Li alloy, the authors observed a low hardening rate with the presence of small T 1 precipitates while a high-hardening rate for thicker T 1 precipitates. The difference between the two behaviours has been linked to the dislocations/precipitate interaction nature: shearing or bypassing, respectively [START_REF] Deschamps | The influence of precipitation on plastic deformation of Al-Cu-Li alloys[END_REF][START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF].

It is then important to realize that the evolution of the shape of the strain-stress curves clearly indicates a modification of the microstructural mechanisms activated during the deformation in the 2050 alloy between the as-received state and the thermally aged state, while for the 2219 alloy, only a few, if any, microstructural changes should be expected.

Microstructure description

Alloys have been first observed at a medium scale using Optical and Scanning Electron Microscopes (SEM), with the main objectives to measure the grains sizes and to identify the relevant microstructural features.

The grain sizes were measured using the intercept method on optical micrographs. Straight lines were drawn on the three characteristic directions, Rolling Direction (RD), Transversal Direction (TD) and Short Direction (SD), relative to the hot-rolling directions of the plates, to determine the mean intercept value and then the mean grains length and width. Average grain size and related form factors are reported in Table 3. In the following, only the dimension of grains observed on the TD-SD plane, that is perpendicular to the rolling direction, are reported. Notice that due to the frequent observations of sub-grain boundaries in the 2050 alloy and in order to confirm the presence of sub-grains boundaries in this alloy, EBSD SEM technique has been used. Since this technique was only employed to discriminate the grain boundary nature, the results are not presented here.

For the 2219 alloy (Figure 2) in the as-received state, the mean grain dimension is approximately 290 µm with a form factor of 1.84 (Table 3). Similar values have already been reported in the literature [START_REF] Naga Raju | Microstructure and high temperature stability of age hardenable AA2219 aluminium alloy modified by Sc, Mg and Zr additions[END_REF]. On the as-received 2050 alloy, the grain dimensions are smaller. The mean grain dimension is approximately 120 µm with a form factor of 1.5. The etching initially performed to allow a better observation of the grain boundaries also revealed a precipitate free zone (PFZ) near the grain boundaries in the 2050 alloy (Figure 3a.). This precipitate free zone can be evidenced using SEM-CBS. On the Figure 3a. and in the proximity of a grain boundary, noted A for example, it is possible to observe less white signal compared with the grain centre. As the Keller reactant is sensitive to the hardening precipitates present in the aluminium matrix, this would tend to indicate that, in the 2050 alloy, diffusion is accelerated in the vicinity of grain boundaries, preventing precipitation from occurring or promoting the dissolution of close precipitates. Such PFZ has also been observed elsewhere [START_REF] Proton | The influence of artificial ageing on the corrosion behaviour of a 2050 aluminiumcopper-lithium alloy[END_REF].

In order to confirm the nature of the coarse precipitates present in both 2219 and 2050 alloy microstructures, SEM-EDX analysis and image analysis have been conducted. On the 2219 alloy, results confirmed the large presence of coarse precipitates of Al-Cu within the microstructure. In order to discriminate the nature of the analysed precipitates, surface density measurements have been made. The quantification made with ImageJ software indicates the presence of Al-Cu precipitates larger than 7,5 µm 2 . Other smaller precipitates, less than 7,5 µm 2, are present and are mostly composed by Al-Cu-Mn-Fe. For the 2050 alloy, fewer precipitates can be observed. The larger ones, bigger than 7,5 µm 2 , are Al-Cu-Mn-Fe precipitates while the smaller ones, less than 7,5 µm 2 are Al-Ti-Mg-Mn precipitates. The analysis of the precipitate surface measured by image analysis emphasize the difference between the two alloys, where in the 2219 alloy, the precipitates have a measured surface density of 2,39%, compared to the 0.99% measured in the 2050 alloy.

For both alloys, the ageing treatment does not induce drastic changes in the mean grain size (Table 3). With the SEM observations and for the 2050 alloy, the ageing treatment results in an increased precipitation in grains and sub-grain boundaries. In addition, the precipitate free zone along the grain boundaries is still present (Figure 3). On the microstructure of the 2219 alloy, no significant change can be identified at this scale of observation.

The modification induced by the ageing treatment on the tensile properties observed previously in both alloys might have been related to the Hall-Petch effect via a grain size modification. In this case, this effect cannot be addressed:

-The microstructure of the 2050 alloy is not stable throughout the ageing treatment,

-The introduction of deformation during the elaboration processes of the two alloys, -The absence of significant change on the 2219 alloy microstructure explaining the tensile property variations.

With these points, the observed increase of this mechanical property cannot be ascribed to grain size variation. It is believed that the origin of these modifications requires a study at a finer scale, with TEM observations.

Evolution of the Fine Scale Microstructure

Conventional TEM observations clearly show that the microstructure of both as-received alloys is already filled with numerous hardening (nano)precipitates (Figure 4). These fine precipitates are known to impede the motion of dislocations upon stress, thus giving rise to a substantial increase in yield stress compared to pure aluminium [START_REF] Develay | Aluminium et alliages d'aluminium corroyés : propriétés métalliques Aluminium et alliages d'aluminium corroyés : propriétés métalliques[END_REF].

In both alloys, these hardening precipitates are plate-like shaped. In the 2050-T84 alloy (Figure 4a.), the precipitates can be indexed using Selected Area Electron Diffraction (SAED, Figure 5a.) pattern as T 1 -Al 2 CuLi precipitates. Figure 6 has been used for the SAED indexation, as it presents the diffraction pattern of the precipitates found in 2219 and 2050 alloys when orientated in a <001> Al direction. These precipitates have been previously reported in this alloy [START_REF] Lequeu | Aluminum-copper-lithium alloy 2050 developed for medium to thick plate[END_REF] as well as in other Al-Cu-Li alloys, such as in 2198 [START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF] or 2055 alloy [START_REF] Balducci | Effects of overaging on microstructure and tensile properties of the 2055 Al-Cu-Li-Ag alloy[END_REF]. They have been widely studied in the literature of Al-Cu-Li alloys, and, as observed here, are known to form platelets lying in the {111} Al planes of the aluminium matrix: this is why they appear edge-on in the <110> Al observation directions (Figure 4a.). During deformation, these 3 atomic layers wide precipitates [START_REF] Donnadieu | Atomic structure of T1 precipitates in Al-Li-Cu alloys revisited with HAADF-STEM imaging and small-angle X-ray scattering[END_REF][START_REF] Deschamps | The influence of precipitation on plastic deformation of Al-Cu-Li alloys[END_REF] are usually sheared by gliding dislocations. It is worth emphasizing that while these precipitates have a small thickness, their density here is inversely very high. This has a direct consequence on the mechanical properties of the 2050 alloy: as the efficiency of the precipitates to block the dislocations is a function of the distance between precipitates, a high density of precipitates well explains the high level of yield stress observed for this alloy. The overlapping of stress fields surrounding the precipitates has an experimental drawback: the difficulty to observe well resolved contrasts in conventional TEM.

Compared to the microstructure of the as-received 2050 alloy observed by TEM, the as-received 2219 one (Figure 4b.) contains different larger precipitates. The plate-like precipitates observed in this alloy microstructure areprecipitates lying in the {100} Al planes. This is why they appear edge-on with a 90° symmetry when observed along a <001> Al direction (Figure 4b.) Even though the exact structure is still being discussed, the observed SAED pattern (Figure 5b.) matches with the precipitate structure proposed by [START_REF] Bourgeois | Structure and energetics of the coherent interface between the θ′ precipitate phase and aluminium in Al-Cu[END_REF]. Due to their larger size, these precipitates are more difficult to shear and then mostly bypassed by the Orowan mechanism [START_REF] Costa Teixeira | The effect of shearresistant, plate-shaped precipitates on the work hardening of Al alloys: Towards a prediction of the strength-elongation correlation[END_REF]. Between these thicker precipitates, smaller precipitates can also be observed, areas are pointed with the C arrow (Figure 4b.). To confirm the nature of these precipitates, the selection of a small area to have a representative SAED pattern is experimentally very difficult. Nevertheless, considering the precipitate orientation and the alloy chemistry, it is possible to assume that these are or precipitates as the GP zones are nearly impossible to observe with conventional TEM. These smaller precipitates play an important role in the as-received alloy, as they decrease the mean spacing value between precipitates and thus contribute to the enhanced mechanical properties.

After the 1000h ageing treatment at 200°C, the fine microstructure of both alloys has changed (Figure 7). In the 2050 microstructure (Figure 7a.), T 1 -Al 2 CuLi precipitates can still be observed but they appear thicker compared to the as-received alloy, precipitates pointed by B arrows (Figure 7a.). Also, a new precipitate family can be observed, the precipitates are indicating with A arrows on Figure 7a. Using SAED, this new precipitate family can also be indexed asprecipitates. Figure 7a. presents the SAED pattern orientated in a <001> Al direction show strikes with thicker parts where the diffraction of the phase occur [START_REF] Li | Influence of Mg Contents on Aging Precipitation Behavior of Al-3.5Cu-xMg Alloy[END_REF]. Due to the crystal orientation <110> Al , they are not being seen edge-on as they are lying in the {100} Al planes. Thus, they appear thicker than the T 1 precipitates in the Figure 7a. Notice that it is not possible to know if these precipitates, as well as some precursors such as GPI zones or GPII zones/ precipitates, appear during the ageing treatment or if they were too small to be seen by conventional TEM. In fact, streaking along the <100> Al directions which can be observed on the SAED pattern of the asreceived 2050-T84 (Figure 5b.) is known to be due to the presence of Cu-rich GP zones in the {100} Al planes [START_REF] Mukhopadhyay | Nucleation of Ω phase in an Al-Cu-Mg-Mn-Ag alloy aged at temperatures below 200[END_REF]. With further ageing, the streaking presents brighter points near the diffraction of the phase.

From this observation, it results that the density of hardening precipitates has largely felt down after ageing of the 2050 alloy, resulting in a consequent increase in the mean distance between precipitates, thus lowering the yield stress by promoting Orowan process instead of shearing of precipitates. This is fully in accordance with the reduction of the yield stress, and the variation of the work-hardening rate (Figure 1b.), which is low for as-received, thus connected with shearing of precipitates, and higher after ageing, thus related to more frequent Orowan mechanism [START_REF] Costa Teixeira | The effect of shearresistant, plate-shaped precipitates on the work hardening of Al alloys: Towards a prediction of the strength-elongation correlation[END_REF]. This can explain the wide presence of T 1 precipitates in the as-received 2050 alloy.

As mentioned before, precipitates have been reported in the 2050 alloy. However, as noted by different authors, the SAED pattern of the and of the T 1 phases are too close to be distinguished. For example, SAED pattern of an Al-Cu-Li without magnesium or silver [START_REF] Yoshimura | Transmission electron microscopy study of the evolution of precipitates in aged Al-Li-Cu alloys: the θ′ and T1 phases[END_REF] is very close to an SAED pattern of an Al-Cu-Mg-Mn-Ag alloy [START_REF] Mukhopadhyay | Nucleation of Ω phase in an Al-Cu-Mg-Mn-Ag alloy aged at temperatures below 200[END_REF]. The precipitates have not been found in Al-Cu-Li alloys so far, and in a recent study highlighting the effect of magnesium in these alloys, it has been concluded that magnesium allows a selective precipitation of the T 1 -Al 2 CuLi precipitates [START_REF] Gumbmann | The effect of minor solute additions on the precipitation path of an Al-Cu-Li alloy[END_REF].

On the 2219 alloy, theprecipitates are also thicker after the ageing treatment. This is also confirmed by the SAED pattern observations. The Figure 7b. which corresponds to the diffraction of the 2219 aged microstructure, is quite similar to the Figure 5b., the as-received one.

However, and even if it appears that the small-sized precipitates have disappeared or at least became very small and by consequence hardly visible, the important point is that there is not a drastic change of the mean distance betweenprecipitates before and after ageing. It thus results that the yield stress of the 2219 alloy after ageing, while reduced compared to asreceived alloy, has essentially retained its magnitude or only slightly decreased. Regarding the hardening rate, as the mechanism by which dislocations overcome the precipitates is also essentially conserved, it is not abnormal to observe equivalent hardening rates before and after ageing for the 2219 alloy.

It is worth emphasizing that a detailed statistical description of the distances between precipitates, correlated with the nature of the dislocation/precipitate interaction, would allow estimating the theoretical yield strengths, to be compared with the measured ones (work in progress).

At this point, the precipitates stability can be discussed. The presented results have shown, for the ageing behavior of the 2020-T84 alloy, the precipitation of the phase. This phase appears with the ageing treatment, sign of the thermal stability at this temperature when compared to the phase. First, the precipitation temperature can be addressed. These two phases present very close precipitation temperature: from 280°C to 300°C at 5°C.min -1 [START_REF] Härtel | Effects of particle reinforcement and ECAP on the precipitation kinetics of an Al-Cu alloy[END_REF][START_REF] Bassani | Thermal analysis of Al-Cu-Mg-Si alloy with Ag/Zr additions[END_REF][START_REF] Oguocha | Precipitation and dissolution kinetics in Al-Cu-Mg-Fe-Ni alloy 2618 and Al-alumina particle metal matrix composite[END_REF] for the phase and 280°C and 325°C at 50°C.min -1 [START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF] for the phase. Due to the heating rate variation, the precipitation temperature of the phase is lower than the presented values when the heating rate is 5°C.min -1 . Thus, phase exists at lower temperature than phase. Secondly, the growth mechanism can be addressed. It is believed that after approximately 80 hours at 190°C, the growth of the phase has stopped with a mean thickness of 2 nm [START_REF] Dorin | Strengthening mechanisms of T1 precipitates and their influence on the plasticity of an Al-Cu-Li alloy[END_REF]. For the phase, platelets have been measured up to approximatively 14 nm (24 with [START_REF] Bourgeois | Structure and energetics of the coherent interface between the θ′ precipitate phase and aluminium in Al-Cu[END_REF]) after 24 hours placed at 200°C. Due to the close precipitation temperatures and growth possibility of the and phases, the apparition of the during the ageing treatment at 200°C indicates its stability when compared to the phase.

Conclusions

In order to consider the use of aluminium alloys at an intermediate temperature for a structural application, mechanical test followed by the characterization of the microstructural evolution has been conducted on two 2xxx aluminium alloys, in the as-received T8 state and after a supplementary ageing treatment of 1000 hours at 200°C.

600

  Nanolab. SEM samples were smoothed out with graduated SiC papers and finally polished with a 1 µm suspended diamond solution. The samples were observed with a 20 kV acceleration tension and a current of 1.4 nA. A circular backscatter (CBS) detector was used to highlight the presence of intermetallic precipitates. To complete the study, size characterization of the coarse precipitates was conducted by Image J software (imagej.nih.gov/ij/) on SEM-CBS images.-Transmission Electron Microscopy observations were conducted on a JEOL JEM2010 HC (with an LaB6 filament) at 200 kV. Microstructure characterization was carried out on thin foils prepared by twin-jet electro-polishing in a 2/3 vol. % Methanol and 1/3 vol. % Nitric Acid solution at 20 V and -15°C on samples mechanically polished until reaching a thickness of 200 µm. Square samples (approximately 2 x 2 mm 2 ) glued with silver lacquer to a copper grid were prepared for classical TEM imaging.

Figure 1 .

 1 Figure 1. a. Engineering stress-strain tensile tests performed at room temperature and b) the corresponding work-hardening rates (in a log-log scale).

Figure 2 .

 2 Figure 2. Optical micrographs of the 2219 alloy a. as-received state and b. after ageing. Rolling direction and short transverse directions are noted on the micrographs RD and ST respectively. Dark dots are related to the presence of coarse precipitates.

Figure 3 .

 3 Figure 3. SEM-CBS micrographs of the 2050 alloy a. as-received state and b. after ageing where A arrows indicate grain boundaries with the presence of a precipitate free zone from either side, and B arrows are pointing towards possible sub-grains boundaries.

Figure 4 .

 4 Figure 4. TEM observations on the as-received state alloys. a. The 2050 alloy observed in <110>Al direction and b. 2219 alloy observed in <001>Al direction. A arrows show precipitates in (111) Al planes, B arrows the precipitates in (100) Al planes and C arrows show areas with small precipitates.

Figure 5 .Figure 6 .

 56 Figure 5. SAED patterns of a. the 2050-T84 and b. the 2219-T851 captured with an electron beam parallel to the <001> Al

Figure 7 .

 7 Figure 7. TEM observations of the aged alloys. a. The 2050 alloy in <110>Al direction and b. 2219 alloy in <001>Al direction. A arrows show precipitates in (100) Al planes and B arrows precipitates in (111) Al planes.

Figure 8 .

 8 Figure 8. SAED patterns of a. the aged 2050-T84 and b. the aged 2219-T851 captured with an electron beam parallel to the [001] Al .

Table 1 .

 1 The results have shown a drop of tensile properties of both alloys but with very different magnitudes. After the ageing treatment, the 2219 yield stress only drops of about 80 MPa while the 2050 yield stress drops of more than 200 MPa (tests performed in the rolling direction). This work demonstrates that these changes are linked with the evolution of the microstructure at a fine scale. Indeed, 2219 alloy presents only a slight change in size and distribution of the hardening nanoprecipitation. For the 2050 alloy, the disappearance of the smallest precipitates, which were responsible for the superior mechanical properties in the as-received state, is at the origin of its great strength decay. It is then envisioned that when the mechanical properties stability with temperature is sought, microstructures with more stable precipitates, such as theones, should be favoured, compared to T 1 -Al 2 CuLi precipitates giving rise to better mechanical properties but less stable in time with temperature. Nominal composition of the studied 2219 and 2050, as per AMS4262 and AMS4413respectively.

	Alloy Constituents (wt.%)
	Cu	Fe	Mg	Mn	Si	Ti	V	Zn	Zr	Li	Ag
	2219 5.80-6.80 <0.30 0.02	0.20-0.40 <0.20 0.02-0.10 0.05-0.15 <0.10 0.10-0.25 -	-
	2050 3.20-3.90 0.1	0.20-0.60 0.20-0.50 0.08 -	-	0.25 0.06-0.14 0.70-1.30 0.29

Table 2 .

 2 Summary of the tensile tests realized at room temperature for the as-received alloys and Thermal Aged alloys (noted 'aged') (Yield Tensile Strength (YS) and ultimate (US) strength values and the elongation ( ) at breaking are reported in the Rolling Direction (RD) and the Transverse Direction (TD) of the plates).

	Alloy	2050		2219
	Heat treatment	T84	aged	T851	aged
	Ftu (MPa)
	RD	535 ± 2	327 ± 3	457 ± 2	378 ± 5
	TD	517 ± 4	330 ± 2	431 ± 7	373 ± 5
	Fty (MPa)
	RD	502 ± 2	231 ± 3	358 ± 1	252 ± 3
	TD	463 ± 3	231 ± 2	350 ± 5	245 ± 3
	(%)
	RD	9.7 ± 0.8	12.4 ± 0.7	11.2 ± 0.4	13.4 ± 0.4
	TD	7.4 ± 0.9	10.0 ± 0.6	8.0 ± 0.3	9.8 ± 0.4

Table 3 .

 3 Grains dimensions and aspect ratio of the 2050 and 2219 alloys in as-received states (T84 and T851) and after the Thermal Ageing (TA1). Results are given for Rolling Direction (RD), Transverse Direction (TD) and Short Direction (SD).

	Alloy	2050		2219
	Heat treatment	T84	TA1	T851	TA1
	Measured
	dimension
	RD, µm	127± 13	132 ± 35	299 ± 32	234 ± 29
	Aspect ratio
	RD-TD	1.6	2.3	1.9	1.8
	RD-SD	3.2	4.0	4.0	3.4
	TD-SD	2.0	1.8	2.2	1.9


Samuel Beaussant

Sébastien Lengagne

Benoit Thuilot

Olivier Stasse
email: olivier.stasse@laas.fr

Delay Aware Universal Notice Network: Real world multi-robot transfer learning

Keywords: Transfer learning, Sim2real, Reinforcement learning, Robotic task, delay

General purpose simulators provide cheap training data to learn complex robotic skills. However, the transition from simulation to reality is often very challenging for the agent. One major issue is the delay on the physical robot that may deteriorate the performance of the deployed agent. Furthermore, once a successfully trained learning-based control policy is available, re-purposing the knowledge acquired by the agent to enable a structurally distinct agent to perform the same task is hazardous if done naively. In this work, we address the above issues with a single method, the DA-UNN (Delay Aware Universal Notice Network), which decomposes the knowledge into robot-specific and task-specific modules for fast transfer. Our framework deals with delays immanent to physical systems in order to improve sim2real transfer. We evaluate the efficiency of our approach using simulated and actual robots on a dynamic manipulation task where delay management is crucial.

I. INTRODUCTION

The Reinforcement Learning (RL) field has been successfully applied to a wide range of problems in the past years, demonstrating both its versatility and efficiency. From reaching human-level control on Atari games [START_REF] Mnih | Playing Atari with Deep Reinforcement Learning[END_REF] to beating world class champions at complex games [START_REF] Silver | A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play[END_REF], [START_REF] Openai | Dota 2 with Large Scale Deep Reinforcement Learning[END_REF], the RL field is full of promises. The robotic domain also benefited from the tremendous progresses made in RL as shown in recent work for quadruped robots [START_REF] Tsounis | DeepGait: Planning and Control of Quadrupedal Gaits using Deep Reinforcement Learning[END_REF] even succeeding in very intricate manipulation tasks such as solving a rubik's cube with a shadow hand [START_REF] Openai | Solving Rubik's Cube with a Robot Hand[END_REF]. However, despite great achievements, RL suffers from very low sample efficiency, which means that a large amount of interactions with the environment is needed to obtain a high-performance policy. One way to mitigate this issue is through the use of transfer learning which alleviates the burden of training models from scratch by re-purposing knowledge acquired on another domain.

Nevertheless, in the context of RL, the neural network architectures are too shallow and do not encourage knowledge segmentation as it is often the case in large Computer Vision models [START_REF] Donahue | A deep convolutional activation feature for generic visual recognition[END_REF]. As a consequence, if no particular precaution is taken, the unconstrained backpropagation procedure may result in an entangled knowledge representation. In this setting, it is difficult to determine which part of the network is relative to the task or the robot, making a partial or total transfer of the policy network hazardous if done naively and with very low chances of success. Fig. 1: Transfers considered. UNN module is trained with the BAM robot and then transferred to all robots. Vanilla agents are trained directly on the simulated robots and then transferred on the corresponding physical robots.

The UNN (Universal Notice Network) framework introduced in [START_REF] Mounsif | Universal Notice Network: Transferable Knowledge Among Agents[END_REF] tackles this issue by implementing the idea of knowledge segmentation between the agent and the task. More precisely, the purpose of the UNN framework is to enable multi-task and multi-robot transfer by creating a reusable and robot-agnostic module of skills. Previous work [START_REF] Mounsif | Universal Notice Network: Transferable Knowledge Among Agents[END_REF] already demonstrated the efficiency of UNN transfer in simulation on a wide variety of tasks and robots morphology, exhibiting zero-shot performance in some cases. By applying the same transfer method on real world robots, we wish to find out if results obtained in simulation generalize well to the real world.

However, simulation to real world transfer (also called sim2real in the literature) is still an open problem. In general, simulations are imperfect and difficult to calibrate. The resulting modeling discrepancies cause a reality gap, which makes the transfer of RL policies from simulation to the real-world non-trivial. Sim2real is especially appealing as it offers an efficient alternative to expansive real world data collections for learning complex robotic skills by training in a general-purpose simulator. However, most of the time, methods focus solely on domain adaptation between the real world and the simulation [START_REF] Bousmalis | Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping[END_REF], [START_REF] Arndt | Meta Reinforcement Learning for Sim-to-real Domain Adaptation[END_REF]. They tend to ignore troublesome hardware specific issues such as control latency induced by medium of data transmission, computation delay, sensor sampling rates (etc.) unmodeled in the simulator. Consequently, the policy obtained by training in simulation could be drastically disturbed once transferred in the real world if the task requires short reaction time.

In this paper, we consider the time delay associated with the physical system as another model's input by including it in the observed state. At training time, we randomize the value of the delay and show that the agent is able to adapt to multiple delays on a dynamic and delay-sensitive manipulation task. Our contributions are as follows:

1) We present and evaluate a delay-aware method to deal with the immanent delay on real hardware, thus furthering the adaptation capabilities of the UNN. 2) We evaluate the benefits of the UNN multi-robot transfer method over a vanilla transfer on real world robots.

A pool containing four differently shaped real and virtual robots will solve a dynamic manipulation task they have not been trained on, by using the knowledge created by another agent as depicted in Figure 1.

II. RELATED WORK

Transfer learning in RL has been recognized as an important direction towards building more sophisticated agents. For instance, multi-task learning aims at improving robots versatility via methods such as meta-learning [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF], [START_REF] Duan | RL 2 : Fast Reinforcement Learning via Slow Reinforcement Learning[END_REF]. Another interesting approach is to hierarchically decompose complex problems into tractable, simpler and reusable modules of skills through the use of concept networks [START_REF] Gudimella | Deep Reinforcement Learning for Dexterous Manipulation with Concept Networks[END_REF].

Transfer between morphologically distinct robots on the other hand, is currently less studied in the scientific literature. A method proposed in [START_REF] Mounsif | CoachGAN: Fast Adversarial Transfer Learning between Differently Shaped Entities[END_REF] encapsulates and leverages skills learned by a task expert by using GAN's discriminators as support for the knowledge transfer. Work by Gupta and Devin [START_REF] Gupta | Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning[END_REF] presents a method to learn an invariant feature space for transferring skills between different robots. Other work by the same authors [START_REF] Devin | Learning modular neural network policies for multi-task and multi-robot transfer[END_REF] uses a modular approach by training policy modules that are decomposed over robots and tasks. By doing so, they address both multi-robot and multi-task transfer by re-combining these modules to form a policy, with minimal additional re-training. Multiples combinations of robot-task pairs are trained together to create a latent space common to all modules. The UNN method used in this work, while very similar, differs from the prior method by explicitly defining the state shared between the task-specific and robotspecific modules. This choice suppresses the need of relying on a shared latent space between robot and task which may be prone to over-fitting if the number of available modules is too low. Furthermore, in the UNN framework, the task module can be trained separately and only once, thus creating a truly robot-agnostic module while saving the time and trouble of having to train multiple possible pairs of the training set.

Several methods has been proposed to deal with delays in robotic. In [START_REF] Behnke | Predicting away robot control latency[END_REF], the authors proposed a neural network based method to address the control delay issue. A predictor, approximated by a neural network, must infer the current state of a fast moving robot by observing a vector of stacked outdated states. To account for the imperfect actuators of the real hardware, authors of [START_REF] Openai | Solving Rubik's Cube with a Robot Hand[END_REF] introduced action delays with a probability of 0.5 at the beginning of every simulation training episode of a neural network with a LSTM layer. By doing so, they force the memory enhanced neural network to adapt to action delays. A cornerstone paper in RL with delay [START_REF] Walsh | Learning and planning in environments with delayed feedback[END_REF] learns a model of the undelayed Markov Decision Process (MDP) to simulate the most likely state in which the agent currently is, given the last observed (delayed) state and the k last actions taken since, k being the delay in timestep. This allows the agent to take decisions based on the expected current state rather than an outdated state, effectively undoing the harmful effect of delays. They also introduced the concept of Constant Delayed MDP used in section IV-A. Ramstedt and Pal [START_REF] Ramstedt | Real-Time Reinforcement Learning[END_REF] studied real-time RL by taking into account the computation time needed to select an action (supposed inferior to one time step). Their proposed algorithm, Real-Time Actor-Critic, additionally takes as input the action from the previous step in order to compensate the one-step action delay. However, previous methods are not suited for transfer between system with different delays. The agent must learn again from scratch whenever the system delay changes contrarily to our proposed method. III. UNIVERSAL NOTICE NETWORK Fig. 2: Schematic representation of the UNN [START_REF] Mounsif | Universal Notice Network: Transferable Knowledge Among Agents[END_REF] .

A. The UNN pipeline

Instead of learning a single policy that will have to handle both robot control and task resolution, the UNN method [START_REF] Mounsif | Universal Notice Network: Transferable Knowledge Among Agents[END_REF] relies on an explicit decomposition between task-specific and robot-specific knowledge as depicted in Figure 2. In this modular approach, a model of the task, the UNN (Figure 2) is created in such a way that any robot, regardless of its morphology, number of articulations or actuators can efficiently benefit from it. This is similar to creating a notice containing a set of high level instructions, that any kind of robot could follow to solve a given task. It is primary to ensure that the robot has the mobility and capacities required to comply with the UNN instructions and accomplish the task. Defining R the set of feasible actions the robot can produce and U the set of actions required by the UNN to perform a task, we assume in this paper that the robot can perform the required actions, as sum up in:

U ⊆ R (1)
Once this assumption is verified, two conditions are essential:

• First, the UNN module must be robot-agnostic to enable multi-robot transfer. This implies that robot-specific observations have to be translated into a feature space shared by the considered robots before being fed to the UNN module. • Secondly, it is necessary to design a controller that will map the UNN commands from the shared feature space into low-level, robot-specific actions that the robot can execute. These two requirements are handled by two additional modules inherent to the robot morphology called the bases. They are paired with the UNN and serve as an interface between the robot and the task module. More formally, the three modules form a pipeline composed of the input base m r i and the output base m r o , specific to the robot, which handle respectively the first and the second conditions, and the UNN m T u robot-agnostic and specific to the task only. In this setting, the state vector that is provided by the environment at each timestep can be split into two parts s r , s T , respectively holding data intrinsic to the considered robot and task-related information, independent from the agent. The input base m r i receives s r to compute:

s U = m r i (s r ) (2)
which can be considered as robot-agnostic. The input base m r i is thus responsible for mapping the robot space to the shared feature space where the UNN operates. The next processing stage is the UNN module (i.e. the task module), conditioned by the task related observation s T and the processed agent representation vector s U . It then computes:

o out = m T u (s T , s U ) (3)
where o out is the command vector in the shared feature space. Finally, o out is then re-mapped to the robot space by the output base with the following transformation:

a r = m r o (o out , s r ) (4)
which yields a r the effective action taken by the robot. In other words, the UNN module focuses solely on solving the task at hand, ignoring low level considerations such as the robot's DoF and shape, handled by the bases. This approach makes it possible to create a reusable module of skills that can be transferred to structurally different robots as long as their bases are available. It is then possible to build a library of UNN modules and robot's modules, draw any subset of interest from it and combine a UNN/Bases pair into a novel fully functional policy.

B. Modules training

In practice, each of the three sub-modules m r i , m T u , m r o can be either learned or obtained via analytical methods.

1) Bases modules: In the case where bases are obtained using neural networks, they can be trained on a suitable primitive task to acquire basic motor skills. Another alternative is to collect a dataset of trajectories of the robot and fit a regression model with supervised learning techniques. A last alternative, used in this work, consists to use analytical models for the robots bases.

2) UNN module: The UNN module (or task module) can be trained with or without the bases modules. In the first case, the UNN is coupled with a robot and its associated bases. The UNN interacts with the environment through the bases and its error on the task is back propagated through the network. In this case, we affect only the UNN module weights. However, the UNN module may then take advantage of the robot hardware structure to achieve the task (for instance, blocking an object between two articulations). As a consequence, the UNN may favor certain body configurations which may be detrimental for transfer. This issue is solved by using the Base Abstracted Modeling (BAM) method [START_REF] Mounsif | Bam ! base abstracted modeling with universal notice network : Fast skill transfer between mobile manipulators[END_REF]. It assimilates the robot to its effector by setting m r i and m r o to identity mappings, thus making no assumption on the robot's constitution and preventing any bias related to the bases. This is equivalent to considering a purely virtual and free-flying robot. Using BAM enables faster convergence of the policy and a more defined knowledge segmentation, which in turns improves UNN transfer.

IV. DELAY AWARE UNIVERSAL NOTICE NETWORK

A. Constant Delayed Markov Decision Process

The standard UNN proved its efficiency and versatility on a broad panel of tasks in simulation. However, these results were obtained with perfect robots (e.g no offset and no delay) acting in a standard Markov Decision Process (MDP). Traditionally it is assumed in RL that at every timestep, the environment pauses while the agent receives the current observation, in order to derive an action that will be executed without delay. Of course, things do not behave this way in the real world. All agent observations and actions are delayed by an amount depending on the hardware used for the task. Therefore, an agent trained in simulation without exposition to delays will perform worse or even fail in the real world if no precaution is taken.

This brings up the need to adopt a different decision process modeling to solve tasks in the presence of delay. As we consider the delay to be constant, we found the Constant Delay MDP formulation introduced in [START_REF] Walsh | Learning and planning in environments with delayed feedback[END_REF] to be well suited. A CDMDP defines the delay d as the number of timesteps between an agent occupying a state and receiving its feedback from the environment, where d ∈ R + . The delay is assumed to be part of the environment. A known result in CDMPD is that observation delay and action delay are equivalent from the agent's point of view [START_REF] Katsikopoulos | Markov decision processes with delays and asynchronous cost collection[END_REF]. Hence, we treated the total delay as being entirely caused by observation delay (see Figure 3).

B. Delay Aware UNN

A CDMDP can be transformed into a regular MDP by enlarging the state space with a history of the d last actions taken since the last observation. This transformation allows theoretically to derive an optimal policy for the CDMDP considered [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF]. However, this approach does not allow direct transfer between systems with different delays as the input dimension depends on d. In this work we address the delay issue by augmenting the state space of the UNN module with the estimated delay of the system and by training the agent on a corresponding delayed environment as depicted in Figure 3. A key feature of the UNN is its ability to adapt to any robot regardless of its morphology. To keep this idea of "universality", the delay was randomized during training to ensure that the UNN can adapt to a wide range of delay. By giving it access to the immanent delay, we enable the UNN to act accordingly and to develop predictive capabilities. Thus, we add d to the task specific observations.

During training, the delay is sampled regularly from a discrete uniform distribution as U(d min , d max ) where d min and d max are respectively the minimum and the maximum delay considered for the environments. Since there is no assumption about the systems, we assumed a uniform distribution of the delay. But any knowledge could be used to deduce a better delay distribution. When deployed, the identified delay of the system is fed to the UNN, so it can act accordingly. While our approach can only yield sub-optimal CDMDP policies due to the incomplete state space considered, we believe that it represents an interesting trade-off between optimality and flexibility. This very simple method can improve drastically the performance of an agent on a delayed MDP as presented in section VI, given that the delay has been accurately determined and is suited for transfer on systems with different delays.

V. EXPERIMENTAL SETUP

A. System Architecture and Robots

In this section, we briefly present the different robots adopted throughout these experiments. We tested our method on both physical and simulated robots to demonstrate its efficiency and versatility. The physical robots used were a serial arm braccio robot with 5 DoF and a 4 DoF serial arm (see Figure 4b). These DIY robots are cheap and usually hard to work with, given their low reliability. Still, we manage to use them efficiently in our experiments. We also considered their simulated counterparts (see Figure 4a).

To sum things up, 5 different kinds of robots were used:

• BAM: the virtual BAM robot with the identity bases.

• Robot 1: the virtual braccio robot.

• Robot 2: the virtual 4 DoF robot.

• Robot 3: the physical braccio robot.

• Robot 4: the physical 4 DoF robot. Significant offset was present in the robots joints, making each movement inaccurate. In this regard, the offsets first needed to be identified, in order to use analytical models efficiently on both physical robots. A fixed webcam was used to obtain the required pose estimations with OpenCV. The control frequency was 10 Hz, which means the agent was observing the environment state and acting every 0.1 second. The nominal delay was in average 300 ms on the physical systems. We identified the delay by measuring the time between a command send to the robot and the observation by the agent that the robot moved.

On the simulation side, agent's training was performed in simulation using the Unity physic simulator with the ML-agent package introduced in [START_REF] Juliani | Unity: A General Platform for Intelligent Agents[END_REF], a set of convenient tools for RL with a complete and reliable implementation of several RL algorithms. The PPO algorithm [START_REF] Schulman | Proximal Policy Optimization Algorithms[END_REF] was used to create the neural network policies, as it provides a monotonous performance improvement while being perfectly adapted to continuous action spaces. On-policy algorithm are also known to deal better with delays. We trained four kinds of agents:

• Delay Aware UNN Agent: The BAM virtual robot is trained in simulation with exposition to randomized delays to create the UNN. • Delay Aware Vanilla Agent: The agent is trained from scratch directly on the simulated robot, with exposition to randomized delays. • Finally, we also considered their delay unaware counterparts, trained without exposition to delays, in order to display the benefits of our delay management approach. These agents will be used for the transfers detailed in section VI-B (see Figure 1).

B. Task description

We display our method benefits on a 2D manipulation task (planar task), where a robot needs to keep a ball at a desired position on a gutter. In this regard, only 3DoF were required for the physical robots (base rotation and wrist roll unused). To further increase the gap between both robots, the Robot 4 was used as a 2 DoF robot (wrist pitch unused). The gutter is fixed at one end and held at the other end by the robot's effector which therefore decides of its orientation and, as a consequence, of the position of the ball (see Figure 5). This task can be formalized with the following MDP:

State: s t ∈ R 4+1 : the ball position and velocity on the gutter, the effector height, the desired ball position and the system delay d for the delay aware agents.

Action: a t ∈ R n is the target joints position (n being the number of considered joints). However, the vanilla agent was not making any progress with a full access to the action space. Indeed, to balance the ball on the gutter, it is first needed to hold it properly. These desired body configurations are just a fraction of the full state space and it is very unlikely to discover them without any prior knowledge of the task. To ease the vanilla agent learning process, its action space was constrained to output joints offsets values w.r.t a reference joints position which maintained the gutter in an equilibrium position.

Reward:

r t = r -β|θ e | if d des,b < δ -αd des,b -β|θ e | else ( 5
)
where r is a small positive reward, δ is the positive reward area and d des,b is the distance between the ball and the desired ball position. |θ e | is the angle between the effector pose and the vertical plane, α and β a weighting constant. This penalty ensures that the effector is in the right orientation to hold the gutter properly for the vanilla agent. The effector orientation constraint for the UNN is handled by the output base, which means that β is set to zero when training the UNN.

C. Delay Aware UNN creation

Creating the UNN module means training an agent to balance the gutter and keep the ball at the desired position. As the BAM method showed better transfer results [START_REF] Mounsif | Bam ! base abstracted modeling with universal notice network : Fast skill transfer between mobile manipulators[END_REF], we decided to use it to create the UNN module. In this setting the robot is assimilated to its effector and the UNN output o out ∈ R is a single value indicating at what height below or above the horizontal reference position of the gutter the effector should be. To take actions, the agent observes s T , specific task information, as well as the effector height given by m r i (s r ). For this task, we chose the intermediate state s u and o out shared between the UNN module and the bases to be the effector position. The UNN module was receiving an extra input d ∼ U(0.1, 1) representing the current delay of the system during training. A delay range between 0 and 1 is recommended as it corresponds to a normalized input. In our case, it also corresponds to our actual delay in second, with 0.1s being the smallest delay possible for our control frequency. The delay was created on the simulator by stacking the observations in a FIFO buffer, before feeding them to the UNN module.

VI. RESULTS

In this section, we present our results both on training and transferring on the chosen manipulation task. In particular, we compare the UNN agents with the vanilla agents with and without delay awareness. For further experiments, delay was added artificially to the real system with the same FIFO method seen in section V-C. All the hyper-parameters used during training are available in appendix A. A description of the robot morphology is presented in B. Code can be found at github.com/sabeaussan/DelayAwareUNN. Videos showing our results are available here.

A. Training

During the training, the desired ball position and system delay (for delay aware agents) were regularly changed to improve the adaptive capabilities and re-usability of the UNN. More precisely, a new delay d was sampled from U(0.1, 1) every 15 episodes. The desired ball position given to the model, varying between 20% and 80% of the gutter length, was also sampled from a uniform distribution U(0.2, 0.8) every 1000 training steps. Both the BAM agents and the Vanilla agents were trained for 4 millions steps. Figure 6 shows the cumulative reward obtained per episode. Only the term d des,b (distance between the ball and the desired position) common to both reward functions was considered for the comparison, as it reflects the agent overall progression on the task. As shown in Figure 6, the BAM agents in both settings converge slightly faster than their vanilla counterparts. The BAM agents focus solely on the task, leaving robot specific considerations to their bases. This decomposition of the learning problem similar to hierarchical RL eases the learning process. It is also worth noting that introducing varying delay during training reduces the convergence speed, as the task becomes more challenging. However, in the UNN framework, this training overhead is outweighed by the increased reusability of the UNN module.

B. Transfer

There is two kinds of transfer to consider: simulation to real robot transfer and robot to robot transfer. The UNN framework mitigates the sim2real transfer problem by considering the real robot and the simulated one as two different robots, each one with its own bases, thus partially addressing the sim2real transfer as a robot to robot transfer. In this section we evaluate two methods of transfer

• UNN transfer: Once trained to convergence with the BAM robot, the UNN module is transferred to each robot of the set. • Vanilla transfer: The vanilla agents trained on the simulated robot are directly transferred to their physical counterpart. This will serve as a baseline to study the UNN benefits for sim2real transfer. The performance metric used was the integral of the absolute value of the error between the ball position and the desired ball position over time:

t d des,b dt (6)
Where d des,b is the distance between the ball and the desired ball position. This metric has the advantage of taking into account both settling time and the steady state error (the closer to 0, the better). For a fair comparison, each experiment has been conducted with the same settings (same initial ball position and desired ball position). Performances displayed in Tables I and II were averaged over 50 episodes.

1) Influence of delay:

In this part, we evaluate the first contribution of this work: our delay management method, on both simulated and physical robots. Three delays were considered for the experiments: 300 ms (corresponding to the delay on the physical system), 500 ms and 700 ms. Figure 7 shows ball trajectories for the three delays considered, obtained by the UNN agents on robot 1 (virtual braccio robot) and 3 (physical braccio robot). On the simulation side, we added an optimal trajectory obtained with the delay unaware UNN agent acting on an undelayed environments to serve as a reference (see Figure 7a). The same agent was then exposed to the delays considered to study how quick performance deteriorates for unaware agents as the delay increases. As shown, agents not exposed to delays during training completely failed and systematically overshot when trying to get the ball at the required position in delayed environment. Figure 7b emphasizes the inability of the delay unaware agents to cope with the physical system immanent delay (300 ms) as the ball starts oscillating. Moreover, as the delay increases, the delay unaware agents tend to become unstable. As for delay aware agents, in the simulation, they still manage to follow closely the optimal trajectory.

Table I shows the performances obtained in sim2sim transfer with the UNN agents on both delay aware and unaware settings. It is shown that delay aware agents perform from 3.5 to 5.72 times better than their unaware counterparts. It is also clear from looking at Figure 7b and Table II, which shows the average performance after sim2real transfer, that dealing with delay in simulation greatly improves the results of the UNN agents once deployed on the physical robots. Vanilla agents also benefited from this delay management method, as shown in Table IIb, demonstrating the versatility of the proposed method.

2) sim2sim transfer: In this paragraph, we discuss the results obtained when transferring the delay aware UNN module from the BAM robot to robots 1 and 2 in simulation. We also compare the performance obtained against delay aware vanilla agents which learned the task from scratch on robots 1 and 2. As shown in Tables I and IIb, UNN-based approachs slightly outperform the policy of the vanilla agents for the robots and delays considered. We want to emphasize that the UNN module has been trained only once and on only one robot, the BAM robot, but still performs better than the vanilla agents specifically trained on robots 1 Results are displayed as delay aware method on the left / delay unaware method on the right. and 2. These results demonstrate the appealing re-usability and effectiveness of the UNN module. In some cases, the delay aware UNN agents achieve zero-shot performances (e.g robot 1 on delay 0.5). In the worst case, the transfer efficiency is 90.9% (3.12/3.43), 100% being the performance obtained by the UNN module on the BAM robot. In average, the transfer efficiency is 97.7 % for robot 1 and 93.7% on robot 2. Ideally, the UNN module paired with any of the robots would yield similar performance as with the BAM robot if equation (1) was respected. However, in some cases the body configurations required to comply with the UNN commands are not precisely achievable by the robot. For instance, the desired effector position need some of the joints to rotate beyond their limits. This also explain why the UNN transfer is less efficient on the 2 DoF robot, as it is less expressive and has a harder time following UNN commands.

3) sim2real transfer: In this paragraph, we study the UNN methodology as a sim2real transfer tool. More specifically, we compare the performance obtained after transfer on the physical robots for Vanilla agents and UNN agents. In this case, both the UNN module obtained on the BAM robot and the vanilla agents obtained on the simulated robots, were transferred to the physical robots. As shown in Figure 7, the UNN agent on robot 3 still manages to put the ball at the desired positions without too much overshooting. Table II shows the results obtained. As usual, the agents trained in simulation and transferred to the real world show lower performance than their virtual counterparts due to the reality gap. However, they still manage to obtain decent performances. One notable result is that the delay unaware agents transferred to the physical robot obtain very poor performance unlike their delay-aware counterpart. Once again, UNN based agents outperform vanilla agents. Moreover, the UNN based transfer reaches up to 78% (100% corresponds to the BAM performance) in the best case, while the vanilla transfer reaches 70.6% (100% corresponds to the vanilla agent on robot 1). In average, the UNN sim2real transfer efficiency is 68% on robot 3 and 54% on robot 4, against 63% on robot 3 and 52.7% for robot 4 for the vanilla sim2real transfer. As mentioned earlier, this slight sim2real improvement can be attributed to the robot-agnostic nature of the UNN module. Indeed, even if the vanilla agents were trained in simulation with a virtual copy, it remains an inaccurate model of the physical robot. The UNN on the other hand ignores those discrepancies by considering the physical robot and the virtual one as two different robots, each with their own bases.

C. Discussion and perspectives

From the previous results, it appears clearly that the delay management method used considerably improves the performances when working with delayed environment, as its often the case on the real world. Moreover, the UNN approach not only achieves very efficient transfer between robots in simulation, but slightly improves sim2real transfer over vanilla transfer. However, the zero-shot sim2real transfer efficiency is nowhere near what was obtained for the sim2sim transfers but further training could be done on the physical robots to achieve better performance. As aforementionned, the UNN mitigates the sim2real transfer by considering the physical system as just another robot that can be interfaced with the UNN module. Nevertheless, the UNN module which was trained in simulation can still overfit on its environment. As a result, the instructions given can be unsuitable if it is placed in a new domain with a slightly different state distribution, e.g the real world. Fortunately, the UNN approach can be combined with state-of-the art sim2real methods such as automatic domain randomization [START_REF] Openai | Solving Rubik's Cube with a Robot Hand[END_REF] to improve sim2real transfer.

VII. CONCLUSION

In this work, we studied the benefits of the UNN transfer for a sim2real application. More specifically, we addressed the delay management problem that occurs when working with a physical system by making the UNN "aware" of the latency of the system it is working with. By doing so, we extended the versatility of the UNN method and the range of compatible systems. We demonstrated this method efficiency by solving a dynamic manipulation task where delay management is paramount and showed that transfer across systems with heterogeneous delays and structurally distinct robots is possible. However, the UNN approach only is not sufficient for efficient sim2real transfer, but could be enhanced with other sim2real methods. This work empirically demonstrated the feasibility of our approach on a low dimensional task. Future work will investigate the efficiency of our delaymanagement method on a higher dimensional task.

Fig. 3 :

 3 Fig. 3: Schematic representation of the delay aware UNN. Observations are queued into a pile of length d and each timestep, the observation at the top is fed to the agent (first in, first out).

Fig. 4 :

 4 Fig. 4: Robots considered for the experiments

Fig. 5 :

 5 Fig. 5: Physical experiments setup. Left robot will perform the task, while the right robot is used only to hold one end of the gutter.

Fig. 6 :

 6 Fig. 6: Training curves. All the agents were trained for 400000 steps.

Fig. 7 :

 7 Fig. 7: Ball trajectory with 0.3, 0.8 and 0.5 as desired ball position.

TABLE I :

 I Sim2sim transfer. Performances obtained for the UNN transfer on the simulated robots. Results are displayed with delay aware method on the left / delay unaware method on the right.

TABLE II :

 II Sim2real transfer. Performances obtained for the vanilla transfer and the UNN transfer on the physical robots.

APPENDIX

A. Neural network parameters

The neural networks used for the UNN module were feedforward networks with 2 hidden layers of 128 units each. The activation function used was the Swish function [START_REF] Ramachandran | Searching for Activation Functions[END_REF]. For the PPO algorithm, the discount factor γ is 0.99, the clipping hyperparameter is 0.2, the learning rate was 3.10 -4 with a linear decay. The batch size was 1024, the buffer size was 12000 and the number of epoch was 5. The Generalized Advantage Estimation [START_REF] Schulman | High-Dimensional Continuous Control Using Generalized Advantage Estimation[END_REF]