R F A B B R O

Laboratoire Pimm

Developments in Nd:YAG laser welding

Keywords:

Laser welding in keyhole (KH) mode using solid-state laser emitting in near infrared at nearly 1 micron wavelength is discussed in this chapter. The main physical processes involved in laser welding are presented and the reasons for using this laser wavelength are shown. Section 3.2 describes the KH geometry and related physical mechanisms controlling its stability. The role of the main operating parameters is also presented. Section 3.3 shows examples of the evolution of keyhole and weld pool behaviour for various welding speeds illustrating the mechanisms discussed in Section 3.2. Finally in the conclusion, expected diagnostics improvements necessary for supporting adapted numerical simulations of this laser welding process are discussed.

Introduction

The recent advent of the new generation of solid-state lasers based on Nd-YAG, disk or fibre technology, emitting in the near infrared region in the range of 1.03-1.07 micron, has renewed and strongly amplified the interest in using these types of lasers for the welding process. Compared to CO 2 lasers, their greater ability to transport this wavelength with a fibre offers an incomparable advantage for industrial applications particularly when anthropomorphic robots are used. Also, these lasers now have a very high level of reliability, with high efficiencies and presently achieve CW powers up to several tens of kW, with an outstanding beam quality reaching quite near diffraction limit. So, for these reasons they have become a very popular tool for joining technology where high performance and high quality with good flexibility are required for the welding of many different materials.

One usually defines two different types of welding. The first one is called the conduction mode: in that case, the absorbed intensity is lower than a given threshold, which depends on the material and the welding speed, where evaporation inside the focal spot is not sufficienthy intense to generate a keyhole. The size of the corresponding melt pool is then mainly controlled by the heat conduction from the focal spot and welding speed. Keyhole (KH) welding or deep penetration welding is the second type of laser welding, which occurs when the laser intensity inside the focal spot is high for generating a high evaporation rate, and so the corresponding recoil pressure depresses the melt pool surface and allows the laser beam to penetrate deeply inside the material. KH generation is therefore the basic mechanism of this mode of welding characterized by a combination of complex physical mechanisms such as laser absorption, conductive and convective heat transfer, a complex hydrodynamic flow around the KH and inside the melt pool, and a strong coupling of the vapour plume with this melt pool.

If conductive welding appears at first glance rather simple to describe, the dominant mechanisms being laser absorption at the melt pool surface, and conductive heat flow inside the melt whose hydrodynamics is controlled by Marangoni surface effect, this is not at all the case with KH welding. One can easily understand that there are many unknowns concerning these very different complex mechanisms that occur during that process and great efforts to understand them are still being undertaken in many laboratories in order to master this important laser process, to make it more reliable and also to improve it in order to achieve better performance. In the first part of this chapter, we will review the main phenomena that are believed to control this process: they concern basically the keyhole (KH) formation and its dynamics, and the resulting hydrodynamics of the melt pool that will define the final weld seam quality. The effect of operating parameters, such as welding speed, laser intensity or power and of course the nature of the material used are also of primary importance for understanding this global behaviour. In the second part, a typical example of evolution of the resulting weld seam is presented when two of the main operating parameters, the welding speed and the incident power, are modified. The chapter concludes by presenting possible directions for improvement of the present understanding necessary for future adaptations to numerical simulations of that process.

3.2

Basics of laser welding in keyhole (KH) mode with solid-state lasers

Main geometrical KH characteristics: wall inclination and KH depth

The basic phenomenon that is at the origin of the KH formation is the recoil pressure that pushes the liquid generated during the surface laser heating, on the sides of the focal spot and allows further penetration of the laser beam inside the material. This 'piston effect', which has been described by [START_REF] Semak | The role of recoil pressure in energy balance during laser materials processing[END_REF] for welding conditions, can be easily understood in the case of pulsed laser welding, or of course during drilling processes that occur at much higher intensities than for welding, i.e. when there is no displacement of material during the laser pulse: the vapour/liquid interface moves into the laser direction, inside the material with a 'drilling velocity' V d and the melt is ejected laterally with a velocity V m mainly controlled by the recoil pressure P evap resulting from the strong surface evaporation. When there is a relative displacement between the laser beam and material, defined by the welding speed V w , the previous scheme is greatly modified: the KH geometry no longer has an axial symmetry: it is then defined by a keyhole front wall (KFW) and a keyhole rear wall (KRW), characterized by their respective inclinations along the welding direction and the side walls can be considered as quite vertical (see Fig. 3.1(a)).

For this configuration, the main process of laser energy deposition results from the incident laser beam that impinges the KFW under a large angle of incidence. Depending on the partial or full penetration regime of the KH, its aspect ratio L/D (L: KH depth, D: KH diameter), and its KFW inclination , the local piston effect on the KFW ejects the melt sideways around the KH and downwards along the KFW. This KFW inclination angle is a very important parameter: first, it defines the main absorptivity A( ) of the incoming laser beam on this surface. As the local angle of incidence is rather high, the absorptivity can be important if one considers the Fresnel equation behaviours and their dependence with the beam polarization [START_REF] Dausinger | Energy coupling efficiency in laser surface treatment[END_REF] that can be modified by local surface roughness [START_REF] Bergström | The absorption of light by rough metal surfaces -a three-dimensional ray-tracing analysis[END_REF]. Also, macroscopic or microscopic surface deformations resulting from surface instabilities generating humps or ripples [START_REF] Golubev | Melt removal mechanisms in gas-assisted laser cutting of materials[END_REF]) have a non-negligible effect on the level of local absorptivity. Direct measurements of KFW absorptivity by using full penetration welding experiments have shown levels of absorptivity in the range of 60-70% [START_REF] Fabbro | Study of keyhole behavior for full penetration Nd-Yag CW laser welding[END_REF] for non-polarized beams. As a consequence, one should expect that multiple reflections of the laser beam inside the KH are not important for the process of energy distribution inside the KH when medium range incident laser intensities are used.

The second consequence that implies the KFW inclination is that it controls the penetration depth. From geometrical considerations (see Fig. 3.1(b)), it is easy to see that the contribution on the penetration depth L of the impingement of the incident beam with a diameter D, when the entire beam is intercepted, is given by L = D/tg . Physical analysis of the equilibrium of the KFW has allowed us to determine the main basic parameters of the KH geometry, for stationary conditions (Fabbro, 2010b). One can find that the KFW inclination is given by a rather simple relation:

tg ≈ V w /(kI 0 A 0 ) [3.1]
In Eq. [3.1], the parameter k is a proportionality factor between the drilling speed V d and the absorbed intensity: V d ≈ k . I abs . This factor k is representative of some energy balance of the process that depends mainly on the workpiece material. It can be determined experimentally for similar Incident beam has a uniform intensity I 0 , with a diameter D. Material is moving from right to left with the welding speed V w . Under these conditions, the bottom of the KH can be enlarged by the reflected beam.

conditions of incident laser intensity (typically k ≈ 3 ¥ 10 -11 m 3 /J for 304 stainless steel) or estimated from the results of the 'piston model' of [START_REF] Semak | The role of recoil pressure in energy balance during laser materials processing[END_REF]. So the dependence on penetration depth L with the main operating parameters will be:

L ≈ k I 0 A 0 D/V w = (4A 0 k/ ) • P/(D • V w ) [3.2]
where P is the incident laser power. It is interesting to notice that this scaling dependence on the penetration depth L with the ratio P/(D • V w ) has been verified in many experiments [START_REF] Beyer | High power laser materials processing -new developments and trends[END_REF][START_REF] Dausinger | Laser welding of aluminium alloys: problems, approaches for improvement and applications[END_REF]Verhaege and Hilton, 2005), particularly at high welding speeds V w . At low welding speeds, because of the small inclination of the KFW, it is easy to show that the resulting absorbed laser intensity on the KFW becomes very low and only depends on the welding speed, with the relation:

I abs = V w /k [3.3] Equation [3.3
] is interesting because it shows that the absorbed intensity depends only on the welding speed, and not on the incident intensity. This is a very simple self-regulating mechanism: for a given welding speed V w , as the incident laser intensity I 0 increases, the KH depth L also increases (Eq. [3.2]), but the absorbed intensity on FKW is constant, due to the decrease of the FKW inclination (Eq. [3.1]). Now, let's consider the KH stability at low welding speed. The KH will be maintained open, if the excess of recoil pressure P evap over the ambient pressure P a counterbalances the surface tension pressure P s = 2 /D inside the KH [START_REF] Hirano | Experimental determination of temperature threshold for melt surface deformation during laser interaction on iron at atmospheric pressure[END_REF]; as this opening pressure is increasing with the absorbed intensity, Eq. [3.3] shows that for low welding speeds, this excess of recoil pressure should become lower than the characteristic KH closing pressure controlled by surface tension (Fabbro, 2010b). So, for these conditions, the KH cannot be stationary; it closes itself quite periodically, and is continuously re-opened, as in a drilling process, by the continuous incoming incident laser beam. From these considerations, one can estimate that the maximum penetration depth L max for achieving a stable KH regime is given by:

L max ≈ (2CA 0 / )P [3.4]
where C is a constant defined by the relation between the evaporation pressure P evap with the absorbed intensity: P evap -P a ≈ CI abs , where C is in the order of 1 bar/(MW/cm 2 ) for SUS304 (Fabbro et al., 2006a). For greater penetration depths, one can expect strong fluctuations that may lead to some defects such as trapped porosity from the KH bottom inside the weld seam, strong spatter generation and, as a final consequence, some limitation of the penetration depth. Typically, for SUS304, one can estimate that (Fabbro, 2010b):

L max (mm) ≈ 1.5P (kW) [3.5]
The last important effect of the KFW inclination concerns the direction and the level of the dynamic pressure of the vapour emitted by the ablation process on the KFW. We have seen that for the range of incident intensities characteristic of KH generation, the process generates a recoil pressure that can be estimated to be linearly dependent on the absorbed intensity on the KFW. So, as the welding speed increases, the KFW inclination also increases as well as the absorbed intensity on the KFW. Consequently, the recoil pressure also increases and typically can reach a few bars for an absorbed intensity of about 1 MW/cm 2 on steel material (Fabbro et al., 2006a). As the direction of expansion of the vapour is perpendicular to the local surface of the KFW, this vapour jet impinges on the KRW with a dynamic pressure P d that can be similar to the recoil pressure, if the distance between the KFW and KRW is small, which is usually the case. It is the impact pressure P d of this vapour on the KRW that maintains the KH opened, balancing the sum P s + P m of the 'closing' pressures of the KRW that result from the surface tension pressure P s = 2 /D and from the dynamic pressure P m ≈ 0.5 m V 2 m due to the liquid melt flowing laterally around the KH with the speed V m , as a result of the recoil pressure applied on the KFW. (One must mention that if the welding velocity is not too low, the laser beam does not hit the KRW; so we do not have the term P evap for the previous pressure balance on KRW.) For these conditions, one can show that the equilibrium of the KRW cannot be stable (Fabbro and Chouf, 2000a, b), and so the KRW shows many fluctuations that are likely to generate some defects inside the melt pool. Moreover, these fluctuations can be even more easily triggered, if one considers that surface instabilities such as humps or ripples are present on the KFW surface, this effect being reinforced at low welding speeds [START_REF] Golubev | Melt removal mechanisms in gas-assisted laser cutting of materials[END_REF]. In that case, important localized evaporation on these humps located on the KFW produces strong jets of vapour that can easily destabilize the KRW. As a result, bubbles of vapour can be generated inside the melt pool. Depending on the fraction content of the trapped shielding gas inside the KH (which may not be negligible, if one considers the very unstable behaviour of the KH in this regime), their position along the KH and the velocity field inside the melt pool, these bubbles can be evacuated at the melt pool surface, or if not, may result as porosity whose final size depends mainly on their shielding gas content when they are initially generated. The dynamic behaviour of these bubbles, whether or not it leads to porosity formation, has been remarkably demonstrated by the various X-ray shadowgraphy experiments realized by the Osaka team [START_REF] Katayama | Understanding and improving process control in pulsed and continuous wave laser welding[END_REF] for the last 20 years and also more recently at IFSW in Stuttgart [START_REF] Abt | Novel X-ray diagnostic for in-situ diagnostics of laser based processes -first approach[END_REF]. Also, some rather recent 2D and 3D numerical simulations have been able to reproduce the complex mechanisms of the interaction of the vapour plume with the melt pool [START_REF] Fabbro | Modelling of the melted bath movement induced by the vapour flow in deep penetration laser welding[END_REF][START_REF] Amara | Modelling of gas jet effect on melt pool movements during deep penetration laser welding[END_REF][START_REF] Zhang | An investigation on the effects of side assisting gas flow and metallic vapour jet on the stability of keyhole and molten pool during laser full-penetration welding[END_REF].

The previous considerations about the KH geometry (and more precisely its KFW inclination defined by the welding speed and the incident laser intensity), and the corresponding direction and dynamic pressure of the emitted vapour allow us to have a general view of the behaviour of the coupling between the KH and the surrounding melt pool: ∑ At low welding speed, the KH is rather deep, quite vertical and very unstable, leading to irregular collapses. The laser beam impinges on the collapsed liquid wall and pushes it downwards generating bubbles inside the melt pool that are not easily evacuated (with possible pore generation). Upward expansion of the vapour plume induces droplets and spatter generation. ∑ When the welding speed increases, KFW inclination increases and correspondingly the KH depth decreases. The evaporation on the KFW generates vapour jets that impinge on the KRW, producing different types of defects that may or may not be evacuated, depending on their size and the local induced upward flow of the liquid behind the KRW. So, bubbles leading to pores can be created and also droplets are emitted from the top part of the KRW and/or from the rear rim of the KH aperture. For some range of welding speeds, adapted conditions may lead to quite a stable configuration where the upper part of the KH is opened enough to allow a correct vapour ejection with a quasi-stationary velocity field inside the melt pool. In that case, porosity formation can be suppressed. ∑ At high welding speed, the KH depth still decreases and becomes rather small. The KFW inclination and therefore the absorbed laser intensity are important, so the resulting vapour jet is intense but it is emitted quite vertically; so, it no longer interacts with the KRW, and the previous defects cannot appear. However, as the melt flow expelled sideways from the KFW has a high velocity under these conditions, the resulting melt pool may present strong undercuts or even humping defects at welding speeds. This evolution of the KH and melt pool behaviour as a function of the welding speed will be detailed in Section 3.3.

Spatter formation

In addition to the porosity formation mechanism, the vapour jet issued from the KFW also plays a very important role in the spatter generation mechanism. From Eq. [3.1], we are able to know the mean direction of the ejected metal vapour, because it is emitted perpendicularly to the KFW whose inclination increases when the welding speed increases or the incident laser intensity decreases. But when increases, the vapour jet impinges on the upper part of the KRW, with an increase of its dynamic pressure due to higher absorbed intensity. Therefore, under these conditions, spatters can be generated and are ejected rearwards, in the direction of the vapour jet. By using high-speed video camera, the analysis of the maximum ejection angle of the trajectory of droplets emitted from the rear part of the KH rim confirms the linear relation between the KFW inclination and this maximum angle of droplet trajectory [START_REF] Weberpals | InfIuence of inclination angle on spatter behaviour at welding with lasers of strong focusability[END_REF]. In fact, the direction of this spatter emission is deviated from the direction of the vapour jet. This is because the final velocity of the spatters is given by a sum of momentum of the vapour jet emitted perpendicularly from the KFW and an upward directed melt flow occurring on the KRW [START_REF] Weberpals | Nutzen und Grenzen gutter Fokussierbarkeit beim Laserscheweinssen[END_REF]. This deviation is more important in partial penetration than in full penetration. This spatter ejection can even be partly controlled and reduced: for similar operating conditions, by inclining the laser axis in the forward direction, the resulting reduction of the KFW inclination makes the vapour jet rather collide with the central part of the KRW, instead of its upper part, which is more sensitive to droplet ejection. In that case, the KH becomes more elongated, and spatter formation or induced defects inside the melt pool are minimized.

There is also another method for reducing spatter emission and for improving the final weld seam quality. It is usually observed that when the laser focal spot is located inside the workpiece, at a distance from the workpiece surface of about 1-3 mm (depending on the corresponding beam waist and beam intensity profile), the weld seam quality is significantly improved and even a greater penetration depth results. This effect is likely to result from the localization of the most intense zone of evaporation in the beam, which corresponds to the minimum focal spot, which is then deep inside the workpiece. At this depth, the KRW and behind it the melt pool, can sustain more easily the impact of the vapour jet, than in the wider upper part of the KRW. So, one can understand that the vapour jet is the main mechanism that can perturb and efficiently modify the hydrodynamics of the melt pool. Its control, if possible, is of primary importance during laser welding. Additional examples will be shown in Section 3.3.

A similar, and even stronger, perturbation is observed when Zn-coated sheets are welded in overlap joint configuration without a gap between the sheets. In that case, hot zinc vapours can only emerge inside the keyhole through a channel that is localized at the sheet interface and emerging at the KFW. Due to their high dynamic pressure (roughly corresponding to the saturation pressure of Zn at steel melting temperature), Zn vapours expel violently the liquid and large amounts of the rear melt pool can be blown away leading to cavities and blowholes in the weld seam. It has been shown experimentally that, depending on the relative position of this sheet interface with the upper surface, this liquid perturbation can be minimized in order to favour the Zn vapour expansion (see Fig. 3.2). The same effect is also obtained when a top-hat intensity distribution is used compared to a 'peaked' intensity distribution, because a clear aperture is similarly generated at the bottom of the KRW. Another possible solution consists in using an elongated focal spot along the welding direction that maintains the KRW at enough distance, at about 1 mm, in order to allow the dynamic pressure of Zn vapour to decrease. However, this solution requires a rather large incident laser power ( [START_REF] Kielwasser | Physical processes during pulsed Nd:YAG laser and CO 2 laser welding of zinc coated steel[END_REF]Fabbro et al., 2006b).

It is also well known that an efficient solution to this problem is to maintain a gap of about 0.1-0.2 mm depending on the thickness of the sheet material and the coatings, which allows Zn vapour venting between the two sheets. Some recent simulations of this effect show the resulting modification of the melt flow field [START_REF] Geiger | A 3D transient model of keyhole and melt pool dynamics in laser beam welding applied to the joining of zinc coated sheets[END_REF]. For generating such a gap between the two sheets, many techniques have been designed, but an elegant solution was recently proposed by pre-processing dimples on the upper surface of the second sheet before the final assembly [START_REF] Gu | Remote laser welding of Zn-coated sheet metal component in a lap configuration utilizing humping effect[END_REF]. These dimples, with controlled height, can be easily generated using a laser weld seam realized in the humping regime, i.e. when the weld seam is obtained at very high welding speed, typically 60-80 m/min. This operation is then very rapidly obtained on these parts, using high-speed remote scanning optics and high beam quality lasers.

At low welding speed, the inclination angle of the KFW is very small, so the keyhole is quite vertical. It is also rather unstable, continuously re-opened and consequently the metal vapour plume is mainly ejected upwards. Under these conditions, this vapour plume has much less impact on the KRW, but interacts mainly by its friction effect along the KH walls. The corresponding shear stress g and induced upwards melt flow velocity U 0 can be estimated using g ≈ 8 g V g /D and U 0 ≈ 3 ( g L/ m m ) 1/3 . Typically g ≈ 100-200 N/m 2 and the resulting induced upward melt flow velocity U 0 ≈ 5-10 m/s (Fabbro, 2010a). This upwards melt flow is very important and, when it reaches the melt pool surface, it spreads along the radial direction. It is also responsible for the large bumps that are generally observed around the KH rim for these operating conditions. Kelvin-Helmholtz instability can also appear for these high speeds of ejected vapour plume; the previous process is then amplified and may lead to spatter and droplet ejection [START_REF] Golubev | On possible models of hydrodynamical nonstationary phenomena in the processes of laser beam deep penetration into materials[END_REF]. It is likely that high beam quality lasers, which generate small focal spot diameters, easily induce these effects. On the other hand, one can also understand the quality improvement generally observed when large keyhole sections such as those obtained with large or elongated focal spots are used [START_REF] Hohenberger | Laser welding with Nd:Yag multi-beam technique[END_REF][START_REF] Luft | New applications for welding with diode lasers[END_REF]. The use of large focal spot diameters is a general rule that can be applied for improving the KH stability.

KH opening stabilization using a side gas jet

We have seen that for low welding speeds, in stationary conditions, the evaporation pressure is not sufficiently intense to maintain an opened keyhole. It is then possible to maintain this opening by using a neutral gas jet delivered inside the keyhole by a small side nozzle [START_REF] Kamikuki | Prevention of welding defect by side gas flow and its monitoring method in continuous wave ND:Yag laser welding[END_REF]Fabbro et al., 2006c) (see Fig. 3.3(a)). If the gas is correctly delivered, the dynamical pressure of the gas jet can be adjusted in order to balance the closing pressure controlled by its surface tension effects. Figures 3.4 (a) and (b) show characteristic images of the melt pool with and without using the gas jet. The effect of this additional gas flow is very interesting: It maintains the KRW wall at a large distance from the KFW and so the laser beam is no longer intercepted by the KRW fluctuations. Therefore the laser beam can penetrate deeper inside the material. Also, the induced vapour plume is much less important and is more stable compared to conventional welding. Finally the hydrodynamic flow of the melt pool is very different from what is usually observed without the gas jet and it can be controlled: melt flow is ejected rearwards in a continuous, stationary and quite laminar flow. As a result of the use of the side gas jet, a gain of 40-50% in the KH penetration is observed, without any porosity. Also, the wine-cup-shaped weld seam usually observed at these low welding speeds is suppressed (see Fig. 3.3(b, c)) because the flow is no longer directed only upwards but also rearwards by the gas jet. This modified flow also explains the increase of the height of the observed weld seam reinforcement (see Fig. 3.3(c)). 3D modelling of the interaction of a side gas jet with the melt pool was also realized (L. Zhang et al., 2011): it was concluded that an improvement in the stability of the molten pool and a reduction in spatters and pores could be obtained. This technique can be used for severe conditions: using this gas jet-assisted laser welding technique, X. Zhang et al. (2011) have been able to weld 316L stainless steel plates up to 40 mm thick, in two passes with an 8 kW disk laser and at 0.3 m/min welding speed without any porosity or other defects.

Vapour plume behaviour

Compared to the CO 2 laser wavelength, the 1.06 m of the Nd-Yag laser has a very different behaviour concerning its interaction with the vapour plume. Firstly, as the Inverse Bremsstrahlung absorption coefficient of an ionized gas with a given electronic density follows a 2 scaling law, one can consider that this absorption mechanism is not at all relevant at 1.06 mm, for incident laser intensities characteristic of welding processes. As a consequence, the temperature of the vapour plume follows the temperature of the surface where the evaporation process occurs; in fact, this plume temperature may be even lower, due to its expansion into the ambient atmosphere and because its reheating cannot occur due to the very low absorption coefficient, as discussed above. So these plume temperatures are rather low, in the range and above the evaporation temperature T v of the material at atmospheric pressure. Spectroscopic measurements have confirmed this rather low temperature of the vapour plume and correspondingly its very low ionization state (Greses, 2003; [START_REF] Greses | Plume attenuation under high power Nd:Yttritium-Aluminum-Garnet laser welding[END_REF][START_REF] Katayama | Understanding and improving process control in pulsed and continuous wave laser welding[END_REF]. This explains that, unlike CO 2 laser welding where the plume temperature in the range of 6,000-10,000 K results from its reheating by the Inverse Bremsstrahlung absorption, the nature of the shielding gas used, such as He, N 2 or Ar, has no effect on the final result of welding when 1 mm laser wavelength is used.

The second aspect of the laser-plume interaction concerns the important role of particles ejected back towards the laser along the direction of the beam. These particles have a very wide range of radius r that can vary from a few nanometres when they result from clusters of atoms, aggregates or ultra-fine particles generated by some complex mechanism of local condensation of these metallic atoms, to micron size or above, when they result from liquid droplets detached from the liquid wall by this violent expanding vapour jet as discussed in Section 3.2.2. Because this range of radius particles is typically smaller or about the 1.06 mm wavelength, it is expected that scattering and absorption by these small particles may play a much more important role than with the CO 2 10.6 mm laser wavelength. It is known that Mie scattering becomes effective when the ratio r/ is about one and Rayleigh scattering scales with (r/ ) 4 , so these mechanisms must be very efficient with short laser wavelengths. The analysis of the relative contribution of the beam absorption (that produces heating of these particles) and of the scattering terms (that induces a significant defocusing effect of the laser beam) shows that absorption is dominant for small particles, while scattering becomes important for larger ones (2 r/ > 1).

Nevertheless, the application of these theories to experimental conditions is not satisfying due to the lack of precise knowledge of experimental parameters such as the volume ratio (ratio of particle volume to plume volume) or particle radius distribution along the vapour plume. However, [START_REF] Greses | Plume attenuation under high power Nd:Yttritium-Aluminum-Garnet laser welding[END_REF] has shown that for welding conditions at 1.06 mm, the particle average size inside the vapour plume varied from 20 to 50 nm when He or Ar shielding gas was used, while for CO 2 laser welding, this size was typically 10 times smaller; this likely results from the strong difference of the plume temperature between these two types of welding. Moreover, attenuation and scattering of a probe beam, of different wavelengths, transverse to the vapour plume, confirm the importance of this mechanism occurring at shorter laser wavelengths [START_REF] Katayama | Understanding and improving process control in pulsed and continuous wave laser welding[END_REF]. In fact, the real importance of the perturbing effect of these scattering mechanisms can really be emphasized when the vapour plume is blown away by an intense transverse gas jet located very near the workpiece surface. Typically, the penetration depth and weld seam area are improved by 15-20% when the vapour plume is correctly 'cut'. It is likely that the use of a side gas jet, as discussed in Section 3.2.3, also induces such effect by dispersing the vapour plume and therefore reducing its scattering and/or absorbing effect. One must also add that, apart from these localized scattering mechanisms, radial index variation on the shielding gas above the surface induced by thermal heating from the workpiece or from the vapour plume itself, can perturb beam propagation and modify the focal spot intensity distribution, particularly for fibre remote welding [START_REF] Mitzutani | Numerical calculation of laser beam path influenced by high temperature gas above specimen during laser welding[END_REF]. These results clearly show that the control of the vapour plume and more particularly its size reduction or its suppression by adapted techniques, or of the shielding gas environment, improves the quality and the performance of the welding process. Moreover, due to the large relative variation of the focal spot by these perturbing mechanisms when small focal spots (and/or small apertures of focusing optics) are used, one may expect that these improvements should be very effective when high beam quality lasers are operated.

Welding under vacuum conditions

Welding under vacuum conditions is also another way of reducing the perturbing effects induced by the vapour plume previously discussed. This improvement has already been analysed for high power CO 2 laser welding [START_REF] Poueyo-Verwaerde | Experimental study of laser induced plasmas in welding conditions with CW CO 2 laser[END_REF]. At low ambient pressure, the Inverse Bremsstrahlung process of laser absorption by the plasma plume is less efficient due to the decrease of its electronic density as a consequence of the reduction of ambient pressure. Therefore, plume temperature decreases and more laser energy can be transmitted into the KH; typically, on 35NCD16 steel, at a low welding speed of 0.3 m/min and P laser = 7.5 kW, the KH penetration depth was increased by about 40% when ambient pressure was reduced to about 0.6 kPa. Simultaneously, the characteristic 'wine cup' shape of the weld seam top disappeared and large bulging of the melt pool was observed behind the KH.

At 1.06 mm laser welding, if very similar resulting effects are also observed [START_REF] Abe | Deep penetration welding with high power laser under vacuum[END_REF], the physical mechanisms are rather different. We have seen in Section 3.2.4 that the Inverse Bremsstrahlung absorption process is not effective at this wavelength; therefore the only mechanism that is likely to occur when the ambient pressure is reduced is the strong decrease of the vapour plume density during its expansion outside the KH into this reduced atmosphere. The size of the plume is efficiently reduced due to this 3D expansion just outside the KH aperture and the strong decrease of the observed plume luminosity and its steady behaviour confirm the reduction of perturbing effects. One can also expect that inside the KH, the vapour plume density is also decreased due to this easy and rapid expansion into vacuum. This point also explains the corresponding suppression of spatter emission and the much more stable behaviour of the KH and the melt pool. At low welding speeds, the KH being quite vertical, the vapour plume is then ejected vertically along the KH axis and so perturbs more efficiently laser beam propagation. Improvements observed for welding in vacuum conditions are therefore much more effective at low welding speeds. The consequence of the strong reduction of the absorption and scattering mechanisms in low vacuum conditions, with a more stable KH, is that more laser energy can be injected inside the KH, which then has a greater depth with a narrower size of the weld seam. [START_REF] Abe | Deep penetration welding with high power laser under vacuum[END_REF] obtained an increase of KH depth typically from 20 to 43 mm when the ambient pressure was reduced to about 1 kPa, on 304 stainless steel, at a welding speed of 0.3 m/min (with an incident laser power of 16 kW on a 0.4 mm focal spot diameter). For higher welding speeds, 3-6 m/min, the corresponding increase was only of 15-20%.

One must add that evaporation under vacuum conditions also modifies the temperature threshold for KH generation. As a general condition of KH equilibrium, we have previously seen that it is the excess of the recoil pressure P evap due to evaporation, over the ambient pressure P a , which must counterbalance the KH closure pressure P s due to surface tension, at low welding speeds. Therefore, for vacuum conditions, this condition is realized for a surface temperature of the KH walls with a lower temperature than the usual evaporation temperature T v . As this reduction of surface temperature can be about 10% of T v [START_REF] Hirano | Experimental determination of temperature threshold for melt surface deformation during laser interaction on iron at atmospheric pressure[END_REF], by considering the energy balance of the process, a significantly greater penetration depth has to be expected for similar operating parameters.

Examples of weld speed variation on global behaviour of keyhole (KH) and melt pool

The previous discussion will be illustrated now by some examples of evolution of the melt dynamics, analysed with a high-speed video camera, for variable welding speeds. Incident laser power and laser spot diameter were kept constant at 4 kW and 0.6 mm, respectively. 304 Stainless steel samples were used, in partial penetration mode. This analysis allows the definition of five main characteristic contiguous regimes, which are only controlled by the welding speed and where the action of the vapour plume on the melt pool is emphasized. This action is controlled by the geometry of the keyhole front, itself defined by the welding speed and incident laser intensity. Although the following observations are done for specific conditions of P = 4 kW and 0.6 mm focal spot diameter, it must be emphasized that these results are general and can be classified as a function of the FKW inclination angle

Welding speeds below 5 m/min: 'Rosenthal' regime

This regime, occurring for welding speeds lower than 5 m/min, is characterized by a rather large melt pool, even in front of the keyhole, because of the rather low welding speed. The melt pool surface shows many chaotic surface fluctuations and large swellings of liquids fluctuating around the keyhole aperture, which is rather well defined and remains circular (see Fig. 3.5(a)).

Because of these fluctuations, there is no clear laminar flow around the keyhole and the Marangoni effect is unlikely to occur for these conditions. Many spatters are emitted mainly from the keyhole rim, and particularly on its front side. On this range of welding speeds, one can see a rapid decrease of the penetration depth as well as large fluctuations of ejection direction of the vapour plume. These fluctuations can be correlated with the previously described melt pool swelling fluctuations. Macrographs of cross sections of the weld seam show a gradual transition of the shape from the well-known 'wine cup shape' characteristic of a low welding speed (typically about 1-3 m/min) to a more slender shape for 5 m/min. From the analysis of the keyhole tilting for these low welding speeds, discussed in Section 3.2.1, we know that the keyhole is quite vertical. Therefore, despite these fluctuations, one can schematize this regime by a vertical cylindrical keyhole surrounded by a large melt pool with very limited hydrodynamics. So the description of this situation as a kind of 'Rosenthal' heat flow regime, where the keyhole surface is uniformly heated at some temperature close to the evaporation temperature, could be used.

Welding speeds between 6 and 8 m/min: 'single wave' regime

This regime is observed for welding speeds ranging from 6 to 8 m/min. It is characterized by the presence of a rather large single swelling generated near the top of the rear keyhole wall (see Fig. 3.5(b)). It is only from this region that large melt droplets are emitted. This large wave is ejected rearwards quite periodically due to back and forth oscillations of the melt pool, leading to quite periodic closures of the keyhole. It is the vapour plume, which is emitted rather deeply inside the keyhole and which collides with the melt pool that triggers these oscillations.

One can observe that only the tilted KFW is clearly heated by the incident laser beam (unlike the 'Rosenthal' regime where the luminosity characterizing the laser heated surface was more uniformly and randomly distributed all around the keyhole surface). More interestingly, when the ejected vapour plume collides with the melt pool and lifts it, a corresponding local heating of the liquid surface results from the impact of this energetic heated vapour plume. So the vapour plume not only transfers impulse momentum, but also a non-negligible amount of energy due to its rather high temperature.

It is interesting to note that when high beam quality lasers are used, i.e. with smaller focal spots and correspondingly rather greater penetration depths obtained for this range of rather low welding speeds, one also observes a 'humping regime' characterized by very periodic humps generated at the rear of the KH on the weld seam [START_REF] Behler | Melt pool dynamics in high speed welding with modern high power solid state lasers[END_REF][START_REF] Miyamoto | High-speed microwelding by single-mode fiber laser[END_REF][START_REF] Thomy | Humping in welding with single mode fiber lasers[END_REF]. This results from the friction of the ejected vapour acting all along the deep KRW that drags and lifts this part of the melt pool, and whose effect is dominant compared with its colliding effect that occurs when larger focal spots are used.

Welding speeds between 9 and 11 m/min: 'elongated keyhole' regime

This regime, which is observed for welding speeds ranging from 9 to 11 m/ min, is characterized by an elongated keyhole whose maximum length of about 2 mm, is obtained at 11 m/min (see Fig. 3.5(c)). This regime also shows some fluctuations, but with much less intense liquid oscillations than in the previous regime, and smaller induced swelling heights. This elongated keyhole shows two characteristic zones that are heated: the first one corresponds, of course, to the more inclined KFW, common to all regimes, and the second one is located at the rear end of this elongated keyhole, inside the melt pool. Moreover, vapour plume seems to be also emitted from the second heated spot and so directed frontward. As in the previous regime, heating by the collision of the vapour plume emitted from the keyhole front is also possible. One could also add that weld seams obtained with this regime have rather good quality, and do not show characteristic defects.

Welding speeds between 12 and 19 m/min: 'pre-humping' regime

This regime is observed for welding speeds ranging from 12 to 19 m/min. It is characterized by a rather strong sideward melt flow that contours the KFW, and an important tilting angle. The KH aperture is elongated (but with a shorter length than in the previous regime) and the surface of the following melt pool shows fluctuations characterized by only surface waves with rather small amplitudes. There is also a central melt flow along the KFW that emerges from the bottom of this very stable KFW and is deflected rearwards with its surface melt pool reaching a level close to the initial sample surface (see Fig. 3.5(d)). No more spatters or droplets are emitted, even from the rear rim of the keyhole as in the previous regime. Melt pool surface reaches the level of the surface sample and wets both sides. But above 15-16 m/min welding speeds, undercuts are observed on the sides of the seam. These undercuts begin to be important because the two side flows emitted from the KFW collide with the central emerging flow and press it towards its centre, slightly lifting it. Finally, the vapour plume is very stable; its direction no longer fluctuates and is precisely ejected perpendicularly to the KFW interacting with the top part of the KRW.

Welding speeds above 20 m/min: 'humping' regime

For welding speeds above 20 m/min, a very characteristic melt flow appears which corresponds to the humping regime, where the weld seam has very strong undercuts, with solidified large swellings of quite ellipsoidal shape, separated by smaller valleys.

The main central flow always emerges from the bottom of the KFW (see Fig. 3.5(e)); it is strongly deflected rearwards and it rises up to a level that is much lower than the surface sample. So the mean melt velocity inside it is very high. It stays attached to the bottom of the resulting groove along a distance of about 2 mm and then a liquid jet is detached from the central part of this flow and forms a thin vertical strip propagating rearwards at high velocity. The hump appears at a certain distance from the detachment point along this vertical strip of liquid jet [START_REF] Fabbro | Experimental study of the humping process during Nd:Yag CW laser welding[END_REF]. It is the shrinkage of this melt jet strip, due to the Rayleigh instability driven by surface tension, at about 2 mm from the detachment point that generates the instability. This shrinkage cools locally this strip of melt jet and so attaches it to the sample. The fluid flowing inside this strip of melt jet is then stopped at this point, and the hump can grow. The growth of this hump stops when a next shrinkage occurs at about a similar distance from the detachment point that allows the growth of the next hump. Of course, as this thin melt jet strip is located at the centre of section of the groove, very severe undercuts are generated even near the humps. For higher welding speeds, this scheme is not modified.

Analysis of the transition thresholds between these different regimes

The five previous regimes have been observed for a given incident laser power of 4 kW and focal spot of 0.6 mm, for different welding speeds.

In fact, one can say that this is a very general result: any set of operating parameters will produce behaviour that corresponds to one of these five regimes. These experiments have been reproduced by using incident laser powers varying between 2.5 and 4 kW and in order to have a greater range of incident intensities, the focal spot was reduced to 0.45 mm. The same regimes were observed but with modified welding speed thresholds. These results are reported in Fig. 3.6 on a laser power-welding speed diagram.

Welding is obtained with a KH mode in the region labelled K. Conduction mode is only observed in the region labelled C. Typically, for a welding speed of 25 m/min, KH regime is obtained for incident laser power greater than 0.5 kW. Of course, this power increases with the welding speed.

For incident power varying from 2.5 to 4 kW, we have reported the welding speed thresholds between the 'Rosenthal'/'Single-wave'/'Elongated'/'Prehumping'/'Humping regimes (these five corresponding regimes are labelled, respectively R, S, E, P, H). Except for the P/H transition, all the other characteristic thresholds increase with the incident laser power. Best-fit lines passing through the origin of these three series of experimental points of these consecutive regimes have also been drawn.

For the P/H transition, it is interesting to note that for incident laser powers smaller than 2.5 kW, no humping instability is observed, even at very high welding speed. Also, when the incident power increases, the threshold for the P/H transition decreases; this behaviour is similar to previous experimental results obtained for different conditions [START_REF] Thomy | Humping in welding with single mode fiber lasers[END_REF]. The different behaviour of this P/H transition, compared to the three previous ones, results from the very different mechanisms driving this humping instability. It is also interesting to notice that if we extrapolate the P/H transition to higher incident powers, the pre-humping regime should disappear.

From our previous analysis of the keyhole front equilibrium, one can relate the slope of the three best-fit lines plotted in Fig. 3 experimental thresholds of the corresponding transitions, with the mean inclination of the KFW. The combination of Eqs [3.1] and [3.2] gives a linear relation between incident laser power P and welding speed V w : [3.6] where the slope B of this linear relation between P and V w , is given by:

P = B • V w ,
B LD kA D kA = 4 = 4 t kA an 0 A A 2 0 A A p p LD a [3.7]
Using Eq. [3.7], the KFW inclination angles that would reproduce the slopes of these three best-fi t lines in Fig. 3.6 are 15°, 30° and 45° for respectively the R/S, S/E and E/P transitions. In fact, these angles are in fair agreement with the corresponding experimental measurements obtained by using the high speed videos. This model shows that this slope B is proportional to the penetration depth L (Eq. [3.7]). So, for a given transition (R/S, S/E or E/P), all the welding speed thresholds corresponding to that transition will occur at the same penetration depth or KFW inclination. Of course, as expected, when one goes from R to P regimes, the corresponding penetration depths decrease as the welding speeds increase.

So these results indicate that the transition between the fi rst four regimes is basically controlled by a characteristic tilting angle of the KFW. As we have seen that this angle also defi nes the direction and the dynamic pressure of the emitted vapour plume (this vapour plume being directed perpendicularly to its emitting surface), these results clearly confi rm that it is the level of interaction of the vapour plume with the rear melt pool that defi nes the type of hydrodynamic regime. At low welding speeds, the KFW is slightly tilted; the absorbed intensity is then rather small, as well as the dynamic pressure of vapour. The vapour plume is directed quite horizontally inside the melt pool and so perturbs the melt pool from its bottom. This would correspond to the occurrence of the single wave regime. As the welding speed increases, the tilting angle increases, as well as the absorbed intensity and correspondingly the dynamic pressure of the vapour plume. But this vapour plume is more emitted quite vertically and therefore perturbs more effi ciently the top part of the melt pool as in the pre-humping regime.

Conclusion and future trends

We have described in this chapter the main physical mechanisms that control the quality of the process of laser welding in KH mode. The main conclusion that results is the very important role played by vapour plume generated by the evaporation process that leads to the KH formation. As important is the direction of this vapour plume defi ned by the KFW inclination. In fact, two main operating parameters, the incident laser intensity and the welding speed, control both this direction and the dynamic pressure of the vapour plume. We have seen that this vapour plume can perturb the welding process in two ways: first by perturbing the beam propagation, by scattering and/or absorbing the incident incoming beam, particularly when small focal spots or low aperture optics are used. As a general rule, it is recommended to blow away the vapour jet each time it is possible, in order to get reproducible, constant and non-perturbed conditions of irradiations. The vapour jet can also strongly perturb the melt pool geometry and its hydrodynamics by its impact on the KRW. The final quality of the weld seam is a sensitive function of the method used to attenuate these perturbations. We have seen that enlarging the focal spot, for example by using a side gas jet that repels the KRW to larger distances from the KFW, or finding an adapted defocused position, or inclining the incident laser beam, or even welding in vacuum conditions may be possible solutions that should improve the final quality, but of course their possible use must be balanced with the induced constraints. However, there are still many unknowns related to different mechanisms and future experiments should be focused on them. One can mention several points that should be improved in the near future. A precise knowledge of absorption mechanisms on the KFW is necessary. This could be helped by determining the KFW surface geometry by adapted experiments with high time and space resolution, improving, for example, present high-speed video imaging and/or X-ray shadowgraphy techniques. This problem of surface instabilities generated by non-linear coupling of thermal field and hydrodynamic response should also be more deeply analysed on a theoretical basis. The knowledge of resulting recoil pressure is also rather poor; as we have seen, this parameter is very important because it controls melt side-flow velocity and dynamic pressure of the vapour jet. Presently, very few experiments have been done to determine this recoil pressure for this range of absorbed intensity. Moreover, the effect of several parameters such as the ambient pressure and the nature of alloying elements of the workpiece make the determination of recoil pressure rather complex. Also, the thermodynamic state of metal vapour (mean density, temperature, cluster or particles distribution, flow field, etc.) is completely unknown at present. We have seen that this point is very important for the beam propagation into the vapour plume outside the KH; but for inside the KH, beam scattering should control its propagation and so the resulting beam homogenization should interfere with the previous surface instabilities and absorption mechanisms.

Concerning the melt pool, its geometry, and particularly the shape of its KRW, and also its dynamics flow field, should be analysed with greater spatial and temporal precision than presently. High-speed X-ray shadowgraphy, possibly using tracers, will be an indispensable tool. One should also add that a better knowledge of the surface tension or the dynamic viscosity of the workpiece material dependence with temperature would be useful. Finally, one could also remark that there is an important thermodynamic parameter, which is the temperature, which is usually poorly measured during experiments and which is always a direct output of numerical simulations. Having diagnostics able to give time and space resolved temperature distribution in a sub-melting to evaporation temperature range is still a challenge.

So, the precise determination of all these very different parameters should be important data that must be compared with corresponding results of numerical simulations. It is interesting to note that already, several publications begin to show the possibility to realize 3D numerical simulations of laser welding where these main various mechanisms can be described self-consistently, for non-stationary conditions, and where the hydrodynamics of the melt pool and vapour/free surface interaction are also taken into account (Ki et al., 2002a, b;[START_REF] Dasgupta | Physics of zinc vaporization and plasma absorption during CO 2 laser welding[END_REF][START_REF] Otto | Numerical simulations -a versatile approach for better understanding dynamics in laser material processing[END_REF][START_REF] Zhang | An investigation on the effects of side assisting gas flow and metallic vapour jet on the stability of keyhole and molten pool during laser full-penetration welding[END_REF][START_REF] Pang | A three-dimensional sharp interface model for self-consistent keyhole and weld pool dynamics in deep penetration laser welding[END_REF]. Depending on their initial various hypotheses and also computational facility used, the main phenomena involved in the coupling between the KH and the melt pool dynamics discussed in that chapter, have already been correctly reproduced [START_REF] Pang | A three-dimensional sharp interface model for self-consistent keyhole and weld pool dynamics in deep penetration laser welding[END_REF]. So, the numerical results of simulations are presently very encouraging. We are convinced that with continuous efforts on improvements to these numerical simulations on one hand, and, on the other, with the corresponding experimental progress on several experimental diagnostics as mentioned previously, one should be able to have, within a few years, very powerful simulation tools that are able not only to reproduce precisely the main observed characteristics, but also to optimize operating parameter selection for a targeted goal, by avoiding long series of experiments.

3.5

3.6

Appendix: list of symbols

  (a) General skech of laser welding: A: Laser beam; B: Welding speed; C: KH front; D: Humps; E: Vapour jets; F: FKW inclination; G: Melt flow; H: Chevron structure; I: Solidification front; J: Melt pool; K: Weld seam cross section; L: Resolidified material. (b) Scheme of the longitudinal section of a keyhole in a case of full penetration.

  Computed keyhole profiles when two Zn-coated sheets of different thicknesses are welded in overlap configuration. The RKW is deformed by the Zn vapour jet escaping from a channel at the sheet interface: (a) 0.44 mm/0.90 mm and (b) 0.90 mm/0.44 mm. Experimentally, one observes that case (b) is much more stable than case (a) because the greater enlargement of the KH bottom allows the Zn vapour to escape more easily. A: Sheet interface; B: Zn vapour channel; C: Zn vapour jet; D: Flow followed by the Zn vapours (incident laser power: 4 kW, V w : 3.5 m/min, top-hat intensity distribution, focal spot diameter: 0.45 mm, 5 ms time interval between each profile for visualizing their fluctuations)(Fabbro et al., 2006b).

  a) Scheme of the opening of the KH aperture using a longitudinal side gas jet. The two positions, front or rear, can be used. Weld seam without (b) and with (c) a side gas jet. The weld seam contours have been highlighted (P laser : 4 kW, V w : 1 m/min, focal spot diameter: 0.6 mm). A flow rate of 20 l/min of Argon gas was delivered through a 2 mm diameter nozzle. It generates a dynamical pressure of 20 kPa at 5 mm from the nozzle aperture on its axis. By adjusting the argon flow rate, any dynamical pressure can be obtained inside the keyhole in order to avoid its closure.

  Melt pool behaviour and KH aperture without (a) and with (b) a side gas jet that delivers a dynamical pressure of 8 kPa inside the KH. The resulting elongated KH obtained in that case has a length of about 2 mm. (Welding speed: 3 m/min; P laser : 3 kW; Spot diameter: 0.6 mm). White vertical arrows show the axis of incident laser beam location.

  (a) Rosenthal regime: V w < 5 m/min (b) Single wave regime: 6 V w < 8 m/min (c) Elongated KH regime: 9 < V w < 11 m/min (d) Pre-humping regime: 12 < V w < 19 m/min (e) Humping regime: V w > 20 mCharacteristic views of KH with surrounding melt pool with corresponding weld seam cross section, and corresponding sketches of the five described regimes obtained for various welding speeds. (Focal spot diameter: 0.6 mm, stainless steel material.) Scale of melt pool: 1 mm.

  Location of different regimes respectively labelled R, S, E, P, H, inside the incident laser power-welding speed plane. Regions corresponding to keyhole (K) and conduction (C) regimes are also indicated. Linear best-fits for the R/S, S/E and E/P transitions have been drawn, with corresponding FKW tilting angle a. (Top-hat intensity distribution; Focal spot diameter: 0.45 mm; Stainless steel material.)

  , m dynamic viscosity of metal vapour and melt (Pa.s)

	A( )	angular dependence of absorptivity
	A 0	absorptivity under normal incidence
	B	slope of relation between laser power and welding speed (J/m)
	C	slope of relation between evaporation pressure and absorbed intensity
		(s/m)
	D	spot diameter (m)
	I abs I cond I 0 k	absorbed laser intensity (W/m 2 ) conduction loss inside solid (W/m 2 ) incident laser intensity (W/m 2 ) constant (m 3 J -1 )
	L	keyhole depth (m)
	L max	maximum depth of a quite stable keyhole (m)
	P	incident laser power (W)
	ambient atmospheric pressure (N/m 2 ) dynamic pressure of ejected vapour plume (N/m 2 ) P evap evaporation pressure (N/m 2 ) P a P d P s keyhole closing pressure due to surface tension (N/m 2 )
	r	particle radius inside the vapour plume (m)
	T v	workpiece evaporation temperature (K)
	V d	drilling velocity (m/s)
	V g	vapour plume velocity (m/s)
	V m	melt velocity (m/s)
	V w	welding speed (m/s)
		inclination of the front keyhole wall (rd)
		laser wavelength (m)
	m	melt density (kg/m 3 )
		surface tension of metal liquid (N/m)
	g	shear stress (N/m 2 )

g


Ali El Amine
email: ali.el-amine@imt-atlantique.fr

Paolo Dini
email: pdini@cttc.es

Loutfi Nuaymi
email: loutfi.nuaymi@imt-atlantique.fr

Reinforcement Learning for Delay-Constrained Energy-Aware Small Cells with Multi-Sleeping Control

Keywords: Energy saving, delay, multi-level sleep mode, 5G, Q-learning

HAL is

I. INTRODUCTION

In order to support future traffic requirements (highspeed mobile data services and better coverage), next generation cellular networks are expected to deploy denser mobile access networks with different Base Station (BS) sizes (large and small cells). Although this improves the capacity and coverage of these networks, it also brings new challenges such as pushing the limits of energy consumption and problems related to interference management. In order to meet these challenges, there is a growing interest to steer research efforts towards energyefficient solutions, with particular emphasis on BS Sleep Mode (SM) techniques.

BS sleep scheme is considered among the best methods to save energy, since it does not require changes to current network architecture, and it is easy to implement [START_REF] Wu | Energy-efficient base-stations sleep-mode techniques in green cellular networks: A survey[END_REF]. We distinguish between two types of sleep schemes: binary SM level (ON and OFF) and multi-level SM. In binary SM, the BS shuts down completely when the traffic is low to save energy. The literature is rich and many surveys exist on binary SM [START_REF] Wu | Energy-efficient base-stations sleep-mode techniques in green cellular networks: A survey[END_REF]- [START_REF] Feng | Base station on-off switching in 5G wireless networks: Approaches and challenges[END_REF]. On the other hand, multi-level SM splits the sleep states into several levels with different characteristics on the activation/deactivation period and the power consumption [START_REF] Debaillie | A flexible and futureproof power model for cellular base stations[END_REF]- [START_REF] Lähdekorpi | Energy efficiency of 5G mobile networks with base station sleep modes[END_REF]. The latter adds flexibility to cope with the traffic requirements to further enhance the system performance by reducing the energy consumption of future networks while minimizing the impact on the Quality of Service (QoS). In this regard, there exist several studies that consider the problem of Energy-Delay Tradeoff (EDT) using multi-level SM scheme [START_REF] Ezzahra | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF]- [START_REF] El-Amine | Location-Aware sleep strategy for Energy-Delay tradeoffs in 5G with reinforcement learning[END_REF]. By applying tools from machine learning, and in particular reinforcement learning, these works study the multi-objective function of the EDT problem for different SM policies depending on the service use case, allowing the operator to tune the network parameters towards more energy savings or less service delay.

Coupled with the problem of energy consumption in dense Heterogenious Networks (HetNets) is the cochannel interference, where the unplanned deployment of small cells makes the interference management more complex. In [START_REF] De Domenico | Reinforcement learning for interference-aware cell dtx in heterogeneous networks[END_REF], the authors designed a centralized Fuzzy Q-Learning (FQL) algorithm controller for small cells to minimize the energy consumption while respecting the users' QoS. The proposed solution manages the interference between the small cells in order to limit its effect and to maximize the energy efficiency of the network.

In this paper, we study the problem of EDT in a Het-Net where small cells can switch to different SM levels in order to save energy. We propose a distributed Qlearning algorithm controller for small cells that adapts their activity based on the level of interference, the traffic load and the size of the buffered data in the cell. Different from the works in [START_REF] Ezzahra | Reinforcement learning approach for advanced sleep modes management in 5G networks[END_REF]- [START_REF] El-Amine | Location-Aware sleep strategy for Energy-Delay tradeoffs in 5G with reinforcement learning[END_REF], in this paper we take into account the co-channel interference between the cells to minimize its effect to enhance the network's performance. Different from [START_REF] De Domenico | Reinforcement learning for interference-aware cell dtx in heterogeneous networks[END_REF], we propose a distributed Qlearning algorithm where each cell makes autonomous decisions according to the Decentralized SON (D-SON) paradigm. Furthermore, we extend the model of [START_REF] De Domenico | Reinforcement learning for interference-aware cell dtx in heterogeneous networks[END_REF] to multi-SM scheme. Our numerical analysis shows that multi-level SM scheme outperforms binary sleep scheme in terms of energy savings and added delay. We also focus on the convergence analysis, and the impact of The remainder of the paper is organized as follows. Section II details the system model understudy with the 5G multi-level SM scheme. In Section III, we present our distributed Q-learning algorithm for multi-level SM scheme. We discuss some performance results in Section IV. In Section V, we draw our conclusions and discuss future research directions.

II. SYSTEM MODEL

We consider a two-tier heterogeneous multi-cell network composed of a Macro Base Station (MBS) and Small Base Stations (SBSs). Together, these BSs serve a total of K users. The SBSs are densely deployed in the coverage area of the macrocell and operate in a dedicated carrier to create a local hotspot that will increase the system capacity in the designated area, where needed. In order to enhance the system energy efficiency, these SBSs can dynamically switch to different SM levels. This will not only save energy and limit the CO 2 footprint, but it will also enhance the system capacity by limiting the interference between neighboring cells. A distributed local network controller (e.g., SDN) is in charge of managing the activity of these cells (see Fig. 1). We note that for binary SM levels (ON and OFF), finding the optimal set of active SBSs is a combinatorial problem with high computational complexity. In this work, we further add another layer of complexity by considering multi-level SMs. In order to solve this problem, the local controller manages a given SBS by means of interacting with the environment in order to map what to do (action) when faced with a current situation (state). This method is known as Q-Learning. In particular, each SBS at each iteration, measures the expected average interference from neighboring cells and the SBS state in the previous iteration. The local controller uses this information along with the SBS buffer and traffic load states to control its activity, as we detail later in this section.

A. Multi-Level Sleep Modes

In [START_REF] Debaillie | A flexible and futureproof power model for cellular base stations[END_REF], GreenTouch identified four distinct SM levels by grouping sub-components with similar transition latency when being activated or deactivated. The presented model enables to quantify the power consumption of the BS in each of the four SMs. These are:

-SM 1: It considers the shortest time unit of one OFDM symbol (i.e., 71µs) comprising both deactivation and reactivation times. In this mode only the power amplifier and some processing components are deactivated. -SM 2: It corresponds to the case of sub-frame or Transmission Time Interval (TTI) (i.e., 1 ms). In this SM, more components enter the sleep state. -SM 3: It corresponds to the frame unit of 10 ms.

Most of the components are deactivated in this mode. -SM 4: This is the deepest sleep level. Its unit corresponds to the whole radio frame of 1s. It is the standby mode where the BS is out of operation but retains wake-up functionality. Higher energy savings can be achieved when switching BSs to a deeper SM, since more components will be deactivated. However, this will be associated with longer transition latency which may impact the QoS of the system. In Table I, we present the SM levels characteristics.

Along with SM and users' dynamics, the BS has to wake up periodically to send signalling bursts. In contrast to Long Term Evolution (LTE) systems where each antenna must transmit every 0.2 ms a unique Cell Reference Signals (CRS) for channel quality estimates and mobility measurements among other Synchronization Signaling (SS), no CRS is required for 5G [START_REF] Gpp | On requirements and design of ss burst set and ss block index indication[END_REF]. Instead, SS and Physical Broadcast CHannel (PBCH) are transmitted in SS/PBCH block periodically. It has been agreed in Third Generation Partnership Project (3GPP) [START_REF] Gpp | On requirements and design of ss burst set and ss block index indication[END_REF] that this periodicity can be set to any value among [5, 10, 20, 40, 80, 160 ms]. With these values, SM 4 cannot be used. Hence, we limit our work to the first three SM levels.

B. Network Operation

In our model, we consider connected users continuously requesting packets during a fixed period of time. We use the term block of data to describe the overall packets requested by users during a time slot t (ms) and contained in one block. In this work, we do not focus on the inter-packet dynamics. We rather consider a continuous flow of data with an average size (kbits) that is first buffered in the SBS buffer before transmitted to the designated users.

At each time slot t, a block of data (composed of several packets) is buffered and stacked in the designated cell buffer. The size of this block n depends on the number of users being served in the cell and users' requirement based on the service. Furthermore, this block of data is characterized by a Time-To-Live (TTL) l (ms), which is the latency constraint above which the remaining data are lost. Here, we consider that the block of data can be partially served in case the serving time was not enough to deliver it all, due to high interference and/or inactivity of the cell (i.e., in sleep mode). Furthermore, the cell buffer operates in FIFO (First Input First Output) fashion. One can relate such service to a current or future 5G delay-tolerant application where the order of the packets does not matter.

If a user requests a packet from a SBS in a SM state, e.g., SM 1, SM 2, or SM 3, it will be buffered until the SBS activates in a future time step. Thereafter, the SBS manages the available radio resources to serve the users during the transmission time slot. The state of the SBS has an impact on the latency added to the system. The deeper the SM level is, the more time the user will have to wait until the SBS reactivates.

C. Problem formulation

Following the above system description, we can model the problem as a discrete-state Markov Decision Process (MDP), where a SBS m in a state s m (t) takes an action a m (t) and transitions to another state s m (t + 1). We denote by S = {R, B, I l } the state space of the SBS, where R, B and I l are the sets related to the cell throughput, buffer state, and the estimated spectral efficiency loss (bit/s/Hz) due to the interference from nearby active SBSs, per SBS, respectively. In this work, we assume that the users request a Near Real Time Service (NRTS) traffic with an average packet size [kbits]. The time needed to complete this service depends on two factors: The buffering time until the SBS re-activates, and the capacity of the serving SBS that is highly affected by the co-channel interference from neighboring cells.

At each time slot, the local controller decides an action a m (t) ∈ A = {Active, SM 1, SM 2, SM 3} for a SBS after observing its state. In order to evaluate the cost associated with each action-pair, we use the function that takes into account the power consumption of the SBS, and the users' QoS:

c(s, a) = (1 -w) • p(s, a) + w • d(s, a) (1)
where p(s, a) is the power consumption of the SBS, w is a weighting factor that prioritizes between the power consumption and the QoS, and d(s, a) indicates the data lost due to exceeding the delay constraint either from the operational state of the cell (i.e., in sleep) or the high interference level limiting the cell throughput.

In order to solve the above MDP, we first need to derive the transition probabilities and cost from one state to another. Then, a backward dynamic programming needs to be performed to evaluate the optimal value function, in order to derive the optimal policy. However, this requires a complete knowledge of the system information. To alleviate the non-causal system information, we rely on QL to find the optimal policy by only exploring and interacting with the current state of the system.

III. DISTRIBUTED Q-LEARNING A. Preliminaries on Reinforcement Learning

Distributed Q-learning is an online optimization technique that aims at controlling multi-agent systems, i.e., a system featuring M BSs which take decisions (select the appropriate SM level) in an uncoordinated fashion. Each BS has to learn independently a policy (SM 1, SM 2 or SM 3) through real-time interactions with the environment. Q-learning finds the optimal policy in the sense that it maximizes the expected value of the total reward (Q-value) over all successive episodes. The agents (i.e., BSs) have a partial view of the overall system, and their actions may differ since the users are unevenly distributed over the network. In particular, the decision of a BS to choose a SM level is affected by how many users it has to serve, and on how delay-tolerant these users are.

In Q-learning, each agent takes an action a t m from an action set A, then moves to a new state s t+1 m while receiving a reward r t m . This reward is then used to update the Q-value locally, Q(s t m , a t m ), indicating the level of convenience of selecting action a t m when in state s t m . The Q-value is updated following the update rule:

Q(s t m , a t m ) ← Q(s t m , a t m )+ α[r t m + γ max a∈A Q(s t+1 m , a t+1 m ) -Q(s t m , a t m )] ( 2
)
where α is the learning rate that represents the speed of convergence, and γ ∈ [0, 1] is the discount factor that determines the current value of the future state costs.

During the learning phase, each agent selects the corresponding action based on the -greedy policy, i.e., it selects with probability 1the action associated with the maximum Q-value, and with probability selects a random action (here, y ∈ [0, 1] is a random variable):

a t m =    argmax a∈A Q(s t m , a t m ), if y > rand(A), otherwise m = 1, . . . , M.
(3) By implementing the -greedy policy, the BS would have explored all possible actions and avoided local minima. For more details on Reinforcement Learning (RL) and Q-learning the reader is referred to, e.g., [START_REF] Sutton | Introduction to Reinforcement Learning[END_REF].

B. Q-Learning Algorithm

At time slot t, the local state of the small cell m is s m (t) = {r m (t), b m (t), i m l (t)}. The states have been quantized each into 3 levels. We chose these levels of quantization to balance between performance and complexity. Due to the limited number of state-action pairs, Q-learning is able to pave the way towards optimality, thus there is no need for Deep Q-learning. In order to explore different combinations of states, we make the users move inside their serving cell at each time slot. We define an episode as a fixed simulation run where the users in motion request data continuously with a given minimum rate requirement. During this episode, the users move in a predefined and deterministic pattern, and the algorithm explores different states to find the optimal policy that minimizes the cost function in [START_REF] Wu | Energy-efficient base-stations sleep-mode techniques in green cellular networks: A survey[END_REF]. After the training phase is complete, the system goes online and operates in real-time. The Q-Learning algorithm is described in Algorithm 1.

Algorithm 1 : Q-Learning Algorithm 1: procedure Training (Q m T (s, a)) 2:
Initialize the positions of the users, their trajectory and q m (s, a) = 0, ∀m ∈ M, ∀s ∈ S and ∀a ∈ A. Visit state s m .

7:

Select an action a m using -greedy rule in (3). Update the Q-value q m (s, a) from (2).

11:

end for 12: end while 13: end procedure

1: procedure Online 2: for m ∈ M do 3: Q m (s, a) = Q m T (s, a) 4:
Run Q-Learning. 5: end for 6: end procedure

IV. NUMERICAL ANALYSIS

A. Convergence analysis

We analyze the convergence of the proposed Qlearning algorithm for different values of w during the training phase in figures 2 and 3. During this phase, the system is fed with a training episode that describes the dynamics of the network: the users are moving in the designated area, they are requesting service packets, and the small cells are reacting (i.e., taking action to switch to different modes) based on their partial view of the environment that defines their local states (cell throughput, cell buffer size and estimated spectral efficiency loss). By repeating this episode a number of epoch of times, each cell should have learned how to map its observable states to actions. In order to maximize the efficiency of the algorithm to choose the appropriate action when facing a given state, the training episode should cover as much states as possible so that when the system goes online, the probability of visiting an unvisited state is minimized. The design of the training episode is important so that the agent gathers enough experience from the environment, which is critical in stabilizing the Q functions, thus forging the best policy.

In Fig. 2, we observe that the system converges after few iterations (around 10 iterations for most values of w). When w = 0 and w = 1, the equation in (1) simplifies to minimizing energy consumption and service delay only, respectively. In this special case, the agent (i.e., small cell) is able to find the optimal policy relatively fast compared to other values of w. When w ∈ ]0, 1[, the agent needs more time to capture the tradeoff between energy and delay, thus requiring more epochs to finally converge.

In Fig. 3, we illustrate an example of the convergence behavior of the system, where we plot the evolution of the average network cost function in (1) and the average cells states, on a per epoch basis. The later corresponds to the average cells states distribution over each epoch. We take the example where w = 0. During the first iterations of the training, the system explores different policies. This can be observed by almost having a uniform cells states distribution. With the evolution of epochs, each cell learns the best policy, which is in this example biased to switching to the deepest SM level in order to save the highest amount of energy.

Since the convergence time of the training phase is small as previously shown in Fig. 2, it is possible to skip the independent training phase and merge it with the online one. In this case, the system starts online with no previous knowledge about the environment and learns to adapt on the spot. The main advantage of this approach is that it does not require the prerequisite training step. However, it extends the time until the system converges. This can be observed in Fig. 4 where we notice an increase in the number of iterations required for the Qfunctions to stabilize. Nevertheless, this increase is of couple of iterations, and thus insignificant. The major reason behind this increase is that with the online mode, the episode is changing in each iteration in contrast to the training phase where the training episode is fixed. This dynamic nature of the iteration makes it harder for the agent to learn as opposed to static episode.

B. Policy analysis

In Fig. 5, we report the different policies of the SBSs for different values of the tuning parameter w. We observe that the system tends to save more energy when w approaches zero by switching to the deepest SM most of the times, while a gradual shift to active mode is observed when increasing w. However, it is interesting to highlight the behavior for w = 1. For this case, the cells are required to be active for a minimum period of time to minimize the delay (around 37% of the time). During the remaining time and after satisfying the agent's goal, the cell switches randomly to any of its states. This can be observed by the uniformly distributed state modes for SM 1, SM 2 and SM 3, while the active mode has a higher probability in order to satisfy the delay constraint of 100 ms. In Fig. 6, we illustrate the average percentage cell energy consumption normalized to the case where the cell is always on (i.e., without sleep mode) and the average small cell delay as a function of w with packet delay constraint of 100 ms. We compare our proposed Q-learning algorithm (QL) with another QL algorithm that considers two states only (active and SM 3). This algorithm is referred to as binary SM-QL algorithm. We observe that our multi-SM-QL algorithm outperforms the binary QL algorithms for the different values of w. This is due to the added flexibility with multi SM levels on saving energy. By tuning the value of w, the operator can shift the performance of his network towards either energy saving or QoS.

In order to analyze the performance of the policies, we report in Fig. 7 the average data lost due to the sleep scheme policy. Even though the average delay of the requested data falls under the delay constraint limit of 100 ms (as shown in Fig. 6), around half of these requested data are dropped for small values of w. This is because the Q-learning algorithm optimizes the data dropped that is indirectly related to the served delay. Hence, it is important to link served data delay with their drop rate.

V. CONCLUSIONS

Network densification with small cells is a key enabler in future 5G networks to support high data rate demands. However, it is critical to manage the activity of these cells in order to limit the network energy consumption. This paper has investigated an approach based on reinforcement learning to manage the activity of small cells to reduce the energy consumption while maintaining a good QoS. In this regard, we proposed a Q-learning algorithm that controls the states of these cells over different sleep mode levels that are compliant with 5G requirements with respect to the synchronization signals periodicity. The proposed algorithm allowed the operator to freely manage the tradeoff between energy saving and delay. Our simulations showed that given a delay constraint, the network is capable of saving at least 50% energy. Future work will focus on extending the model to include users offloading between cells including the macro cell.

Fig. 1 :

 1 Fig. 1: Network architecture with a distributed controller.

3 :

 3 Set the weight w, and the average user velocity. 4: while Learning do 5:for m ∈ M do 6:

8 : 9 :

 89 Calculate the cost c m . Observe next state s m . 10:

Fig. 2 :Fig. 3 :

 23 Fig. 2: Evolution of the convergence of the Q-Learning offline algorithm.

Fig. 4 :

 4 Fig. 4: Evolution of the convergence of the Q-Learning online algorithm.

Fig. 5 :Fig. 6 :Fig. 7 :

 567 Fig. 5: Different SM policies as a function of w.

TABLE I :

 I BS Sleep Modes Characteristics[START_REF] Debaillie | A flexible and futureproof power model for cellular base stations[END_REF].

	Sleep	Deactivation	Minimum	Activation
	level	duration	sleep duration	duration
	SM 1	35.5 µs	71 µs	35.5 µs
	SM 2	0.5 ms	1 ms	0.5 ms
	SM 3	5 ms	10 ms	5 ms
	SM 4	0.5 s	1 s	0.5 s