A Rapetti

F Christien

F Tancret

P Todeschini

S Hendili

J Stodolna

Surfactant effect of impurity sulphur in ductility dip cracking of a high-chromium nickel model alloy

Keywords: Interface segregation, grain boundary embrittlement, impurity embrittlement, welding, ductility dip cracking

The sensitivity to ductility dip cracking was measured in two model high chromium nickel alloys, with the same composition, apart from different sulphur contents. The newly developed Refusion Cracking Test was used, that consists in repetitive refusion lines conducted at the specimen surface. Grain boundary cracks develop in the heat affected zone near the refusion line. Cracking is much more pronounced in the sulphur-enriched alloy,

using WDS and STEM-EDS revealed no segregation of sulphur at grain boundaries. In contrast, grain boundary fracture surfaces are covered with sulphur. This suggests a dynamic type of grain boundary embrittlement where sulphur acts as a surfactant, facilitating crack opening. Sulphur is efficiently provided to the crack tip as it propagates, due to accelerated diffusion by plastic deformation. This allows crack growth rates higher than 10 µm/s. Austenitic materials are prone to "ductility trough" at temperatures between 0.5 and 0.8T M (melting temperature), which manifests itself in grain boundary (GB) cracking and limited elongation to fracture. This phenomenon is also termed "intermediate temperature embrittlement", "hot-shortness" or "ductility dip cracking" (DDC). The detrimental effect of trace elements, especially sulphur, in DDC of steels [1,2], nickel [3][4][5] and copper [6] alloys was demonstrated in early works, using model alloys of well controlled composition at the ppm level, so as to unambiguously identify the impact of particular trace elements. However, some examinations conducted in more recent studies [7,8] of DDC cracked specimens of different materials using advanced characterisation techniques, like Transmission Electron Microscopy or Atom Probe Tomography, failed in demonstrating the segregation of sulphur at GBs. In view of those studies, the role of sulphur in DDC can indeed be questioned.

High chromium nickel alloys are widely used in the primary circuit of pressurized water nuclear reactors. Numerous studies have shown that those alloys can be sensitive to DDC, especially in welding conditions (see for example the background section in [9]). In the present work, DDC is studied on model alloys of identical composition with controlled additions of sulphur in order to unambiguously demonstrate the potentially detrimental effect of this element. Complementary chemical analyses of uncracked and cracked grain boundaries are conducted to investigate the influence of sulphur on embrittlement mechanisms. The sensitivity to DDC is determined using the newly developed RCT test (Refusion Cracking Test) [9,10], that is well adapted to the study of model alloys, where only limited amounts of materials are available. Post-mortem chemical analyses of GBs were conducted in order to question the occurrence of sulphur GB segregation. A control specimen with a known amount of sulphur GB segregation was used to validate the characterization methodologies. The GB cracking mechanism is discussed in the frame of a dynamic type of GB embrittlement due to sulphur.

The materials used in this study are two model alloys of controlled purity with similar composition as 690 alloy (Table 1). 0.5 kg of each alloy was obtained using vacuum melting in a cold silver crucible. After solidification, the ingots (25×25×100 mm) were transformed into plates (7×50×170 mm) by hammering at room temperature. The first alloy (Base) displays a very low sulphur content whereas the second alloy (Base+S) is enriched in sulphur.

Apart from sulphur, the composition of the two alloys is the same. In addition, pure nickel containing 5.4 wt ppm of sulphur was used as a control specimen for sulphur segregation measurements. RCT was conducted in order to assess the sensitivity to DDC of the two alloys. It consists in multiple GTAW (Gas Tungsten Arc Welding) refusion passes performed between two initial pre-passes. The procedure is detailed in [9,10] and schematically reminded in Figure 1a). The procedure ensures that the Heat Affected Zone (HAZ) of the central passes are located in the welding solidification microstructure, which makes the test representative of actual welding conditions prone to DDC [11]. The sample dimensions are 50407 mm. A welding speed of 11 cm/minute, a voltage of 10V and an intensity of 120A were used. Eight and four refusion passes were carried out on the Base and Base+S alloys respectively. After each pass, a binocular magnifier was used to measure the number of cracks and the cumulative crack length at the specimen surface. Cracks are located at distances ranging approximately from 0.4 to 2.2 mm from the melt pool, i.e. in the HAZ. Figure 1b) shows the typical temperature cycle measured during a pass using a thermocouple located at the centre of the cracking region. Thermomechanical simulation of the RCT test was conducted in [START_REF] Rapetti | Fissuration à Chaud Par Chute de Ductilité Dans Les Métaux d'apport Pour Le Soudage d'alliages à Base de Nickel[END_REF] on a 52M

Inconel alloy, having a composition very close to that of the materials studied here. The simulation showed that the maximum tensile stress that develops in the HAZ perpendicularly to the welding direction during a pass is about 300 MPa. In addition, the cumulated plastic deformation is as high as 25% in the HAZ after five passes.

Sulphur GB segregation was measured using WDS (Wavelength Dispersive X-ray Spectroscopy) in the SEM (Scanning Electron Microscope) according to the methodology developed by Christien and Risch [START_REF] Christien | [END_REF]. This methodology consists in acquiring concentration line scans of the segregated solute across the boundary. The apparent concentration peak obtained is a convolution of the actual concentration profile (here extremely narrow, i.e.

~ 1 nm) with the probe function (~ 1 µm). Quantification of the segregated element is obtained from integration of the concentration peak area and is expressed as a mass of sulphur per unit GB area (nanograms/cm 2 ). More details about the methodology and the system used can be found in [START_REF] Christien | [END_REF]. Additional sulphur segregation observations were conducted using EDS (Energy-Dispersive X-ray Spectroscopy) on FIB (Focused Ion Beam) lamellas in a FEI Osiris S/TEM (Scanning Transmission Electron Microscope) equipped with the SuperX system consisting of four EDS spectrometers. The lamellas were prepared in a FEI Helios Nanolab.

The final ion thinning was obtained in a PIPS II system from Gatan at 500 V. Lamella thicknesses below 70 nm were obtained. The STEM beam voltage and current were 200 kV and 2 nA respectively and the probe size was approximately 0.5 nm. Sulphur mapping was conducted in the scanning mode at a resolution of ~500×1000 pixels and the exposure time per pixel was 10 ms. WDS line scans were performed across eleven GBs in the cracked area (red rectangle in Figure 3b)). Only uncracked GBs were selected for WDS analysis. As WDS is not among the common techniques to study monolayer segregation, a control specimen was used to ascertain the capability of the methodology to detect sulphur segregation in the submonolayer range.

This control specimen is pure nickel containing 5.4 wt ppm of bulk sulphur. It was heat treated 100 days at 550°C to obtain a segregation level of some tens of ng/cm 2 . Sulphur GB segregation in nickel was extensively studied in previous works using SIMS [14], Auger [15,16] and WDS [START_REF] Christien | [END_REF]17,18]. Figure 4a) presents examples of GB sulphur profiles obtained in the Base+S alloy (after four RCT passes) and in the control specimen. The sulphur segregation peak is clearly observed in the control specimen as expected. In contrast, no sulphur peak is obtained for the GB analysed in the Base+S alloy after four RCT passes.

Figure 4b) presents the GB sulphur concentrations obtained using the quantitative methodology described in [START_REF] Christien | [END_REF] for the control specimen and for the Base+S model alloy after four RCT passes. Sulphur is detected in all the GBs analysed in the control specimen. The average sulphur GB concentration is 54 ng/cm 2 , i.e. ~10 15 S atoms/cm 2 or 0.55 monolayer, considering that a "monolayer" is a (111) close-packed plane of nickel. In contrast no sulphur segregation is detected in any of the GBs analysed in the Base+S alloy after four RCT passes.

This means that no sulphur segregation exists in that specimen or it is below the limit of detection (previous studies have shown that the limit of detection of the WDS methodology is of a few percents of a sulphur monolayer [START_REF] Christien | [END_REF]18]). This oxide layer is located on top of the sulphur segregation layer, which suggests that oxidation occurred after sulphur segregation. RCT tests conducted on model alloys clearly demonstrated that the sulphur-enriched alloy is much more sensitive to cracking (Figure 2). It should be noted that model alloys were used in this study, differing only in their sulphur content, so that there cannot be any ambiguity on the detrimental effect of this element. As the cracks obtained are all intergranular, it is quite natural to suspect that the metallurgical cause of cracking is the GB segregation of sulphur.

However the numerous GB analyses (Figure 4 and Figure 5b)) conducted on the post-mortem RCT specimen have revealed no sulphur segregation at all in uncracked boundaries. Similar attempts to demonstrate the GB segregation of sulphur on DDC cracked specimens of nickel alloys were made in previous studies by Noecker et al. [7]. As no sulphur GB segregation was found, those authors concluded that sulphur was not involved in the mechanism of DDC. However their conclusion can be questioned in view of the results obtained in the present study: despite the absence of sulphur segregation in uncracked GBs, the detrimental effect of sulphur is not in doubt here, considering the results of Figure 2.

In contrast to uncracked GBs, the STEM-EDS analysis conducted on a cracked GB has shown the strong segregation of sulphur at the cracked surfaces ( Figure 5 c)). This suggests that the mechanism of cracking is of dynamic type, i.e. the cracked surfaces are gradually covered with sulphur at the same time as the crack propagates. Sulphur segregation to the surfaces of the opening crack results in a reduction of the surface energy, which facilitates crack opening, according to the Griffith criterion for example [19]. This mechanism was rationalised by the "slow fracture" approach [20,21] in which the chemical potential of the segregating solute is considered constant as the GB opens. This implies a change in composition of the two surfaces created by the propagating crack. The sulphur acts here as a "surfactant" facilitating the creation of new surfaces [22]. Of course this scenario is possible only if the transport of sulphur is fast enough to "feed" the propagating crack tip.

crack growth rate da/dt (m/s) has to be below a certain value given by Eq.1:

1.

where is the solute diffusion coefficient (m 2 /s), is the solute content in solid solution (m -3 ), is a threshold surface solute concentration (m -2 ) needed to allow the dynamic opening of the grain boundary. Eq.1 can be adapted to the non-isothermal conditions of the RCT tests conducted here:

2.
a is the increase in crack length obtained in one RCT pass, t p is the duration of the RCT pass.

Considering the temperature cycle shown in Figure 1b) (corresponding to the centre of the cracking region), the evolution with time of the diffusion coefficient D(t) can be determined using , and (bulk diffusion coefficient of sulphur in nickel [23]). With (for the Base+S alloy) and , the estimated increase in crack length a for one RCT pass is about 1 µm. The value considered for  is somewhat arbitrary but this is not going to affect significantly the order of magnitude obtained for a.

On the other hand, the actual increase in crack length per pass can be roughly estimated from the RCT data shown in Figure 2. For the Base+S alloy, the number of cracks (Figure 2a))

does not change between passes 3 and 4. Consequently, the corresponding increase in cumulative crack length (Figure 2b)) is due to propagation of existing cracks, not to the onset of new ones. The increase in crack length can then be simply estimated by dividing the increase in cumulative crack length (+ 40 mm) by the number of existing cracks (~300), which results in an individual increase in crack length of ~130 µm per RCT pass. This indicates that the cracks are actually propagating faster by two orders of magnitude than predicted by Eq.2 (~1 µm). Assuming that the time allowed for crack propagation during one RCT pass is roughly about 10 s (i.e. the time spend above ~500°C in Figure 1b)), the average crack growth rate is at least 13 µm/s. This is not unreasonable since previous works on sulphur-induced dynamic embrittlement of steels and nickel alloys [24][25][26][27] have already demonstrated crack growth rates as high as about 10 µm/s in the temperature range 500°C-800°C. It has to be kept in mind that the plastic deformation in the cracking region can strongly accelerate the transport of sulphur to the crack tip. Allart et al. [28] have demonstrated an acceleration by several orders of magnitude of sulphur diffusion during hot deformation of nickel. The high crack growth rate observed here, compared to that estimated from Eq.2, suggests a strong acceleration of sulphur diffusion to the crack tip, due to plastic deformation. This accelerated diffusion efficiently provides the crack tip with sulphur, which allows fast crack propagation. Similar observations were recently obtained in a study on DDC of an austenitic Fe-Ni alloy, where the accelerated sulphur diffusion to the crack tip was explained by sulphur dragging by moving dislocations [27].

In summary, using model alloys of controlled composition, it was shown that sulphur has a strong deleterious effect on ductility dip cracking of a high chromium nickel alloy. On the other hand, post-mortem analyses using WDS and STEM-EDS showed no GB segregation in cracked specimens, although a control specimen was used to validate the methodology of analysis. The presence of sulphur covering the surfaces created by GB crack propagation suggests a dynamic type of embrittlement, where sulphur feeds the crack tip as it propagates.

The surfactant effect of sulphur decreases the surface energy, which facilitates crack opening.

The relatively high crack growth rate observed (>~10 µm/s) requires a fast transport of sulphur to the propagating crack tip, which can be achieved by accelerated diffusion of sulphur due to plastic deformation. Considering this mechanism, there is no contradiction

Figure 1 .

 1 Figure 1. a) Principle of Refusion Cracking Test. Small red lines represent cracks. The length

Figure 2

 2 Figure2presents the results obtained for the two alloys tested using RCT, i.e. the number of

Figure 2 .

 2 Figure 2. RCT results for Base and Base+S alloys. a) Number of cracks. b) Cumulative crack

Figure 3 (

 3 Figure 3 (a) presents a SEM surface observation of the Base+S sample after four RCT passes.

Figure 3 .

 3 Figure 3. Observations of Base+S sample after four RCT passes. a) SEM surface observation,

Figure 4 .Figure 5 )Figure 5 Figure 5 Figure 5

 45555 Figure 4. a) Examples of GB sulphur line scans obtained using the WDS methodology in the

Figure 5 .

 5 Figure 5. STEM-EDS mapping on GBs. a) control specimen, b) uncracked GB and c) cracked GB in Base+S RCT specimen. The dashed red line indicates the GB location. Blue rectangles in images indicate the mapped areas. The images were obtained using STEM high-angle anunular dark field for a) and bright field TEM for b) and c).

Table 1 .

 1 Composition of the model alloys used in this study.

		Ni	Cr	Fe	Mn	Si	Ti	Al	C	N	O	S
	Alloy				Wt %					Wt ppm
	Base	*	29.9 8.42 0.85 0.14 0.21 0.13	11	1	7	4.7
	Base+S	*	30.0 8.44 0.84 0.14 0.21 0.13	16	1	7	52

DATA AVAILABILITY STATEMENT

The raw and processed data required to reproduce these findings are available from the corresponding author upon request.

between the deleterious effect of sulphur on ductility dip cracking and the absence of sulphur segregation in uncracked GBs of cracked specimens.

ACKNOWLEDGMENT

This research was mainly funded by EDF (Electricité de France) with the support of ANRT (Association Nationale Recherche Technologie). The authors thank Claude Varillon for the preparation of model alloys.


Andrii Karpus

Dr Simon Harrisson

Rinaldo Poli

Stéphane Mazières

Dr Eric Manoury
email: eric.manoury@lcc-toulouse.fr

Mathias Destarac
email: destarac@chimie.ups-tlse.fr

Stéphane Mazières

Dr A Karpus

Prof R Poli

Dr S Mazières

Prof M Destarac

Stephane Mazières

Well-Defined P III -Terminated Polymers from Phosphorylated Carbodithioate RAFT Agents

come   L'archive ouverte pluridisciplinaire

Introduction

Reversible addition-fragmentation chain transfer (RAFT) [START_REF] Moad | Living Radical Polymerization by the RAFT Process -A Third Update[END_REF][START_REF] Perrier | 50th Anniversary Perspective: RAFT Polymerization-A User Guide[END_REF] polymerization is a widelyapplied tool for macromolecular design. The essential component of this type of reversibledeactivation radical polymerization 1 is a chain transfer agent (CTA) with the general formula Z-C(S)S-R0. The CTA efficiency is determined by the reactivity of the C=S double bond, which is strongly affected by the nature of Z-group [START_REF] Chiefari | Thiocarbonylthio Compounds (SC(Z)S-R) in Free Radical Polymerization with Reversible Addition-Fragmentation Chain Transfer (RAFT Polymerization). Effect of the Activating Group Z[END_REF][START_REF] Destarac | Dithiocarbamates as Universal Reversible Addition-Fragmentation Chain Transfer Agents[END_REF] and by the lability of the S-R0 bond [START_REF] Chong | Thiocarbonylthio Compounds [SC(Ph)S-R] in Free Radical Polymerization with Reversible Addition-Fragmentation Chain Transfer (RAFT Polymerization). Role of the Free-Radical Leaving Group (R)[END_REF] in the initiation step. Therefore, for the polymerization of a particular monomer, it is of great importance to select a CTA with appropriate Z and R0 groups in order to moderate the reactivity of the carbodithioate group, with Z playing a key role in the stability of the intermediate radical.

The most successful families of RAFT agents [START_REF] Keddie | RAFT Agent Design and Synthesis[END_REF][START_REF] Destarac | On the Critical Role of RAFT Agent Design in Reversible Addition-Fragmentation Chain Transfer (RAFT) Polymerization[END_REF] are dithioesters [START_REF] Chiefari | Living Free-Radical Polymerization by Reversible Addition-Fragmentation Chain Transfer: The RAFT Process[END_REF][START_REF] Moad | Mechanism and Kinetics of Dithiobenzoate-Mediated RAFT Polymerization -Status of the Dilemma[END_REF] (Z = alkyl or aryl), dithiocarbamates [START_REF] Mayadunne | Living Radical Polymerization with Reversible Addition-Fragmentation Chain Transfer (RAFT Polymerization) Using Dithiocarbamates as Chain Transfer Agents[END_REF][START_REF] Destarac | Dithiocarbamates as Universal Reversible Addition-Fragmentation Chain Transfer Agents[END_REF] (Z = NR2), dithiocarbonates or xanthates [START_REF] Chiefari | Thiocarbonylthio Compounds (SC(Z)S-R) in Free Radical Polymerization with Reversible Addition-Fragmentation Chain Transfer (RAFT Polymerization). Effect of the Activating Group Z[END_REF][START_REF] Destarac | Xanthates as Chain-Transfer Agents in Controlled Radical Polymerization (MADIX): Structural Effect of the O-Alkyl Group[END_REF] (Z = OR) and trithiocarbonates [START_REF] Chiefari | Living Free-Radical Polymerization by Reversible Addition-Fragmentation Chain Transfer: The RAFT Process[END_REF][START_REF] Fuchs | Stability of Trithiocarbonate RAFT Agents Containing Both a Cyano and a Carboxylic Acid Functional Group[END_REF] (Z = SR). Several hundred of these compounds have been developed and their efficiency in RAFT polymerization has been systematically studied. Nevertheless, each CTA is suitable to control the polymerization of only a narrow range of monomers except for isolated cases like N-methyl-N-(4-pyridinyl) [START_REF] Benaglia | Switchable) RAFT Agents[END_REF][START_REF] Keddie | Switchable Reversible Addition-Fragmentation Chain Transfer (RAFT) Polymerization in Aqueous Solution, N,N-Dimethylacrylamide[END_REF] and N-aryl-N-(4-pyridinyl) [START_REF] Moad | Advances in Switchable RAFT Polymerization[END_REF] dithiocarbamate switchable RAFT agents or CTAs of intermediate reactivity such as O-alkyl xanthates [START_REF] Destarac | How Can Xanthates Control the RAFT Polymerization of Methacrylates? In Reversible Deactivation Radical Polymerization: Mechanisms and Synthetic Methodologies[END_REF][START_REF] Destarac | On the Importance of Xanthate Substituents in the MADIX Process[END_REF] or pyrazole-based dithiocarbamates. [START_REF] Gardiner | Dithiocarbamate RAFT Agents with Broad Applicabilitythe 3,5-Dimethyl-1H-Pyrazole-1-Carbodithioates[END_REF][START_REF] Gardiner | 4-Halogeno-3,5-Dimethyl-1H-Pyrazole-1-Carbodithioates: Versatile Reversible Addition Fragmentation Chain Transfer Agents with Broad Applicability[END_REF] A few examples of more "exotic" CTAs based on Zgroups involving other elements including heteroelements have also been described: fluorine [START_REF] Theis | A Synthetic Approach to a Novel Class of Fluorine-Bearing Reversible Addition-Fragmentation Chain Transfer (RAFT) Agents: F-RAFT[END_REF] , selenium [START_REF] Matioszek | Reversible Addition-Fragmentation Chain-Transfer Polymerization of Vinyl Monomers with N,N-Dimethyldiselenocarbamates[END_REF][START_REF] Demirci | A New Selenium-Based RAFT Agent for Surface-Initiated RAFT Polymerization of 4-Vinylpyridine[END_REF][START_REF] Zeng | Selenium-Substituted Carbonates as Mediators for Controlled Radical Polymerization[END_REF] , tin [START_REF] Kulai | Alkyl Triarylstannanecarbodithioates: Synthesis, Crystal Structures, and Efficiency in RAFT Polymerization[END_REF][START_REF] Kulai | RAFT Polymerization with Triphenylstannylcarbodithioates (Sn-RAFT)[END_REF] or phosphorus. [START_REF] Laus | Controlled Radical Polymerization of Styrene with Phosphoryl-and (Thiophosphoryl)Dithioformates as RAFT Agents[END_REF][START_REF] Alberti | Direct ESR Detection of Free Radicals in the RAFT Polymerization of Styrene[END_REF][START_REF] Mazières | Phosphinoyl and Thiophosphinoylcarbodithioates: Synthesis, Molecular Structure, and Application as New Efficient Mediators for RAFT Polymerization[END_REF][START_REF] Geagea | Chromium and Molybdenum Pentacarbonyl Complexes of Phosphinocarbodithioates: Synthesis, Molecular Structure and Behaviour in RAFT Polymerisation[END_REF][START_REF] Kulai | Manganese Phosphinocarbodithioate for RAFT Polymerisation with Sunlight-Induced Chain End Post-Treatment[END_REF][START_REF] Kulai | Enhanced Control of Phosphinoylcarbodithioate-Mediated RAFT Polymerization: Key Role of Substituents at the Phosphorus Center[END_REF][START_REF] Glassner | Ultra)Fast Catalyst-Free Macromolecular Conjugation in Aqueous Environment at Ambient Temperature[END_REF] The introduction of heteroelements with nuclei of halfinteger spin allows the use of NMR spectroscopy to study the CTA consumption, kinetics and mechanisms at the early stages of the polymerization process, as well as for the characterization of the ω-chain end functionality of the obtained polymers. [START_REF] Demirci | A New Selenium-Based RAFT Agent for Surface-Initiated RAFT Polymerization of 4-Vinylpyridine[END_REF][START_REF] Zeng | Selenium-Substituted Carbonates as Mediators for Controlled Radical Polymerization[END_REF][START_REF] Laus | Controlled Radical Polymerization of Styrene with Phosphoryl-and (Thiophosphoryl)Dithioformates as RAFT Agents[END_REF][START_REF] Alberti | Direct ESR Detection of Free Radicals in the RAFT Polymerization of Styrene[END_REF][START_REF] Mazières | Phosphinoyl and Thiophosphinoylcarbodithioates: Synthesis, Molecular Structure, and Application as New Efficient Mediators for RAFT Polymerization[END_REF][START_REF] Geagea | Chromium and Molybdenum Pentacarbonyl Complexes of Phosphinocarbodithioates: Synthesis, Molecular Structure and Behaviour in RAFT Polymerisation[END_REF] Phosphorus-containing CTAs (P-RAFT) offer a possible reactivity modulation by coordination of the P atom to a metal (for P III agents, Z = P III (R)2Mt/L) [START_REF] Geagea | Chromium and Molybdenum Pentacarbonyl Complexes of Phosphinocarbodithioates: Synthesis, Molecular Structure and Behaviour in RAFT Polymerisation[END_REF][START_REF] Kulai | Manganese Phosphinocarbodithioate for RAFT Polymerisation with Sunlight-Induced Chain End Post-Treatment[END_REF] or by varying the electron-withdrawing strength of the phosphorus atom substituents (for P V agents, e.g. Z = P(E)R2, E = O, S). [START_REF] Mazières | Phosphinoyl and Thiophosphinoylcarbodithioates: Synthesis, Molecular Structure, and Application as New Efficient Mediators for RAFT Polymerization[END_REF][START_REF] Glassner | Ultra)Fast Catalyst-Free Macromolecular Conjugation in Aqueous Environment at Ambient Temperature[END_REF] Despite simple and effective syntheses, P-RAFT CTAs are still rather scarce. Nevertheless, interest in such compounds has grown because of the good control obtained for more-activated monomers such as styrene (St) and n-butyl acrylate (nBA).

However, to the best of our knowledge RAFT polymerizations with a free (i.e. not coordinated) P III as a function in the CTA Z group have not been described. Indeed, a comparison of the reactivity of either P III and P V , or free and coordinated P III CTA derivatives in polymerization would significantly aid the fundamental understanding of the Z-group impact. A potential switch between P V and P III would lead to an increase in the efficiency spectrum of these agents from more-activated monomers to less-activated monomers. In addition, a free P III ω-chain endfunctionality for the obtained polymers may open new applications via its ligand properties, for instance coordination to metal cations or to metal nanoparticles. [START_REF] Skaff | Reversible Addition Fragmentation Chain Transfer (RAFT) Polymerization from Unprotected Cadmium Selenide Nanoparticles[END_REF] We have recently reported the simple and effective gram-scale synthesis of various S-(1-phenylethyl)(di-tert-butylphosphorylated)carbodithioates with different substituents on phosphorus (1-3 in Figure 1), explored their spectral properties and described simple chemical transformations between members of the series. [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] In particular, we have obtained for the first time thiocarbonylthio compounds with phosphine-borane (1) or free phosphine (3) as Z groups.

In this contribution, we describe the reactivity of these phosphorus-containing RAFT agents, as well as two new related compounds, in the RAFT polymerization of monomers with different reactivity and propose methods for the preparation of free phosphine-terminated polymers.

Results and discussion

Figure 1. P III -P V CTA derivatives evaluated in RAFT polymerization.

The synthesis of the new CTAs 4 and 5 (Figure 1), containing an S-bonded 1methoxycarbonylethyl group (acrylate-type radical) as R0, was similar to that previously described for the 1-phenylethyl derivatives 1 and 3, using methyl 2-bromopropionate as electrophile in the alkylation step instead of 1-bromo-ethylbenzene (see Scheme 1 and experimental part). [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] Scheme 1. Synthesis of 4 and 5.

The efficiency of the P-RAFT agents 1-4 was evaluated in the polymerization of nBA and St (Scheme 2). CTA 5 was used as a model for polyacrylate-P III chain end in our [START_REF] Kulai | Manganese Phosphinocarbodithioate for RAFT Polymerisation with Sunlight-Induced Chain End Post-Treatment[END_REF] P NMR investigations (vide infra). All polymerizations reported in Figures 2345and Table S1 were carried out in sealed glass tubes at 60°C with azobisisobutyronitrile (AIBN) as thermal initiator and toluene as solvent. Polymerization of n-butyl acrylate. Linear semi-logarithmic plots (Figure 2) were observed in the early stages of the nBA polymerizations in the presence of compounds 1, 3 and 4, corresponding to pseudo-first-order kinetics with downward deviation from linearity at higher reaction times, which is due to initiator decay. The apparent rate constants (kp app =kp[P°]) vary as follows: 1 ~ 4 > 3.

The BH3-derivative 1 and its homologue 4 (differing by the R0 group) afford a good level of control, with Mn close to the targeted one and present moderate dispersities (Đ = 1.25-1.59 for 1, entries 1-4 of Table S1 and Đ = 1.25-1.49 for 4, entries 19-24, SEC-RI chromatograms in Figure S13 for 1 and in Figure S15 for 4) which tend to decrease during the first half of the polymerization and then slightly increase mainly due to the increasing fraction of AIBN-derived chains (Figure 3). Interesting information can be extracted from the 31 P NMR monitoring. The continuous disappearance of the broadened peaks at 69-70 ppm, corresponding to the phosphorus linked to the BH3 unit (Figures S2 (1) and S6 (4)) and the observed bleaching of the intensively pink-colored initial solution with time may be due to a gradual thermal degradation of the dormant chain-ends.

With the free phosphine (P III ) derivative 3, a poor level of control was observed for the polymerization of nBA, with much higher Mn values than expected (Table S1, entries 9-13) and also dispersities ranging between 1.48 and 1.80 (SEC-RI chromatograms, Figure S17).

Additionally, the 31 P{ 1 H} NMR of the reaction mixtures shows the disappearance of the CTA peak at 67.4 ppm after 1 h of polymerization and 2.3% of monomer conversion (Figure S10) with the formation of multiple products. This observation allows us to suppose a non-selective decomposition of the free phosphine thiocarbonylthio chain-end under the polymerization conditions, even though compound 3 was proven to be thermally stable since no changes in the 31 P{ 1 H} NMR spectrum were observed in toluene-d8 at 60°C up to 48 h of heating. [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] The change in color of the polymerization mixture from bright pink to pale orange is another indication of the degradation of the phosphinocarbodithioate group during the polymerization. In addition, when a toluene solution of 3 with 300 eq. of n-butyl acrylate was heated at 60°C for 24 h without AIBN, the 31 P NMR analysis of the resulting solution confirmed the presence of numerous peaks, indicating the instability of 3 under these conditions.

As the ionic CTA 2 was insoluble in toluene, the polymerization of nBA with 2 was attempted in acetonitrile at 60°C. No polymer was formed. Instead, a loss of color, the precipitation of insoluble products coming from the decomposition of the functionalized chainends, and the disappearance of the 31 P{ 1 H} NMR resonance (Figure S23) were observed after 1 h. Both phosphine-borane derivatives 1 and 4 led to a much better control for the polymerization of St than for that of nBA, with low dispersities of 1.04 -1.11 (1) and 1.07 -1.10 (4) for the highest conversions (Table S1, 1 entries 6-8, 4 entries 26-29). The SEC analyses also presented the characteristics of well-controlled polymerizations (Figure S14 for 1 and Figure S16 for 4). The 31 P{ 1 H} NMR spectra recorded for the polymerization mixtures are given in Figure S4 and Figure S8, respectively. CTA 4 is fully consumed at < 10% conversion, indicating that it has a high transfer constant for St. CTA 1 is fully consumed at ca. 20% conversion. Both 1 and 4 present a similar level of control, with only slightly different dispersities after 120 h of reaction. Contrary to the nBA polymerizations (see above), neither loss of color nor disappearance of the phosphine-borane peak were observed during St polymerizations, which lasted up to 120 h. Thus, the thermal stability of the BH3-CTA derivatives is in accordance with our previous observations [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] and the slow decomposition of the phosphinocarbodithioate end groups during the polymerization of nBA may result from a specific reactivity between the phosphinocarbodithioate functionality and nBA or the propagating radical, such as the reduction of the nBA ester function by borane.

A similar level of control for the polymerization of St was observed with the free phosphine derivative 3, with dispersities of 1.09 -1.15 (Table S1, entries 14-18). The results of the SEC analysis are presented in Figure 6. Contrary to the other CTAs of the series, a small peak is visible in the higher molar mass region for the highest reaction times, which is reflected in the slightly higher dispersity values observed (1.15 for 3 compared to 1.10 or lower for 1 and 4). However, this corresponds to the first example of an efficient RAFT polymerization mediated with a P III phosphino-RAFT agent, thereby allowing the direct preparation of wellcontrolled PSt samples with free phosphine ω-chain ends. Figure 6. Overlay of the RI-SEC chromatograms of PSt samples obtained in the polymerization mediated by P-RAFT 3.

To better explore the impact of the electronic difference in the Z-group, a series of St polymerizations were carried out with CTAs 1 and 3 in sealed NMR tubes and stopped at less than 4% monomer conversion (Figures S19-S20). In the case of 3 it was possible to follow the CTA consumption using 31 P{ 1 H} NMR as previously reported, but not for 1 due to broad, overlapping peaks. [START_REF] Kulai | Enhanced Control of Phosphinoylcarbodithioate-Mediated RAFT Polymerization: Key Role of Substituents at the Phosphorus Center[END_REF] CTA 3 was completely consumed at monomer conversions < 2% (Table S2, Figure S20). Similar monitoring by 1H NMR spectroscopy, on the other hand, did not allow the same level of precision. This study confirms that 31P{1H} NMR is a powerful tool to quantitatively assess the ability of the P-RAFT agents to transfer to PSt oligoradicals. Analyses performed by SEC at ~1% monomer conversion suggested a more selective formation of oligomers with CTA 3 than for 1 (see figures S21-S22 for the SEC-RI of compounds 1 and 3, respectively).

Vinyl acetate. Compounds 1-3 were not tested as possible CTAs to control the polymerization of vinyl acetate (VAc) due to the very low rate of 1-phenylethyl leaving radical addition to VAc. [START_REF] Benaglia | Polystyrene-Block-Poly(Vinyl Acetate) through the Use of a Switchable RAFT Agent[END_REF] In contrast, compounds 4 and 5 possess a 1-methoxycarbonylethyl leaving group which adds rapidly and efficiently to VAc. [START_REF] Girard | Influence of Macromolecular Characteristics of RAFT/MADIX Poly(Vinyl Acetate)-Based (Co)Polymers on Their Solubility in Supercritical Carbon Dioxide[END_REF][START_REF] Moad | A Critical Survey of Dithiocarbamate Reversible Addition-Fragmentation Chain Transfer (RAFT) Agents in Radical Polymerization[END_REF] Both 4 and 5 completely inhibited the polymerization, similarly to the behavior previously reported for phosphinoyl CTAs. [START_REF] Mazières | Phosphinoyl and Thiophosphinoylcarbodithioates: Synthesis, Molecular Structure, and Application as New Efficient Mediators for RAFT Polymerization[END_REF][START_REF] Kulai | Enhanced Control of Phosphinoylcarbodithioate-Mediated RAFT Polymerization: Key Role of Substituents at the Phosphorus Center[END_REF] This is not surprising for CTA 4 but counter-intuitive for 5, as this compound, like the xanthates and dithiocarbamates that are efficient RAFT agents for VAc, [START_REF] Destarac | Dithiocarbamates as Universal Reversible Addition-Fragmentation Chain Transfer Agents[END_REF] possesses a lone pair in its Z-group that is available for conjugation with the C=S bond of the thiocarbonylthio function. This result, together with the rather unexpected high reactivity of P(III) RAFT agent 3 in St polymerization led us to consider a theoretical calculation approach.

Theoretical calculations. Theoretical calculations were carried out on compound 3 and on related model compounds. In particular we wished to compare related R2EC(S)SR0 systems (E = N, P), since dithiocarbamates (E = N) are well-known to provide suitable control for the RAFT polymerization of VAc. [START_REF] Destarac | Dithiocarbamates as Universal Reversible Addition-Fragmentation Chain Transfer Agents[END_REF] The first investigated systems, for the purpose of computational efficiency, had the small CH3 as R in the Z group and R0 = CHMe(O2CMe), which represents a good model of the PVAc chain. The optimizations revealed different configurations at the E atom for these two CTAs: planar for N (sum of the three bond angles = 359.98°) and pyramidalized for P (301.98°), as can be seen in Figure 7 The possible engagement of the P lone pair in a resonance with the CS2 group could be affected by further conjugation and by steric hindrance. These possibilities were probed by optimizing the geometries of the CTAs with R = Ph (R0 = CHMe(O2CMe)) and tBu (R0 = CHMePh), respectively. The resulting geometries are shown in Figure 8. For both systems, the N atom in the dithiocarbamate system has an essentially planar geometry, whereas the P atom in the phosphinocarbodithioate remains markedly pyramidalized. It is remarkable that the large steric encumbrance of the two tBu groups hampers a full conjugation between the tBu2N and CS2 groups, since the two planes in tBu2NC(S)SCHMePh are skewed by a dihedral angle of 57°C, but the N atom does not greatly lose its planarity (sum of the bond angles = 350.08°, vs 359.99° for the Ph system). On the other hand, the same steric bulk slightly attenuates pyramidalization of the P atom in the phosphinocarbodithioate system, but only from 306.74° for R = Ph to 316.96° for R = tBu. Well-controlled polymers with P III -chain ends. While well-controlled PSt samples can be prepared directly by RAFT polymerization with agent 3 (Scheme 3, top), the same polymer may be also accessible via a two-step polymerization-deboranation procedure with the use of the borane protected RAFT agent 1 (Scheme 3, bottom), followed by treatment of PSt-1 by triethylamine to remove the borane caps, as previously described for the deboranation of CTA 1. [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] For the nBA polymerization, the P III -RAFT CTA 3 showed limitations with no control and NMR evidence for chain-end degradation due to the incompatibility of the P III moiety and nBA (see above). To overcome this obstacle in the control of the acrylate monomer polymerization with a free phosphine P-RAFT agent, the two-step polymerization-deboranation procedure was repeated with the use of the borane-protected RAFT agent 4, which involves an acrylate type leaving group suitable for nBA polymerization. This strategy allowed us to achieve the generation of a well-controlled PnBA-4 (-BH3) bearing also P III -chain ends (Scheme 4). Scheme 3. Preparation of polystyrene with P III ω-chain ends by two pathways: direct application of P III -RAFT agent (top), two-step polymerization-deboranation procedure (bottom). Scheme 4. Preparation of poly(n-butyl acrylate) with P III ω-chain ends by a two-step polymerization-deboranation procedure.

The two polymer samples PSt-1 and PnBA-4 were respectively prepared from CTAs 1 (Scheme 3) and 4 (Scheme 4) in toluene at 60°C. The polymerization was stopped after 24 h for compound 1 and 10 h for compound 4, yielding PSt-1 with Mn = 7.1 kg/mol and Đ = 1.06 (Table 1, entry 1) and PnBA-4 with Mn = 28.8 kg/mol and Đ = 1.34 (Table 1, entry 3).

Conclusions

In the present contribution, we have described investigations on RAFT agents involving various types of phosphorylated functions in the Z-group, including the free phosphine tBu2P-, analogue, the P atom is reluctant to rehybridize and achieve a planar configuration, as required to engage the lone pair in electronic conjugation with the CS2 unsaturation. Therefore, P V and P III RAFT agents exhibit similar reactivities, precluding the use of a P V -P III chain-end transformation as a switchable RAFT agent system. Despite the degradation of the macro-RAFT agent obtained from 3 during the nBA polymerization, a well-controlled PnBA with a reasonably low dispersity around 1.35 could be obtained by an indirect two-step procedure, which involves nBA polymerization via the corresponding P-BH3 chain transfer agent, followed by a facile deboranation step. All these chain-end-functionalized polymers may be useful for further applications.

and

Experimental Section

General

All reactions were carried out under an argon atmosphere using standard Schlenk techniques. Melting points were determined on the Boetius apparatus and were uncorrected.

Solvents were carefully dried by conventional methods and distilled under argon before use.

Column chromatography was performed on Carlo Erba silica gel 60, 0.035-0.070 mm.

Compounds 1 -3 [START_REF] Karpus | Synthesis of S -Alkyl Phosphinocarbodithioates with Switch between P(III) and P(V) Derivatives[END_REF] and di-tert-butylphosphine borane [START_REF] Frew | Palladium(II) Complexes of New Bulky Bidentate Phosphanes: Active and Highly Regioselective Catalysts for the Hydroxycarbonylation of Styrene[END_REF] were prepared according to literature procedures. Di-tert-butylchlorophosphane (96%, Alfa Aesar), S57).

St and nBA polymerization procedure

A solution containing the monomer (4.10 g for nBA, 3.34 g for St), the RAFT agent (1.1 × 10 -4 mol) and AIBN (1.8 mg, 1.1 × 10 -5 mol) in toluene (1.42 g for nBA, 2.02 g for St) was prepared in a 15 mL tube. Further, the solution was transferred into 6 sealing tubes, degassed by three freeze-pump-thaw cycles and sealed under vacuum. The tubes were heated at 60°C in a thermostated heating block. At the desired time, each polymerization was quenched by freezing in liquid nitrogen and the obtained solution was analyzed by NMR to estimate the monomer conversion. Then the excess amounts of monomer and solvent were removed by evaporation at ambient temperature under vacuum and the residues were analyzed using SEC.

VAc polymerization procedure

The solution that contained the monomer (2.76 g), RAFT agent (1.1 × 10 -4 mol) and AIBN (3.6 mg, 2.2 × 10 -5 mol) in EtOAc (2.76 g) or without solvent was prepared in a 15 mL tube. Further, the solution was transferred into sealing tubes, degassed by three freeze-pump-thaw cycles and sealed under vacuum. The tubes were heated at 70°C in a thermostated heating block for the requisite times. The polymerization was quenched by freezing in liquid nitrogen and the obtained solution was analyzed with NMR to estimate monomer conversion. Then the excess amounts of monomer and solvent were removed by evaporation at ambient temperature under vacuum and the residues were analyzed using SEC.

Procedure for the kinetic measurements

A solution containing the monomer (2.78 g, 0.027 mol), the RAFT agent (8.9 × 10 -5 mol), AIBN (1.5 mg, 8.9 × 10 -5 mol), 1,4-dioxane (0.078 g, 8.9 × 10 -4 mol) and triphenylphosphine sulfide (0.025 g, 8.9 × 10 -5 mol) in C6D6 (1.76 g, 0.021 mol) was prepared in a 15 mL Schlenk tube with a PTFE needle valve and degassed by three freeze-pump-thaw cycles. This solution was then transferred into 8 NMR tubes in the glove box under an argon atmosphere and sealed with rubber septa. The tubes were heated at 60 °C in a thermostated heating block. At the desired time, each tube was quenched by freezing in liquid nitrogen and the obtained solution was analyzed by NMR. Then the excess amounts of monomer and solvent were removed by evaporation at ambient temperature under vacuum and the residues were analyzed using SEC.

Procedure for the preparation of PSt-1, PnBA-4

The solution (5 mL) that contained the monomer (3.33 g, 0.032 mol for St; 4.10 g, 0.032 mol for nBA), RAFT agent (1.1 × 10 -4 mol), AIBN (1.1 × 10 -4 mol), in toluene (1.42 mL for nBA, 2.10 mL for St) was prepared in a 15 mL Schlenk tube with a PTFE needle valve and degassed by three freeze-pump-thaw cycles. Then the solution was heated at 60°C in a thermostated heating block for the requisite times (24 h for PSt-1, 10 h for PnBA-1). The polymerization was quenched by freezing in liquid nitrogen and the obtained solution was analyzed with NMR. Then the excess amounts of monomer and solvent were removed by evaporation at ambient temperature under vacuum and the residues were dissolved in CHCl3 and precipitated by cold CH3OH. Obtained solids (pink PSt-1 and PnBA-4) were dried under vacuum and analyzed using SEC.

Polymer ω-chain ends deboranation

A solution of the desired polymer (0.2 g of PSt-1 or PnBA-4) in THF (10 mL) was prepared in a 15 mL Schlenk tube with a PTFE needle valve and degassed by three freezepump-thaw cycles. Degassed Et3N (0.05 mL) was added dropwise under Ar. The resulting mixtures were stirred for 24 h at ambient temperature. All volatiles were then removed by evaporation at ambient temperature under vacuum. The obtained pink solids (PSt-1 (-BH3) and

PnBA-4 (-BH3)) were dried under vacuum and analyzed using 31 P NMR and SEC.

Computational details

The computational work was carried out with the Gaussian09 suite of programs. [START_REF] Frisch | Gaussian 09 Revision D.01[END_REF] Gasphase geometry optimizations were executed without any symmetry constraint using the M06-2X functional [START_REF] Zhao | The M06 Suite of Density Functionals for Main Group Thermochemistry, Thermochemical Kinetics, Noncovalent Interactions, Excited States, and Transition Elements: Two New Functionals and Systematic Testing of Four M06-Class Functionals and 12 Other Functionals[END_REF] and the 6-31G(d,p) basis set for all atoms. All final geometries were characterized as stationary points by verifying that all second derivatives of the energy were positive. Thermochemical corrections were obtained at 298.15 K on the basis of frequency calculations, using the standard ideal gas, rigid rotor and harmonic oscillator approximations.

A further correction of 1.95 Kcal/mol was applied to bring the G values from the gas phase (1 atm) to the solution (1 mol/L) standard state. [START_REF] Bryantsev | Calculation of Solvation Free Energies of Charged Solutes Using Mixed Cluster/Continuum Models[END_REF]

Scheme 2 .

 2 Scheme 2. A general representation of P-RAFT mediated polymerization of St and nBA in

Figure 2 .

 2 Figure 2. Semi-logarithmic kinetic plot for P-RAFT-mediated polymerization of nBA in

Figure 3 .

 3 Figure 3. Evolution of Mn and Đ vs monomer conversion in P-RAFT-mediated

Figure 4 .

 4 Figure 4. Semi-logarithmic kinetic plot for P-RAFT-mediated polymerization of St in

Figure 5 .

 5 Figure 5. Evolution of Mn and Đ vs monomer conversion in P-RAFT-mediated

  . The intermediate of the degenerate exchange process was simulated by the Me2EC • [SCHMe(O2CMe)]2 radical. For E = N, the calculated free energy of this intermediate is nearly the same as that of the separate Me2EC(S)SCHMe(O2CMe) and (MeCO2)CH • Me fragments, as expected for an efficient degenerative transfer, assuming that the addition and fragmentation barriers are low. The loss of resonance stabilization for the intermediate in the dithiocarbamate system is evidenced by the slight pyramidalization of the N atom (355.24°) and by the loss of its coplanarity with the CS2 plane. For E = P, on the other hand, the intermediate is more stable than the separate fragments by 10.8 kcal mol -1 , suggesting a difficult fragmentation process. The P atom pyramidalization is essentially unchanged in the intermediate adduct (303.32°), confirming the absence of resonance stabilization in the starting P-RAFT CTA.

Figure 7 .

 7 Figure 7. Energy profile (relative ΔG in kcal mol -1 ) and views of the optimized

Figure 8 .

 8 Figure 8. Views of the optimized geometries for (a) Ph2EC(S)SCHMe(O2CMe) and (b)

Figure 9 :

 9 Figure 9: Overlay of the 31 P{ 1 H} NMR spectra of polystyrene samples prior to (PSt-1) and

Figure 10 :

 10 Figure 10: Overlay of 31 P{ 1 H} NMR spectra of poly(n-butyl acrylate) samples prior to

4 and 5 ,

 45 their efficiency in the control of the St and nBA polymerizations. Extension of the already established synthetic methodology for tBu2P(X)-C(S)SR0 with R0 = CHMePh (X = BH3, 1; X = nothing, 3; Figure1) allowed access to two new phosphinoylcarbodithioate RAFT agents, with R0 = CH(CH3)C(O)OCH3. The combination of 1 H and 13 P NMR was crucial to monitor the polymerization and to provide direct information on the reactivity of the CTA. Raft agents 1 and 4 with a phosphorus atom capped with a BH3 moiety provide excellent control with nearly perfect chain-end fidelity for the polymerization of St. For nBA polymerization the control is efficient in terms of overall kinetics, control of macromolecular characteristics, and chain-end fidelity. Compounds 3 and 5 are the first examples of P-RAFT agents bearing a free P III Z-group. CTA 3 can be directly used in RAFT polymerization to prepare well-controlled polystyrenes with a low dispersity of 1.09 -1.15 with free phosphine ω-chain ends as potential macromolecular ligands. CTAs 3 and 5 do not provide a controlled polymerization of VAc but rather complete inhibition of polymerization because, unlike the N III (dithiocarbamate)

Table 1 .

 1 Macromolecular characteristics of polymers prior to and after deboranation of the ω-chain ends.

	Entry	Compound	Mn,SEC, kg/mol a	Đ b
	1	PSt-1	7.1	1.06

volatile products, gave pure polystyrene samples PSt-1 (-BH3) with Mn = 7.1 kg/mol and Đ = 1.08 (Table

1

, entry 2) and PnBA-4 (-BH3) with Mn = 28.4 kg/mol and Đ = 1.35 (Table

1

, entry 4).

Acknowledgements

We thank the Agence Nationale de la Recherche for support of this work through grant ANR-16-CE29-0014-RAFTSWITCH. Additional support from the CNRS (Centre National de la Recherche Scientifique) is also gratefully acknowledged. We also thank the CALMIP mesocenter of the University of Toulouse for the allocation of computational resources. The authors thank Dr. Ihor Kulai for fruitful discussions.

Associated content:

The Supporting Information is available free of charge on the ACS Publications website at DOI: . Tables S1-S3 and Figures S1-S32 ( 1 H and 31 P NMR, SEC-RI chromatograms).

Synthesis of 4:

A solution of di-tert-butylphosphine borane, tBu2PH•BH3 (1.00 g, 6.24 mmol) in THF (100 mL) under Ar atmosphere was treated with a slight excess of n-butyllithium (4.1 mL, 1.6 M in hexane) at -40°C. The resulting bright yellow mixture was allowed to warm up to 0°C and stirred for 1 h. Then a large excess of CS2 (2.2 mL, 33.6 mmol) was added dropwise at 0°C to yield a dark red solution. The reaction vessel was covered with aluminum foil and was allowed to warm up to 12°C and stirred for 15 min. Methyl 2-bromopropionate (1.10 mL, 9.87 mmol) was then added, and the dark red solution was stirred for an additional 4 h at 12-13°C. After removal of the solvent under reduced pressure, the residual product was subjected to column chromatography (silica gel, hexane/toluene (1:1 v/v)). Removal of the solvent resulted in 1.025 g (51% yield) of the product as a pink amorphous solid. S54).

MP

Synthesis of 5:

A solution of the carbodithioate 7 (0.474 g, 1.47 mmol) in toluene (30 mL), under Ar atmosphere in a Schlenk tube covered with aluminum foil, was treated with DABCO (0.357 g, 2.92 mmol). The reaction mixture was stirred for 6 h at 60°C. After removal of the solvent under reduced pressure, the residual product was subjected to column


Yue-Cai Huang

Jie -5607-885x]

Zhang

Siyuan Yu
email: s.yu@bristol.ac.uk

Self-Learning Routing for Optical Networks

Keywords: Optical networks, Routing, Self-learning

It is generally very difficult to optimize the routing policies in optical networks with dynamic traffic. Most widely-used routing policies, e.g., shortest path routing and least congested path (LCP) routing, are heuristic policies. Although the LCP is often regarded as the bestperforming adaptive routing policy, we are often eager to know whether there exist better routing policies that surpass these heuristics in performance. In this paper, we propose a framework of reinforcement learning (RL) based routing scheme, that learns routing decisions during the interactions with the environment. With a proposed self-learning method, the RL agent can improve its routing policy continuously. Simulations on a ring-topology metro optical network demonstrate that, the proposed scheme outperforms the LCP routing policy.

Introduction

Routing of optical networks has been under research for more than two decades [START_REF] Chatterjee | Routing and spectrum allocation in elastic optical networks: A tutorial[END_REF][START_REF] Klinkowski | Survey of resource allocation schemes and algorithms in spectrally-spatially flexible optical networking[END_REF][START_REF] Zang | A review of routing and wavelength assignment approaches for wavelength-routed optical WDM networks[END_REF]. There are basically three approaches: fixed routing, fixed-alternative routing [START_REF] Harai | Performance of alternate routing methods in all-optical switching networks[END_REF][START_REF] Ramamurthy | Fixed-alternate routing and wavelength conversion in wavelength-routed optical networks[END_REF], and adaptive routing [START_REF] Chan | Analysis of least congested path routing in WDM lightwave networks[END_REF][START_REF] Li | Dynamic wavelength routing using congestion and neighborhood information[END_REF]. For the fixed routing, one predetermined route is always chosen for a given source-destination pair. If the route is unavailable, the call request is blocked. For the fixed-alternative routing, there is an ordered list of fixed candidate routes for each source-destination pair. Upon call request, each route is tried following the order until a route is found available; if no route is available, the request is blocked. For the adaptive routing, the path is chosen dynamically depending on the network status upon the request arrival. Among all routing policies, adaptive routing generally gives the best performance [START_REF] Zang | A review of routing and wavelength assignment approaches for wavelength-routed optical WDM networks[END_REF].

Most well-known adaptive routing policies are heuristic policies. For example, the adaptive shortest-cost-path routing tries to minimize the resources used for each connection upon its arrival, aiming to hold more simultaneous connections.

The least congest path (LCP) routing balances the load over the network, so as to avoid bottlenecks. Among the adaptive routing policies, the LCP routing is generally regarded to have the best performance [START_REF] Li | Dynamic wavelength routing using congestion and neighborhood information[END_REF]. These heuristic adaptive routing policies can be regarded as to capture some features of the network status and then intuitively exploit the features to achieve good performance. For instance, the LCP policy considers the feature of load on each available path and select the least congested path. As known, besides the load, there are many other features that may affect the network performance, e.g., the number of hops, the link availability, the traffic in service, the topology, etc. From this perspective, A good adaptive policy needs to include enough features and properly exploit them, while on the other hand, in a realistic situation, the variations of network status are often too many to be summarized. These variations come from the change of spectrum availability of each link, the change of remaining holding time of each call, etc. Even if we only consider a simplified version of network status with only the spectrum availability and source-destination pairs, a huge number of network status still exit. Consider the following example. For a five-node bi-directional ring-topology network, with five wavelengths on each link, and only four sourcedestination pairs (1-2, 2-1, 1-3, 3-1), there are over four billion simplified network status. Therefore, making good adaptive routing policies is challenging. In our opinion, that is a major reason that heuristic algorithms dominate the dynamic routing policies over the past decades.

Recently, Google successfully applied reinforcement learning in playing Atari games and achieved above-human-level performance [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF]. The RL-based agent can derive efficient representations of the environment, and use these to generalize past experience to new situations. This gives us a possible way to overcome the difficulty of feature extraction for adaptive routing. In [START_REF] Alyatama | Dynamic routing and wavelength assignment using learning automata technique [all optical networks[END_REF][START_REF] Kiran | A reinforcement learning framework for path selection and wavelength selection in optical burst switched networks[END_REF][START_REF] Koyanagi | A reinforcement learning-based lightpath establishment for service differentiation in all-optical WDM networks[END_REF][START_REF] Pointurier | Reinforcement learning based routing in all-optical networks[END_REF], the authors apply reinforcement learning to the routing of optical networks. These methods allow the RL-based agent to learn a routing policy during the interaction with the network environment. In a more recent work [START_REF] Chen | Deep-RMSA: A deepreinforcement-learning routing, modulation and spectrum assignment agent for elastic optical networks[END_REF], a Deep Q-Network (DQN) algorithm is used to capture the features inspired by [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] and obtain some performance improvement compared to the shortest path routing policy. While we still have the following open question: whether there are better routing policies than the existing ones, and how can we find the best routing policies. This paper tries to answer these questions.

In this paper, we propose a RL-based self-learning method. It is based on the following observation: suppose we already have one routing policy, if we can change this policy with just one better action under one specific circumstance, we should have a better policy. With our proposed method, the RL agent changes its current policy according to the competition result of the current policy and a reference policy. In this way, the RL agent should be able to learn for a policy that is no worse than the reference policy. Then, the reference policy is periodically updated with the learnt policy. By repeating the above learning process iteratively, the RL agent can improve its policy continuously.

In this section, we first model the optical network routing problem into an RL problem with self-learning (Subsection 2.1 and 2.2). Then, we introduce how to apply DQN algorithm to the routing problem in Subsection 2.3. Due to space limitation, the basic knowledge of RL will not be covered in this paper, and readers are strongly recommended to refer [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] for better understanding of this paper.

Mathematical representation of state and action

We consider the RL state only at the time when a new connection request arrives. That means at time t, t+1, t+2, . . ., there is one and only one connection request arriving, and there is no connection request arrivals in the time periods between.

The

S t net =       b 11 (t), • • • , b 1w (t), • • • , b 1W (t) • • • b i1 (t), • • • , b lw (t), • • • , b lW (t) • • • b L1 (t), • • • , b Lw (t), • • • , b LW (t)       , (1)
where,

b lw (t) =          1, if wavelength w of link l is available at time t -1, if wavelength w of link l is unavailable at time t. (2)
From Eq. ( 1) and Eq. ( 2), network state S t net is an L × W matrix, with element b lw (t) denoting the availability of wavelength w on link l. W is the total number of wavelengths and L is the total number of links.

The arrival traffic state at time t (i.e., upon new traffic arrival) is S t tra , which is defined as Eq. ( 3). K is the total number of paths for the newly arrived traffic under study.

S t tra =       b 11 (t), • • • , b 1k (t), • • • , b 1K (t) • • • b i1 (t), • • • , b lk (t), • • • , b lK (t) • • • b L1 (t), • • • , b Lk (t), • • • , b LK (t)       , (3)
where,

b lk (t) = 1, if link l is not included by path k -1, if link l is included by path k. (4)
S t net is an L × W matrix, and S t tra is an L × K matrix. S t is to connect the two matrix along the rows, and therefore an L × (W + K) matrix.

The action of the routing problem is to choose one path from the K paths under study. Therefore, action A t ∈ {1, 2, . . . , K}.

Reward

The choice of reward is essential for RL, since the objective of RL is to maximize the expected cumulative reward. Well-designed reward setting should be in consistent with the objective of the routing, i.e., to minimize the call blocking probabilities. As for the routing problem, the evaluation of a routing policy is not determined by one routing decision, but by a sequence of routing decisions. Therefore, the reward should reflect the performance of an action within a sequence. Although we have feed-back from future rewards by discounting, this kind of reward cannot fit well with the objective to minimize the blocking probabilities. In this paper, the reward is set based on the concept of self-learning, which is inspired by the method of "self-play" used by Alpha Zero [START_REF] Silver | Mastering the game of go without human knowledge[END_REF]. The reward is set by a comparison of the network running a learning routing policy with a reference network running a reference routing policy, as illustrated in Fig. 1. It includes three parts: the main network, the reference network and the comparator.

The main network is the optical network adopting the learning routing policies, with -greedy approach used to balance the exploration and exploita-tion. When an arrival is blocked, the network state is recorded and the traffic till the next block is also recorded. The reference network is a virtual optical network for comparison. It is configured following the recorded network state from the main network, and the recorded traffic is injected to this network. A reference routing policy is adopted. The performance of the saved network can be evaluated by simulations from the initial network state and traffic injected with the given policy. The comparator compares the two networks with its two components: sequence comparator and action comparator:

1. Sequence comparator compares the sequence of actions, by considering which sequence incurs a blocking state earlier. There are three kinds of comparison outputs: "better", "same", and "worse". If the main network incurs blocking earlier than the reference network, we regard the sequence of actions from the learning routing policy performs worse than the reference, then the comparator outputs "worse". Similarly, if the main network incurs blocking at the same time with the reference network, the comparator outputs "same". Finally, if the main network incurs blocking later than the reference network, the comparator outputs "better". 2. Action comparator compares each action in the sequence. The sequence comparator gives a general impression of the performance of the actions within a sequence, while not every action is better than (worse than / same with) the actions in the reference policy. Therefore, we should compare them one by one. The action comparator considers each stateaction pairs from the reference network. It compares the same state but with action from the learning policy. It outputs two results: same action or different action.

Finally, based on the comparison results, the reward can be set, as given by Table 1. An intuitive explanation on this kind of setting is as follow. (1) If action comparator outputs "same" or the sequence comparator outputs "same", the actions of the main network and that of the reference network tend to have little difference in affecting the performance, therefore, a relatively small and positive reward 0.1 is given. (2) If actions of the two networks are different, and the sequence in the main network performs better, the action of the main network is potentially a key action to lead to a better sequence, therefore, a relatively big and positive reward 1.0 is given. (3) If actions of the two networks are different, and the sequence in the main network performs worse, this action is potentially a key action to lead to a worse sequence, therefore, a relatively big but negative reward -1.0 is given. (4) If blocking occurs, give a very big negative reward -10. Since the rewards are set by comparisons, policies better than the reference policy can be obtained, when maximizing the cumulative reward.

Another question is how to choose the reference policy. It can be a wellknown policy, e.g., shorted path routing or least congest path routing, or it can be the learning policy as well. When the reference policy is some known policy, the learner is learning a policy no worse than the known policy. If the reference policy is the learning policy itself, then it becomes "self-learning". In this case, the reference policy is the learnt policy with greedy action, and in the main network, the -greedy policy can be used to explore the state space. After some time, the reference policy can be updated with the newly learnt policy. In this way, the RL-agent improves itself continuously.

Learning with Deep Q-Network

Due to the curse of dimensionality, we use a neural network (NN) shown by Fig. 2 as the Q-value function approximator. The input of the neural network is the state at each time step, S t , and the output is the predicted action-value (Q-value) for each individual action for the input state. In this way, the Q-values of all actions is computed with a single forward pass at the same time for a given state. The neural network with weights Θ is used as the Q-network. Denote Q(s, a; Θ) as the parameterized Q-value for state s and action a. Denote Q(s , a ; Θ) as the parameterized Q-value for the next state s and the next action a . The NN can be trained by adjusting Θ i at iteration i to reduce the mean-squared error in the Bellman equation, where the optimal target value is given by: Target: y = r + γ max a Q(s , a ; Θ), r is the instantaneous reward, γ is the discount factor , γ < 1.

(5) Then, the loss function at each iteration i is given by the mean-squared error of between the Q-network and the Q-learning targets,

L i (Θ i ) = E s,a,r,s (y i -Q(s, a; Θ i )) 2 = E s,a,r,s r + γ max a Q(s , a ; Θ - i ) -Q(s, a; Θ i ) 2 . (6)
A gradient descent can be applied to minimize the loss function. Besides, experience replay [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF] is adopted, to randomly sample previous transitions, so as to smooth the training sample distribution over past behaviors. Moreover, several techniques are also used to facilitate the learning, including mini batch gradient descent, dropout in NN, and Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. The DQN algorithm is shown by Algorithm 1. The experience replay technique is shown. For simplicity, the reward setting part is ignored. To get a full picture, Algorithm 1 and Fig. 1 need to be combined.

In the Algorithm 1, two steps are taken recursively: the sampling and the training. During the sampling, samples of transitions are stored to the replay memory. During the training, samples are randomly chosen from the replay memory to train the neural network. Since the samples are chosen with reward given by self-comparing, the samples are generally better than the current routing policy. Therefore, the routing policies becomes better and better. The learnt policy is embedded in the trained Q-networks, arg max a Q(s, a; Θ). The policy in the reference network is updated periodically.

Simulations

A discrete event simulator written in Python is used to simulate the traffic and the resource provision of the optical network. The DQN reinforcement learning part is written based on TensorFlow [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF].

Parameter settings

Consider a five-node bidirectional-ring-topology metro network. The key notations are listed in Table 2 for convenience, and they are also explained in the context. The total number of wavelength is five. The traffic pattern is non-uniform with traffic matrix given as follows:

T = λ      
0, 1, 1, 0, 0 1, 0, 0, 0, 0 1, 0, 0, 0, 0 0, 0, 0, 0, 0 0, 0, 0, 0, 0

      . (7)
This means, traffic only exists for source-destination pair (1, 2), (

. The arrival processes are independent Poisson processes, with arrival rate λ, and the service time distributions follow exponential distributions with average service time µ. In this bidirectional ring topology, routing can be done clockwise or anti-clockwise. Therefore, for each accepted request, only two actions can be chosen: routing clockwise or anti-clockwise. For wavelength assignment, the first-fit strategy is used. A five-layer forward feed fully-connected neural network is chosen as the function approximator for the state-action value function. The neural network is illustrated in Fig. 2. Recall that the state of the RL algorithm is a L × (W + K) matrix. With total number of link L = 10, total number of wavelength W = 5, and total number of paths K = 2, the state is a 10 × 7 matrix. This matrix is reshaped to a 70 × 1 array and forms the input layer of the neural network. Then it is followed by three hidden layers with number of nodes H 1 , H 2 , H 3 . The ReLU activation function is chosen for all hidden layers. Besides, not shown in Fig. 2, the dropout technique is introduced in the neural network to avoid overfitting, with the dropout probability 0.5. Besides, the discount rate γ is 0.99, the explore rate is 0.1, and the learning rate α is 1 × 10 -5 .

Learning for a reference policy

With the aim of outperforming the existing routing policies, we should check first whether the RL-routing can learn to be a given policy, i.e., policy-fitting. Therefore, we run a set of simulations to fit the RL-routing to be the bestperformed LCP routing. We compare the actions given by the Q-network and those given by the LCP policy. If the actions are the same, then give reward 1, otherwise, give reward -1. The simulation results are shown in Fig. 3 and Fig. 4. In the two figures, the x-axis is the number of arrivals batches (1 arrival batch includes 1000 arrivals), i.e., the results are collected every arrival batches (every 1000 arrivals). Figure 3 shows the blocking probabilities of LCP and three RL-based policies with different number of nodes in the hidden layer of the Qnetworks. The accuracy shown by Fig. 4 representing the portion that the actions from RL routing are the same with those from LCP policy. From Fig. 3, we can see that the blocking probabilities become similar with LCP soon (after 2000 bathes) for all Q-networks. While from Fig. 4, it is revealed that only the case with number of hidden layer nodes 1024 fits the LCP after 4000 batches. We can make two points here: (1) To fit even a simple policy such as LCP, we need a large enough neural network. (2) There are many different policies that give similar performance with the LCP.

Self learning

Taking the RL-routing policy that is already fit to the LCP, we apply the selflearning process to pursue better policies. The simulation results are shown in Fig. 5. It is demonstrated by Fig. 5 that the RL-routing, incurs some degradation at the beginning of the training, gradually it outperforms LCP and maintains the advantage over LCP during the training. This demonstrate that, by the selflearning method, indeed better routing policies can be learnt. We should also notice that, the performance of the learnt policies incurs some variations. These variations may come from the exploration and the variations of the reference policy to be compared.

Discussion

This is an early work for applying reinforcement learning to the optical network resource allocation, there are some limitations that need to be investigated for future research. One challenge is the scalability. For the simulated 5-node-5wavelength ring metro network, a five layer fully-connected neural network with 1024 nodes per hidden layer is already required. Some efforts need to be paid to reduce the complexity of the neural networks as the size of the optical network scales up. One potential approach is to use convolutional neural networks, leveraging the locality of the network status. There are also plenty of open problems for future research, including the impact of traffic variations and network topologies, and the spatial/spectral resource assignment.

Conclusion

This paper provides a new direction to optimize the routing of optical networks. The proposed reinforcement learning algorithm is able to continuously improve its routing policy with self learning method. Simulation on a ring-topology metro network demonstrate that the learnt policy outperforms the least congest path routing policy which was regarded the best routing policy. With the computation power increasing rapidly, this RL method can be a scalable approach for dynamic resource allocation and control for optical networks. It can also be combined with software defined network to achieve an agile control plane.

Fig. 1 .

 1 Fig. 1. The illustration of self learning.

Fig. 2 .

 2 Fig. 2. The NN for Q-value approximation (dropout ignored).

Algorithm 1 : 6 while not policy update criterion do 7 #SAMPLING 8 while not sampling stopping criterion do 9 17 while24

 1678917 Reinforcement learning algorithm 1 #INITIALIZATION 2 Create and initialize the optical network 3 Create and initialize the neural network 4 Initialize replay memory D 5 while not stopping criterion do Upon traffic coming, store state to St 10 With probability select a random action At 11 Otherwise select At = arg max A Q(St, A; Θ) 12 Execute action At and wait until new traffic coming, get new state St+1 13 Get reward rt+1 14 Store transition (St, At, rt+1, St+1) in D not training stopping criterion do 18 Sample a random minibatch of transitions (s, a, r, s ) from D 19 Set y = r + γ max a Q(s , a ; Θ) 20 Perform a gradient step on (y -Q(s, a; Θ)) 2 with respect to the neural network parameters Θ Update policy in the reference network 25 end while 26 Policy is arg max a Q(s, a; Θ)

  node per hidden layer = 32 RL, #node per hidden layer = 256 RL, #node per hidden layer = 1024

Fig. 3 .

 3 Fig. 3. Performance of RL-routing with LCP fitting.

Fig. 4 .

 4 Fig. 4. The accuracy of fitting LCP by RL routing.

Fig. 5 .

 5 Fig. 5. Performance of RL-routing with self-learning.

  state at time t, denoted by S t is composed of two parts: the network state S t net and the arrival traffic state S t tra . S t = S t net , S t tra .

	The network state S t	net , representing the resource occupation state at time
	t, is defined as,

Table 1 .

 1 The setting of reward

		sequence comparison comparison action	reward
		better than ref	= =	1.0 0.1
	accept	same with ref	= =	0.1 0.1
		worse than ref	= =	0.1 -1.0
	block	NA	NA	-10.0
	= means same actions, = means different actions.

Table 2 .

 2 Key parameters

		Notation	meaning	value
		N	number of nodes	5
	optical	L	number of links	L = 2N = 10
	network	W	number of wavelengths	5
		K	number of routing paths	2
	Traffic	λ µ	arrival rate average service time	0.4 1.0
		γ	discount rate	0.99
		α	learning rate	1 × 10 -5
	RL		explore rate	0.1
	agent	B	number of hidden layers of the Q-network	3
		H	number of neurons for each hidden layer {32, 256, 1024}
			same number for each hidden layer
		P	dropout probability for NN	0.5

Acknowledgment

This work is supported by: National Natural Science Foundation of China (61490715, U1701661); Local Innovative and Research Teams Project of Guangdong Pearl River Talents Program (2017BT01X121); Fundamental Research Funds for the Central Universities of China (SYSU: 76120-31610003).