Elena Esther

Romero Fandino

Christophe Bérenguer
email: christophe.berenguer@grenoble-inp.fr

John Jairo

Martinez Molina
email: john-jairo.martinez-molina@grenoble-inp.fr

Gain-scheduling wind-turbine control to mitigate the effects of weather conditions on the drive-train degradation

Keywords: Wind Turbine Control, Gain-Scheduling control, Degradation Management, Wind Conditions

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Gain-scheduling wind-turbine control to mitigate the effects of weather conditions on the drive-train degradation Elena E. Romero * Christophe Bérenguer * John J. Martinez *

INTRODUCTION

The development of renewable energy technologies has been strongly encouraged with the progression of the environmental crisis. Thus, wind power generation plays a central role, covering 13% of the electricity demand in Europe with 220 GW and 8.44% in the United States with 123 MW. Moreover, it should continue to grow to cover 25% of Europe's electricity demand in 2030. Therefore, optimizing wind turbines operation is of high interest to the industry. (American Clean Power Association (2021); [START_REF] Komusanac | Wind energy in Europe 2020 statistics and the outlook for 2021-2025[END_REF]; [START_REF] Lee | GWEC -global wind report[END_REF]).

However, different factors affect the profitability of this technology. The random nature of wind speed conditions represents a challenge in optimizing the lifetime of wind turbines. In particular, if we consider that high wind speed variations are one of the principal factors of increasing degradation rate. [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF]; [START_REF] Ma | Flexible wind speed generation model: Markov chain with an embedded diffusion process[END_REF]; [START_REF] Bianchi | Wind Turbine Control Systems[END_REF]. The uncontrolled weather conditions (e.g., wind turbulence intensity) can deteriorate the mechanical transmission in a wind turbine. This could be due to persistent variations in the radial and angular shaft deflections when the system is submitted to wind speed with high variances. Besides, the shaft deterioration increases the maintenance and energy costs. [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF]; [START_REF] Bianchi | Wind Turbine Control Systems[END_REF]; [START_REF] Ma | Flexible wind speed generation model: Markov chain with an embedded diffusion process[END_REF]).

⋆ Elena E. Romero, John J. Martinez and Christophe Bérenguer are with the Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab, 38000 Grenoble

Motivated by the benefits of the improvement of this technology, numerous control approaches to reduce the loads by the wind variation in the wind turbine have been proposed in the literature [START_REF] Li | Fatigue reliability analysis of wind turbine drivetrain considering strength degradation and load sharing using survival signature and FTA[END_REF]; [START_REF] Pan | Performance degradation assessment of wind turbine gearbox based on maximum mean discrepancy and multi-sensor transfer learning[END_REF]). Nevertheless, most of the scientific efforts are focused on the deterioration of individual components of the turbine [START_REF] Sanchez | Healthaware model predictive control of wind turbines using stiffness degradation approach[END_REF]; [START_REF] Castro | Effect of tunneling cracks on structural property degradation of wind turbine blades[END_REF]). However, changes in the wind conditions and their interactions with the control system are not yet considered to analyze the drive-train deterioration.

Thus, considering the effect of the random nature of wind speed represents a challenge in developing future control strategies. Besides, it is expected that wind turbines are operated under a not well-adapted control strategy during significant periods, affecting the efficiency of the turbine [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF]).

This paper aims to present a gain-scheduling wind-turbine control strategy that can mitigate the effects of adverse weather conditions by choosing a suitable feedback control gain depending on the wind turbulence conditions. The approach allows establishing a trade-off between generated and dissipated energy to maximize the efficiency of a wind turbine by taking into account the actual wind conditions. The drive-train is simulated as a flexible shaft by considering a nonlinear mass-spring-damper model. Furthermore, the dissipated energy in the mechanical transmission is used to model the degradation of the shaft. Finally, the performance of the proposed strategy is illustrated through several numerical experiments by considering different wind turbulence scenarios.

MODELLING OF DRIVE-TRAIN DEGRADATION

Wind Turbine Dynamical Model

The representation of the drive-train in a variable speedfixed pitch (VS-FP) wind turbine can be made in a simplified model that considers the flexible transmission that connects two rigid bodies as presented in [START_REF] Bianchi | Wind Turbine Control Systems[END_REF]. Considering the dynamic of the drive-train system: the shaft is deformed with an angle θ s due to the difference between the generator speed ω g and the rotor angular speed ω r . The representation of two rigid bodies allows us to embrace the parts and mechanical devices on each side of the shaft, including terms such as the rotor inertia I r , generator inertia I g , the damping of the transmission B s and the stiffness of the transmission K s .

In this paper, the description of the wind turbine will be limited to the dynamics of the drive-train, reducing the dynamical equations as those presented in [START_REF] Bianchi | Wind Turbine Control Systems[END_REF] in a first-order LPV system. That is:

  θs ωr ωg   = 0 1 -1 -Ks Ir -Bs Ir Bs Ir Ks Ig Bs Ig -Bs Ig θs ωr ωg + 0 0 1 Ir 0 0 -1 Ig ( τr τc ) (1)
where τ r is the aerodynamic torque, which captures the torque produced from the wind observed in the mechanical shaft. τ r can be calculated by considering the wind speed (V ), the air density (ρ), the rotor radius (R), as follows:

τ r = 1 2 ρπR 3 C p (λ, β) λ V 2 (2)
The parameter C p (λ, β) corresponds to the power coefficient, which allows knowing a wind turbine's ability to capture the useful wind power. In general, C p is modeled using numerical approximations, taking into account the pitch angle β and the tip speed ratio λ. See for instance [START_REF] Dai | Research on power coefficient of wind turbines based on SCADA data[END_REF][START_REF] Saint-Drenan | A parametric model for wind turbine power curves incorporating environmental conditions[END_REF]).

The tip speed ratio λ is defined in term of ω r , R and V . In addition, it is possible to calculate a maximum power coefficient C pmax that concerns with the maximum conversion efficiency, achieved at the optimal operation point λ = λ 0 (the optimal λ value) and optimal pith angle (e.g. β ≈ 0).

The variable-speed wind turbines can work with maximum efficiency over a wide wind speed range of rated power.

The benefits of the operation in the VS-FP turbines can be achieved if ω r is modified in proportion to the wind speed to maintain an optimum tip-speed ratio [START_REF] Bianchi | Wind Turbine Control Systems[END_REF]). For control purposes, the generator torque control τ c can be considered as a function of the rotor speed, (see for instance [START_REF] Johnson | Control of variable-speed wind turbines: Standard and adaptive techniques for maximizing energy capture[END_REF]), as :

τ c = K opt c (ω r ) 2
(3) where K opt c is the theoretical optimal feedback control gain calculated as a function of C pmax , λ 0 , and the rotor swept area, denoted A, as follows:

K opt c = 1 2 AR 3 C pmax λ 0 3 (4)
and, the generated energy during the time t will be:

E g = t 0 P g dt = t 0 τ c ω r dt (5)

Degradation Model

During the wind turbine operation, the drive-train is one system that suffers the impact of changing loads that can increment its failure rate. It is commonly accepted that torsional vibrations, bearing friction and impacts, or radial and angular torque fluctuations contribute to the degradation of the drive-train and, consequently, they reduce the useful life of the wind turbine [START_REF] Rahimi | Drive train dynamics assessment and speed controller design in variable speed wind turbines[END_REF], [START_REF] Bianchi | Wind Turbine Control Systems[END_REF], [START_REF] Cetrini | On-line fatigue alleviation for wind turbines by a robust control approach[END_REF]).

Here we consider a degradation model of the drive-train presented in [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF]. The model is based on contact mechanics principles, and it has been used to simulate degradation in the transmission by using dissipated energy E d as an indicator.

The simplified model of a drive-train shown in Fig. 1 was used, considering that the two rigid bodies represent the rotor components (low-speed mass) and the generator components (high-speed mass) connected by a flexible shaft modeled as a spring and damper. Thus, based on contact mechanics, the damping coefficient, denoted B s , can be modeled as a nonlinear function of the angular deformation and can be written as a function of the torsion angle θ s as follows:

B s (θ s ) = 3 2 θ s αK s (6)
where K s is the constant stiffness of the transmission, and α is a constant parameter that depends on the material.

For computing the damping torque, it is necessary to consider the rotational speed difference between both rotational devices. It can be obtained as:

τ d = B s (θ s )(ω g -ω r ) (7)
Therefore, the dissipated power due to angular deformations will be:

P d = τ d (ω g -ω r ) (8)
Taken together, ( 7) and ( 8) show that the power dissipated by the drive-train is a function of the angular shaft torsion and the square of the relative velocity. The amount of dissipated energy is then: Fig. 2. Scheme of the off-line optimization process for designing feedback control gains.

E d = t 0 P d dt = t 0 B s (θ s )(ω g -ω r ) 2 dt (9)

PROPOSED GAIN-SCHEDULING CONTROL STRATEGY

In a wind turbine, the control gain of the generator is usually adjusted without considering the changes in the wind flow conditions. However, in [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF], it has been shown that when the control system ignores wind conditions, the efficiency and the degradation of the turbine are significantly affected. As stated in [START_REF] Romero | Degradation of a wind-turbine drive-train under turbulent conditions: effect of the control law[END_REF], the four following observations can be made:

• The dissipated energy increases when the wind conditions are of high turbulence ; • In addition, the chosen control gain impacts the degradation (which is considered here to be proportional to the dissipated energy E d ), for instance:

E d = Increases when K c > K opt c Decreases when K c < K opt c
where K c stands for the chosen control gain and K opt c is the theoretical optimal feedback control gain ; • A control gain chosen higher than the optimal one leads to a greater energy production than a control gain smaller than the optimal one, i.e consider two possible control gain choices K a c and K b c , around the optimal one, such that

K a c < K opt c < K b c , (10
) then, choosing K b c leads to a higher energy production than choosing K a c ; • However, even if the choice of a higher gain K b c leads to a higher energy production, in the case of high turbulent flow, this increase in the energy production is not that significant. Considering these observations, a trade-off can be found between the dissipated and the generated energy by a proper choice of K c , and the problem is thus to design a control strategy which allows the selection of a suitable control gain according to the wind conditions. Here, we will assume that those wind conditions are identified, online, by a given algorithm that is outside the scope of this paper.

This work proposes a gain-scheduling wind-turbine control strategy to manage the degradation and to optimize the lifetime of the mechanical transmission components while maintaining an acceptable efficiency of a wind turbine under varying weather conditions. In the next section, the proposed control architecture will be presented.

Gain-Scheduling Control Architecture

The proposed control architecture is shown in Fig. 1. That architecture uses the wind speed information for identifying the wind speed conditions w c (laminar or turbulent flow, for instance), as follows:

w c = 1 if v corresponds to turbulent flow 0 if v corresponds to laminar flow
the index w c can be considered as a gain-scheduling variable. The feedback gains K c , stored into a previously designed look-up table, will be selected according to the wind conditions w c as it is illustrated in Fig. 3. Thus, the control torque τ c is then obtained by using (3).

Remark that the dynamics (1), the varying parameter (6) and the control law (3), can be rewritten as a Linear Parametric Varying (LPV) control system. Then, stability guarantees of the proposed control architecture can be obtained by using today available LPV and polytopic analysis tools, see for instance [START_REF] Apkarian | Selfscheduled H ∞ control of linear parameter-varying systems: a design example[END_REF].

The values of the control gains K c can be obtained off-line by solving an optimization problem as it is explained in the next section.

Optimization for gain-scheduling control design

The interest of adopting a gain-scheduling control approach is the possibility to select suitable control gains K s c for each wind condition, which allows to reach a tradeoff between dissipated and generated energy. Hence, those gains can be obtained as the solution of an optimisation problem.

For each case (laminar or turbulent, for instance), the value of the gain is obtained as the one that minimizes the ratio of the dissipated energy E d over the generated energy E g under the considered wind conditions:

K * c = arg min Kc E d (K c , w r ) E g (K c , w r ) (11)
All the gains obtained for each possible situation are stored into a look-up table. The control system will recover these gains, online, to implement the control loop.

Simulation Setting

For VS-FP turbines, a reduced model is regularly used, where the drive-train system is represented by two rigid masses connected by a flexible shaft. In this work, we simulate a VS-FP turbine with a horizontal-axis and fixed gearbox. The rated power of the turbine is 2 MW and 100 m rotor diameter.

For the case of the chosen turbine, the power coefficient curve, C p versus λ : the value of C pmax is 0.4615 at λ 0 equal to 6.4. Therefore, using (4), the theoretical optimal feedback control gain, will be K opt c = 9.5065e5.

Wind Speed Generation

Uncontrolled weather conditions such as the wind speed increase the deterioration of the drive-train in a wind turbine. It is thus necessary to obtain a model to reproduce wind speed under different flow conditions (laminar and turbulent), in order to simulate the wind turbine degradation.

For the laminar wind speed case, the real measurement of the wind speed over a period of approximately 7 hours (27000s) was used to feed the model (Fig. 4a). In the turbulent case, different types of models exist to predict wind speed [START_REF] Hanifi | A critical review of wind power forecasting methods-past, present and future[END_REF]; [START_REF] Manwell | Wind Energy Explained[END_REF]; [START_REF] Liu | A new hybrid ensemble deep reinforcement learning model for wind speed short term forecasting[END_REF]). However, the approach proposed in [START_REF] Ma | Flexible wind speed generation model: Markov chain with an embedded diffusion process[END_REF] allows generating different classes of wind speed by using a stochastic equation ( 12). In this work, this latter model was used to simulate a data set of turbulent wind speed:

dV (t) = -â(V (t) -û)dt + b dW (t) (12)
where the terms a(V (t), t) and b(V (t), t) are the drift and diffusion terms, and dW is a continuous process whose increments are normally distributed, i.e. a standard Wiener process.

The parameters â, b, and û were chosen from real wind speed records for periods as presented in [START_REF] Ma | Flexible wind speed generation model: Markov chain with an embedded diffusion process[END_REF]. Thus, the equation for reproducing turbulent wind speed is:

dV (t) = -0.0314(V (t) -5.5135)dt + 0.2517 dW (t) (13)
For the turbulent wind speed, the simulation was performed during the same period length as in the laminar case (27000s) and is presented in Fig. 4b.

However, this works aims to evaluate how an suitable control gain can mitigate the effects of adverse weather conditions. For this purpose, a third data set was reproduced for 12 hours (t = 43200s), with periods in laminar flow conditions and others in turbulence, organized randomly, during a minimum of 1 hour. In Fig. 5 the alternation between laminar and turbulent periods are represented with lines to identify each flow regimen.

Gain-scheduling Control Strategy Setting

In this work, we considered two scenarios of wind speed conditions (laminar and turbulent) to validate the proposed Gain-scheduling control strategy. Each case was evaluated, separately, in order to obtain two possible suitable control gains, as is show below:

K s c = K turbulent c optimal K c for turbulent flow K laminar c optimal K c for laminar flow
The values of K laminar c and K turbulent c are solution of an off-line optimisation problem as it is illustrated in Fig. 2. As result the optimal control gains are:

• K laminar c
which can take values until 11% above K opt c . This gain is optimal for laminar wind conditions. which can take values until 15% below K opt c . This gain is optimal for turbulent wind conditions.

Besides, we consider that the wind conditions are identified and the change of flow regimen is made every hour (3600 seconds) as is indicated in Fig. 5.

For validation of the proposed gain-scheduling control strategy, the same changing wind condition scenario was used for testing and comparing four different choices of the control gain:

• Case with the proposed gain-scheduling K s c : during all the simulation, K c is switched between K laminar c and K turbulent c depending on the wind conditions;

• Case with the theoretical optimal control gain K opt c : during all the simulation, K c is set at the value of K opt c = 9.5065e5, computed using (4); • Case with K laminar c : during all the simulation, K c is set at a value obtained as optimal under laminar conditions, determined using (11);

• Case with K turbulent c : during all the simulation, K c is set at a value obtained as optimal under turbulent conditions, determined using (11).

Results and discussion

This work used the dynamical system presented in (1) to simulate the generated energy and the dissipated energy (i.e. degradation) in the transmission shaft by considering the necessity to adapt the control gain depending on the variations in the wind nature. During all the simulations, the wind speed used as an input to the model is presented in Fig. 5. The change of flow regimen between laminar and turbulent is indicated in all figures by dotted vertical gray lines. Nevertheless, to find the optimal K laminar c and K turbulent c , the evaluation was made using the wind speed data presented in Fig. 4, giving, as a result, a value for each case. In this work, the values of K laminar c and K turbulent c are 1.0552e06 and 1.4260e5 respectively.

The dynamical system presented in (1) was used to validate the gain-scheduling control strategy, simulating the variations in θ s , switching the K c according to the wind conditions. Fig. 6 shows a comparison of the behavior in the variation of θ s using the optimal case as a reference. The results show that the wind conditions affect the variation of the torsion angle, and it is possible to minimize the impact if a suitable control gain is used. Fig. 7 shows the generated energy for different K c : with K opt c it is possible to generate more energy, followed by the case where K laminar c is used, because a greater K c leads to a higher generation of energy. Nevertheless, the gain-scheduling control strategy allows improving energy generation when the wind exhibits periods of turbulent conditions. Fig. 8 illustrates the dissipated energy for the period of evaluation. The case with K laminar c dissipated a higher amount of energy. Moreover, in the case of K turbulent c , a K c lower than the optimal leads to minor degradation. Nevertheless, it can be seen that the gainscheduling control strategy allows to follow the behavior of the case with K opt c in terms of energy dissipation.

To further discuss these results, let consider the generated and dissipated energy and compare with the K opt c scenario:

• Generated energy: The optimal case always leads to a significant amount of generated energy, and greater K c increases the energy generation. However, if the control gain is switched according to the wind conditions (K s c ), it is still possible to increase the generation of energy almost at the level reached under laminar ideal conditions, see Fig. 9.

• Dissipated energy: regarding the generated energy, a higher generation leads at the same time to a significant level of dissipated energy, and hence degradation. For this reason, the dissipated energy in the case with K laminar c is above the optimal ones. However, the K turbulent c is below the other cases with a significant difference, while the case with suitable control gain K s c allows to keep a low level of dissipated energy, dissipating less energy than in the cases with K laminar c and even with K opt c , see Fig. 10. We can thus conclude that the proposed control adaptation strategy allows to increase energy generation in turbulent cases and decreases dissipated energy in the drive-train system (when compared to situation where K c is higher than optimal). As a consequence, the system can profit of both a higher generation of energy and a lower dissipated energy.

CONCLUSIONS

This paper proposes a gain-scheduling control strategy to optimize the efficiency of a wind turbine under varying weather conditions, finding an optimal trade-off between the generated energy and degradation (due to dissipated energy) in the drive-train. The proposed strategy considers the variation of the wind conditions to alternate between different suitable control gains, maximizing the generated energy and decreasing the dissipated energy.

The proposed strategy was tested using different wind speed scenarios to consider a more complete panorama about the possible wind conditions affecting the turbine. Real data measurement was used to simulate the wind conditions for laminar flow, while for turbulent wind conditions, a stochastic model was used to simulate it.

The results show that it is possible to maximize the generated energy and decrease the dissipated energy by switching the control gains depending on wind flow conditions, decreasing the variations in the shaft angle, and getting closest to the theoretical optimal behavior. Stability conditions of the proposed control scheme can be obtained by expressing the whole dynamical control system as a Linear Parametric Varying system and, by using suitable available tools in this area. This is aspect will treated in a future work.

Fig. 3 .

 3 Fig. 3. Example of gain-scheduling depending on two wind conditions.

Fig. 4 .

 4 Fig. 4. Considered wind speed conditions: (a) laminar and (b) turbulent 4. SIMULATION OF THE PROPOSED APPROACH

Fig. 5 .

 5 Fig. 5. Simulated wind speed.

Fig. 6 .Fig. 7 .

 67 Fig. 6. Comparison of the relative torsion shaft angle for different feedback gains K c .

Fig. 8 .Fig. 9 .

 89 Fig. 8. Dissipated energy for different feedback gains K c .

Fig. 10 .

 10 Fig. 10. Comparison of the relative dissipated energy with respect to the optimal case.


Marthe Rousseau

Anders Meibom

Marc Gèze

Xavier Bourrat

Martine Angellier

Evelyne Lopez

Anders Meibo

Dynamics of sheet nacre formation in bivalves

Keywords: Biomineralization, Calcium carbonate, Nacre, SEM, SIMS

Introduction

Nacre, or mother-of-pearl, is an aragonite-dominated structure produced by bivalves, gastropods, and cephalopods as an internal coating of their otherwise calcitic shells. Because of its highly organized internal structure, chemical complexity, mechanical properties and optical effects, which create a characteristic and beautiful lustre, the formation of nacre is among the best-studied examples of calcium carbonate biomineralization ( [START_REF] Wada | Crystal growth of molluscan shells[END_REF], [START_REF] Bevelander | An electron microscope study of the formation of nacreous layer in the shell of certain bivalve molluscs[END_REF], [START_REF] Mutvei | On the internal structures of the nacreous systems tablets in molluscan shells[END_REF], [Lowenstam, 1981], [START_REF] Levi-Kalisman | Structure of the nacreous organic matrix of a bivalve mollusk shell examined in the hydrated state using Cryo-TEM[END_REF], [START_REF] Currey | Hierarchies in biomineral structures[END_REF], [START_REF] Snow | The mineralogical structure of shells: Part 2. The iridescence colors of abalone shells[END_REF] and [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF]). The high-strength mechanical properties of nacre and the fact that the world pearl jewellery represents a global market worth about 5 billions US$ annually [START_REF] Anon | Anon, Golay's global view[END_REF] have prompted efforts to mimic nacre formation by industrial processes [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF]. Such efforts require a high level of understanding of the biochemistry and the growth dynamics of nacre formation in nature.

Among the nacre-forming organisms, great diversity is observed with regard to the threedimensional arrangement of this highly organized structure [START_REF] Mutvei | On the internal structures of the nacreous systems tablets in molluscan shells[END_REF]. This structural diversity implies substantial variation in the process of nacre formation among different organisms. Here we focus on nacre structure and formation in the pearl-producing bivalve Pinctada margaritifera. Fig. 1 shows the characteristic brick-and-mortar structure of the nacreous layer in P. margaritifera. The primary structural component of mature nacre is a pseudohexagonal tablet, about 0.5-1 micrometer (μm) thick and about 5-10 μm in width, consisting primarily of aragonite: a polymorph of CaCO 3 . At the nanometer (nm) length scale, the aragonite component inside individual tablets is embedded in a crystallographically oriented foam-like structure of intra-crystalline organic materials in which the mean size of individual aragonite domains is around 50 nm, ( [Rousseau et al., 2005a] and [START_REF] Stolarski | Nanostructure of biogenic versus abiogenic calcium carbonate crystals[END_REF]). The crystallographic properties of nacre are that of single crystal aragonite with the c-axis oriented perpendicular to the plane defined by individual tablets. Locally, the a and b axes in the plane of the tablet layers show long-range (i.e. mm-scale) orientation but can also be rotated differently (up to about 45°) around the c-axis in different parts of the shell ( [START_REF] Wada | The crystalline structure on the nacre of the pearl oyster shell[END_REF], [START_REF] Weiner | Design strategies in mineralized biological materials[END_REF], [START_REF] Caseiro | Evolution of deposit thickness of organic and aragonitic material during the growth of P. margaritifera pearls[END_REF], [START_REF] Checa | Selforganisation of nacre in the shells of Pterioida (Bivalvia: Mollusca)[END_REF], [Rousseau et al., 2005a] and [START_REF] Checa | Organization pattern of nacre in Pteriidae (Bivalvia: Mollusca) explained by crystal competition[END_REF]). It is thought that crystallographic orientations are defined by the inter-lamellar, foam-like network of organic material and transferred from tablet to tablet by 'bridges' of similar material (Rousseau et al., 2005a). [START_REF] Checa | Selforganisation of nacre in the shells of Pterioida (Bivalvia: Mollusca)[END_REF] and [START_REF] Checa | Organization pattern of nacre in Pteriidae (Bivalvia: Mollusca) explained by crystal competition[END_REF] have suggested a model for how crystal orientation is established in sheet nacre, in which the b-axis becomes gradually aligned with the radial nacre growth direction. In any case, mature tablets are organized in layers in which each tablet is in physical contact with its neighbour, only separated by a nm thick, inter-tabular matrix consisting primarily of aragonite-nucleating proteins and carboxylates, ( [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF], [START_REF] Nudelman | Mollusk shell formation mapping the distribution of organic matrix components underlying a single aragonitic tablet in nacre[END_REF] and [START_REF] Dauphin | Structure and composition of the nacre-prisms transition in the shell of Pinctada margaritifera (Mollusca, Bivalvia)[END_REF]). Individual layers of tablets are separated by a 50-100 nm layer consisting mainly of chitin and organic material [START_REF] Levi-Kalisman | Structure of the nacreous organic matrix of a bivalve mollusk shell examined in the hydrated state using Cryo-TEM[END_REF]. This is the inter-lamellar matrix. Several layers of nacre tablets form at the same time across a growth front, with subsequent layers arranged as 'stair-cases' in which each step is typically 10-15 μm wide, (Rousseau et al., 2005b). In this way, up to 3-4 layers of nacre tablets are formed per day [START_REF] Caseiro | Evolution of deposit thickness of organic and aragonitic material during the growth of P. margaritifera pearls[END_REF]. It has been established that nacre tablet formation takes place in the space defined by the two-top layers of inter-lamellar matrix [START_REF] Bevelander | An electron microscope study of the formation of nacreous layer in the shell of certain bivalve molluscs[END_REF]. We refer to the layer of organics that separates the animal from the growing nacre as the 'membrane'. This highly cross-linked protein-rich membrane [START_REF] Nudelman | Environmental-and cryo-scanning electron microscopy study of the forming nacreous layer from the shells of the bivalves Atrina rigida and Pinctada margaritifera[END_REF] is in direct contact with the epithelial cellstructure of the overlying organism, which is referred to as the 'mantle' (Fig. 1). In the space beneath the membrane, tablets growth proceeds from nucleation sites, which are complex structures consisting of different layers of organic molecules [START_REF] Nudelman | Mollusk shell formation mapping the distribution of organic matrix components underlying a single aragonitic tablet in nacre[END_REF]. The distribution of nucleation sites is dictated by the geometry of the underlying layer of tablets, each nucleation site is often situated above a triple-junction of underlying tablets (Rousseau et al., 2005b). The mantle delivers specific organic molecules (including proteinaceous materials -the so-called 'silk-phase') and carbonate components to the closed compartment between the membrane and the first inter-lamellar matrix-layer, which was the top membrane during the formation of the previous layer of nacre tablets [START_REF] Bevelander | An electron microscope study of the formation of nacreous layer in the shell of certain bivalve molluscs[END_REF]. It has been speculated that growth of individual nacre tablets occur by nucleation and crystallization of amorphous calcium carbonate (ACC) [START_REF] Towe | Ultrastructure and inferred calcification of the mature and developing nacre in bivalve mollusks[END_REF]). However, because our knowledge about the formation of the nacre structure is largely based on studies of mature nacre structures, little is known about the growth dynamics of individual tablets. The small dimensions of the closed compartment, in which nacre formation takes place [START_REF] Bevelander | An electron microscope study of the formation of nacreous layer in the shell of certain bivalve molluscs[END_REF], and the tight physical relationship between the growing nacre structure and the mantle render direct studies difficult. Here, we circumvent these difficulties with a simple preparation technique and by applying new ion microprobe technology to the study of developing nacre tablets prepared with a minimum of alteration of inter-lamellar and intertablet organics.

Materials and methods

We have studied six month old P. margaritifera oysters grown in their natural habitat, at depths of 7-12 m, under carefully monitored conditions at the Marine Station of Marutea in French Polynesia. The animals were fixed with ethanol initially at 70 vol%. The water content was gradually decreased in increasingly ethanol-rich solutions (3 day baths in 80%, 90%, and 100% ethanol, respectively) and, after complete dehydration, the shells were opened, allowing the soft tissue of the mantle to be separated from the growing nacre precisely at the top surface of the membrane that separates the surface of the growing nacre from the mantle, allowing a study of the interface between the mantle and the shell (see Supplementary Material, SM). The sample was then dried by air. This preparation, which preserves even the most delicate structures of the membrane and the nacre surface (Figs. S1 andS2), reveals several important aspects of nacre growth dynamics. After gold coating, observations were performed at the Common Service of Electronic Microscopy of the laboratories of the Life Sciences, National Museum of Natural History (France) with a JEOL JSM-840A scanning electron microscope (SEM) operating at 15 kV. Following SEM observations, the samples were analyzed by the NanoSIMS in order to image the distribution of elements primarily associated with organic materials in the top-most layer of the growing nacre structure.

NanoSIMS: Using a focused (primary) beam of Cs + with a current of ~1 pA on the sample, secondary ions were sputtered from the sample surface. Ions of hydrogen (H -), nitrogen (N, measured as 12 C 14 N -) and sulfur ( 32 S -) were detected simultaneously (multicollection-mode) in electron-multipliers at a mass-resolving power of ~ 4000 (M/ΔM). At this mass-resolving power, the measured secondary ions are resolved from potential molecular interferences. During Cs + bombardment, charge compensation was achieved by simultaneous delivery of electrons to the surface. Note that because N is measured as the 12 C 14 N -ion detection of N can only be made from regions containing carbon. However, because the nacre structure consists primarily of CaCO 3 and organic matter there are no regions in which carbon is not available to generate the 12 C 14 N -ion. Thus the variations recorded in 12 C 14 N -are representative of variations in the N concentration across the imaged structures. Images were obtained from a pre-sputtered surface by rastering the primary ions across the sample surface.

Results and discussion

Fig. 2 shows the surface membrane, observed from the point of view of the nacre layer, and corresponding features in the growing nacre surface, respectively. Several important observations can be made. (i) For each nacre tablet, there exists in the membrane a circular structure (Fig. 2A andC). The circular membrane structure displays higher topography than the adjacent surface membrane. We refer to this circular structure as the 'ring'. (ii) This ring makes direct physical contact with the growing nacre tablets. Each nacre tablet features a similar ring, several hundred nm wide, with a complementary (i.e. lower) topography (Fig. 2B andD). (iii) Neighbouring tablets can merge during growth by lateral expansion that leads to a 'collision'. In this process, ring structures surrounding each tablet first merge together, then disappear (Fig. 2D). Similarly, the ring structures in the membrane first merge and then disappear in the membrane during tablet collision (Fig. 2C).

These structural observations raise questions about the compositional properties of the ring around individual nacre tablets, which clearly plays an essential role in the growth process. Because the growing tablets are forming in a closed compartment underneath the membrane (Fig. 1 and Fig. 2), direct chemical analysis is problematic. We circumvent these difficulties by taking advantage of recently developed ion microprobe technology, the NanoSIMS. The NanoSIMS allows chemical composition to be imaged with extremely high spatial resolution (on the order of 100 nm), while depth-profiling through the inter-lamellar matrix and into the nacreous structure with a depth resolution of about 10 nm per image ( [START_REF] Meibom | Distribution of magnesium in coral skeleton[END_REF] and [START_REF] Meibom | Biological forcing controls the chemistry of reef-building coral skeleton[END_REF]). The 3-dimensional distribution of H, N and S are diagnostic of organic compounds in which their concentration is greatly increased over that in the aragonitic nacre tablets. Fig. 3A shows the 3-D reconstruction of a depth-sequence of H images from the top-most layers (in the membrane), via intermediate levels to an estimated depth of around 0.2 μm in the tablet structure, where the overlying membrane has been sputtered away completely and the distribution of organics around individual tablets becomes visible; see SD for 3-dimensional reconstructions of the corresponding N and S distributions. The distribution of organics in the ring surrounding each nacre tablet corresponds to the characteristic ring in the overlying mantle; see Fig. 2D. Fig. 3C shows in greater detail the distribution of organics around individual tablets. Importantly, concentrations of H, N and S are clearly confined to the rim around each tablet. In the ring, the signal from N is almost two orders of magnitude higher than in the surrounding aragonitic tablets. Sulfur and Hydrogen are enriched by factors of about 10 and 7, respectively over the signal observed from inside the tablet. The ring structures are flat structures (Fig. 2) and completely resolved by the primary beam of the NanoSIMS, indicating that the enhanced concentrations in H, N and S cannot be ascribed to topographic, i.e. edge effects. The observations demonstrate that the ring around individual aragonite tablets is a 3-dimensional structure that is greatly enriched in organic molecules. Good candidates for these compounds are carboxylates, sulphated sugars, and aragonite-nucleating proteins, which also characterize the nucleation sites and the thin inter-tabular matrix separating mature tablets ( [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF], [START_REF] Nudelman | Mollusk shell formation mapping the distribution of organic matrix components underlying a single aragonitic tablet in nacre[END_REF] and [START_REF] Nudelman | Environmental-and cryo-scanning electron microscopy study of the forming nacreous layer from the shells of the bivalves Atrina rigida and Pinctada margaritifera[END_REF] Nudelman, E. Shimoni, E. Klein, M. Rousseau, X. Bourrat, E. Lopez, L. Addadi and S. Weiner, Environmental-and cryo-scanning electron microscopy study of the forming nacreous layer from the shells of the bivalves Atrina rigida and Pinctada margaritifera, J. Struct. Biol. 162 (2008), pp. 290-300; The high content of N in the ring is consistent with the presence of aragonite-nucleating proteins [START_REF] Nudelman | Mollusk shell formation mapping the distribution of organic matrix components underlying a single aragonitic tablet in nacre[END_REF].

From these observations, we propose that the nacre tablet formation takes place via the formation of a growth-ring that surrounds each tablet (Fig. 2 and Fig. 3). This model is illustrated in Fig. 4. The proposed model is built on several recent observations, in combination with those presented here. It has been established that nacre tablet formation proceeds from a hydro-gel loaded with calcium carbonate precursor material, acidic proteins and the silk-phase [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF]. The acidic proteins in the hydro-gel can either inhibit calcium carbonate crystallization, as demonstrated by in vitro experiments or stimulate aragonite crystallization, depending on the nature of the locally dominating acidic protein fraction ( [Cohen, 2003] and [START_REF] Addadi | Mollusk shell formation: a source of new concepts for understanding biomineralization processes[END_REF]). Polyclonal antibody labelling of aragonitenucleating acidic proteins has shown that a specific component of the acidic protein fraction is concentrated in the nucleation sites, which initiate the formation of individual nacre tablets [START_REF] Nudelman | Mollusk shell formation mapping the distribution of organic matrix components underlying a single aragonitic tablet in nacre[END_REF]. These nucleation sites are in the majority of cases situated at a triplejunction of three nacre tablets in the previously formed layer (Fig. 4) and are therefore supposed to be inherited by mechanical transfer of material displaced upward during nacre tablet collision in the layer below. Once aragonite nucleation is triggered at these nucleation sites, the growth of the nacre tablet proceeds upward to fill the space between the membrane and the underlying inter-lamellar matrix, then laterally until collision with neighbouring tablets take place (Fig. 4). The bulk of this growth phase is mediated by the growth-ring exhibited in Fig. 2 and Fig. 3. The calcium carbonate precursor is most likely amorphous ( [START_REF] Istin | Dynamic state of calcium reserves in freshwater clam mantle[END_REF] and [START_REF] Addadi | Taking advantage of disorders: amorphous calcium carbonate and its roles in biomineralization[END_REF]) consistent with the observation of a thin (about 5 nm) veneer of amorphous calcium carbonate (ACC) surrounding mature nacre tablets ( [START_REF] Gehrke | Retrosynthesis of nacre via amorphous precursor particles[END_REF] and [START_REF] Nassif | Amorphous layer around aragonite platelets in nacre[END_REF]). In contrast, the end-product in bulk nacre tablets is crystallographically oriented aragonite crystals with an average size around 50 nm, essentially assembled into a tablet meso-crystals ( [Rousseau et al., 2005a], [START_REF] Wohlrab | Crystalline, porous microspheres made from amino acids by using polymer-induced liquid precursor phases[END_REF] and [START_REF] Kulak | Continuous structural evolution of calcium carbonate particles: a unifying model of copolymer-mediated crystallization[END_REF]). The growth-ring structures observed in this study most likely plays a key role in the transformation from ACC precursor in the surrounding hydro-gel to the formation of aragonite nano-domains as postulate earlier [START_REF] Weiss | Mollusc larval shell formation: amorphous calcium carbonate is a precursor phase for aragonite[END_REF]. The foam-like, intra-crystalline organic material, in which the aragonite meso-crystals are embedded, must be similar in composition to the organic material that makes up the growth-ring, from which it is inherited. We postulate that the growth dynamics of nacre tablets is controlled by spatially distinct components of acidic proteins in the growth-ring and in the adjacent hydro-gel reservoir supplying the ACC precursor component. In the hydro-gel, proteins that inhibit calcium carbonate crystallization are dominating, whereas in the growth-ring, aragonitenucleating proteins are dominant. In the growth-ring, the ACC precursor phase crystallizes into nano-crystals (i.e. about 50 nm in size) and are added to the growing tablet. This dynamic situation poses several challenges. First, the hydro-gel precursor phase has to be continuously produced from the mantle and transferred to the constantly emerging medium beneath the top membrane where new tablets are about to form. Observations of mineral (presumably ACC) containing granules in the cytoplasm of the mantle [START_REF] Neff | Ultrastructure of the outer epithelium of the mantle in the clam Mercenaria mercenaria (L) in relation to calcification of the shell[END_REF] suggest that ACC, indeed perhaps the entire hydro-gel phase, might be synthesized in the mantle and secreted to the space below the membrane. Second, because the aragonite-nucleating proteins and other organic materials in the growth-ring are transferred to the tablets in which they embed the nano-scale aragonite crystals, the growth-ring is continuously consumed and has to be replaced during tablet growth. This requires either that (1) the aragonite-nucleating proteins of the growth-ring are scavenged from the surrounding silk-phase, presumably by electrostatic forces that can preferentially attract molecules with high acidity, or (2) that these components are delivered directly from the mantle to the growth-ring. Isotopic labelling experiments designed to trace different organic constituents in the mantle and in the nacre-forming compartment might help distinguish between these two different scenarios.

Conclusion

A growth-ring structure, rich in organic materials, surrounds each growing nacre tablet during formation of sheet nacre in P. margaritifera bivalves. This structure disappears as nacre tablets grow laterally and collide with adjacent tablets. It is conceivable that this organic ring structure acts to nucleate aragonite into the highly oriented nano-crystals (~ 50 nm in size) that make up the meso-crystal (i.e. μm sized) nacre tablets [START_REF] Wohlrab | Crystalline, porous microspheres made from amino acids by using polymer-induced liquid precursor phases[END_REF] and [START_REF] Kulak | Continuous structural evolution of calcium carbonate particles: a unifying model of copolymer-mediated crystallization[END_REF]). This adds support for a highly dynamic biomineralization process during which organic materials and carbonate precursor phases (likely amorphous) are delivered to the site of nacre tablet formation from the overlying mantle with a high degree of spatial control.

FiguresFig. 1

 1 Figures

  Fig. 1. The shell of the pearl-producing Pinctada margaritifera oyster. (A) An opened shell in which the animal is still in place. The brighter nacreous layer of aragonite is clearly visible next to the animal. The nacreous layer coats the darker, prismatic shell of calcite that forms the outer shell. During nacre formation the tissue of the animal, the so-called 'mantle', reaches out to the transition between nacre and calcitic shell where new nacre is formed. (B) Schematic representation of the shell structure and the overlying mantle of the animal. (C) Careful removal of the mantle, which is in direct contact with the shell growing surface reveals the surface membrane covering the forming nacre tablets (white arrow) after mechanical fractures of the nacre. (D) Fracture of the shell reveals the brick-and-mortar structure of the nacreous layer. Each layer is about 0.5-1 μm thick.

Fig. 2 .

 2 Fig. 2. The mantle-nacre interface. Electron microscopy images of structures on the mantle side (A and C) and corresponding structures on the surface of the growing nacre tablets (B and D). These two surfaces are in direct contact during nacre formation. Distinct ring structures surround each nacre tablet (D, white arrows) and have morphologically complementary features in the membrane on the mantle side (C, white arrows). This ring disappears when tablets collide during lateral expansion (C and D). See text for discussion.

Fig. 3 .

 3 Fig. 3. Distribution of organics around nacre tablets. (A) A 3-dimensional reconstruction of the H distribution at different depths in the surface layers of the forming nacre. The data were obtained with the NanoSIMS ion microprobe. The direction of sputtering is indicated by a white arrow. The sputtered volume is about 10 μm × 10 μm × 0.2 μm. For greater clarity, the thickness of this volume (blue axis) has been expanded by a factor of 6. The surface membrane is visible in the first images (white arrow) but deeper in the structure organics are concentrated only in the growth-rings surrounding each nacre tablet. (Full 3-D reconstructions of the H, N and S distributions are available in SD as movies). (B) NanoSIMS image showing the distribution of N, which is also preferentially concentrated in the growth-ring around each nacre tablet. (C) Line-profile (track indicated in B) showing the enhanced concentrations of H, N, and S in the growth-ring. See text for discussion. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Acknowledgments

We thank Tahiti Perles for technical and financial support. This work was granted by the Agence Nationale de la Recherche Française under the awards ANR Blan06-3_135480: Nacre Fluid and ANR-05-JCJC-0158: NanoGeoBio. We thank Gérard Mascarel for his help in the electron microscopy work (Service commun de microscopie électronique des Sciences de la Vie du MNHN). Lia Addadi, Steve Weiner (Weizmann Institute of Science) and Rory Arrowsmith are thanked for constructive discussions. Thanks are also due to Francine Lallier and Sandrine Borzeix for their assistance in preparing this manuscript. Two anonymous reviewers are thanked for highly constructive criticism.

Supplementary Material for

Dynamics of Sheet Nacre Formation in Bivalves

Material and methods

Scanning Electron Microscopy

Observations were performed at the Common Service of Electronic Microscopy of the laboratories of the Life Sciences, National Museum of Natural History (France) with a JEOL JSM-840A operating at 15kV, at the Weismann Institute of Science, Rehovot (Israel) with a high-resolution Ultra 55 Zeiss using an in-lens secondary electron detector, at the University of Orléans (France) with a Hitachi 4500 FEG SEM at 3kV with an in-lens detector.

Supp-

SEM, NanoSIMS

Amira

Image 3D reconstruction and visualisation of 25 stacked planes of NanoSIMS images was produced with Amira software (ver 4.1.1) from TGS Mercury Computer System using the voltex module. This direct volume rendering technique is an intuitive method for visualizing 3D scalar fields in which each point in the data volume is assumed to emit and absorb light.

The amount and colour of emitted light and the amount of absorption are determined from the scalar data by using a colourmap that incorporates a factor to change the overall transparency of the object independent of the data value.

NanoSIMS sputter 3D volume reconstruction (movies in mpg format):

-Rousseau_3D_H.mpg -Rousseau_3D_N.mpg


W Puech
email: william.puech@lirmm.fr

Image Encryption and Compression for Medical Image Security

Keywords: Image protection, compression, encryption, watermarking and data hiding

This tutorial presents the problem of protecting the transmission of medical images. The presented algorithms will be applied to images, videos and 3D objects. The main keywords are compression, encryption, watermarking and data hiding.

I. INTRODUCTION

The amount of digital visual data (image, video and 3D object) has increased rapidly on the Internet. Image, video and 3D object security becomes increasingly important for many applications, e.g., confidential transmission, video surveillance, military and medical applications. For example, the necessity of fast and secure diagnosis is vital in the medical world. Nowadays, the transmission of visual data is a daily routine and it is necessary to find an efficient way to transmit them over networks.

Two main groups of technologies have been developed for this purpose. The first one is based on content protection through encryption [START_REF] Schneier | Applied cryptography[END_REF], [START_REF] Uhl | Image and Video Encryption: From Digital Rights Management to Secured Personal Communication[END_REF]. In this group, proper decryption of data requires a key. The second group bases the protection on digital watermarking or data hiding, aimed at secretly embedding a message into the data. In order to not increase the processing time, these two approaches must be combined with the compression stage. Nowadays, the challenge is to perform simultaneously for example image encryption and compression.

The works presented in this tutorial show how encryption algorithms provide security to medical imagery. The main objective is to guarantee the protection of medical images during transmission, and also once this digital data is archived. The subsequent challenge is to ensure that such coding withstands severe treatment such as compression. When a physician receives a visit from a patient, he often requires a specialist opinion before giving a diagnosis. One possible solution is to send images of the patient, along with a specialist report, over a computer network. Nevertheless, computer networks are complex and espionage is a potential risk. We are therefore faced with a real security problem when sending data. For ethical reasons, medical imagery cannot be sent when such a risk is present, and has to be better protected. Encryption is the best form of protection in cases such as this. Many different techniques for the encryption of text already exist.

In this tutorial, in a first part, I will recall the standard information of encryption (block cipher, stream cipher, asymmetric and symmetric encryption) and I will show the application of standard algorithms to images. In a second part I will present combination of image encryption and compression. To finish this second part I will talk of selective encryption methods. In this part I will address the problem of simultaneous partial encryption (PE), selective encryption (SE) and image compression. Indeed, in order to visualize on line images in real time, they must be quickly transmitted and the full encryption is not really necessary. To finish this tutorial I will present some watermarking and data hiding algorithms developped by our team applied to visual data and I will show how these can be suited to medical imagery.

II. A NEW CRYPTO-WATERMARKING METHOD FOR

MEDICAL IMAGES SAFE TRANSFER [START_REF] Puech | A New Crypto-Watermarking Method for Medical Images Safe Transfer[END_REF], [START_REF] Puech | Chapter 10: Coding: Encryption-Watermarking-Compression for Medical Information Security[END_REF] The use of encryption schemes to assure confidentiality in digital networks is a common routine. The drawback of the traditional symmetric ciphering is the risk involved in sending the secret key. This work presents an autonomous method for safely transferring digital images. It is based on three steps. The first one is the encryption of the image with a new asynchronous stream cipher that is resistant to noise. The second step is the ciphering of the secret key with an asymmetric algorithm. The last step is the embedding of the enciphered key into the previous encrypted image. We fully present the proposed asynchronous stream cipher as well as the used data hiding method. The most important features of the proposed procedure is that the proposed stream cipher method is robust to noise and that we send the coded image and the secret key concomitantly. In this way the proposed method is autonomous and ensures the integrity because the key is spread in the whole image. We have applied and shown the results of our method to more than 100 medical images for statistically representative results [START_REF] Puech | A New Crypto-Watermarking Method for Medical Images Safe Transfer[END_REF], [START_REF] Puech | Chapter 10: Coding: Encryption-Watermarking-Compression for Medical Information Security[END_REF]. The keywords of this work are safe image transmission and combination of watermarking and encryption algorithms.

III. A NEW FAST REVERSIBLE METHOD FOR IMAGE SAFE

TRANSFER [START_REF] Puech | A New Fast Reversible Method for Image Safe Transfer[END_REF] In this section a novel reversible method for fast and safe image transfer is proposed. The method combines compression, data hiding and partial encryption of images in a single processing step. The proposed approach can embed data into the image according to the message size and partially encrypt the image and the message without changing the original image content. Moreover, during the same process the image is lossless compressed. Nevertheless, the compression rate depends on the upper bound of message size to embed in the image. The main idea is to decompose the original image into two sub-images and to apply various processes to each sub-image in order to gain space and increase the amount of embedded data. The two sub-images are then scrambled and partially encrypted. The most significant characteristic of the proposed method is the utilization of a single procedure to simultaneously perform the compression, the reversible data hiding and the partial encryption rather than using three separate procedures. Our approach reduces then the computational effort and the required computation time. This method is specially suited for medical images where one can associate the patient diagnostic to the concerned medical image for safe transfer purpose [START_REF] Puech | A New Fast Reversible Method for Image Safe Transfer[END_REF]. The keywords of this work are fast and safe image transmission, lossless compression, reversible data hiding, partial encryption, image protection and real time image processing.

IV. A REVERSIBLE DATA HIDING METHOD FOR

ENCRYPTED IMAGES [START_REF] Puech | A Reversible Data Hiding Method for Encrypted Images[END_REF] Since several years, the protection of multimedia data is becoming very important. The protection of this multimedia data can be done with encryption or data hiding algorithms. To decrease the transmission time, the data compression is necessary. Since few years, a new problem is trying to combine in a single step, compression, encryption and data hiding. So far, few solutions have been proposed to combine image encryption and compression for example. Nowadays, a new challenge consists to embed data in encrypted images. Since the entropy of encrypted image is maximal, the embedding step, considered like noise, is not possible by using standard data hiding algorithms. A new idea is to apply reversible data hiding algorithms on encrypted images by wishing to remove the embedded data before the image decryption. Recent reversible data hiding methods have been proposed with high capacity, but these methods are not applicable on encrypted images. In this paper we propose an analysis of the local standard deviation of the marked encrypted images in order to remove the embedded data during the decryption step. We have applied our method on various images, and we show and analyze the obtained results [START_REF] Puech | A Reversible Data Hiding Method for Encrypted Images[END_REF]. The keywords of this work are image encryption and noise removing.

V. CONCLUSION

In this tutorial we have presented several methods in order to protect the transmissions of medical images. The protection of visual data can be done by using encryption or watermarking algorithms or by combining these two approaches.


Jianyong Xue
email: jianyong.xue@inria.fr

Frédéric Alexandre
email: frederic.alexandre@inria.fr

Multi-task learning with modular reinforcement learning

Keywords: Multi-task learning, Modular reinforcement learning, Incomparable reward scale, Compositionality policy, Model-based reinforcement learning

The ability to learn compositional strategies in multi-task learning and to exert them appropriately is crucial to the development of artificial intelligence. However, there exist several challenges: (i) how to maintain the independence of modules in learning their own sub-tasks; (ii) how to avoid performance degradation in situations where modules' reward scales are incompatible; (iii) how to find the optimal composite policy for the entire set of tasks. In this paper, we introduce a Modular Reinforcement Learning (MRL) framework that coordinates the competition and the cooperation between separate modules. A selective update mechanism enables the learning system to align incomparable reward scales in different modules. Furthermore, the learning system follows a "joint policy" to calculate actions' preferences combined with their responsibility for the current task. We evaluate the effectiveness of our approach on a classic food-gathering and predator-avoidance task. Results show that our approach has better performance than previous MRL methods in learning separate strategies for sub-tasks, is robust to modules with incomparable reward scales, and maintains the independence of the learning in each module.

Introduction

Multi-task learning is popularly observed in humans and several other intelligent animal species, and the ability to tackle with diverse tasks has gradually increased with the enrichment of skills that are learned from experience. For example, in a wild natural environment, most animals must pursue two tasks simultaneously: look for food and avoid predators. Meanwhile, long-time evolution has allowed animals to generate a more sensitive olfactory system to find food rapidly and to be more alert to predators, as well as to run faster.

A hallmark of multi-task learning benefits from the reusage of similar patterns underlying a set of regularities across tasks, in order to improve performance on any other single task [START_REF] Wang | Learning to reinforcement learn[END_REF][START_REF] Sodhani | Multi-task reinforcement learning with contextbased representations[END_REF]. The goal of multi-task learning is to find an optimal solution for solving the entire set of tasks in parallel. However, there exist several challenges: (i) how to maintain the independence of modules in learning their own sub-tasks; (ii) how to avoid performance degradation in situations where modules' reward scales are incompatible; (iii) how to find the optimal composite policy for the entire set of tasks.

These challenges have inspired the emergence of modular reinforcement learning (MRL) approaches, which decompose a multi-task problem into a collection of concurrently running RL modules, each of which learns a separate policy (or sub-policy) to solve a portion of the original problem [START_REF] Simpkins | Composable modular reinforcement learning[END_REF][START_REF] Gupta | Action selection for composable modular deep reinforcement learning[END_REF]. Accordingly, these joint modules implicitly form a larger composite RL problem, the goal is to find the optimal policy for this composite RL [START_REF] Sprague | Multiple-goal reinforcement learning with modular sarsa[END_REF]. Considering the performance degradation in the composability of modules that have incomparable reward scales, [START_REF] Simpkins | Composable modular reinforcement learning[END_REF] introduced a special module named "command arbitrator", and proposed the architecture Arbi-Q to reformulate the MRL. Specifically, the arbitrator's policy assigns modular preferences given the observation of the state, then the selected module's preferred action will be used to interact with the environment. After each interaction, the state-abstraction function in the selected module transforms the world observation into a module-specific subset of the world states, in which modules are associated with the world that they operated in and not coupled to other modules or to an arbitrator.

Nevertheless, the Arbi-Q model has the following limitations: (i) all the modules need to be evaluated at each time-step without considering prior knowledge of modular selection, which slows down the learning process and the decisionmaking; (ii) learning system's action preference always comes from one single module, which may lead to dictatorship in decision-making; (iii) function approximation needs to be improved for complex continuous tasks (or the complex hybrid of discrete and continuous tasks).

Focused on these limitations, in this article, we adopt the idea of multiple model-based reinforcement learning from [START_REF] Doya | Multiple model-based reinforcement learning[END_REF] and present a new MRL architecture named "Inverse Arbi-Q" to coordinate the competition and the cooperation between separate modules. A selective update mechanism enables the learning system to align incomparable reward scales in different modules. Furthermore, the learning system follows a "joint policy" to calculate actions' preferences combined with their responsibility for the current task.

Related work

Growing evidence from behavioral and biological neuroscience notes that most animals' brain parses ongoing behavior into discrete, bound segments [START_REF] Gatti | The human cerebellum as a hub of the predictive brain[END_REF]. Specifically, internal models in separated areas of the cerebellum are responsible for given tasks via a competitive and cooperative way [START_REF] Smith | Modeling incentive salience in pavlovian learning more parsimoniously using a multiple attribute model[END_REF][START_REF] Bernard | Don't forget the little brain: A framework for incorporating the cerebellum into the understanding of cognitive aging[END_REF]. In cases when a new task appears, multiple models learn it through competition, but only one or a small part of them will be recruited for this task. Meanwhile, human behaviors are hierarchically organized: actions cohere into sub-task sequences, which fit together to achieve overall task goals [START_REF] Logan | Hierarchical control of cognitive processes: The case for skilled typewriting[END_REF][START_REF] Botvinick | Hierarchical models of behavior and prefrontal function[END_REF].

In order to scale up the RL to address multiple goals at once, an intuitive approach is to train one RL module to handle each of the sub-goals, then the learned policies for each module can be used to fairly distribute control among the modules. However, these component policies are usually sub-optimal in the context of the composite tasks. Focusing on this problem, [START_REF] Sprague | Multiple-goal reinforcement learning with modular sarsa[END_REF] replace the Qlearning rule within each module with a SARSA(0) learning rule, and propose a model of GM-Sarsa(0) for finding approximate solutions to multiple-goal RL problems. In order to better coordinate competitions between modules, the mixture of experts architecture from [START_REF] Nowlan | Evaluation of adaptive mixtures of competing experts[END_REF][START_REF] Jacobs | Adaptive mixtures of local experts[END_REF] introduced a learning system that is composed of a list of "experts" network, plus a gating network that determines which of the experts should be responsible for the training case. However, this modular architecture relies heavily on the performance of the gating network.

An alternative family of approaches [START_REF] Narendra | Adaptation and learning using multiple models, switching, and tuning[END_REF][START_REF] Doya | Multiple model-based reinforcement learning[END_REF][START_REF] Samejima | Inter-module credit assignment in modular reinforcement learning[END_REF] employs a state predictor within each expert (or module), which enables the environmental dynamics to be predictable. The controllers associated with these experts are decided by the prediction error: the one that has the smallest prediction error will be recruited at any particular moment. Nevertheless, models based on this idea mainly emphasize the competition between modules to choose one single expert that is the most suitable for a specific task (or sub-task), but the cooperation between modules as a compositional strategy for composable complex tasks is relatively lacking. As an improvement, [START_REF] Doya | Multiple model-based reinforcement learning[END_REF] propose a model of multiple model-based reinforcement learning (MMBRL). In the MMBRL, a complex task is decomposed into multiple modules, particularly, the "responsibility signal" was introduced to weight the outputs of multiple modules, as well as to gate the learning of the prediction models and the RL controllers. However, the parallel update strategy degrades its performance when modules has incomparable reward scales, and the composite policy will be far from optimal as the complexity of the learning problem increases.

Generally, the original MRL supports not only the decomposition of complex tasks into modules, but also the composability of separately learned modules as new strategies for tasks that were never solved before [START_REF] Esteban | Hierarchical reinforcement learning for concurrent discovery of compound and composable policies[END_REF][START_REF] Simpkins | Composable modular reinforcement learning[END_REF]. Focusing on the optimality of the composite strategy for the entire task and the independence of learning in separate modules, [START_REF] Samejima | Inter-module credit assignment in modular reinforcement learning[END_REF] introduced the specific concept of "modular reward", which comes from the actual reward after each interaction plus a bonus for passing the task on a proper module. Specifically, this bonus is calculated from the modular value function and the temporal difference in the module gating signal, which propagates the reward toward the entire task achievement between modules. In situations where the tasks require to perform the sub-tasks concurrently, [START_REF] Esteban | Hierarchical reinforcement learning for concurrent discovery of compound and composable policies[END_REF] propose an hierarchical RL approach to learn both compound and composable policies within the same learning process, by which exploiting the off-policy data generated by the compound policy. The results show that the experience collected with the compound policy permits not only to solve the complex task but also to obtain useful composable policies that successfully perform in their corresponding sub-tasks.

Aiming at modules that have conflicting goals and misaligned reward scales, [START_REF] Gupta | Action selection for composable modular deep reinforcement learning[END_REF] presented the framework GRACIAS that assigns fine-grained importance to different modules and enables composable decision making based on modern deep RL methods. More precisely, the arbitrator learns the relative importance of each module in a given state, combined with action-value of the individual modules to calculate the joint action preferences. However, since the number of modules in this architecture is preset in advance, which will gradually limit its performance when the complexity of the task continues to increase. Moreover, when the arbitrator is trained with little prior knowledge, the performance will be initially far from optimal.

In the present work, the idea of Inverse Arbi-Q model is trying to address all the major concerns mentioned above. First, it's different from [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF] where experts are determined by a gating network. Instead, modules in the present architecture learn their own behavioral policy and solve their own sub-tasks by competition. Second, it's also different from [START_REF] Simpkins | Composable modular reinforcement learning[END_REF] where the action selection follows a "joint policy". Here action preferences come from the combination of all MBRL modules, rather than from one single preferred module. Furthermore, different from [START_REF] Doya | Multiple model-based reinforcement learning[END_REF] where not all modules are involved in the update procedure after each interaction, instead, only the most related module will be targeted for amelioration. Moreover, the update of parameters in the selected module is gated by a dynamic signal, the "responsibility signal" [START_REF] Doya | Multiple model-based reinforcement learning[END_REF][START_REF] Samejima | Inter-module credit assignment in modular reinforcement learning[END_REF] , which comes from the gaussian softmax function of prediction error in each module and determines how much the module is responsible for the current situation, rather than only according to a fixed parameter of learning rate as in [START_REF] Simpkins | Composable modular reinforcement learning[END_REF][START_REF] Esteban | Hierarchical reinforcement learning for concurrent discovery of compound and composable policies[END_REF] as we will explain in section 5.1.

Preliminaries

Reinforcement Learning

Reinforcement Learning (RL) paradigm focuses on learning an optimal policy π * that enables the agent to gain the maximum accumulated rewards from future steps [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. At each time step t, the agent stays in state s t ∈ S and follows the current policy π(a|s) to execute an action a t ∈ A; as a result, it receives an immediate feedback r t+1 from the environment and transitions stochastically to the next state s t+1 ∈ S. Note that performing this policy requires either knowing the underlying state transition model P (s t+1 |s t , a t ) and reward function R(s t , a t ), or estimating the state value V (s t ) through rewards received in trajectories.

Based on differences in learning optimal policy, RL approaches could be divided into two categories: (a) model-free RL, which focuses on learning a policy or value function, and (b) model-based RL, which aims at learning a dynamic model and a reward function. In the model-free RL approaches, policy learning only focuses on interaction trajectories, and their parameters are mainly updated by the TD-error between state-values within two (or several) steps. Obviously, it requires a large amount of samples to achieve good performance, and typically only learns a single task at a time; also it usually suffers from high sample complexity.

However, in the model-based RL system, a dynamic model is used to make predictions of the probabilities of transition to the next state s t+1 , and of a reward function to provide the expected values r ′ , given the current state s t and the action a t selected by the current policy π(a|s). Model-based reinforcement learning algorithms are generally regarded as being more sample efficient [START_REF] Nagabandi | Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning[END_REF]. However, to achieve good sample efficiency, model-based RL algorithms conventionally use relatively simple function approximators, which fails to generalize well in complex tasks, or with probabilistic dynamics models in complicated and high-dimensional domains.

Modular reinforcement learning

The Modular Reinforcement Learning (MRL) framework is composed of several concurrently running RL modules, each of which learns a specific policy to solve a simpler sub-problem of the main RL problem. In particular, each module has access to the observation of the environment and shares a common action space [START_REF] Sprague | Multiple-goal reinforcement learning with modular sarsa[END_REF]. During each interaction, all modules are required to execute the same action, while observing the state transition and a reward signal specific to the module [START_REF] Simpkins | Composable modular reinforcement learning[END_REF]. The goal of MRL is to learn the optimal policy for the entire problem, in order to maximize the accumulative global reward in the long run.

Furthermore, action selection in MRL follows a "joint policy", in which action preferences come from the combination of all (or partially related) RL controllers in each module. At each time step, the candidate actions are evaluated by the weighted summation of their value expectations of future rewards from the RL controller in each module.

Q(s, a) = n i=1 Q i (s, a)w i (1)
where the weight w i assigned to each module indicates the responsibility of the module in the current task.

4 The Inverse Arbi-Q architecture

The basic idea of the Inverse Arbi-Q architecture lies in the learning system that decomposes a multi-task learning problem into a collection of RL modules, each of which learns a separate strategy for a specific sub-task concurrently. Figure 1 presents the overall organization of Inverse Arbi-Q architecture, which is composed of a list of n model-based RL (MBRL) modules. Each of MBRL modules shares a common inputs (state space and action space) and has an identical structure that consists of three components: the reward predictor, the RL controller and the state predictor, which provide the elementary realization of the module for learning its own behavioral policy and its own designated sub-task.

Implementation

In this section, we explain the details for the implementation of Inverse Arbi-Q architecture. Subsection 5.1 introduces the state predictor and the calculation of responsibility signal for each module. Subsection 5.2 is about the reward predictor, we used a specific "modular reward" [START_REF] Samejima | Inter-module credit assignment in modular reinforcement learning[END_REF] for rewarding the module that is more suitable for the current sub-task, which encourages the architecture to be more concentrated to update to modules that better fit the current task. In subsections 5.3 and 5.4, we are more focused on the state-value function and the state-action function in the RL controller, as well as the action selection based on these two functions. Additionally, in order to make decisions more coherent, we adopted eligibility traces for each state, which are also introduced in this subsection.

State predictor

The dynamic function F i (s t+1 , s t , a t ) in the state predictor of module i gives the probability distribution of the newly observed state s t+1 based on the previous state s t and the action a t performed by the agent at step t.

F i (s t+1 , s t , a t : θ i ) = P i (s t+1 |s t , a t : θ) (2)
where s t ∈ {1, . . . , N } and a t ∈ {1, . . . , M } are discrete states and actions, i ∈ {1, . . . , n} is the index of modules. Specifically, this dynamic function is realized through a neural network with three fully-connected layers, where the vector of parameters θ represents the weights of the network. This network was trained by a collection of pairs of inputs (s t , a t ) and their corresponding output labels s ′ t+1 from trajectories τ = {s 0 , a 0 , . . . , s T -2 , a T -2 , s T -1 } of length T [START_REF] Nagabandi | Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning[END_REF]. The initial parameters in this neural network were set as a small random value with uniform distribution between 0.0 and 1.0. With the newly observed state s t+1 , the state prediction error δ i st of module i is calculated as follows:

δ i st = F i (j, s t , a t : θ i ) -c(j, s t+1 ), j ∈ {1, . . . , N } (3) c(j, s t+1 ) = 1 j = s t+1 , 0 otherwise. ( 4
)
Along with state prediction errors in each module, the responsibility signal λ i t of each module for the current sub-task is formulated as the gaussian softmax function,

λ i t = λi e -1 2σ 2 δ i st 2 n j=1 λi t e -1 2σ 2 δ j st 2 ( 5
)
λi t = λ i t-1 ρ n j=1 λ j t-1 ρ (6)
where the responsibility predictor λi t represents the prior knowledge or belief about module selection, which is used to maintain the temporal continuity of the selected module. Parameter ρ (0 < ρ < 1) controls the effects of past module selections on current decision-making. Parameter vector θ is updated as follows:

θ i = θ i + αλ i t δ i st ∂F i (s t+1 , s t , a t : θ i ) ∂θ i ( 7
)
where 0 < α < 1 is the learning rate.

Reward predictor

The reward function R i (r ′ , s t , a t ) provides an expected reward r ′ t based on the previous state s t and the action a t selected at step t. Traditionally, parameters in the reward predictor are updated by the reward prediction error, which comes from the error between reward prediction r ′ t and actually received immediate reward r t after each interaction.

As described in the previous section, the responsibility signal represents how responsible the modules are for the current situation; the temporal difference of the responsibility signals between two steps could be used to reflect whether the current module is more sensitive than the previously selected one. Based on this idea, we utilize a "modular reward" [START_REF] Samejima | Inter-module credit assignment in modular reinforcement learning[END_REF], which comes from the immediately received reward and an extra bonus for rewarding the module that better fits the current sub-task. Specifically, this bonus is calculated from the conduct of modular value function and the temporal difference in the responsibility signal, which encourage the learning system to be more focused on the modules they are responsible for, and lead to the optimality of the composite strategy for the entire task. Basically, the modular reward is calculated as:

ri t = r t + (λ i t+1 -λ i t )V i (s t+1 ) (8)
and we have the reward prediction error δ i rt as follows:

δ i rt = ri t -R i (r ′ , s t , a t ) (9)
Weighted by the responsibility signal and the learning rate, parameters in the reward predictor will be updated as follows:

R i (r ′ , s t , a t ) = R i (r ′ , s t , a t ) + αλ i t δ i rt (10)

RL controller

The RL controller in module i provides two functions: the state-value function V i (s t ) and the state-action-value function Q i (s t , a t ). The value function V i (s t ) gives the reward expectation of state s t at step t and the state-actionvalue function Q i (s t , a t ) provides the reward expectation of action a t in state s t . Technically, the goal of reinforcement learning is to improve the policy so that a maximal accumulative reward will be obtained in the long run [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. The basic strategy of reinforcement learning is to use this value function to estimate the state value under the current policy, and then to improve the policy based on the value function. We define the value function of the state s(t) under the current policy π as:

V i π (s t ) = E π ∞ k=0 γ k r(t + k) = E π (R t+1 + γV i π (s t+1 )|s t ) (11)
where γ is the discount factor that controls the effects of future rewards. Traditionally, R t+1 + γV i π (s t+1 ) is also called the "target value" of state s t . Combined with the state transition function F i (s t+1 , s t , a t ) and the reward function R i (r ′ , s t , a t ), we have the full Bellman equation [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] of value function:

V i π (s t ) = at∈A π i t (a t |s t )[R i (r ′ t , s t , a t ) + γ st+1∈S F i (s t+1 , s t , a t : θ i )V i π (s t+1 )] (12)
In order to make decisions in each step more coherent with previous ones, we recorded the eligibility traces e i t s for each state s, which are updated as:

e i t (s) = ηγe i t-1 (s) if s ̸ = s t ηγe i t-1 (s) + 1 if s = s t (13)
The temporal-difference (TD) error δ i vt comes from the deviation between the target value and the evaluated value:

δ i vt = ri t + γV i (s t+1 ) -V i (s t ) (14)
where ri t is the modular reward and the state value V i (s t ) will be updated with responsibility signal as:

V i (s t ) = V i (s t ) + αλ i t δ i vt e i t (s) (15)

Action selection

At each step, the action preferences follow a "joint policy", in which state-action values derive from the combination of RL controllers in each module. Specifically, action is decided based on the weighted summation of its value expectation Q i (s t , a t ) of future rewards in each module at state s t . For each candidate action a j ∈ A, (j = 1, . . . , M ), their values are calculated as:

Q(s t , a j ) = n i=1 λ i t [R i (r ′ , s t , a j ) + γ j∈X F i (s j , s t , a j : θ i )V (s j )] ( 16
)
where X is the set of possible states that the agent observes after taking action a j in state s. In a greedy policy, the action that has the largest value will be selected:

a t = arg max aj ∈A Q(s t , a j ) (17)
Furthermore, we use a softmax function to explore action space in a stochastic way, where the action a t is selected by

P i (a j |s t ) = e βQ(st,aj ) M k=1 e βQ(st,a k ) (18)
where β is set as trials/500 and controls the stochasticity of action selection.

6 Simulation

Settings

In order to investigate the effectiveness of Inverse Arbi-Q architecture, we use a food-gathering and predator-avoidance task derived from [START_REF] Sprague | Multiple-goal reinforcement learning with modular sarsa[END_REF][START_REF] Simpkins | Composable modular reinforcement learning[END_REF]. In this carrotrabbit-predator task, the environment is designed as a 5x5 grid world (as shown in Figure 2. The blue rectangle , purple triangle and green circle represent the rabbit, the predator, and the carrot respectively. Specifically, the rabbit searches carrots with one of eight possible one step actions: {north (N), east (E), south (S), west (W), northeast (NE), southeast (SE), northwest (NW), southwest (SW)} while avoiding being caught by the predator. When the rabbit moves in the location of any of the carrots, it receives a reward of 1.0 and a new carrot will be allocated to an otherwise position. In each step where the rabbit does not find any carrot, it gets a reward of -0.1 to represent increasing hunger. Meanwhile, the predator moves one step directly toward the rabbit while the rabbit moves two steps. If the rabbit avoids the predator, it will receive a reward of 0.5, and a reward of -1.0 if it is caught by the predator, which also means the termination of this trial. In this environment, the observation state of the rabbit and the predator are their absolute coordinates respectively. agent food predator Fig. 2. The food-gathering and predator-avoidance world.

Results

At the begining, we'd like to compare the performance of Inverse Arbi-Q with previous MRL approaches. Fig. 3(a) shows the average number of steps during 5000 trial epochs in 20 simulation runs. It can be seen that the Inverse Arbi-Q model works significantly better than Arbi-* approaches (Arbi-Q, Arbi-MBRL and Arbi-SARSA(0)) and GM-MBRL. Furthermore, we utilize another performance metric of "score" from [START_REF] Simpkins | Composable modular reinforcement learning[END_REF] to compare Inverse Arbi-Q with different models. Specifically, the calculation of a score follows this method: for each time that the rabbit finds a carrot, the score is defined as 1.0, however, if it was caught by the predator, the score is 0.0, in situations that the rabbit avoids the predator while no carrot was eaten, the score is set as 0.5. Fig. 3(b) shows the scores between Inverse Arbi-Q with other approaches. We can find that Inverse Arbi-Q receives more scores that other models.

Moreover, we'd like to know the performance of Inver Arbi-Q with incomparable reward scales. Here we increase the avoidance reward by 10 times to observe the performance of Inverse Arbi-Q in the incomparable rewards. As shown in Fig. 4, Inverse Arbi-Q is robust across different incomparable reward scales.

Conclusions and future work

In summary, we present the simple yet effective architecture Inverse Arbi-Q to scale up the RL to learn compositional strategies and to exert them appropriately in multi-task learning. Particularly, the responsibility signal, the outcome of a gaussian softmax function of prediction error in each module, was used to weight the outputs of multiple modules, as well as to gate the learning of each module. Furthermore, action preferences are calculated by a "joint policy", which comes from the combination of state-action values in each module multiplied by its responsibility signal. Moreover, the selective update mechanism enables the learning system to align incomparable reward scales between different modules.

Our ultimate goal with modular reinforcement learning is to facilitate the integration of multiple modules into a developmental way, in order to learn compositional strategies, as well as to generate appropriate behaviors according to the specific contexts. Nevertheless, a number of questions stands out as important targets for the next stage of research, such as: (i) the improvement of planning strategy or the proactive way in the evaluation of modular reward; (ii) enabling module selection and responsibility calculation in high dimensionality situations; (iii) the design of bidirectional (top-down and bottom-up) solutions for task decomposition and compositionality. Inspirations from continual learning and meta-learning approaches will be a promising direction for learning tremendous tasks effectively without creating additional modules.

Fig. 1 .

 1 Fig. 1. Schematic diagram of Inverse Arbi-Q architecture.

  (a) Averaged step per trial.(b) Averaged score per trial.

Fig. 3 .

 3 Fig. 3. Performance comparison with other MRL approaches.

Fig. 4 .

 4 Fig. 4. Performance of Inverse Arbi-Q with incomparable reward scales.