Andrei Mitriakov

Panagiotis Papadakis

Sao Mai Nguyen

Serge Garlatti

Staircase Traversal via Reinforcement Learning for Active Reconfiguration of Assistive Robots

Keywords: Cognitive robotics, learning-based control, obstacle negotiation, active stability, reinforcement learning, neural networks

Assistive robots introduce a new paradigm for developing advanced personalized services. At the same time, the variability and stochasticity of environments, hardware and unknown parameters of the interaction complicates their modelling, as in the case of staircase traversal. For this task, we propose to treat the problem of robot configuration control within a reinforcement learning framework, using policy gradient optimization. In particular, we examine the use of safety or traction measures as a means for endowing the learned policy with desired properties. Using the proposed framework, we present extensive qualitative and quantitative results where a simulated robot learns to negotiate staircases of variable size, while being subjected to different levels of sensing noise.

I. INTRODUCTION

Among a number of applications for service robots, there is a growing interest towards assisting activities of daily living, for instance via assistive robots [START_REF] Riek | Healthcare robotics[END_REF]. To be able to palliate the loss of autonomy of elderly or frail people, in the context of ambient assisted living (AAL), a robot is required to transport objects or humans (see Fig. 1 for a representative example) in an environment populated with various 3D obstacles such as steps and stairs. To enable mobility in 3D environments, a 2D environment structure typically used in path planning is overly restrictive if the robot requires to undertake novel tasks. The first possible application relates to an autonomous mobile robot (agent) navigating in indoor, usually multi floor, environments while accomplishing different tasks provided by a human or an algorithm. Such a robot can be integrated into a smart home with connected objects and mobile robots such as presented in [START_REF] Nguyen | Plug and play your robot into your smart home: Illustration of a new framework[END_REF] in the context of the leadership (Chaire) research program M@D whose research goal concerns the creation of a living environment for elderly or frail people whose loss of autonomy would be palliated by the use of connected objects and mobile robots (https://chaire-mad.fr/ la-chaire-md/).

A major challenge then amounts to autonomously and safely traversing (negotiating) staircases which constitutes the scope of this paper. Thanks to task similarity, once the first use-case has been successfully addressed, we could transfer the developed behavior to the second use-case of person transportation. Earlier works addressing the stair traversal problem [START_REF] Helmick | Multi-sensor, high speed autonomous stair climbing[END_REF], [START_REF] Hesch | Descending-stair detection, approach, and traversal with an autonomous tracked vehicle[END_REF] were largely based on solutions customized to specific, previously known robot kinematics and based on accurate stair parameter estimates. Broadly speaking, this complicates portability to platforms with different or unknown kinematics or poor environment observability, weakly taking into consideration the dynamics of physical interaction. On the other hand, learning-based control of flippers [START_REF] Paolo | Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning[END_REF] allows to make less restrictive hypotheses with respect to variation in the environment or the robotic kinematics and focuses more on task constraints such as safety [START_REF] Pecka | Fast simulation of vehicles with non-deformable tracks[END_REF]. A more recent approach [START_REF] Ilhan | Autonomous stairwell ascent[END_REF] for navigation in multi floor environment, that has shown good performance on more than hundreds of staircase instances, relies on the passive adaptation to obstacles which cannot be generalized to assistive robots due to high movement stochasticity.

Following up on earlier work [START_REF] Mitriakov | Learningbased modelling of physical interaction for assistive robots[END_REF], we present in this article a learning-based control approach for the problem of staircase negotiation (see Fig. 2), applied to an articulated, tracked robotic vehicle. Employing a reinforcement learning-based (RL) pipeline, our focus here is on the constraints of the control policy that we wish to satisfy, such as safety, traction, execution speed. To demonstrate the generalization ability of the framework, the policy is learnt and tested for a variety of staircases of different step heights and angles, while evaluating its capacity to cope with different levels of noise in its sensory data. This initial work on simulation can serve as paradigm for transfer to different platforms but it can also provide insights when modelling control for newly encountered situations.

The remaining of the paper is organized as follows. Section II presents related works for the problem of stair negotiation together with stability assessment techniques. In section III the proposed reinforcement learning framework is presented. In section III-B we formulate the notion of safe stair traversal and design the reward function. Finally, in section IV the experimental setup is presented together with the obtained results.

II. RELATED WORK

Following various attempts for the kinematic modelling of tracked robots, the authors of [START_REF] Martínez | Approximating kinematics for tracked mobile robots[END_REF] were among the first to propose an effective motion controller. To improve the capability of negotiating more complex obstacles, such robots are often equipped with articulated front and/or rear flippers. These flippers improve obstacle negotiation skills but they inevitably increase the complexity of robot control due to the addition of degrees of freedom while accounting for surface geometry. We can broadly distinguish two main approaches for the control of such robots, namely, learning-free and learningbased.

A. Learning-free approaches

Among learning-free approaches, a terrain traversability (we refer the reader to [START_REF] Papadakis | Terrain traversability analysis methods for unmanned ground vehicles: A survey[END_REF] for an extensive review on the notion of traversability) study was performed in [START_REF] Xie | Motion control technology study on tracked robot with swing arms[END_REF] as well as a motion control algorithm for a tracked robot with two front flippers. This work assumes knowledge of the exact robot kinematics which is not always feasible for commercial robots, while motion control is only applied to the traversal of a palette. The authors of [START_REF] Nagatani | Semi-autonomous traversal on uneven terrain for a tracked vehicle using autonomous control of active flippers[END_REF] elaborate the idea of tangential orientation of flippers during obstacle traversing along with a motion planning framework. They suggest that this allows to negotiate any uneven structure within the limits of robot capabilities, yet based on exhaustive laser sensory data about the environment. In [START_REF] Suzuki | Remote control system of disaster response robot with passive sub-crawlers considering falling down avoidance[END_REF] the authors introduce a framework capable of dealing with obstacles exceeding obstacle negotiation capabilities of [START_REF] Nagatani | Semi-autonomous traversal on uneven terrain for a tracked vehicle using autonomous control of active flippers[END_REF]. They endow a tracked robot with passive flippers, which apply force against the traversing obstacle, and a warning system based on the normalized energy stability margin (NESM). This stability criterion, which estimates deviation from the most stable position, became widely used in robotics while keeping comprehensibility and will be discussed later in detail. The previous framework considers precise knowledge of robot dynamics and supposes that a human expert controls the robot.

To the best of our knowledge, the most elaborate and recent learning-free method was proposed in [START_REF] Ilhan | Autonomous stairwell ascent[END_REF] and delegates the problem of stair negotiation to passive adaptation of the mechanical platform. The latter comprises six arc-shaped legs rotating under a certain control algorithm used both in outdoor and indoor multi floor navigation. This platform is more suitable for search and rescue missions for what it was initially devoted to [START_REF] Saranli | Rhex: A simple and highly mobile hexapod robot[END_REF]. Obviously, its utility is limited in the context of assistive robotics, due to the stochasticity of its movement that could cause considerable slipping and shaking.

B. Learning-based approaches

Employing learning-based control becomes more relevant in cases where the true kinematics of such robots are harder to obtain or approximated in simulation [START_REF] Papadakis | 3D Mobility Learning and Regression of Articulated, Tracked Robotic Vehicles by Physics-based Optimization[END_REF]. The authors of [START_REF] Paolo | Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning[END_REF] were among the first who used the learning-based approach and applied deep deterministic policy gradient (DDPG) to the staircase negotiation problem in an end-to-end fashion. They mapped input data from an inertial measurement unit, front and back cameras with flipper commands using a convolutional neural network (CNN). The main shortcoming of this work is the high processing cost induced from the employed CNN and slow convergence to the optimal policy. One of the most elaborate approaches for flipper control in the scenario of palette traversal is proposed by [START_REF] Pecka | Autonomous flipper control with safety constraints[END_REF]. The main contribution concerns the use of contextual relative entropy policy search [START_REF] Kupcsik | Model-based contextual policy search for data-efficient generalization of robot skills[END_REF] by introducing safety constraints into the optimization problem. A small number of episodes was deemed sufficient for learning to negotiate a palette using a RL algorithm, estimating safety by a physics-based simulator. Often, knowledge of forward/inverse robot kinematics cannot be obtained for commercial robots. In such cases, a learning-based approach appears more suited for obtaining the policy of flipper control for the task of obstacle negotiation.

C. Safety assessment

Moreover, the policy used for traversing terrains has to ensure safety of the robot motion in some manner. For example, safety of actions could be estimated and a robot could be programmed to avoid such unsafe actions [START_REF] Suzuki | Remote control system of disaster response robot with passive sub-crawlers considering falling down avoidance[END_REF]. Another method is to incorporate a safety criterion inside a RL algorithm [START_REF] Pecka | Autonomous flipper control with safety constraints[END_REF]. However, this does not generalize well across all RL algorithms. Another alternative considers integration of penalty terms into the reward function [START_REF] Coraluppi | Brief risk-sensitive and minimax control of discrete-time, finite-state markov decision processes[END_REF] allowing to obtain a policy with desired properties with a wider generalization across RL algorithms. In this paper, we consider this method and incorporate stability and traction criteria into the reward

D. Reinforcement learning preliminaries

In this section, we concisely recall the main components of reinforcement-based learning.

In a typical RL framework [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF], at each discrete time step t and a given state s ∈ S, an agent selects and applies an action a ∈ A with respect to its policy π : S -→ A, obtains a scalar reward r ∈ R and observes a new state s ∈ S. Although A can be discrete or continuous, the latter choice seems more adapted to use-cases requiring variable accuracy depending on the state. For instance, in our use-case, the robot may encounter situations where it is necessary to adapt flippers drastically avoiding step-wise approaching to a desired flipper angle as it would happen with the use of discrete action space. The trajectory τ = {s 0 , a 0 , ..,

s T -1 , a T -1 }, called the roll-out
where T is the number of time steps in the roll-out, defines the set of consecutive state-action pairs. A policy can either be stochastic a t ∼ π θ (a t |s t ) or deterministic a = π(s), depending on the problem.

RL methods could be divided into two groups which are value-based and Policy Search (PS) methods. The first makes an agent evaluate the quality of an action taken in some state by calculation of a value function in accordance with the expected cumulative reward.

Methods of the second group [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] tend to be more advantageous in robotic applications [START_REF] Deisenroth | A survey on policy search for robotics[END_REF]. The reasons for this are ease of implementation thanks to reduced complexity of policy approximation and the fact that small changes in policy do not lead to a drastic changes in behaviour in contrast to value-based methods where an agent can start to exploit a significantly less favorable policy after policy update.

In previous years, policy gradient (PG) algorithms attracted significant attention because of their direct policy optimization in contrast to value-based methods where the policy is obtained via its relation with the learnt approximator. The main idea behind these algorithms is the gradient ascent over the policy approximator parameters θ using the policy gradient ∇ θ J(π θ ),

θ k+1 = θ k + α∇ θ J(π θ k ) ( 1
)
where α is the learning rate and J(π θ ) is the expected return that could be written as follows:

J(π θ ) = E τ ∼π θ [R(τ )] (2)
R(τ ) is the cumulative reward over a trajectory τ .

R(τ ) = T -1 t=0 r t (3)
Finally, r t is the reward obtained by the robot during a time step t, its value depending on the reward function design and will be discussed in the following section III-B.

Among various possible policy gradient algorithms, we have selected Proximal Policy Optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF], [START_REF] Hill | Stable baselines[END_REF] as a state-of-the-art algorithm that exhibits good trade-off between ease of tuning, sample complexity, ease of implementation and good performance [START_REF] Kidziński | Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments[END_REF]. This algorithm updates the policy with stochastic gradient ascent while keeping a new policy close to the previous one that could be completed either by penalisation of Kullback-Leibler divergence or specialized clipping in the objective function.

III. REINFORCEMENT LEARNING FRAMEWORK

A. Flipper Control Problem Formalisation

Robot control could be divided into reactive main track and flipper control [START_REF] Suzuki | Remote control system of disaster response robot with passive sub-crawlers considering falling down avoidance[END_REF], [START_REF] Pecka | Autonomous flipper control with safety constraints[END_REF]. Reactivity implies that the controller directly maps sensor input to desired actions. Hereafter, we assume an independently developed main track controller (i.e. a global path planner) and focus only on the development of flipper control whose objective is to perform a mapping between the pose of a robot on a staircase and the appropriate flipper commands.

In relation to the addressed problem, we consider the robot to select two rotation angles ψ f ront , ψ rear forming an action vector a = (ψ f ront , ψ rear ) ∈ [ψ min , ψ max ] 2 for the front and rear flippers while constant effort is applied to each track moving it forward. The state vector s = (p x , p y , φ f ront , φ rear ) ∈ S consists of the distances p x , p y of the robot centroid to the next step nosing along the axes X and Y (see Fig. 3) where the origin corresponds to the step center. It further includes the flipper angles φ f ront and φ rear in each flipper frame which belong to [φ min , φ max ] where φ min < ψ min < ψ max < φ max .

In accordance to the reactive control architecture, the task to be learned consists of flipper reconfiguring so as: (i) to enable the robot to reach the top of the staircase and (ii) to fulfil certain policy constraints such as traction and stability. This is performed with application of PPO where the agent's goal is to maximize the cumulative return (Eq.3). Learning efficiency is sensible to the reward function design as well as learning policy that can incorporate desired or unpleasant characteristics by mistake. We introduce such reward functions which bring safety characteristics to the policy in the next section and, afterwards, analyze their impact on stability in comparison with the "default" reward function elaborated for stair negotiation within discussed problem formulation.

Our contribution of this paper is summarized as follows: (i) we apply Proximal Policy Optimization (PPO) to learn flipper control for the stair traversal task in simulation (see Fig. 2) and (ii) we examine the use of stability and traction measures during policy learning and (iii) we show the generalisation capability of our policy by varying the terrain parameters and simulating sensory noise.

B. Reward function design

1) Baseline/default reward function: The first and most simple examined reward function remunerates the robot according to the travelled distance on the staircase. It receives the positive reward r t = ∆x t / D max where ∆x t is the travelled distance to the goal on the ground, stair or upper landing at time step t, and D max is the total distance from the starting position to the goal. We do not assign the positive reward twice

R(τ ) = T -1 i=0 ∆x t D max (4)
That total reward hence varies from 0 to 1 accordingly to an applied policy. If the robot gets closer than D 2 without appropriate flipper actions, it could be stuck (cumulative reward 0), thus, we start to provide the reward to evaluate policy performance once the robot is close enough to the first step. The episode is considered terminated if the distance D 4 is crossed on the upper landing (cumulative reward 1). D 4 is chosen to be equal to d+ l 2 , because we want to assure that the robot loses the contact with the stair at the end of the episode, and the traversal is successfully accomplished.

2) Normalized Energy Stability Margin (NESM): Following the idea of applying negative rewards to unsafe states in Markov Decision Processes [START_REF] Coraluppi | Brief risk-sensitive and minimax control of discrete-time, finite-state markov decision processes[END_REF], we incorporate a negative penalty term by introducing a stability criterion into the reward function. In detail, we employ the Normalized Energy Stability Margin (NESM) [START_REF] Hirose | Normalized energy stability margin and its contour of walking vehicles on rough terrain[END_REF], [START_REF] Garcia | A comparative study of stability margins for walking machines[END_REF] that is based on the fact that the robot rotates around a support line when tumbling/tipping over. This margin E is usually deduced as the difference between the maximum centroid height and its current height [START_REF] Suzuki | Remote control system of disaster response robot with passive sub-crawlers considering falling down avoidance[END_REF].

= H max -H (5)
Based on this measure, we associate robot instability I as equal to the current height H over the surface A (see Fig. 4). Thus, when I = 0 the robot is considered as maximally safe, whereas when I = H max = cos(β) d 2 + ( l 2 ) 2 -ld cos(π -|φ min |) -h where β is the stair inclination, the robot is maximally unsafe. The value H max is the distance of the robot centroid to the surface A Fig. 4: NESM notations when the robot is pivoting over the lower support polygon foothold and φ rear = φ min . We consider the negative reward based on NESM to be -I/1m where division by 1m ensures independence from the chosen units of measurement. Thus, the reward per time step r t = ∆x t / D max -I. Every time step the robot can obtain a positive reward along with a negative one if its gravity center G deviates from the surface A , whenever the robot centroid projection is located within the first and last nosings on the surface A .

3) Support polygon (SP):

The idea of maximizing the contact of flippers to the traversing surface is commonly employed as a measure of traversability. Authors of [START_REF] Nagatani | Semi-autonomous traversal on uneven terrain for a tracked vehicle using autonomous control of active flippers[END_REF] suggested to set the orientation of flippers tangential with respect to the traversed surface during obstacle negotiation in order to maximize traction. In [START_REF] Suzuki | Remote control system of disaster response robot with passive sub-crawlers considering falling down avoidance[END_REF] flippers are even pushed against the traversed surface in order to enforce contact.

It is reasonable to assume that a minimal contact surface area should be maintained, between the flippers and the obstacle being negotiated, otherwise the exerted control actions might not have the desired effect. On the other hand, depending on the geometry of the robot and the staircase, maximizing surface contact might drive the robot to poses from which it would be impossible to successfully traverse the staircase.

Knowing the actual size proportions between a conventional staircase and commercial tracked robotic vehicles, maximizing traction while traversing a staircase indeed amounts to moving at an angle equal to the inclination of the staircase. Therefore, we assume that the maximum traction with the surface is achieved when the robot maximizes its SP. The maximum SP surface area P max is then simply the projected surface area of the robot on the surface A when the robot is located on the staircase with a flat configuration, namely, with flat flippers.

Letting the actual SP surface area be P , in order to favor a control policy where the robot has increased traction, a negative reward is considered at each time step r p = -(P max -P )/(P max ). Thus, the overall reward per time step is r t = ∆x t / D max + r p .

IV. EXPERIMENTS

In this section, we present the experiments that we conducted to evaluate the RL control pipeline as was described earlier in Section III, quantitatively as well as qualitatively, by varying the form of staircases and in the presence of different levels of simulated noise.

Main experimentation protocol: We simulate the controllearning pipeline of the robot in the Gazebo environment (gazebosim.org). The tracked robot is endowed with front and rear flippers and its task is to learn how to mount a five-step staircase whose tread and rise are randomly chosen. Among various alternatives for imitating mobile robot tracks, we employ the common approximation of representing tracks by an ensemble of overlapping, non self-colliding, tracked wheels (cf. [START_REF] Pecka | Fast simulation of vehicles with non-deformable tracks[END_REF]).

Experiments were performed repeatedly for a given configuration of a staircase. At each time step the robot observes the state vector s. The state vector comprises relative distances to the next step nosing p x and p y . The former varies from the minimum distance p min The robot starts at a front-parallel position with respect to the staircase at a distance of D 1 + D 2 . Once the robot travels D 1 , upon application of an action the robot obtains a reward r proportional to the travelled distance and potentially penalized depending on the chosen reward function (see section III-B). A traversal experiment is deemed as successful if the robot succeeds in traversing over D 4 on the upper landing from the last nosing. The finite-horizon undiscounted return of one trajectory represents the proportion of travelled distance and could vary from 0 to 1 if we do not model safety. This sequence is repeatedly executed until the end of an episode that is set to occur after a maximum number of 100 time steps. If the standard deviation of 25 most recent robot position estimations along the X-axis drops under 0.01 m, the episode stops assuming that the robot has been "stuck". All experiments have 10000 time steps that compares with 250 episodes while remaining parameters were chosen in accordance with [START_REF] Fujimoto | Addressing function approximation error in actor-critic methods[END_REF].

Employed policy representation: We wish to map the robot state represented by a vector s to a vector of actions a under the policy π, which requires a good policy function approximation. Artificial neural networks have demonstrated remarkable results in machine learning as well as in rein-forcement learning in particular, and could be well suited for nonlinear function approximation [START_REF] Paolo | Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning[END_REF], [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF], [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF]. Having small input and output vectors of estimated robot state and corresponding actions, we approximate the policy with a multilayer perceptron consisting of two layers where each of them has 32 neurons with tanh activation function, in contrast to [START_REF] Paolo | Towards continuous control of flippers for a multi-terrain robot using deep reinforcement learning[END_REF] which employs a complex CNN because of necessity to treat complex image input data. Judging from the obtained qualitative data, this policy function approximator provided satisfactory results for the task under consideration.

Policy testing: A learnt policy is evaluated on stair configurations that do not necessarily appear during learning. We perform this after the first policy update and, then, every 8 policy updates on three stair configurations with parameters presented in Table I. During this assessment, we calculate mean cumulative reward, stability and traction over 3 cases which correspond to small, medium and big staircases also called configurations.

TABLE I: Stair configurations used in evaluation

Type

Step

A. Reward function evaluation

Along with the quantitative results presented later in this section, we bring a video that provides representative results of the learnt policies for staircases of varying difficulty, accessible in (https://partage.imt.fr/index.php/s/pBSzKaoeDnqFSyA/ download).

For every type of reward function, we performed 3 training repetitions. During experiments, cumulative reward, NESMbased instability measure I and estimated robot projection ratio P/P max were measured every 20 episodes. Fig. 5 presents the smoothed average cumulative reward per learning episode and testing, using min-max bands. Hereinafter referred smoothing is accomplished using an exponential moving average (EMA) with the coefficient 0.95 and 0.8 for learning and testing curves respectively.

With respect to learning performance (Fig. 5 (a)), the NESM-based and SP policies converge as early as the unsafe policy (around 175 episodes) and their mean episode returns vary from 0.5 at the beginning up to 1 at the end. We can observe the presence of non-zero episode cumulative rewards in the beginning that could be provoked by random actions that the robot makes in the beginning of training which allows it to reach half of its chassis length on the staircase. On the other hand, the optimal reward values per episode are slightly lower for policies that incorporate safety, because of unavoidable necessity to perform less safe actions. We can further observe that max bands of learning curves reach the maximum return after 100 episodes for all types of reward function. This means that policies are yet capable to perform appropriate flipper control, but min bands are constantly increasing due to behaviour improvement on unseen staircases. By 200 episodes, the mean curves converge and this could be considered as the end of learning.

With respect to testing performance (Fig. 5 (b)) for stair configurations of varying difficulty (cf. Table I), we observe that the mean reward tends to become higher during learning, which suggests a successful learning progress, independently of the staircase. Moreover, the mean test learning curve for SP policies constantly increases the cumulative reward Rτ that also confirms steady learning of a policy that satisfies safety constraints.

With respect to the results shown in Fig. 6 (a), we can deduce that the policy safety does not change too much in the beginning of the learning due to possible difficulties to negotiate the stair. Instability for all 3 policy types starts to increase after 100 episodes which correlates to learning curves whose max bands reach the maximum cumulative reward 1. We can argue that a safety drop is unavoidable to perform in order to accomplish the main task of stair traversal. By the end of the training, the lowest instability corresponds to NESMbased and default (unsafe) policies, while the instability of SP tends to be higher. Test instability curves (see Fig. 6 (b)) show the same behaviour, the SP policy is less safe by the end of training, while the NESM-based and unsafe policies have the same instability values. One notable difference on test learning curves (Fig. 6 (b)) in the beginning of learning, namely, is that the SP-based policy is considerably more unstable compared to the two other policy types. The projection assessment curves (see Fig. 7 (a)) show that the NESM and SP-based policies tend to better ensure a flat robot configuration compared to the default policy. The NESM and SP policies perform better in most cases than the unsafe policy after episodes, NESM-based and projection penalisation terms could be considered as improving traction. The SP policy tends to increase its projection meanwhile the NESM-based policy decreases it. Thus, the SP policy learns desired behaviour of flippers. In accordance with the tests in Fig. 7 (b), the SP-based policy tends to increase traction with the stair, two other policy types decrease their projection that correlates to learning projection curves. It seems surprising at first inspection that a safety-based NESM and unsafe policies exhibit the same behaviour (see 6 (b), 7 (b)) in terms of instability and traction. However, we penalize elevation of the robot over the stair and at the same time the robot has to be relatively flat in order to negotiate a staircase. Thus, the robot does not visit states which may drive it to unsafe state and this constraint is automatically satisfied.

B. Resilience of policy learning to noise

Our targeted application in the context of AAL where the robot is integrated within a smart space/house [START_REF] Papadakis | System design for coordinated multi-robot assistance deployment in smart spaces[END_REF], suggests a relatively well-controlled environment. The part of the state of the robot concerning its relative position with respect to the Thus, noise should be accounted for during learning at the moment when we provide the robot with estimation of its relative position to the next step. We simulate corrupted estimation of p x , p y via a Gaussian error model, namely, p i ∼ N (p i , σ x ) where p i is the noisy position, p i is the true position given by the simulation environment, σ i = ξ • V max is the chosen standard deviation, i denotes either X or Y axes, V max stands for the maximum possible observation value and ξ is a parameter modelling different noise levels. We experiment with three levels of noise, abbreviated as low, medium and high, that correspond to ξ = {0.1, 0.3, 1.0}.

Firstly, three sets of experiments were conducted for each noise level using the baseline/default reward function. We can observe (see Fig. 8 a)) that policies learned with low or medium noise converge as the default policy. On the other hand, high noise in state estimation decreases the learning convergence rate, but eventually, the same cumulative reward is attained after 250 episodes. During testing (see Fig. 8 (b)), high noise levels seemed to have a more notable impact on the final cumulative reward, yet, we do not expect such extreme

V. CONCLUSION

In this paper we have presented a RL framework for the staircase negotiation. Within 200 learned trajectories which correspond to 60 policy updates, the robot learned how to safely traverse staircases consisting of 5 steps with varying rise and stair angle in representative real-world ranges. We have designed and compared three alternative reward functions. The NESM-based policy did not behave significantly differently with respect to the unsafe policy. Incorporation of the projection maximization into the reward function produced a desired behaviour in terms of flipper sticking to the stair. A policy learnt without safety criteria presented a good generalization capability when applied to newly encountered stair parameters. Finally, while noisy sensory data may decrease convergence rate the final control policy attains the maximum reward in most cases. In our subsequent work, we plan to couple these results with [START_REF] Mitriakov | Learningbased modelling of physical interaction for assistive robots[END_REF], and apply it on a real robot allowing it to autonomously navigate in 3D indoor environments.

Fig. 1 :

 1 Fig. 1: Example platform and application scope: Scewo Bro wheelchair traversing a staircase (https://scewo.ch/en/bro/).

Fig. 2 :

 2 Fig. 2: Stair traversal task illustration

Fig. 3 :

 3 Fig. 3: Relative distance of the robot centroid to the next step edge/nosing

x= 0

 0 to the horizon of step observation p max x = 1m. Relative distance along Y axis has as its maximum value the biggest height p max y which is assumed to be the negotiation robot capability critical , thus, p max y = H critical = h + d • sin π 4 and its minimum p min y = -d • sin π 4 . In every episode, irrespective of the results of the previous episode, a new staircase of 5 steps is generated in the following way. First, a random rise size H rise ∈ [H min rise , H max rise ] is sampled where H min rise = 0.5•H critical and H max rise = 0.8•H critical . Second, the stair angle β ∈ [β min , β max ], where β min = 20 • and β max = 30 • , is randomly sampled. Last, the tread size is calculated as D tread = H rise / tan(β).

Fig. 5 :

 5 Fig. 5: Smoothed cumulative reward R(τ ) during (a) learning and (b) testing

Fig. 6 :

 6 Fig. 6: Smoothed instability I during learning and tests

Fig. 7 :

 7 Fig. 7: Smoothed projection P/P max during learning and tests

Fig. 8 :

 8 Fig. 8: Smoothed cumulative reward R(τ ) during and tests, noising

TABLE II :

 II Resilience to noise for safety constrains during learning

	Measure		Level of noise
	type	Without	Low	Medium	High
	I	0.017 ± 0.021	0.015 ± 0.017	0.013 ± 0.025	0.005 ± 0.009
	P/Pmax	0.962 ± 0.047	0.986 ± 0.007	0.966 ± 0.029	0.967 ± 0.026

TABLE III :

 III Resilience to noise for safety constrains during testing

	Measure		Level of noise
	type	Without	Low	Medium	High
	I	0.018 ± 0.01	0.022 ± 0.008	0.017 ± 0.01	0.015 ± 0.01
	P/Pmax	0.991 ± 0.003	0.99 ± 0.003	0.992 ± 0.002	0.99 ± 0.003

VI. ACKNOWLEDGEMENTS

The present work is performed in the context of the leadership program M@D (Chaire Maintien@Domicile), project RE-ACT (Physical Interaction Models of Companion Robots) financed by Brest Métropole and the region of Brittany (France) and supported by project VITAAL (Vaincre l'Isolement par les TIC pour l'Ambient Assisted Living), cofinanced by European Regional Fund (FEDER).


Benoît Pardoen
email: benoit.pardoen@entpe.fr

Pierre Bésuelle

Stefano Dal Pont

Philippe Cosenza

Jacques Desrues

Effect of claystone small-scale characteristics on the variability of micromechanical response and on microcracking modelling

Keywords: Modelling, Clay rock, Micromechanics, Homogenised response, Microcracking

Argillaceous rocks have a complex and heterogeneous structure at different scales. At the scale of the mineral inclusions embedded in a clay matrix, the deformation generally induces microcracking and material damage. Modelling the latter requires to take into account microscale characteristics and their effect on the micromechanical response. This response can be used in double scale approach to predict material behaviour at larger scale. Thus, heterogeneous microstructures of a claystone are generated with a distribution of morphological properties satisfying experimental observations. The overall microscale material behaviour under solicitation is obtained by numerical homogenisation. Then, the variability of the material response is studied with regard to small-scale characteristics. In terms of deformation and failure, a dominant shear deformation mode and decohesion between grains are observed. The decohesion induces microcracking in the microstructure and strain softening of its overall response.

Introduction

Clayey rocks are considered as host rocks for deep geological repository of nuclear wastes. The role of material microstructure on the mechanical behaviour at macroscopic scale is a crucial issue. Clay rocks have a complex heterogeneous structure over a wide range of scales. For deformation, mechanical damage, and macroscopic failure, the most relevant scale is the one of the mineral inclusions and microcracking, i.e. the micrometric scale [START_REF] Desbois | Deformation in cemented mudrock (Callovo-Oxfordian Clay) by microcracking, granular flow and phyllosilicate plasticity: insights from triaxial deformation, broad ion beam polishing and scanning electron microscopy[END_REF]. In the following, the material behaviour and its characteristics are considered at this scale. The mineralogical properties include grain morphology and mineral phase proportions. Defining realistic microstructure allows to derive the overall microscale behaviour by computational homogenisation. The material behaviour variability is then analysed regarding microstructure variabil-ity, microstructure size, and grain morphology. Hereafter, the studied clay rock and its microstructure are firstly described. Secondly, the generation of representative numerical microstructures is built from realistic characteristics obtained based on experimental characterisation. Thirdly, the used microscale model is briefly presented. Lastly, the clay rock behaviour is modelled leading to the reproduction of mechanical response, deformation, micro-damage, and microcracking that depend entirely on the microstructure. The behaviour variability is also investigated with regard to microscale properties.

Clay rock microstructure

The considered material is the Callovo-Oxfordian claystone (COx) studied by the ANDRA in France for nuclear wastes repository. The COx claystone is a crossanisotropic, low-permeable, and indurated shale with quasi-horizontal bedding planes. The mineralogical characteristics and the generation of representative microstructures are detailed in this section.

At microscale, the structure of the COx claystone is heterogeneous due to the presence of several mineral types. In the following, the considered small scale is the one of the mineral inclusions and experimental evidences (Yven et al., 2007;[START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF][START_REF] Cosenza | Effect of the local clay distribution on the effective elastic properties of shales[END_REF][START_REF] Armand | Fundamental aspects of the hydromechanical behaviour of Callovo-Oxfordian claystone: From experimental studies to model calibration and validation[END_REF] are used to represent the claystone microstructure in a realistic manner. The zone of interest is the COx clay-rich unit (located at 490m depth). It is a zone composed of a high clay mineral content in the form of a porous clay matrix which embedded non-porous mineral inclusions [START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF]. Its main mineralogical characteristics are detailed hereunder with average values summarised in Table 1. These values are derived from 2D vertical material sections (SEM and X-ray micro-CT) perpendicular to the bedding planes [START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF].

-Mineralogical composition: Tectosilicates (mostly quartz) 15-25%, carbonates (mostly calcite) 25-35%, heavy minerals (pyrite) 0-3% and clay minerals 40-60% [START_REF] Armand | Fundamental aspects of the hydromechanical behaviour of Callovo-Oxfordian claystone: From experimental studies to model calibration and validation[END_REF]. -Size of mineral inclusions: It ranges from several to a few hundreds of micrometers [START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF]. Quartz grains are larger than carbonates, in average, and pyrite grains are the smallest. -Morphology of mineral inclusions: In average, carbonate inclusions are more elongated than quartz (2D elongation =1 if round, <<1 if elongated in Table 1 and Fig. 1c). The average preferential orientation is horizontal (0°, Fig. 1d) and thus parallel to the bedding planes. -Characteristic size: For morphological characteristics, the size and length of a Representative Elementary Volume (REV) are V REV ≈0.001mm³ and L REV ≈100μm [START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF][START_REF] Cosenza | Effect of the local clay distribution on the effective elastic properties of shales[END_REF].

Furthermore, deformation and cracking mechanisms occur at microscale. Potential decohesion mechanisms around mineral inclusions and microcracking within the clay matrix develop in the claystone [START_REF] Desbois | Deformation in cemented mudrock (Callovo-Oxfordian Clay) by microcracking, granular flow and phyllosilicate plasticity: insights from triaxial deformation, broad ion beam polishing and scanning electron microscopy[END_REF][START_REF] Bésuelle | Mesure de champs de déformation dans l'argillite du Callovo-Oxfordien à l'échelle du micron[END_REF]. Contacts between clay matrix and non-clay inclusions serve as microcrack attractors. Then, intergranular microcracking paths develop through the clay matrix [START_REF] Desbois | Deformation in cemented mudrock (Callovo-Oxfordian Clay) by microcracking, granular flow and phyllosilicate plasticity: insights from triaxial deformation, broad ion beam polishing and scanning electron microscopy[END_REF]. From the above mineralogical characteristics, it is possible to generate realistic microstructures. 2D microscale elementary areas (EA) are developed to represent the grains and the failure modes at small scale. Heterogeneous microstructures are generated with periodic random Voronoï tessellation. It allows to define the solid grains, each cell representing an inclusion or a clay grain (see Fig. 1a). The generation method is adapted from van den Eijnden et al. ( 2017) to enrich the mineralogical characteristics [START_REF] Pardoen | Accounting for small-scale heterogeneity and variability of clay rock in homogenised numerical micromechanical response and microcracking[END_REF]. A variability of the latter is also considered to reproduce the observed distribution of clay and mineral contents, orientation, and elongation. Examples of realistic microstructures of COx claystone are visible in Fig. 1a and Fig. 3a.

Microscale model

The microscale model contains the details of the material microstructure and is embedded in a double-scale finite element (FE²) approach [START_REF] Frey | A two-scale poromechanical model for cohesive rocks[END_REF]van den Eijnden et al., 2016van den Eijnden et al., , 2017)). The overall response of periodic microscopic EA is derived by microscale finite element computation and then used as a homogenised implicit constitutive model at macroscale (by computational homogenisation). The scale transition is realised by enforcing periodic boundary conditions on the 2D microscale model (under plane strain conditions) which result of the macrostrain. Hereafter, the focus is on the microscale model definition and its ability to reproduce the behaviour of clay rock, including the deformation processes at grain scale.

The generated EAs (Fig. 1a) are defined as assemblies of (isotropic linear) elastic deformable continuous solid grains in interaction with cohesive crack models. The latter are damageable cohesive interfaces which permit material softening due to deformation. Thus, the non-linearity of the material behaviour is concentrated at grain contacts. The cohesive laws in both tangential and normal contact directions are given by: Following the above definitions, large deformations may develop at microscale and are mainly concentrated at grain contacts. Potential microcracks are considered at grain interfaces which allows two failure types as observed in the clay rock [START_REF] Desbois | Deformation in cemented mudrock (Callovo-Oxfordian Clay) by microcracking, granular flow and phyllosilicate plasticity: insights from triaxial deformation, broad ion beam polishing and scanning electron microscopy[END_REF][START_REF] Bésuelle | Mesure de champs de déformation dans l'argillite du Callovo-Oxfordien à l'échelle du micron[END_REF]: decohesion around inclusions (inclusion-clay contacts) and intergranular microcracking within the clay matrix (clay-clay contacts). Grain breakage is not considered. However, numerous possible microcrack paths can develop within the clay matrix. Furthermore, only crack porosity is considered at grain interfaces.

𝑐 𝑡 = 1 𝐷 𝑡 (1 -

Modelling of clay rock behaviour

The micromechanical properties of the heterogeneous EAs are chosen to reproduce the clay rock behaviour (Table 2). Each mineral constituent is assigned linear elastic properties [START_REF] Ahrens | Mineral physics and crystallography: a handbook of physical constants[END_REF][START_REF] Robinet | Effects of mineral distribution at mesoscopic scale on solute diffusion in a clay-rich rock: Example of the Callovo-Oxfordian mudstone (Bure, France)[END_REF]van den Eijnden et al., 2017) and homogeneous interface properties are assumed (van den Eijnden et al., 2017;[START_REF] Pardoen | Accounting for small-scale heterogeneity and variability of clay rock in homogenised numerical micromechanical response and microcracking[END_REF]. The clay rock micromechanical behaviour is analysed through the responses of EAs subjected to loading. A biaxial compression is numerically modelled in 2D plane strain state with a first isotropic confining phase followed by a vertical compression (enforced global homogenised vertical strain ε 1 ) under constant lateral confining pressure σ 3 . The variability of material response and deformation processes under deviatoric solicitation are analysed. Stress-strain deviatoric response curves are detailed in Fig. 2 for σ 3 =12MPa. They present the evolution of the global deviatoric stress q=σ 1σ 3 with the homogenised vertical ε 1 and lateral ε 3 strains of the EAs. Numerical results indicate that the microscale overall response shows some dispersion due to the variability of the microstructures (i.e. variable inclusion positions, mineral contents, and grain morphology). The results in Fig. 2a highlight the response scattering related to the randomness of inclusion positions, morphology, and mineral contents (random EA generation). In Fig. 2b, the mineral contents are fixed with 50% of clay and only the range and the average response curves are detailed. The highlighted variability suggests that the chosen EA size, based on grain morphology, is not sufficient regarding the mechanical response.

The effect of microstructure size is studied for different number of cells in the EAs. 50, 100, and 250 cells lead to mean characteristic lengths of L EA = 50, 70, and 105 μm (Fig. 3a). The other micromechanical parameters are kept from Tables 12, with 50% of clay. For each case, the range of response curves (over 10 EAs) is illustrated in Fig. 3b. The variability of the microscale response reduces with increasing size, which mitigates the partial lack of EA representativeness. A number of 250 grains is kept as a compromise between representativeness and computation cost in double-scale framework. The deformation at small-scale within the microstructure deserves also to be analysed. One microstructure (numbered [1]) with 60% of clay content is firstly studied to enlighten grain movements and micro-damage of interfaces. The response curve and the small-scale deformation are detailed in Fig. 4a-b [1] (up to ε 1 =0.1). A peak deviatoric stress followed by strain softening occurs (Fig. 4a), which is characteristic of dense material failure. The amplitude of material softening depends on the interface properties. Furthermore, microscale deformations are detailed at several stages of the vertical loading (Fig. 4b), before (ε 1 =0.01) and after peak stress (ε 1 =0.05, 0.1). Cohesion sof-tening (D n/t <1) and complete decohesion (D n/t =1) of interfaces (i.e. partially and fully damaged) are indicated by graphical symbols in Fig. 4b. The microstructure deformation remains limited before peak stress. Then, damage increases, coalesces, and forms a microcrack propagating through the entire EA. This induces important grain displacements (mainly sliding, Fig. 4b) and a global softening behaviour of the microstructure (Fig. 4a). Moreover, shearing is the dominant mode of deformation and damage at grain contacts. Only a small proportion of interfaces reaches complete decohesion (Fig. 4b) but contributes significantly to the microscopic localised deformations in the EA.

Conclusions

The realism of the modelling of clay rock microstructural behaviour is improved by considering heterogeneous microscale characteristics and variability. The representativeness of the small-scale behaviour and the influence of property variability are emphasised. Results indicate that the variability of micromechanical response, small-scale deformations, and microcracking patterns are related to: random positions of inclusions, mineral contents, and grain morphology. The observed variability reduces with the microstructure size. Analysis of micro deformation indicate that shearing mode of deformation is dominant at grain contacts under global deviatoric solicitation. The development of damage in the microstructure is localised and engenders intergranular micro faults, as observed experimentally. This, in turn, leads to microscale strain softening. Furthermore, the material behaviour is affected by the geometry of potential cracking paths. Finally, the enhancement of microscale behaviour modelling allows new possibilities for realistic upscaled modelling of heterogeneous rocks.

van den Eijnden, A., Bésuelle, P., Chambon, R., Collin, F. (2016)

Fig. 1

 1 Fig. 1 Representativeness of the microstructures: (a) example of numerical elementary area, (b) distributions of clay content, (c,d) comparison of experimental and numerical morphological characteristics of calcite(Robinet 2008)

Fig. 2 Fig. 3

 23 Fig. 2 Microscale response variability to the randomness of EA generation and mineral contents: (a) variable mineral contents (40% to 60% of clay), (b) fixed contents (50% of clay)

Fig. 4

 4 Fig. 4 Microstructural behaviour variability for various types of microstructures: (a) material response, (b) microcrack appearance, (c) microcracking patterns Let us now consider several microstructures with same size (i.e. same number of cells), 50% of clay content, and identical micromechanical parameters (Table 2). To enlighten the effect of microscale characteristics, the selected EAs represent different degrees of grain elongation, orientation, and angularity. They are presented in deformed state for a global homogenised vertical strain of ε 1 =0.1 in Fig. 4c (microstructures [2]-[3]-[4]). The confining pressures is still of σ 3 =12MPa as previously and the global response curves are illustrated in Fig. 4a. At ε 1 =0.1, all EA responses have reached softening due to grain contact damage. Moreover, the microstructures exhibit various microcracking patterns. The microstructures [1-2] in Fig. 4 have similar grain morphological characteristics but different positions of inclusions and mineral contents. They have similar microcracking patterns but with different locations and inclinations of the micro fault. The other microstructure [3-4] have a different grain morphology.[3] has rounded grains. The results in Fig.4show that grain roundness increases the global shear strength (peak deviatoric stress), reduces the softening and damage, and restricts the microcracking appearance. [4] is anisotropic with elongated grains oriented parallel to the claystone bedding planes (horizontal) and a perpendicular (vertical) loading to them. The results show that the elongation does not affect the microcrack orientation, neither the shear strength. Nevertheless, it increases the overall material softening and shearing of the microstructure.

Table 1

 1 Average mineralogical characteristics of the COx claystone (EST26095)

	Minerals	Orientation	Elongation	Mean area	Area fraction
			[-]	[μm²]	[%]
	Clay matrix	-	-	-	50
	Pyrite	-	-	-	2
	Calcite	horiz.	0.55	45	30
	Quartz	horiz. (variable)	0.64	100	18

  and n subscripts correspond to tangential and normal directions, c n/t are interface cohesion forces, 𝑐 𝑛/𝑡 𝑚𝑎𝑥 are maximal cohesion forces, Δu n/t are interface relative displacements. Then, D n/t are softening parameters representing the interface degradation state 0≤D n/t ≤1 (depending on initial degradation states 𝐷 𝑛/𝑡 =0), and κ is a penalisation term to avoid grain interpenetration (κ >>𝑐 𝑛 𝑚𝑎𝑥 if Δu n <0, κ=0 if Δu n ≥0).

					𝐷 𝑡 )	𝛥𝑢 𝑡 𝛿 𝑡 𝑐 𝑐 𝑡 𝑚𝑎𝑥	(1)
		𝑐 𝑛 =	1 𝐷 𝑛	(1 -𝐷 𝑛 )	𝛥𝑢 𝑛 𝛿 𝑛 𝑐 𝑐 𝑛 𝑚𝑎𝑥 -𝜅 𝛥𝑢 𝑛 2	(2)
	where t 0	and time history of
	Δu n/t ), 𝛿 𝑛/𝑡 𝑐	are critical relative displacements inducing complete decohesion (D n/t =1,
	c n/t

Table 2

 2 Microscale mechanical parameters

	Minerals	ν [-]	E [GPa]
	Clay ma-	0.110	2.3
	trix
	Pyrite	0.154	305
	Calcite	0.317	84
	Quartz	0.074	95
	Interfaces	𝑐 𝑛 𝑚𝑎𝑥 [𝑀𝑃𝑎] 𝑐 𝑡 𝑚𝑎𝑥 [𝑀𝑃𝑎] 𝐷 𝑛/𝑡 0 [-]	𝛿 𝑛/𝑡 𝑐 [-]
		1.0	2.5	0.001	0.1

  . A FE 2 modelling approach to hydromechanical coupling in cracking-induced localization problems. International Journal of Solids and Structures 97-98:475-488. van den Eijnden, A., Bésuelle, P., Collin, F., Chambon, R., Desrues, J. (2017). Modeling the strain localization around an underground gallery with a hydro-mechanical double scale model; effect of anisotropy. Computers and Geotechnics 85:384-400. Yven, B., Sammartino, S., Geraud, Y., Homand, F., Villieras, F. (2007). Mineralogy, texture and porosity of Callovo-Oxfordian argillites of the Meuse/Haute-Marne region (eastern Paris Basin). Bulletin de la Societe Geologique de France 178:73-90.


C Ramoul

N E Beliardouh
email: nebeliardouh23@gmail.com

R Bahi

C Nouveau

A Djahoudi

Michael J Walock

Orcid C Ramoul

Surface performances of PVD ZrN coatings in biological environments

Keywords: PVD coating, ZrN, wear, corrosion, bioadhesion

Zirconium nitride (ZrN) thin films were deposited by reactive RF magnetron sputtering on Ti-6Al-4V and Si (100) substrates for potential use in biomedical applications. The tribological behaviour was evaluated against bovine bone in dry condition using a pin-on-disc apparatus. Abrasion is the primary wear mechanism observed in ZrN/bone contact. The corrosion properties were determined through two electrochemical techniques: potentiodynamic polarization and electrochemical impedance spectroscopy. The coatings with reduced oxygen content provided: (i) good resistance against corrosion when exposed to physiological solution and (ii) better anti-bioadhesion against Staphylococcus aureus bacteria.

Introduction

In the field of orthopaedic implants, the challenge is to develop alloys with high biocompatibility and the requisite material properties to effective serve as substitute bones and joints for a long period without rejection. Here, bone-implant integration is closely related to the biocompatibility (immunology/bioadhesion), wear and corrosion behaviour of both the implant (artificial organ) and surrounding bone tissue.

Transition metal nitrides, oxides and intermetallics, especially Titanium-based alloys, have been widely studied and show great potential for the protection of orthopaedic applications. Diffusion hardened Zr-2.5Nb, known as ('Oxinium'), is another category of surface engineered material used for bio implants. As recently reviewed by Dearnley [START_REF] Dearnley | A Introduction to surface engineering[END_REF], it appears as the better non-coated material.

In order to eliminate the implant rejection risks due to bacterial infections, newer approaches are currently being evolved. The goal is to convert the surfaces of biomedical devices into antimicrobial surfaces while preserving the corrosion and wear resistance. Nanostructured surfaces and antimicrobial agents, such as silver, used to reduce infections are the best coating strategies and could represent an alternative to antibiotics.

Recently, it was demonstrated that bacterial adhesion on Zirconium alloy films was minimized by tailoring the surface with silver ions surrounded by amorphous carbon phases using a sputtering system [START_REF] Calderon | Nano-galvanic coupling for enhanced Ag + release in ZrCN-Ag films: antibacterial application[END_REF]. Unfortunately, accumulation of silver ions (Ag + ) is well known to have negative effects on general health [START_REF] Grischke | Antimicrobial dental implant functionalization strategiesa systematic review[END_REF]. So, the challenge becomes the development of multifunctional surfaces with simultaneous opposite responses; on the one hand towards human cells, i.e. providing adhesion molecules for attaching tissues and cells (osseointegration) and against bacterial adhesion/proliferation (antibacterial activity) on the other hand.

Over the past two decades, ZrN coating has attracting interest for its excellent erosion resistance, biocompatibility, good ductility and excellent chemical stability [START_REF] Menghani | Corrosion and wear behaviour of ZrN thin films tribology[END_REF]. In addition, there is evidence that ZrN coatings have better corrosion resistance than TiN coatings [START_REF] Ningshen | Corrosion study of ZrN deposited on 304L stainless steel[END_REF].

In this work, a ZrN/Zr system was evaluated for its suitability as a protective coating in biological environments. The corrosion resistance in physiological solution (PS) and tribological performances against bovine bone were investigated systematically. This was followed by a study of the material's antibacterial activity against Staphylococcus aureus (S. aureus).

This study was conducted taken into account two fundamental aspects (i) an economic aspect, i.e. a low-cost biomaterial and (ii) an environmental aspect; because recovering (recycling) only one metallic element (Zr) from an oxide is much easier than recovering two metallic elements or more from a complex structure.

Experimental

Coatings deposition

Zirconium Nitride films were realized by the radio frequency magnetron sputtering technique (NORDIKO type 3500). A pure Zirconium (99.98%) disc (101.6 mm diameter, 3 mm thick) was used as the target material. The substrates are Titanium alloys (Ti-6Al-4V) intended for tribological tests and Si (100) wafers (10 × 10 × 0.380 mm 3 ) for corrosion and biological tests.

Metallic substrates (cylinder Ø = 20 mm × 4 mm) were polished by SiC papers grade 800. All substrates were ultrasonically cleaned for 15 min in acetone and dried in air. Before deposition, the target disc and the substrates were in situ etched for 5 min at 10 µbar in an Ar discharge. A pure Zr buffer layer of approximately 100 nm was first deposited (10 min, 5 µbar) to improve adhesion between the ZrN topcoat and the substrates. The target to substrate distance was fixed at 80 mm and the target bias voltage was -900 V (Zr).

In addition to improve the adhesion, a functional buffer layer is an efficient strategy for overcoming corrosion substrate problems (contact between the electrolyte and the substrates surface because of pores and other coating defects) and improving the tribological properties. Four (4) samples (ZrN-01; ZrN-02; ZrN-03 and ZrN-04) were deposited by changing Ar/ (N 2 +O 2 ) ratio, as summarized in Table 1. The total working pressure (5 µbar), deposition time (∼242 min) and substrate temperature (150°C) were kept constant. As-deposited and post-test films were characterized by the scanning electron microscopy (SEM; JEOL JSM-5900LV-10 kV) with energy dispersive X-ray spectroscopy (EDS; Oxford INCA x-act) and optical profilometry (Veeco-Wyko NT1100).

Bone pins preparation

The bone pins were prepared from the femur of a bovine. The complete femur was immersed in distilled water at a constant temperature (100°C) for 2 h to remove residual soft tissue attached to the bone. From this clean bone, the pins were machined as cylinders (cylinder Ø = 6 mm × 20 mm), ultrasonically cleaned (15 min) in acetone, dried in air and finally kept at 3-4°C in a fridge (until ready for use in the pin-on-disc tribometer).

Coating characterizations

Tribology

Dry friction tests were conducted in ambient air at room temperature and relative humidity of 40% ±5 using a pin-on-disc tribometer (CSM Instruments, Switzerland) in accordance to the ISO 7148, ASTM G99-95a and ASTM G133-95 standards to determine the coefficient of friction (COF). Bone pins were used as counterparts against rotating ZrN-coated Ti-6Al-4V discs.

When it is impossible to make a ball geometry, e.g. from organic matter of animal origin (bone, cartilage, etc.), the pin-on-disk configuration provided by © CSM Instrument tribometer is perfectly adapted for this case (Figure 1). The pin holder is positioned at 45°; this allows to make several tests with one pin. The tribological test parameters are listed in Table 2.

Electrochemical analyses

The electrochemical corrosion tests were carried out using a conventional three-electrode cell of 300 ml capacity and Gamry® Potentiostat/Galvanostat Reference 1000. The cell was fitted with a coated Si sample as working electrode (WE), saturated calomel electrode (SCE) as the reference electrode and platinum as a counter electrode (CE). The corrosion behaviour was evaluated by measuring the open circuit potential (OCP) after 3600 s in 150 ml of 0.9% NaCl PS at room temperature with a scan rate s = 1 mV/s. The corrosion potential (E corr ) and the corrosion current density (I corr ) were deduced from the Tafel plot (i.e. log i vs. E plot). Electrochemical impedance spectroscopy (EIS) tests were conducted in a frequency range from 10 5 to 10 -2 Hz at a rate of 5 points/decade, in E ocp vs. SCE. The potentiodynamic polarization test was raised from -1 to +1 V, at a scan rate of 1 mV/s. The surface contact of the sample with the electrolyte was 0.5 cm 2 . EIS parameters were calculated using the ZView software (Scribner Associates, Inc).

Results and discussion

ZrN coatings microstructural characterization

Figure 2 shows SEM images of an as-deposited ZrN coating on a Si substrate. There are some sputtering defects, e.g. pits and droplets, as shown in Figure 2 (a-b). All films have an overall thickness of ∼1.6 to 1.9 µm and exhibit a columnar structure, as shown in Figure 2(c). The surface roughness of the samples is shown in Table 2. No remarkable differences in roughness parameters (R a , R q ) are observed. The ZrN-01 and ZrN-02 samples have a relatively low oxygen content in comparison to ZrN-03 and ZrN-04. During dry sliding tests, the following scenario most probably occurs; at the beginning of the wear test, there is a severe material transfer between the coating and the pins, which leads to an increase of the COF. When the steady state is reached, the material transfer is slowly reduced and stabilized after a large percentage of the wear track area is covered with the transferred material. At the steady state, the highest COF value (∼0.58) was observed for the ZrN-02 coating surface, whereas lower COF values (∼0.38) were obtained for the other three ZrN coatings (Figure 3). Thus, the COF is fairly high. That is the disadvantage of hard ceramic coatings (ZrN), e.g. their brittleness and relatively high COF. The COF of the uncoated substrate at the steady state was ∼0.58. From Figure 3, there is clear evidence that, after 3 m of sliding distance the coatings are destroyed.

Wear behaviour

COF of the ZrN coatings

The wear volume loss of the ZrN coatings/bone couple was not measurable as it was of the order of the surface roughness of the film. In addition, the wear scars on the bone pin could not be estimated because of the low interaction between bone and light in profilometry. So, quantitative data of wear rate were not possible in this study. Only a qualitative analysis of the wear tracks was conducted using SEM/EDS.

Wear mechanisms of the ZrN-coated discs

Figure 4 shows the wear tracks of ZrN coatings on Ti-6Al-4V substrates tested against bone pins as counterparts in dry conditions. The primary wear mechanism appears to be abrasion, as shown in Figure 4(a-d). In addition to grooves and scratches parallel to sliding direction, wear debris accumulated in the wear tracks are also observed as deep furrows from trapped particles (Figure 4(c)). The elemental composition of some isolated particles, analysed by EDS, indicates the presence of both bone elements (Na, Mg, Ca, K, etc.) and coating elements (Zr and oxygen) in the adhered film (points 1, 2, 3, 4 in Figure 4(a-e)). So, the wear debris is a mixture of the antagonist pair components. From this, one can conclude that bone adhesion to the disc is the second wear mechanism observed in the ZrN/bone contact pair. SEM analysis also revealed some areas within the wear track where the coating did not appear to be damaged after the wear tests. No plastic deformation was detected in the wear tracks on any of the coatings.

Wear mechanisms of the bovine bone pins

Bone is a heterogeneous and anisotropic material. It has a hierarchically organized structure and is often classified as a 'composite material' [START_REF] Grimm | Orthopedic biomaterials[END_REF]. At the highest hierarchical level (with feature sizes ∼1 to 5 mm), there are two types of bone: cortical bone (which is the type used in this study) and trabecular bone. Cortical bone consists of repeating units called Haversian systems or osteons. [START_REF] Grimm | Orthopedic biomaterials[END_REF]. The macrostructure of the bone pin specimen (cortical bone) is shown in Figure 5. Figure 6 (a-d) shows micrographs of the bone pin wear scars. It appears that the edges of the bone pin (see Figure 1; analyzed area (Wear track 'T')) become larger after acting as counterparts against ZrN-coated discs. Wear scars also reveal deformation of the Haversian system, lacunae (crevasses indicated by white narrows), isolated wear particles (black narrows) and entrapped particles (ellipse). EDS analyses, carried out at different locations (isolated as well as entrapped particles) within the wear scars, indicate that a mixture of both the antagonist elements in contact was found. Abrasive wear is the only wear mechanism observed on the bone pins.

When sliding against ZrN-coated discs, segments of the bone pins were initially scraped off by surface asperities along the wear track. This is due to the large difference in mechanical properties between the antagonists in contact. A literature study reveals the average hardness (H ) and elastic modulus (E) of cortical bovine bone obtained from nanoindentation in transversal direction between 0.647 and 0.892 GPa and 19.8±1.6 GPa, respectively [START_REF] Wang | Nano-and macroscale characterization of the mechanical properties of bovine bone[END_REF]. ZrN coatings hardness and modulus values range from 15 to 34 GPa and 240 to 320 GPa, respectively [START_REF] Dorcioman | Wear tests of ZrC and ZrN thin films grown by pulsed laser deposition[END_REF]. As the number of sliding cycles increases, the transferred layer onto ZrN surface becomes brittle and is destroyed through fatigue; the resulting wear debris acts as a third body.

Electrochemical behaviour

Potentiodynamic polarization measurements

Figure 7(a) shows the potentiodynamic polarization curves (plots of potential vs. log current density) of the ZrN coatings exposed to the PS for 2 h. All specimens form an anodic domain (labelled '1'), which corresponds to the formation of a self-passivating (Zr, N, O) film after being soaked for a period of time. The size of this domain is indicative of the durability of the selfpassivating film and the corrosion resistance of each coating. For example, the anodic plateau remained flat from -0.4 up to +0.65 V in the case of ZrN-01. The rupture of the passive film occurs in the trans-passivation domain (labelled '2'). In this domain, the current density abruptly increases as the materials start to corrode (labelled '3'). The polarization resistance (R p ) of Tafel curves, which takes into consideration both the corrosion current density and the slopes of polarization curves, can provide a quantitative ranking of coatings resistance to corrosion and can be estimated using the well-known Stern-Geary equation.

Table 3 summarizes the relevant values from the Tafel plots. From these results, it is clear that ZrN-01 is significantly more corrosion-resistant (high value of R p and low value of I corr ) when compared to the other three coatings.

EIS analysis

Each sample's impedance was measured after 1 h of immersion in the PS. Nyquist plots of the real and imaginary parts of the impedance are shown in Figure 8(a). The plots show that all tested samples are characterized by two capacitive loops in the high and medium frequency range. By comparing the diameters of the loops in the Nyquist plot, the sample's corrosion properties can be easily evaluated. Larger diameter loops equate to higher corrosion resistance. This analysis supports the previous conclusion: ZrN-01 has the best corrosion resistance of the four tested samples. An equivalent circuit (EC) model, which is shown in Figure 8(b), was selected for quantitative studies. This EC is extensively used in similar situations [START_REF] Subramanian | Microstructural, tribological and electrochemical corrosion studies on reactive DC magnetron sputtered Zirconium nitride films with Zr interlayer on steel[END_REF]. Figure 8(c) shows good agreement between the experimental and modelled values. The fitted line was plotted as a dashed line. The fitting errors for each configuration in this study were less than 9%.

In the selected equivalent circuit, R s represents the electrolyte resistance. The Constant Phase Elements, CPE1 and CPE2, are the double layer capacitance and the coating capacitance respectively. R ct is the charge transfer resistance and R pore represents the pore resistance (and other fails of coating).

The overall corrosion resistance of the coating can be estimated by the sum of all resistances of the circuit model in accordance with the method described by Subramanian et al. [START_REF] Subramanian | Microstructural, tribological and electrochemical corrosion studies on reactive DC magnetron sputtered Zirconium nitride films with Zr interlayer on steel[END_REF]. Therefore, using the quantitative data given in Table 4, the ZrN-01 coating presents the highest resistance against corrosion in a physiological liquid solution of the four tested samples, as expected.

Biocompatibility

The bactericidal properties of ZrN coatings were assessed using S. aureus bacteria. This bacterium is the main pathogenic microbe responsible for implant associated infections [START_REF] Lauderdale | Biofilm dispersal of community-associated methicillin-resistant Staphylococcus aureus on orthopedic implant material[END_REF]. Two kinds of samples are used: ZrN coating on silicon wafer and an uncoated Ti-6Al-4V substrate. Ti6-Al-4V is one of the most studied biomaterials, and its biocompatibility in various environments has been well documented. This sample acts as a 'reference' material for the comparison of the biocidal properties of ZrN coatings.

Following sterilization in an autoclave, three samples (bare Ti-6Al-4V, ZrN-01/Si and ZrN-03/Si) were exposed to bacteria suspended in a buffered glucose broth and incubated at 37°C for 24 h. To quantitatively assess the antibacterial effect of the three samples, the plate count method, in association with the dilution method, is used [START_REF]Merck KGa A 64271[END_REF]. Specimen A is the liquid solution collected from the suspension which contained the metallic Ti-6Al-4V substrate. Specimen B and specimen C are the liquid solutions which previously contained the ZrN-01/Si and ZrN-03/Si samples, respectively. Twenty microlitres of each specimen are spread over a sterilized Petri dish coated with mannitol salt agar, also known as Chapman medium. The coated Petri dishes were incubated at 37°C for 24 h. Quantitative test results (counts) are given in term of colony forming units per millilitre (CFU/ml).

After incubation, the S. aureus colonies are clearly visible (Figure 9) and were easily counted. The results indicate that S. aureus bacteria were completely killed in specimen B, while the surviving bacterial colonies are 3.2 × 10 5 (CFU/ml) for specimen A and 5.2 × 10 5 (CFU/ml) for specimen C. Specimen B and Specimen C have approximately similar surface parameters (i.e. roughness, morphology, etc.) but there is a noticeable difference in the oxygen content of the samples. So, surface morphology effects cannot explain the difference in the biocidal effect of these two specimens (B and C). In addition, it has been reported that a surface roughness <0.2 μm may not affect bacterial adhesion because most bacteria are >0.2 μm [START_REF] Ji | Evaluation of antibacterial activity and osteoblast-like cell viability of TiN, ZrN and (Ti1-xZrx)N coating on titanium[END_REF].

Figure 1 .

 1 Figure 1. Schematic of the pin-on-disc tests.

Figure 3

 3 shows the friction coefficient curves of the ZrN films sliding against bovine bone in dry conditions. It shows that the friction coefficients of the four coatings possess similar features, i.e. the friction curves of ZrN films are characterized by two different stages. The first stage represents run-in and ends with a rapid increase of the COF; the second stage was the steady state wear regime.

Figure 3 .

 3 Figure 3. COF of the ZrN coatings sliding against bone counterparts.

Figure 2 .

 2 Figure 2. SEM images of the as-deposited ZrN-01 coating on Si substrate: (a) global view of the surface, (b) magnified image of the region enclosed by the white square (area A) and (c) crosssection image of the same sample.

Figure 4 .

 4 Figure 4. Wear mechanism of ZrN coatings: abrasion (scratches-grooves) and bone adhesion wear; (a) wear track of ZrN-01, (b) wear track of ZrN-02, (c) zoom of the square A, (d) wear track of ZrN-03, (e) wear track of ZrN-04 and (f) EDS analysis results at points 1, 2, 3, 4; (S.Dsliding direction).

Figure 5 .

 5 Figure 5. (a) SEM micrograph of a bone pin specimen. (b) A schematic diagram illustrating the macrostructure of cortical bone; O -Osteons (Haversian systems), L mconcentric lamellae, HC -Haversian canal and L Clacunae.

Figure 6 .

 6 Figure 6. SEM images of wear scars of bone pins after pin-on-disc tests for: (a) ZrN-01, (b) ZrN-02, (c) ZrN-03 and (d) ZrN-04 (dash lines indicate scratch marks, SDsliding direction).

Figure 7 .

 7 Figure 7. Potentiodynamic polarization curves of ZrN coatings in physiological liquid solution.

Figure 8 .

 8 Figure 8. Electrochemical test results (EIS): (a) Nyquist plots of tested specimen in physiological liquid solution, (b) equivalent circuit model and (c) example of fitting curve (WE = working electrode and RE = reference electrode).

Figure 9 .

 9 Figure 9. Staphylococcus aureus colonies grown in sterilized Petri dishes of: (a) specimen A, (b) specimen B and (c) specimen C.

Table 1 .

 1 Chemical composition and roughness of ZrN/Zr coatings.

					Chemical
		Ratio Ar/(N+O 2 )	Total thickness	composition (Weight %)	Roughness (μm)
	Sample	(%)	(μm)	N	O	Zr	R a	R

q ZrN-01 80/20 1.82 16.5 1.5 81.9 0.024 0.031 ZrN-02 70/30 1.67 16.4 1.8 81.8 0.027 0.053 ZrN-03 60/40 1.75 12.2 4.9 82.9 0.018 0.024 ZrN-04 50/50 1.72 12.9 6.1 81 0.014 0.021

Table 2 .

 2 Pin-on-disc test conditions.

	Parameters	Value
	Velocity (cm/s)	2
	Applied load (N)	2
	Total sliding distance (m)	5
	Wear track diameter (mm)	4
	Environment	Air
	Temperature (°C)	20 ± 3
	Humidity (%)	40 ± 5

Table 3 .

 3 Potentiodynamic polarization test results.

			I corr
	Samples	E corr (mV)	(μA/ cm 2 )	a (mV/ dec)	c (mV/ dec)	R p (Ω)	χ 2 .10 -4
	ZrN-01 ZrN-02 ZrN-03 ZrN-04	-466.76 -376.53 -612.98 -486.20	0.007 0.042 0.037 0.044	174.8 297.7 512.7 202.4	544.3 274.5 220.2 219.9	8.04.10 6 15.99 1.47.10 6 13.42 1.80.10 6 14.44 1.04.10 6 9.10

Table 4 .

 4 EIS data obtained by equivalent circuit simulation.

	Samples	R s (Ω cm 2 )	CPE1 (μF/cm²)	a 1	R pore (Ω cm 2 )	CPE2 (μF/cm²)	a 2	R ct (Ω cm 2 )
	ZrN-01 ZrN-02 ZrN-03 ZrN-04	163.1 155.3 304.3 365.2	2.68 10 -6 16.23 10 -9 5.11 10 -6 2.46 10 -6	0.91 0.86 0.96 0.78	675 10 3 35.7 10 3 4.6 10 3 362.9 10 3	2.08 10 -6 3.92 10 -6 5.85 10 -6 3.92 10 -6	0.73 0.74 0.72 0.81	205 10 6 31.08 10 6 13.39 10 6 48.85 10 6

Acknowledgements

The authors are thankful to Mr. Denis LAGADRILLERE for SEM/EDS analysis.

However, chemical interactions between microorganisms and biomaterial surfaces can have a significant effect on bioadhesion and the proliferation process [START_REF] Manuel | Bioadhesiona review of concepts and applications[END_REF][START_REF] Vadillo-Rodriguez | Localized attraction correlates with bacterial adhesion to glass and metal oxide substrate[END_REF]. Within liquids, oxide surfaces can be positively charged and bacteria typically are negatively charged [START_REF] Bixer | Biofouling: lessons from nature[END_REF]. Consequently, bioadhesion of bacteria is significantly enhanced in the presence of surface oxides. This result is in accordance with conclusions reported by Vadillo-Rodriguez et al. [START_REF] Vadillo-Rodriguez | Localized attraction correlates with bacterial adhesion to glass and metal oxide substrate[END_REF]. The presence of oxygen, and the development of surface oxides, can also affect the release of Zr ions. Zirconium ions interact with the bacterial cell wall, which may disturb its permeability and/or may inhibit bacterial metabolism [START_REF] Manuel | Bioadhesiona review of concepts and applications[END_REF].

Therefore, oxygen content appears to be a key limiting factor in the biocompatibility of ZrN surfaces. Oxygen content leads to higher bioadhesion aptitude and more colonies (proliferation). As a result, there is a higher probability of S. aureus survival and increased chances for infection.

Conclusion

From the above-mentioned results, it can be concluded that:

(i) All coatings exhibited good wear properties against bone counterparts. The primary wear mechanisms were abrasive wear and bone adhesion for the ZrN-coated discs and thirdbody abrasive wear for the bone counterparts. (ii) All coatings exhibited good corrosion resistance.

However, the oxygen content seems to play a more significant role here than in the wear tests.

The sample with the lowest oxygen content,

Disclosure statement

No potential conflict of interest was reported by the authors.