Sao Mai Nguyen

Pierre-Yves Oudeyer

Active Choice of Teachers, Learning Strategies and Goals for a Socially Guided Intrinsic Motivation Learner

Keywords: strategic learner, imitation learning, mimicry, emulation, artificial curiosity, intrinsic motivation, interactive learner, active learning, goal babbling, robot skill learning

We present an active learning architecture that allows a robot to actively learn which data collection strategy is most efficient for acquiring motor skills to achieve multiple outcomes, and generalise over its experience to achieve new outcomes. The robot explores its environment both via interactive learning and goal-babbling. It learns at the same time when, who and what to actively imitate from several available teachers, and learns when not to use social guidance but use active goal-oriented self-exploration. This is formalised in the framework of life-long strategic learning. The proposed architecture, called Socially Guided Intrinsic Motivation with Active Choice of Teacher and Strategy (SGIM-ACTS), relies on hierarchical active decisions of what and how to learn driven by empirical evaluation of learning progress for each learning strategy. We illustrate with an experiment where a simulated robot learns to control its arm for realising two kinds of different outcomes. It has to choose actively and hierarchically at each learning episode: 1) what to learn: which outcome is most interesting to select as a goal to focus on for goal-directed exploration; 2) how to learn: which data collection strategy to use among self-exploration, mimicry and emulation; 3) once he has decided when and what to imitate by choosing mimicry or emulation, then he has to choose who to imitate, from a set of different teachers. We show that SGIM-ACTS learns significantly more efficiently than using single learning strategies, and coherently selects the best strategy with respect to the chosen outcome, taking advantage of the available teachers (with different levels of skills).

Strategic Active Learning for Life-Long Acquisition of Multiple Skills

Life-long learning by robots to acquire multiple skills in unstructured environments poses challenges of not only predicting the consequences or outcomes of their actions on the environment, but also learning the causal effectiveness of their actions for varied outcomes. The set of outcomes can be in large and high-dimensional sensorimotor spaces, while the physical embedding of robots allows only limited time for collecting training data. The learning agent has to decide for instance in which order he should focus on learning how to achieve the different outcomes, how much time he can spend to learn to achieve an outcome or which data collection strategy to use for learning to achieve a given outcome.

Active Learning for Producing Varied Outcomes with Multiple Data Collection Strategies

These questions can be formalised under the notion of strategic learning [START_REF] Lopes | The Strategic Student Approach for Life-Long Exploration and Learning[END_REF].

One perspective is learning to achieve varied outcomes. It aims at selecting which outcome to spend time on. A typical classification was proposed in [START_REF] Qi | Twodimensional active learning for image classification[END_REF][START_REF] Reichart | Multi-task active learning for linguistic annotations[END_REF] where active learning methods improved the overall quality of the learning. In sequential problems as in robotics, producing an outcome has been modelled as a local predictive forward model [START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF], an option [START_REF] Barto | Intrinsically motivated learning of hierarchical collections of skills[END_REF], or a region in a parameterised goal/option space [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF]. In these works each sampling of an outcome entails a cost. The learning agent has to decide which outcome to explore/observe next. However most studies using this perspective do not consider several strategies. Another perspective is learning how to learn, by making explicit the choice and dependence of the learning performance on the method. For instance, [START_REF] Baram | Online choice of active learning algorithms[END_REF] selects among different learning strategies depending on the results for different outcomes. However most studies using this perspective consider a single outcome. Indeed, these works have not addressed the learning of both how to learn and what to learn, to select at the same time which outcome to spend time on, and which learning method to use. Only [START_REF] Lopes | The Strategic Student Approach for Life-Long Exploration and Learning[END_REF] studies the framework of these questions, and only examined a toy example with discrete and finite number of states, outcomes and strategies. In initial work to address learning for varied outcomes with multiple methods, we proposed the Socially Guided Intrinsic Motivation by Demonstration (SGIM-D) algorithm which uses both:

• socially guided exploration, especially programming by demonstration [START_REF] Billard | Handbook of Robotics, chapter Robot Programming by Demonstration[END_REF], and

• intrinsically motivated exploration, which are active learning algorithms based on measures of the evolution of the learning performance [START_REF] Oudeyer | What is intrinsic motivation? a typology of computational approaches[END_REF] Forward Model to reach goals in a continuous outcome space, in the case of a complex and continuous environment. High-dimensional environments can be handled by SGIM-D, designed for multiple outcomes in a continuous outcome space. In [START_REF] Mai Nguyen | Bootstrapping intrinsically motivated learning with human demonstrations[END_REF], SGIM-D learned to manipulate a fishing rod with a 6-dof arm, i.e. to place the float on the surface of the water, which is described as a 2d continuous outcome space. The robotic arm was controlled by a motor primitive with 24 continuous parameters that determine the trajectory of its joint positions. The robot learned which action to perform for a given goal position on the surface of the water , where the hook should reach when falling into the water. However, the outcomes considered belonged to only one type of outcomes. Moreover, although SGIM-D has 2 learning strategies, it is a passive learner which only imitates when the teacher decides to give a demonstration. SGIM-D does not learn which method enables it to perform best.

In this paper, we address these two limitations. We study how a learning agent can achieve varied outcomes in structured continuous outcome spaces, even with outcomes of different types, and how he can learn for those various outcomes which strategy to adopt among 1) active self-exploration, 2) emulation of a teacher actively selected among available teachers, 3) mimicry of an actively selected teacher. We propose an algorithm for actively choosing the appropriate strategy, among several strategies.

Formalisation

Let us consider an agent learning motor skills, i.e. the mapping between an outcome space and a policy space. As an illustration, let us imagine the agent learning how to play tennis, He maps how the ball behaves (outcome) with respect to the movement of his racket (policy). He thus learns a forward model M to predict where the ball bounces given the movement of his racket. More importantly, he builds an inverse model L -1 to control his racket in order to make the ball bounce at a desired position. A good player knows which outcomes are feasible and knows at least one policy to produce any possible outcome: he can place the ball anywhere on the court. Ideally, he builds an inverse model L -1 such that M(L -1 ) is identity. More formally, we define an outcome space which may comprise of outcomes of different types and different dimensionalities. For tennis, outcomes can be the bouncing positions, spin angles ... We only assume that they can be parameterised by parameters τ ∈ T and that we can define a distance measure J on T × T .

A policy π θ is described by motor primitives parameterised by θ ∈ Π. Its outcome is M(θ), where the mapping M : Π → T describes the environment. For the tennis player, the policy controls the movement of his arm and racket and M represents the physical equations for the ball trajectory. The performance of a policy π θ at completing an outcome τ is measured by the distance between τ and the outcome of π θ : J(τ M(θ)).

The agent focuses on learning the inverse model and builds its estimate L -1 : T → Π. We note that M -1 , the inverse of M might not be a function as M might be redundant, whereas our learner builds a function L -1 that finds at least one adequate policy to complete every outcome τ. In sum, it endeavours to minimise with respect to L -1 :

I = τ∈T P(τ)J(τ M(L -1 (τ))) τ ( 1
)
where P(τ) is a probability density distribution over T . A priori unknown to the learner, P(τ) can describe the probability of τ occurring or the reachable space or a region of interest. We assume that T can be partitioned into subspaces where the outcomes are related, and in these subspaces our parametrisation allows a smooth variation of τ → J(τ M(θ)) ∀θ with respect to τ most of the time. This partition, initially unknown to the agent, needs to be learned. Note that we have described our method without specifying a particular choice of policy representation, learning algorithm, action or outcome space properties. These designs can indeed be decided according to the application at hand. In particular, outcomes can be of different types and dimensionalities. In this case, we note T the subspaces of T corresponding to the different types of outcome and T = ∪T .

Our Approach

To solve the problem formalised above, we propose a system, called Socially Guided Intrinsic Motivation with Active Choice of Teacher and Strategy (SGIM-ACTS) that allows an online interactive learning of inverse models in continuous highdimensional robotic sensorimotor spaces with multiple teachers, and learning strategies. SGIM-ACTS learns various outcomes with different types of outcomes, and generalises from sampled data to continuous sets of outcomes. Technically, we adopt a method of generalisation of policies for new outcomes similar to [START_REF] Da Silva | Learning parameterized skills[END_REF][START_REF] Kober | Reinforcement learning to adjust parametrized motor primitives to new situations[END_REF]. Whereas in their approaches the algorithms use a pool of examples given by the teacher preset from the beginning of the experiment to learn outcomes specified by the engineer of the robot, in a batch learning method; in our case, the SGIM-ACTS algorithm decides by itself which outcomes it needs to learn more to better generalise for the whole outcome space, like in [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Barto | Intrinsically motivated learning of hierarchical collections of skills[END_REF][START_REF] Oudeyer | Intrinsic motivation systems for autonomous mental development[END_REF]. Moreover, SGIM-ACTS actively requests the teacher's demonstrations online, by choosing online the best learning strategy, similarly to [START_REF] Baram | Online choice of active learning algorithms[END_REF], except that we do not learn with a discrete outcome space for a classification problem, but with a continuous outcome space. SGIM-ACTS also interacts with several teachers and uses several social learning methods, in an interactive learning approach.

Our active learning approach is inspired by:

• intrinsic motivation in psychology [START_REF] Ryan | Intrinsic and extrinsic motivations: Classic definitions and new directions[END_REF] which triggers spontaneous exploration and curiosity in humans, which recently led to novel robotic and machine active learning methods which outperform traditional active learning methods [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Lopes | Exploration in model-based reinforcement learning by empirically estimating learning progress[END_REF] • teleological learning [START_REF] Csibra | Teleological and referential understanding of action in infancy[END_REF] which considers actions as goaloriented, and recently led to efficient goal babbling methods in robotics [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF][START_REF] Rolf | Goal babbling: a new concept for early sensorimotor exploration[END_REF] • psychological theories for socially guided learning [START_REF] Call | Imitation in animals and artifacts, chapter Three sources of information in social learning[END_REF][START_REF] Dautenhahn | Imitation in Animals and Artifacts[END_REF][START_REF] Tomasello | Shared intentionality[END_REF], as detailed in the next section.

After this formal description of our approach, we analyse our point of view on social guidance in section 2. Then, we detail the proposed algorithm SGIM-ACTS in section 3, before testing it on a problem to learn how to throw and place a ball (fig. 1) in section 4.

Social Guidance

Interactive Learning

An interactive learner who not only listens to the teacher, but actively requests for the information it needs and when it needs help, has been shown to be a fundamental aspect of social learning [START_REF] Chernova | Interactive policy learning through confidence-based autonomy[END_REF][START_REF] Nicolescu | Natural methods for robot task learning: Instructive demonstrations, generalization and practice[END_REF][START_REF] Thomaz | Socially Guided Machine Learning[END_REF]. Under the interactive learning approach, the robot can combine programming by demonstration, learning by exploration and tutor guidance. Several works in interactive learning have considered extra reinforcement signals [START_REF] Thomaz | Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers[END_REF], action requests [START_REF] Daniel | Incremental learning of subtasks from unsegmented demonstration[END_REF][START_REF] Lopes | Active learning for reward estimation in inverse reinforcement learning[END_REF] or disambiguation among actions [START_REF] Chernova | Interactive policy learning through confidence-based autonomy[END_REF]. In [START_REF] Cakmak | Designing interactions for robot active learners. Autonomous Mental Development[END_REF] the comparison of a robot that has the option to ask the user for feedback, to the passive robot, shows a better accuracy and fewer demonstrations. Therefore, requesting demonstrations when it is needed can lessen the dependence on the teacher and reduce the quantity of the demonstrations required. This approach is the most beneficial to the learner, for the information arrives as it needs it, and to the teacher who no longer needs to monitor the learning process.

For an agent learning motor skills, i.e. the mapping between policies and outcomes, let us examine the type of social guidance that a learner can get as reviewed in [START_REF] Argall | A survey of robot learning from demonstration[END_REF][START_REF] Billard | Handbook of Robotics, chapter Robot Programming by Demonstration[END_REF][START_REF] Lopes | From Motor to Interaction Learning in Robots, chapter Abstraction Levels for Robotic Imitation: Overview and Computational Approaches[END_REF][START_REF] Schaal | Computational approaches to motor learning by imitation[END_REF] with respect to: what, how, when and who [START_REF] Dautenhahn | Imitation in Animals and Artifacts[END_REF]. In this section, we note H the information flow from the human to the robot.

What?

Let us examine the target of the information given by the teacher, or mathematically speaking, the space on which he operates. This can be either the policy or outcome spaces, or combinations of them.

Policy Space

Many social learning studies target the policy parameter space Π. For instance, in programming by demonstration (LbD), H shows the right policy to perform in order to reach a given goal.

As an illustration, when teaching how to play tennis, your coach could show you how to hit a backhand by a demonstration, or even by taking your hand and directing your movement. This approach relates to two levels of social learning: mimicry, in which the learner copies the policies of others without an appreciation of their purpose, and imitation, in which the learner reproduces the policies and the changes in the environment, as formalised in [START_REF] Call | Imitation in animals and artifacts, chapter Three sources of information in social learning[END_REF][START_REF] Lopes | From Motor to Interaction Learning in Robots, chapter Abstraction Levels for Robotic Imitation: Overview and Computational Approaches[END_REF][START_REF] Whiten | Primate culture and social learning[END_REF]. The literature often considers that targeting the policy space is the most directive and efficient method. However, it relies on the human teacher's expertise, which bears limitations such as ambiguity, imprecision, underoptimality or the correspondence problem.

Outcome Space

The second kind of information is about possible outcomes τ ∈ T , and is related to goal-directed exploration, where the learner focuses on discovering different outcomes instead of different ways of entailing the same outcome. Psychologically speaking, this case pertains to the emulation level of social learning, where the observer witnesses someone produce a result on an object, but then employs his own policy repertoire to reproduce the result, as formalised in [START_REF] Call | Imitation in animals and artifacts, chapter Three sources of information in social learning[END_REF][START_REF] Lopes | From Motor to Interaction Learning in Robots, chapter Abstraction Levels for Robotic Imitation: Overview and Computational Approaches[END_REF][START_REF] Chrystopher | Imitation and Social Learning in Robots, Humans and Animals: Behavioural, Social and Communicative Dimensions[END_REF][START_REF] Whiten | Primate culture and social learning[END_REF]. During our tennis training, your coach could ask you to hit with the ball the right corner of the court, wherever you received the ball, whichever shot you use. Goal-directed approaches allow the teacher to reset goal outcomes [START_REF] Argall | Learning robot motion control with demonstration and advice-operators[END_REF], to request the execution of outcomes [START_REF] Thomaz | Socially Guided Machine Learning[END_REF] or to label outcomes [START_REF] Thomaz | Socially Guided Machine Learning[END_REF][START_REF] Thomaz | Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers[END_REF]. The learner can infer from the demonstrations the goal outcome by positional and force profiles to iron and open doors [START_REF] Kormushev | Caldwell. Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input[END_REF], or by using inverse reinforcement learning [START_REF] Lopes | Simultaneous acquisition of task and feedback models. Development and Learning (ICDL)[END_REF]. This approach is essential to learn multiple outcomes, and all the more interesting as it is inspired by psychological behaviours [START_REF] Csibra | Teleological and referential understanding of action in infancy[END_REF][START_REF] Tomasello | Shared intentionality[END_REF][START_REF] Whiten | Primate culture and social learning[END_REF]. The drawback is that the learning needs the actions repertoire to be large enough to be used to reach various goals, before it improves.

As we want the learner to accomplish not only a single outcome but to be efficient on a large variety of goals, we choose to bootstrap its learning with information targeting the outcome space. Furthermore, we also want the learning process to benefit from the social interaction early. So that the learner builds its action repertoire quickly, we choose to target the policy parameter space Π too.

When?

The timing of the interaction varies with respect to its general activity during the whole learning process. The rhythm of social interaction varies considerably among studies of social learning:

• At a fixed frequency: In classical imitation learning, the learner uses a demonstration to improve its learning at every policy it performs [START_REF] Argall | Learning robot motion control with demonstration and advice-operators[END_REF][START_REF] Argall | Teacher feedback to scaffold and refine demonstrated motion primitives on a mobile robot[END_REF][START_REF] Cakmak | Effects of social exploration mechanisms on robot learning[END_REF]. This solution is illadapted to the teacher's availability or the needs of the learner who requires more support in difficult situations.

• Beginning of learning: A limited number of examples are given to initialise the learning, as a basic behaviours repertoire [START_REF] Argall | Learning robot motion control with demonstration and advice-operators[END_REF][START_REF] Argall | Teacher feedback to scaffold and refine demonstrated motion primitives on a mobile robot[END_REF], or a sample behaviour to be optimised [START_REF] Kormushev | Robot motor skill coordination with EM-based reinforcement learning[END_REF][START_REF] Peters | Reinforcement learning of motor skills with policy gradients[END_REF]. The learner is endowed with some basic competence before self-exploration. Nevertheless, if the interactions are restricted to the beginning, the learner could face difficulties adapting to changes in the environment.

• At the teacher's initiative: The teacher alone decides when he interacts with the robot [START_REF] Thomaz | Socially Guided Machine Learning[END_REF], by for instance giving corrections when seeing errors [START_REF] Cakmak | Designing interactions for robot active learners. Autonomous Mental Development[END_REF][START_REF] Koenig | Communication and knowledge sharing in human-robot interaction and learning from demonstration[END_REF]. Nevertheless, it still is time consuming as he needs to monitor the robot's errors to give adequate information to the learner.

• At the learner's initiative: The interactive learner can request for the teacher's help in an ambiguous [START_REF] Cakmak | Designing interactions for robot active learners. Autonomous Mental Development[END_REF][START_REF] Chernova | Interactive policy learning through confidence-based autonomy[END_REF] or unknown [START_REF] Thomaz | Socially Guided Machine Learning[END_REF] situation, or only reproduces the observations when the observed outcome matches its goal during goal-based imitation or mimicking [START_REF] Cakmak | Effects of social exploration mechanisms on robot learning[END_REF]. This approach is the most beneficial to the learner, for the information ar-rives as it needs them, and the teacher needs not monitor the process.

These 4 types can be classified into 2 larger groups:

• batch learning, where the data provided to the learner is decided before the learning phase, and is given independently of the learning progress, generally in the beginning of the learning phase.

• interactive learning, where the user interacts with the incrementally learning robot, either at the teacher's or the learner's initiative.

Who?

While most social guidance studies only consider a single teacher, in natural environments, a household robot in reality interacts with several users. Moreover, being able to request help to different experts is also an efficient way to address the problem of the reliability of the teacher. Imitation learning studies often rely on the quality of the demonstrations, whereas in reality a teacher can be performant for some outcomes but not for others. Demonstrations can be ambiguous, unsuccessful or suboptimal in certain areas. Like students who learn from different teachers who are experts in the different topics of a curriculum, a robot learner should be able to determine its best teacher for the different outcomes it wants to achieve.

In this work, we consider the possibility of a learner to observe and imitate from several teachers, as much like a child in a natural environment would observe and imitate several adults in his surrounding throughout his development. In this case, choosing whom to imitate, recognising who is the expert in the outcomes we need to make progress, constitutes an important strategy choice.

Actively Learning When, Who and What to Imitate

For the model and experiments presented below, our choice of social guidance among this listing of social learning is:

• What: We opted for an information flow targeting both policy and outcome spaces, to enable the biggest progress for the learner. It can imitate to reproduce either a demonstrated policy or outcome. Therefore, our learner can decide whether to mimic and emulate by learning what is the most interesting information.

• When: Interactive learning at the learner's initiative seems the most natural interaction approach, the most efficient for learning and less costly for the teacher than if he would have to monitor the learner's progress to adapt his demonstrations. The robot has to learn when it is useful to imitate.

• Who: Interactive learning where the learner can choose who to interact with and to whom to ask for help, is an important strategy choice in learning.

Thus, it learns to answer the four main questions of imitation learning: "what, how, when and who to imitate" [START_REF] Breazeal | Robots that imitate humans[END_REF][START_REF] Dautenhahn | Imitation in Animals and Artifacts[END_REF] at the same time. We address active learning for varied outcomes with multiple strategies, multiple teachers, with a structured continuous outcome space (embedding sub-spaces with different properties). The strategies we consider are autonomous selfexploration, emulation and mimicking, by interactive learning with several teachers. Hereafter we describe the design of our SGIM-ACTS (Socially Guided Intrinsic Motivation with Active Choice of Teacher and Strategy) algorithm. Then we show through an illustration experiment that SGIM-ACTS efficiently learns to realise different types of outcomes in continuous outcome spaces, and it coherently selects the right teacher to learn from.

Algorithm Description

In this section, we describe the SGIM-ACTS architecture by giving a behavioural outline in section 3.1, before describing its general structure in section 3.2. We then detail the different functions in sections 3.3 and 3.4. The overall architecture is summarised in Algorithm 3.1 and is illustrated in fig. 3 .

Architecture Outline

SGIM-ACTS is an architecture that merges intrinsically motivated self-exploration with interactive learning as socially guided exploration. In the latter case, a teacher performs an observed trajectory ζ which achieves an outcome τ . Note that the observed trajectory might be impossible for the learner to re-execute, and he can only approach it best with a policy π θ .

The agent learns to achieve different types of outcomes by actively choosing which outcomes to focus on and set as goals, which data collection strategy to adopt and to which teacher to ask for help. It learns local inverse and forward models in complex, redundant and continuous spaces. SGIM-ACTS learns by episodes during which it actively chooses simultaneously an outcome τ ∈ T to reach and a learning strategy with a specific teacher (cf. 3.4.3). Its choice σ is selected between : intrinsically motivated exploration, mimicry from teacher 1, emulation of teacher 1, mimicry from teacher 2, emulation of teacher 2 .... In an episode under a mimicking strategy (fig. 3), our SGIM-ACTS learner actively self-generates a goal τ where its competence improvement is maximal (cf. 3.4.3). The SGIM-ACTS learner explores preferentially goal outcomes easy to reach and where it makes progress the fastest. The selected teacher answers its request with a demonstration [ζ τ ] to produce an outcome τ that is closest to τ (cf. 3.3.1). The robot mimics the teacher to reproduce ζ , for a fixed duration, by performing policies π θ which are small variations of an approximation of ζ .

In an episode under an emulation strategy (fig. 3), our SGIM-ACTS learner observes from the selected teacher a demonstration [ζ τ ]. It tries different policies using goal-directed optimisation algorithms to approach the observed outcome τ , without taking into account the demonstrated policy ζ . It re-uses and optimises its policy repertoire built through its past autonomous and socially guided explorations (cf. 3.3.2). The episode ends after a fixed duration.

In an episode under the intrinsic motivation strategy (fig. 3), it explores autonomously following the SAGG-RIAC algorithm [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF]. It actively self-generates a goal τ where its competence improvement is maximal (cf. 3.4.3), as in the mimicking strategy. Then, it explores which policy π θ can achieve τ best. It tries different policies to approach the self-determined outcome τ , as in the emulation strategy (cf. 3.3.2). The episode ends after a fixed duration. The intrinsic motivation and emulation strategies differ mainly by the way the goal outcome is chosen. An extensive study of the role of these different learning strategies can be found in [START_REF] Sao | Properties for efficient demonstrations to a socially guided intrinsically motivated learner[END_REF]. Thus the mimicry exploration increases the learner's policy repertoire on which to build up emulation and self-exploration, while biasing the policy space exploration. Demonstrations with structured policy sets, similar policy shapes, bias the policy space exploration to interesting subspaces, that allow the robot to overcome high-dimensionality and redundancy issues and interpolate to generalise in continuous outcome spaces. With emulation learning, the teacher influences the exploration of the outcome space. He can hinder the exploration of subspaces attracting the learner's attention to other subspaces. On the contrary, he can encourage their exploration by making demonstrations in those subspaces. Selfexploration is essential to build up on these demonstrations to overcome correspondence problems and collect more data to acquire better precision according to the embodiment of the robot. This behavioural description of SGIM-ACTS is followed in the next section by the description of its architecture.

Hierarchical Structure

SGIM-ACTS improves its estimation L -1 to minimise I = τ P(τ)J(τ M(L -1 (τ))) τ by exploring with the different strategies the outcome and policy spaces. Its architecture is separated into three levels:

• A Strategy Exploration level which decides actively which learning strategy to use between intrinsic motivation, emulation and mimicry, and which teacher to ask for demonstrations (Select Goal Outcome and Strategy). To motivate its choice, it maps T in terms of interest level for each strategy (Outcome and Strategy Interest Mapping) to keep track which strategy and which subspace of T leads to the best learning progress.

• An Outcome Space Exploration level which minimises I by exploring T. It decides actively which outcome τ to focus on, to minimise J(τ M(L -1 (τ ))), according to the adopted strategy. In the case of an emulation strategy, it sets the observed outcome of the demonstration τ as a goal. In the case of mimicry and intrinsic motivation strategies, it self-determines a goal τ selected by the Select Goal Outcome and Strategy function.

• A Policy Space Exploration level which explores the policy parameters space Π to improve its estimation of J and estimate the inverse mapping L -1 (τ The exploration in the three levels is the key to the robustness of SGIM-ACTS in high dimensional policy spaces.

Policy Space Exploration

Mimicry

This function tries to mimic a demonstration (ζ τ ) with policy parameters θ = θ +θ with a random movement parameter variation |θ | < and π θ is the closest policy to reproduce ζ . θ is computed by minimising over θ the distance between ζ and the motor primitives π θ . This function thus makes an estimate of J(τ M(θ)) in the locality of θ . After a short fixed number of times, SGIM-ACTS computes its competence at reaching the goal τ .

Goal-Directed Policy Optimisation

This function searches for policies π θ that guide the system toward the goal τ by 1) building local models of J during exploration that can be re-used for later goals and 2) updating its estimated inverse model L -1 . In the experiments below, exploration mixes local optimisation with the Nelder-Mead simplex algorithm [START_REF] Lagarias | Convergence properties of the nelder-mead simplex method in low dimensions[END_REF] and global random exploration to avoid local minima. The measures are used to build memory-based local direct and inverse models, using interpolation and more specifically locally weighted learning with a gaussian kernel such as presented in [START_REF] Atkeson | Locally weighted learning[END_REF].

Strategy and Outcome Space Exploration

Emulation

In the emulation strategy, the learner explores outcomes τ that he observed from the demonstrations: τ ← τ . The learner tries to achieve τ by goal-oriented policy optimisation, which allows data collection and updating of L -1 .

Outcome and Strategy Interest Mapping

T is partitioned according to interest levels. We note = {R T = ∪ R } a partition of T . For each outcome τ explored with strategy σ , the learner evaluates its competence progress, where competence measure assesses how close it can reach τ: γ = J(τ M(L -1 (τ))). A high value of γ means a good competence at reaching the goal by strategy σ . For each episode, it can compute its competence for the goal outcome at the beginning of the episode γ 1 and the end of the episode γ 2 after trying A movements and measure its competence progress:

= 2( (α * γ 1 -γ 2 |T | • A ) -1) with ( ) = + - 2 ( 2
)
where α is a constant and |T | is the size of the subspace T .

T is partitioned so as to maximally discriminate areas according to their competence progress, as described in Algorithm 3.2 and [START_REF] Baranes | Active learning of inverse models with intrinsically motivated goal exploration in robots[END_REF]. For each strategy σ , we define a cost κ(σ ), which are weights for the computation of the interest of each region of the outcome space. κ(σ ) represents the preference of the teachers to help the

R (σ ) = |R | =|R |-δ κ(σ ) if |R (σ )| > then ← Split R .

end if end for return

robot or not, or the cost in time and energy ... of each strategy, and in this study κ(σ ) are set to arbitrary constant values. We compute the interest as the local competence progress, over a sliding time window of the δ most recent goals attempted inside R with strategy σ which builds the list of competence progress measures

R (σ ) = { 1 |R (σ )| }: R (σ ) = |R (σ )| =|R (σ )|-δ κ(σ ) (3)
The partition of T is done recursively and so as to maximally discriminate areas according to their levels of interest. A split is triggered once a number of outcomes has been attempted inside R with the same strategy σ . The split separates areas of different interest levels and different reaching difficulties. The split of a region R into R +1 and R +2 is done by selecting among m randomly generated splits, a split dimension ∈ |T | and then a position (we suppose that R ⊂ T ⊂ T with T a n-dimensional space) such that:

• All the τ ∈ R +1 have a jth component smaller than vj;

• All the τ ∈ R +2 have a jth component higher than vj;

• It maximises the quantity Q ( ) = |R +1 | |R +2 | | R +1 ((σ )) - R +2 (σ )|
, where |R | is the size of the region R ;

Select Goal Outcome and Strategy

In order to balance exploitation and exploration, the next goal outcome and strategy are selected according to one of the 3 modes, chosen stochastically with respectively probabilities p1, p2 and p3:

• mode 1: choose σ and τ ∈ T randomly. It ensures a minimum of exploration of the full strategy and outcome spaces.

• mode 2: choose the region R (σ ) and thus the strategy σ with a probability proportional to its interest value R (σ ):

P (σ ) = R (σ ) -min( R ) |R | =1 R (σ ) -min( R ) (4)
A outcome τ is then generated randomly inside R . This mode uses exploitation to choose the region with highest interest measure.

• mode 3: the strategy and regions are selected like in mode 2, but the outcome τ ∈ R is generated close to the already experimented one which received the lowest competence estimation. This mode also uses exploitation to choose the best outcome and strategy with respect to interest measures.

We illustrate in the following section this hierarchical algorithm through an illustration example where a robot learns to throw a ball or to place it at different angles with 7 strategies: intrinsically motivated exploration, mimicry from 3 teachers and emulation from 3 teachers.

Throwing and Placing a Ball

Experimental Setup

In our simulated experimental setup, we have a 1 degree-offreedom arm place a ball at different angles or throw the ball by controlling its angular acceleration φ (fig. 1). The time evolution of its angular acceleration is described with motor primitives determined by 14 parameters. Π ⊂ R 14 as described in 4.1.1. The outcome space is composed of 2 types of outcomes T = T 1 ∪ T 2, that we detail in 4.1.2 and 4.1.3.

Policy Parameter Space

Starting from angle φ = 0, the robot can control its angular acceleration φ. Its movement is parameterised by ( φ1 1 φ7 7 ) which defines the acceleration of the arm for the 7 durations . It thus defines φ( ) as a piecewise constant function. The policy parameter space is arbitrarily set to a 14 dimensional space. where

T 1 ⊂ R 2 , T 2 ⊂ R and T 3 ⊂ R 3 .
T is partitioned in regions R to which are associated measures of competences γ for each strategy. The "Select Goal Outcome and Strategy" function chooses the (region, strategy) pair that makes the most competence progress.

Throwing Outcomes

The first type of outcomes is the different distance x and height h at which the ball B can be thrown. T 1 = {( )} is a continuous space of dimension 2. The ball, initially in the robot's hand is first accelerated by the robot arm, and then automatically released:

• at position OB =0 which is the position of the tip of the arm,

• with velocity OB =0 which magnitude is the velocity of the arm, and which direction is the tangent of the arm movement.

Then, the ball falls under gravity force, described by the equation:

OB = 2 • 2 + OB =0 • + OB =0 (5)
where is the gravity force. x is therefore computed for , the time when the ball touches the ground, or in other words the solution to the 2nd polynomial equation:

- 2 • 2 + =0 • + =0 = 0 (6)
The maximum height is also directly computed by equation:

= =0 + ( OB =0 ) 2 2 ; (7)
To make the throwing less trivial, we also added a wall as an obstacle at x= 10. The ball can bounce on the wall using an immobile wall model and elastic collision.

Placing Outcomes

The second type of outcomes is placing a ball at different angles φ. Therefore T 2 is of dimension 1. To achieve an outcome in T 2, the robot has to stop its arm in a direction φ before releasing the ball, i.e. it learns to reach φ at a small velocity | | < | |. Any policy would move the arm to a final angle φ, but to "place" the ball at an angle, it also needs to reach a velocity smaller than | |. Therefore placing a ball is difficult. The robot learns which arm movement it needs to perform to either place at a given angle φ or to throw a ball at a given height and distance. Mathematically speaking, it learns highly redundant mappings between a 14-dimensional policy space and a union of a 1D and a 2D continuous outcome spaces. In our experimental setup, the outcome space is thus the union of two continuous spaces of different dimensionalities, related to throwing and placing skills, which makes it complex because of the continuous and composite nature of the space. The complexity of the placing of the ball depends on the physics of the body and on the structure of motor commands. We choose to control the robot by angular acceleration to emphasise the difference in the ease of control between the "throwing outcomes" which require rather a velocity control, and the "placing outcomes" which require rather a position control. Given the motor control by acceleration and the encoding of motor primitives, the placing outcomes are thus more difficult to achieve than the throwing outcomes.

Several Teachers and Strategies

We create simulated teachers by building 3 demonstration sets from which to pick a random demonstration when asked by the learner :

• teacher 1 has learned how to throw a ball with SAGG-RIAC. The teacher 1 has the same motor primitives • teacher 2 is an expert in placing, programmed by an explicit equation to place at any angle with a null velocity. The teacher 2 too has the same motor primitives encoding as the learner, and the robot observes from the demonstrated trajectories directly the demonstrated ( φ1 1 φ7 7 ).

• teacher 3 is an expert in placing, except that in this case the learner faces correspondence problems and misinterprets the two parameters φ6 and φ7 as the opposite values. In this experiment, we do not attempt to solve this correspondence problem. We also note that while the learner has issues mimicking teacher 3, he has no issues emulating teacher 3, as the outcome space parametrisation is the same.

Therefore in our experiment, the interactive learner can choose between 7 strategies : SAGG-RIAC autonomous exploration, emulation of each of the 3 teachers or mimicry of each of the 3 teachers.

Comparison of Learning Algorithms

To assess the efficiency of SGIM-ACTS, we decide to compare the performance of several learning algorithms (fig. 5):

• Random exploration : throughout the experiment, the robot learns by picking policy parameters randomly. It explores randomly the policy parameter space Π.

• SAGG-RIAC : throughout the experiment, the robot uses active goal-babbling to explore autonomously, without taking into account any demonstration by the teacher, and is driven by intrinsic motivation.

• mimicry : at a regular frequency, the learner determines a goal τ where learning progress is maximal, and requests to the chosen teacher a demonstration. The teacher selects among his data set a demonstration [ζ τ ] so that τ = • SGIM-ACTS : interactive learning where the robot learns by actively choosing between intrinsic motivation strategy or one of the social learning strategies with the chosen teacher: mimicking or emulation.

We run simulations with the following parameters. The costs of all socially guided strategies κ(σ ) are set to 2, and the cost of intrinsic motivation is set to 1. The probabilities for the different modes of selecting a region of the outcome space and a strategy are: p1 = 0.05, p2 = 0.7 and p3 = 0.25. Other parameters are = 0 05 = 10, α = 1000 and = 0 01. For each experiment, we let the robot perform 8000 actions in total, and evaluate its performance every 1000 actions, by requiring the system to produce outcomes from a benchmark set that is evenly distributed in the outcome space and independent from the learning data.

Results

The comparison of these four learning algorithms in fig. 6 shows that SGIM-ACTS decreases its cumulative error for both placing and throwing. It performs better than autonomous exploration by random search or intrinsic motivation, and better than any socially guided exploration with any teacher. Fig. 7 details that SGIM-ACTS error rate for both placing and throwing is low. For throwing, SGIM-ACTS performs the best in terms of error rate and speed because it could find the right strategy. We also note that random exploration and SAGG-RIAC also perform well for solving the 2nd degree polynomial equation ( 5) to achieve throwing outcomes. While mimicking and emulating teacher 1 decreases the error as expected, mimicking and emulating a teacher who is expert in another kind of outcomes and is bad in that outcome leaves a high error rate. For placing, SGIM-ACTS makes less error than all other algorithms. Indeed, as we expected, mimicking the teacher 2, and emulating teachers 2 and 3 enhances low error rates, while mimicking a teacher with correspondence problem (teacher 3) or an expert on another outcome (teacher 1) gives poor result. We also note that for both outcomes, mimicry does not lead to important learning progress, and the error curve is almost flat. This is due to the lack of exploration which leads the learner to ask demonstrations for outcomes only in a small subspace. Indeed, we see in fig. 8 which illustrates the percentage times each strategy is chosen by SGIM-ACTS with respect to time, that mimicry of teacher 3, which lacks efficiency because of the correspondence problem, is seldom chosen by SGIM-ACTS. Mimicry and emulation of teacher 1 is also little used because autonomous learning learns quickly throwing outcomes. Teachers 2 and 3 are exactly the same with respect to the outcomes they demonstrate, and are emulated in the same proportion. This figure also shows that the more the learner cumulates knowledge, the more autonomous he grows : his percentage of autonomous learning increases steadily.

Not only does he choose the right strategies, but also the right outcome to concentrate on. Fig. 9 shows that he concentrates in the end more on placing, which are more difficult. Finally, fig. 10 shows the percentage of times over all the experiments where he chooses at the same time each outcome type, a strategy and a teacher. We can see that for the placing outcomes, he seldom requests help from the teacher 1, as he learns that teacher 1 does not know how to place the ball. Likewise, because of the correspondence problems, he does not mimic teacher 3. But he learns that mimicking teacher 2 and emulating teachers 2 and 3 are useful for placing outcomes. For the throwing outcomes, he uses slightly more the autonomous exploration strategy, as he can learn efficiently by himself. The high percentage for the other strategies is due to the fact that the throwing outcomes are easy to learn, therefore are learned in the beginning when a lot of sampling of all possible strategies is carried out. SGIM-ACTS is therefore consistent in its choice of outcomes , data collection strategies and teachers.

Conclusion and Discussion

We presented the SGIM-ACTS (Socially Guided Intrinsic Motivation with Active Choice of Teacher and Strategy) algorithm that efficiently and actively combines autonomous selfexploration and interactive learning, to address the learning of multiple outcomes, with outcomes of different types, and with different data collection strategies. In particular, it learns actively to decide on the fundamental questions of programming by demonstration: what and how to learn; but also what, how, when and who to imitate. This interactive learner decides efficiently and coherently whether to use social guidance. It learns when to ask for demonstration, what kind of demonstrations (action to mimic or outcome to emulate) and who to ask for demonstrations, among the available teachers. Its hierarchical architecture bears three levels. The lower level explores the policy parameters space to build skills for determined goal outcomes. The upper level explores the outcome space to evaluate for which outcomes he makes the best progress. A meta-level actively chooses the outcome and data collection strategy that leads to the best competence progress. We showed through our illustration example that SGIM-ACTS can focus on the outcome where it learns the most, while choosing the most appropriate associated data collection strategy. The active learner can explore efficiently a composite and continuous outcome space to be able to generalise for new outcomes of the outcome spaces. SGIM-ACTS has been shown an efficient method for learning with multiple teachers and multiple outcome types. The number of outcomes used in the experiment is infinite, with a continuous outcome space that is made of 2 types of outcomes, but all the formalism and framework is in principle scalable to a higher number of types of outcomes. Likewise, the method should apply to domestic or industrial robots who usually interact with a finite number of teachers. Even in the case of correspondence problems, the system still takes advantage of the demonstrations to bias its exploration of the outcome space. When the discrepancies between the teacher and the learner are small, demonstrations advantageously bias the exploration of the outcome space, as argued in [START_REF] Sao | Properties for efficient demonstrations to a socially guided intrinsically motivated learner[END_REF]. Future work should test SGIM-ACTS on more complex environments, and with real physical robots and everyday human users. It would also be interesting to compare the outcomes selected by our system to developmental behavioural studies, and highlight developmental trajectories.

Figure 1 :

 1 Figure 1: An arm, described by its angle φ, is controlled by a motor primitive with 14 continuous parameters (taking bounded values) that determine the evolution of its acceleration φ . A ball is held by the arm and then released at the end of the motion. The objective of the robot is to learn the mapping between the parameters of the motor primitive and two types of outcomes he can produce: a ball thrown at distance x and height h, or a ball placed at the arm tip at angle φ with velocity smaller than | |.

Figure 2 :

 2 Figure 2: Representation of the problem. The environment can evolve to an outcome state τ by means of the learner's policy of parameter θ or the teacher's actions ζ. The learner and the teacher have a priori different policy spaces. The learner estimates L -1 : T → Π. By emulation or mimicry, the learner can take advantage of the demonstrations (ζ,τ ) of the teacher to improve its estimation L -1 .

Figure 4 :

 4 Figure 4: The selection of outcome and strategy is based on a partition of the outcome space with respect to different competence progress levels. We illustrate with the case of an outcome space of 3 different types of outcomes. T = T 1 ∪ T 2 ∪ T 3whereT 1 ⊂ R 2 , T 2 ⊂ R and T 3 ⊂ R 3 .T is partitioned in regions R to which are associated measures of competences γ for each strategy. The "Select Goal Outcome and Strategy" function chooses the (region, strategy) pair that makes the most competence progress.

Figure 5 :

 5 Figure 5: Comparison of several learning algorithms encoding as the learner, and the robot observes from the demonstrated trajectories directly the demonstrated ( φ1 1 φ7 7 ).

τ∈{DS

  } ||τ -τ||. The learner mimics the demonstrated policy ζ by repeating the movement with small variations.

Figure 6 :

 6 Figure 6: Mean error for the different learning algorithms averaged over the two sub outcome spaces (final variance value ∆ is indicated in the legend) . • emulation : at a regular frequency, the learner determines a goal τ where learning progress is maximal, and requests to the chosen teacher a demonstration. The teacher selects among his data set a demonstration [ζ τ ] so that τ = τ∈{D S } ||τ -τ||. The learner tries to reproduce the outcome τ .

Figure 7 :

 7 Figure 7: Mean error for the different learning algorithms for each of the throwing outcomes and placing outcomes separately. The legend is the same as in fig. 6.

Figure 8 :Figure 9 :

 89 Figure 8: Strategy chosen by SGIM-ACTS through time: percentage of times each strategy is chosen for several runs of the experiment.

Figure 10 :

 10 Figure 10: Consistency in the choice of outcome, teacher and strategy: percentage of times each strategy, teacher and outcome are chosen over all the history of the robot.

  Figure 3: Time flow chart of SGIM-ACTS, which combines Intrinsic Motivation and Mimicking and Emulation into 3 layers that pertain to the strategy, the outcome space and the policy space exploration respectively.

	Socially Guided
	Exploration
	Strategies
	Autonomous Exploration Strategies	σ = mimic teacher i	Select Goal Outcome &Strategy	σ = explore autonomously	σ = emulate teacher i
		Request to		progress		Request to
	teacher n Level Correspondence Strategy		Outcome & Strategy Interest Mapping		teacher n Correspondence
	Outcome		τg		τg	τd
	Space Exploration		J	J		Emulate τg ← τd
	Policy
	Space
	Exploration	ϴd	ϴr, τr	ϴr, τr	Goal Directed
		Mimic		Policy Optimisation
						SG

). With the mimicry learning strategy, it mimics the demonstrated trajectory ζ by the chosen teacher to estimate J around that locality (Mimicry). With the emulation and autonomous exploration strategy, the Goal-Directed Policy Optimisation function minimises J(τ M(θ)) with respect to θ. It attempts to reach the goals τ set by the Strategy and Outcome Space Exploration level, and gets a better estimate of J that it can use later on to reach other goals. It finally returns to the Strategy and Outcome Space Exploration level the measure of competence progress for reaching τ or τ .

  Find the region R ∈ such that τ ∈ R . Add in R (σ ), the list of competence progress measures of experiments τ ∈ R with strategy σ . Compute the new value of competence progress of R (σ ):

	Algorithm 3.2 [ ] = Update Outcome and Strategy Interest
		Mapping( M	τ	σ )
	input: : set of regions R and corresponding	R (σ ) for each
	strategy σ .
	input: τ	: goal outcome of the episode and its progress
	measure.
	input: M	: the set of all observed outcomes during the episode
	and their progress measures (τ	).
	input: σ : strategy and teacher used during the episode.
	parameter: M : the maximal number of elements inside a region.
	parameter: δ : a time window used to compute the interest.
	for all (τ	) ∈ {M	(τ	)} do

Acknowledgement

This work was supported by the French ANR program (ANR 2010 BLAN 0216 01) through Project MACSi, as well by ERC Starting Grant EXPLORERS 240007.


Y Kervran

S Janfaoui

O De Sagazan

S Crand

N Coulon

J P Gauthier

T Mohammed-Brahim

Microcrystalline silicon gauges on flexible substrates for high deformations with high spatial resolution

Keywords: Microcrystalline silicon, strain gauge, spatial resolution, flexible substrate, resistor, transistor

Microcrystalline silicon films are used as piezoresistive material to fabricate resistor and transistor strain gauges. Very small gauges are fabricated allowing the possibility to measure high deformations with both high sensitivity and spatial resolution. Resistor gauges with 5 x 125 µm² smallest size showed a gauge factor of -24 when applying high strains (0.55%, radius of curvature 5 mm). Field effect thin film transistor (TFT) gauges showed a gauge factor of -85 but under slightly lower strains (0.45%, radius of curvature 10 mm). This value is close to strain gauges made of single crystalline silicon (~ 100, regarding the absolute value), but which are not flexible and then cannot support high deformations. The more complex technological process of TFT gauges could reserve their use to the measurement of low deformations when high resolution is required, while resistor gauges could be chosen when focusing on a mix between size and sensitivity in a large range of deformations.

Introduction

There is now an increasing demand for light and foldable electronics. Numerous worked in robotics [START_REF] Someya | Conformable, flexible, large-area networks of pressure and thermal sensors with organic transistor active matrixes[END_REF], health system [START_REF] Jung | Point of care temperature and respiration monitoring sensors for smart fabric applications[END_REF], aeronautics [START_REF] Casano | Experimental modal analysis of an aircraft model wing by embedded filter Bragg grating sensors[END_REF] and so on are interested in. To ensure the monitoring of such systems, strain gauges are usually designed to study the electrical response of sensors (e.g. force/pressure sensor with a cavity [START_REF] Kwon | Design and fabrication of a flexible three-axial tactile sensor array based on polyimide micromachining[END_REF]) to any deformations.

The sensitivity of a strain gauge is defined by the Gauge Factor (GF), the ratio of relative change in electrical resistance (R/R) to relative change in length of a resistor (L/L, the strain ), R/R = GF x (L/L). Two main types of materials are used as strain gauges: metals and semi-conductors. For metals, the most common ones and commercially available, the GF ~ 2 is weak, and the need of long wires to get measurable resistances make their use prohibitive when a high density is needed, like a deformation mapping with high spatial resolution [START_REF] Yang | Design of Microfabricated Strain Gauge Array to Monitor Bone Deformation In Vitro and In Vivo[END_REF]. In the case of semiconductors, such as single crystalline silicon, the controlled resistivity and its high |GF| ~ 100, due to its high piezoresistivity, can help to reach a high resolution.

The problem is that single crystalline silicon made resistors are rigid even for small sizes, and cannot support high deformations. One solution is to thin enough the silicon wafer on which the resistance is fabricated and to report it on a flexible substrate [START_REF] Won | Piezoresistive Strain Sensors and Multiplexed Arrays Using Assemblies of Single-Crystalline Silicon Nanoribbons on Plastic Substrates[END_REF]. However, the costs of thinning and report operations are prohibitive. Another solution is to fabricate silicon resistors directly on a flexible substrate. Deposited silicon films cannot be single crystalline but their structures are amorphous or polycrystalline with some nanometers to micrometers sized grains. Due to this disordered structure, the GF value is lower than 100 but much higher than 2.

Direct process on flexible substrate needs to take care on its chemical and mechanical characteristics. First, it has to support the different chemical solutions and the maximum temperature of the process. Then, other characteristics can be specific: biocompatibility for biological applications, more or less flexibility (depending on the Young modulus), optical transparency in flexible displays field. Polyimide (PI) such as Kapton ® can hold 400°C during 120 minutes with 1.2% shrinkage but it is not transparent (and usually used at lower temperatures). Transparent plastic that hold the highest temperature is polyethylene naphthalate (PEN). When annealed, it can reach 180°C with 0.2% shrinkage during 30 minutes. Both are produced by DuPont™ Teijin Films.

In this paper, polyimide and PEN are used. To be compatible with the limited maximum temperature reached by PEN, silicon resistors are fabricated on both substrates at a lower temperature than 180°C. This requirement is a challenge to obtain well crystallized silicon. But if the challenge is jumped, gauges made by crystallized silicon deposited on plastics can fulfil all the requirements of high GF and miniaturization using lithography technologies.

Previously, authors [START_REF] Alpuim | Fabrication of a strain sensor for bone implant failure detection based on piezoresistive doped nanocrystalline silicon[END_REF][START_REF] Alpuim | Piezoresistive silicon thin film sensor array for biomedical applications[END_REF] reported gauges based on crystallized silicon films deposited on flexible substrate at low temperature with a highest GF ~ 20/30 (absolute value). As expected, this factor is lower than single crystalline silicon GF but much higher than metal ones. However, it was given only for weak deformations with a strain lower than 0.25%. For practical use, strain gauges have to measure much higher deformations. That is why, small gauges are here studied in this purpose. 2 types are used: the first one is a basic resistor and the second one is a thin film transistor allowing an expected better sensitivity thanks to the field effect.

Resistive Gauges

Process

Microcrystalline silicon (µc-Si) is used as piezoresistive material. The µc-Si films are deposited at 165°C in a Plasma Enhanced Chemical Vapor Deposition (PECVD) reactor using silane, hydrogen and argon. Previously [START_REF] Kandoussi | Nanocrystalline silicon TFT process using silane diluted in argonhydrogen mixtures[END_REF], we have shown that the crystalline fraction in µc-Si films is improved when adding argon in the usual silane and hydrogen gas mixture [START_REF] Matsuda | Microcrystalline silicon: Growth and device application[END_REF]. Here, 1.5 sccm silane, 75 sccm hydrogen and 75 sccm argon (1% dilution of silane in Ar-H 2 mixture) are used. Then, arsine as N-type doping gas is added in the gas mixture to vary the resistivity/conductivity of µc-Si film in a very large range depending on the arsine flow. Thus, the highest conductivity obtained is 12 S.cm -1 for a 50 nm thick N-type µc-Si film, and 3x10 -9 S.cm -1 for the same thickness but undoped film. So, the electrical resistance of the gauge can be adapted depending on the needs.

To study the answer of microcrystalline silicon film based strain gauges, differently sized resistors are fabricated on polyimide sheets covered by 50 nm thick silicon nitride deposited by RF-PECVD at 150°C using silane and ammonia. The slightly N-type doped µc-Si film is deposited and islands patterned by Reactive Ion Etching (RIE) with SF 6 plasma. Aluminium film is then deposited by Joule Effect Evaporation and patterned by wet etching to form different gaps L between electrodes. The total structure uses the known Transmission Line Method (TLM, Figure 1).

TLM

Figure 1:

The TLM structure designed. The width of the resistors is W. The gap L between the electrodes varies between 5 µm to 160 µm. It doubles each time between 2 successive contacts. The pads (1mm²) are designed in order to be able to place the probes during bending tests.

The aim is to separate the effects of the gauge resistances from the contact resistances. Indeed, the total resistance of a gauge with its 2 contacts is given by R = L/(eW) + 2R C where  and e are respectively the resistivity and the thickness of the µc-Si film, L the distance between electrodes (the length of the gauge), W their width and R C is the contact resistance of one metal/semi-conductor interface. The resistance of the aluminium routing is neglected (R Al << R µcSi ). Then plotting the measured R as a function of L leads to a straight line (if contacts are ohmic) with a slope proportional to the resistivity . The piezoresistivity of the present µc-Si material can be then checked during tensile tests by bending the gauges with different radii of curvature using homemade tools (Figure 3). 16 TLM structures with 4 W (1000, 500, 250 and 125 µm) are fabricated on the same 5 x 5 cm² polyimide sheets. Each structure has 6 L (5, 10, 20, 40, 80, 160 µm), for a total of 24 different resistances. In each cell, resistors with same W and different L are measured. This allows checking the linearity of the variation of the resistance when L varies. Figure 2 shows this linearity for 3 cells with different W. The correlation factor of the fit is given in the figure to show the excellent linearity of the curves. This shows that contacts between aluminium electrodes and µc-Si film are ohmic. The slightly lower factor obtained for W = 1000 µm is due to the smallest resistances about the same order than the contact resistance. The resistivity of the µc-Si film can be determined then from the slopes of these straight lines. Its calculated value is 0.137 cm.

Bending Tests

After checking the true value of the resistivity of µc-Si film independently on contacts resistance, the TLM structures have been bent using homemade tools with 5 different radii of curvature (2.5, 2, 1.5, 1 and 0.5 cm, tensile tests). For each radius R, the strain applied to the structure is calculated using a simplified model. The real structure of the resistor is a stack of 3 films: 50 µm polyimide substrate, 50 nm thick top silicon nitride and 100 nm thick microcrystalline silicon films. But the model is constituted only by the µc-Si Al 1 mm substrate and the silicon nitride film due to its much higher Young modulus (270 GPa) than the other films. Thus, the mechanical behaviour of the resistors is mainly defined by the behaviour of the silicon nitride. Equation (1) [START_REF] Gleskova | Failure resistance of amorphous silicon transistors under extreme in-plane strain[END_REF] links the radius of curvature R to the strain  applied to the resistor for the simplified model. The procedure to study the sensitivity of the gauges consists in a first measurement of the 24 resistors without bending (flat substrate). Then, the TLM structure is stuck on the holder with the highest radius of curvature and the resistors measured. After that, the TLM structure is removed and the resistors measured again without bending (flat substrate). The previous steps are renewed by decreasing the radius of curvature. However, it is important to note that a first bending at the smallest radius must be applied to get a good linearity in the following results.

  2 2 1 1 2 1 2                      R d d f S
The most important result for practical applications is the comeback of the resistors to their initial value (when the resistors are flat) after each bending. Figure 4 shows an example of 6 resistances with the same W as a function of the gap L between the electrodes measured after each bending corresponding to decreased radii of curvature from 2.5 cm to 0.5 cm. For each L, the resistances show a relative difference with their original flatten value inferior to 1% and no differences between the linear straights can be highlighted which shows the good comeback. Figure 5 presents the mean relative variation of the 24 measured resistances as a function of the strain  calculated with the equation ( 1). The means are represented because no influence of the size of the resistances on the sensitivity has been noted and for each point, the standard deviation of the 24 data is also represented. Measures are more dispersed for high strains due to the difficulty to place the probes during high bending. The gauge factor, defined by the slope of the fit, is GF ~ -24 (with a minus sign because the resistances decrease during tensile tests). The value is similar to the gauge factor of crystallized silicon films given in references [START_REF] Alpuim | Fabrication of a strain sensor for bone implant failure detection based on piezoresistive doped nanocrystalline silicon[END_REF][START_REF] Alpuim | Piezoresistive silicon thin film sensor array for biomedical applications[END_REF]. But the value is given here for higher strains, with a maximum of 0.55%, corresponding to a curvature radius of 5 mm. Moreover, the small size of the gauges studied in this work (minimum of 5 x 125 µm²) allows the possibility to integrate such strain gauges when high spatial resolution is required. 3 Thin Film Transistor Gauges

Process

Microcrystalline silicon (µc-Si) based N-type Top-Gate TFTs are fabricated on 5 x 5 cm² sheets of PEN at a maximum temperature of 180°C. The sheet of PEN is encapsulated on both sides with 250 nm thick silicon nitride. A 100 nm thick layer undoped µc-Si is deposited in the same conditions as for resistive gauges. After that, a 70 nm thick layer arsenic doped µc-Si is deposited in the same manner as the undoped µc-Si using a mixture of silane, hydrogen and arsine. A first photolithography step is used to define drain, source and channel regions.

The µc-Si doped layer is etched by SF 6 plasma in a RIE reactor to define source and drain regions. Then, a second photolithography step is necessary in the purpose to insulate transistors one from each other. The undoped µc-Si is etched by SF 6 plasma. This step is followed by an RCA cleaning. After that, 300 nm thick silicon nitride film acting as gate insulator is deposited by PECVD at 150°C using mixture of silane, nitrogen and ammonia. A third photolithography step is used to open windows for drain and source contacts. The silicon nitride film is etched by SF 6 plasma. A layer of aluminium is deposited by thermal evaporation, and a fourth photolithography step is used to define the drain, source and gate electrodes. Aluminium is etched by wet way. The final structure is presented in Figure 6.

TFTs sensitivity to mechanical strain

The previous TFTs are electrically characterized when submitting to different curvature radii starting from infinite radius (flat TFTs) to a radius of 1.5 cm. An example of the drain-source current I DS versus the gate-source voltage V GS characteristic, plotted in linear regime under a drain-source voltage of V DS = +1V, is given in Figure 7.

The on-state current at constant V GS increases under tensile stress. To calculate the GF of these transistors, we have to plot this current as a function of the strain which is calculated from the radius of curvature by using, as previously for resistors, a model of the real structure of the TFT (stack of 5 films): 250 nm thick bottom silicon nitride film, 125 µm PEN substrate, 250 nm thick top silicon nitride, 100 nm thick µc-Si film, 300 nm thick silicon nitride gate insulator, 300 nm thick aluminium gate contact. Considering the much higher Young modulus of silicon nitride (270 GPa) and the different thicknesses, the total bended structure is considered constituted by 3 layers only: bottom 250 nm thick silicon nitride, 125 µm PEN substrate and a top 550 nm thick silicon nitride (Figure 8). Figure 9 shows the relative variation of the on-state current (I DS /I DS ), taken at V GS = 30V and V DS = 1V, as a function of the calculated strain. This variation is presented here for 4 differently sized TFTs, varying the width for the same length or varying the length for the same width. Whatever the size is, the plots can be fitted by a straight line with the same slope. For the present TFTs, the GF is -85. The minus sign is due to the definition of the gauge factor which implies ∆R/R, while here I/I is calculated. This value is very important in comparison with the resistive gauges made of the same microcrystalline silicon film.

 

       1 1 1 2 2 1 1 2 1 2 2 1 2 2 1 1 2 2 2 1 2 1 0                                     f f S d d d R R (2)

Conclusion

Two types of strain gauges were studied in this work. Both used the piezoresistivity of microcrystalline silicon films, deposited directly crystallized on plastic substrates at low temperature by using silane, hydrogen and argon gas mixture. The use of silicon films led us to use usual photolithography technologies.

The first type is a µc-Si film resistor with 2 electrodes. The smallest size was 5 x 125 µm², demonstrating a high spatial resolution. The reliability of the technological process was shown and a gauge factor of -24 was demonstrated even when applying high strain of 0.55% (for a radius of curvature of 5 mm).

The second type is a field effect TFT. The size of these TFTs is similar to the resistors (larger by a factor ~2) but the gauge factor of -85 is much higher, close to the value of single crystalline silicon gauges. The maximum strain applied was 0.45% (for a radius of curvature of 10 mm). Then, even if the technological process is more complicated but reliable, TFT gauges could be used to measure low deformations requiring high resolution.

Figure 2 :

 2 Figure 2: Variation of the resistances with 3 W (250, 500 and 1000 µm) when the distance L between the electrodes varies from 5 to 160 µm. The linearity of the curves is shown through the values of the correlation factor R of the linear fit.

( 1 )

 1 In this equation, d S and d f are the thicknesses of the substrate and the silicon nitride films respectively. and  are defined by are the Young modulus of the substrate (2.5 GPa) and the silicon nitride film (270 GPa) respectively.

Figure 3 :

 3 Figure 3: Optical images of homemade tools used to bend the structure with different radii of curvature for tensile tests. The structure is bent in the same direction of the current flow, to study only the longitudinal GF.

Figure 4 :

 4 Figure 4: Variation of the resistance for W = 1000 µm when the distance L between the electrodes varies from 5 to 160 µm. The variation is measured when the substrate is flat and after each successive bending.

Figure 5 :

 5 Figure 5: Relative variation of the gauge resistances as a function of the tensile strain. Each point is the mean of 24 resistances and the standard deviation is represented. The slope of the fit is GF ~ -24.

Figure 6 :

 6 Figure 6: Structure of µc-Si Top-Gate TFT.

Figure 7 :

 7 Figure 7: I DS vs V GS characteristic plotted in linear regime under V DS = +1V. TFT channel width is W = 100 µm and length is L = 20 µm.

Figure 8 :

 8 Figure 8: Mechanical model of the total TFT structure.

  Its value is in the same order than single crystalline silicon resistive gauges.

Figure 9 :

 9 Figure 9: Relative variation of the on-current of differently sized TFTs as a function of the tensile strain. The slope of the fit is GF ~ -85.