Paul Tonelli

Jean-Baptiste Mouret
email: mouret@isir.upmc.fr

On the Relationships between Generative Encodings, Regularity, and Learning Abilities when Evolving Plastic Artificial Neural Networks

Keywords:

On the Relationships between Generative Encodings, Regularity, and

Introduction

A major goal of bio-inspired artificial intelligence is to design artificial neural networks (ANNs) with abilities that resemble those of animal nervous systems [START_REF] Pfeifer | How the Body Shapes the Way we Think[END_REF][START_REF] Floreano | Neuroevolution: from architectures to learning[END_REF][START_REF] Floreano | Bio-Inspired Artificial Intelligence: Theories, Methods, and Technologies[END_REF][START_REF] Mouret | Encouraging behavioral diversity in evolutionary robotics: An empirical study[END_REF]. A promising approach to design such "artificial nervous systems" is to use evolutioninspired algorithms, in particular because Darwinian evolution is regarded as the primary process responsible for shaping their natural counterparts. Despite the large amount of work in this direction, striking differences still separate most artificiallyevolved networks from biological ones: biological nervous systems are much larger [START_REF] Braitenberg | Brain size and number of neurons: an exercise in synthetic neuroanatomy[END_REF], much more organized [START_REF] Meunier | Modular and Hierarchically Modular Organization of Brain Networks[END_REF], much more plastic [START_REF] Abbott | Synaptic plasticity: taming the beast[END_REF] and, overall, much more complex [START_REF]Principles of Neural Science[END_REF].

It is commonly believed that the key for understanding the evolution of large and organized neural networks is the developmental process that links genes to nervous systems [START_REF] Pfeifer | How the Body Shapes the Way we Think[END_REF][START_REF] Hornby | Creating high-level components with a generative representation for body-brain evolution[END_REF][START_REF] Stanley | A taxonomy for artificial embryogeny[END_REF][START_REF] Clune | On the performance of indirect encoding across the continuum of regularity[END_REF]. The genotype of animals does not encode each synapse individually, it instead describes rules of development that are executed multiple times to give birth to networks with regular patterns of connection. Influenced by this concept, many researchers proposed artificial developmental systems with diverse inspirations including chemical gradients [START_REF] Clune | On the performance of indirect encoding across the continuum of regularity[END_REF][START_REF] Stanley | A hypercubebased encoding for evolving large-scale neural networks[END_REF], gene regulatory networks [START_REF] Bongard | Evolving modular genetic regulatory networks[END_REF][START_REF] Mattiussi | Analog Genetic Encoding for the Evolution of Circuits and Networks[END_REF], cell divisions [START_REF] Gruau | Adding learning to the cellular development of neural networks: Evolution and the Baldwin effect[END_REF], computational neuroscience models [START_REF] Mouret | Importing the computational neuroscience toolbox into neuroevolution-application to basal ganglia[END_REF] and L-systems [START_REF] Hornby | Creating high-level components with a generative representation for body-brain evolution[END_REF]. A caged animal must learn to associate stimuli (here lights) to actions (here pushing a lever). The experimenter selects a stimulus/action association, presents it to the animal, record the action, and gives the reward to the animal. The experimenter then chooses another association in the association set and starts the cycle again. The association set is learned once the animal associates the right action to each stimulus. B. Formalization of the Skinner box as a task for an artificial neural network. Each stimulus is an input of the neural network. Positive and negative rewards are two additional inputs. The output is selected according to a softmax function (Methods) and the result of the softmax is looped back to the input layer.

Nonetheless, most networks evolved with developmental systems cannot change during the "lifetime" of the controlled agent, whereas animal nervous systems are continuously changing to enable on-line behavioral adaptation and learning [START_REF] Abbott | Synaptic plasticity: taming the beast[END_REF]. The basis of most of these changes seems to be provided by synaptic plasticity, that is, by the ability of synapses to strengthen or weaken over time [START_REF] Abbott | Synaptic plasticity: taming the beast[END_REF][START_REF] Hebb | The organization of behavior[END_REF]. Several papers report experiments in which neural networks with synaptic plasticity are evolved [START_REF] Floreano | Neuroevolution: from architectures to learning[END_REF][START_REF] Gruau | Adding learning to the cellular development of neural networks: Evolution and the Baldwin effect[END_REF][START_REF] Niv | Evolution of Reinforcement Learning in Uncertain Environments: A Simple Explanation for Complex Foraging Behaviors[END_REF][START_REF] Urzelai | Evolution of adaptive synapses: Robots with fast adaptive behavior in new environments[END_REF][START_REF] Soltoggio | Evolving neuromodulatory topologies for reinforcement learning-like problems[END_REF][START_REF] Soltoggio | Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios[END_REF][START_REF] Soltoggio | Novelty of behaviour as a basis for the neuro-evolution of operant reward learning[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF][START_REF] Risi | Indirectly Encoding Neural Plasticity as a Pattern of Local Rules[END_REF]. Yet, only a handful of them use developmental systems [START_REF] Gruau | Adding learning to the cellular development of neural networks: Evolution and the Baldwin effect[END_REF][START_REF] Soltoggio | Evolving neuromodulatory topologies for reinforcement learning-like problems[END_REF][START_REF] Risi | Indirectly Encoding Neural Plasticity as a Pattern of Local Rules[END_REF].

The present paper shows that these two topicsdevelopmental systems and synaptic plasticity-are actually deeply connected 1 .

One of the main challenge when designing ANNs with learning abilities is to make them capable of learning in a large class of situations, that is, designing them so they can adapt their behavior to maximize a reward signal (or minimize an error) in as many situations as possible. For instance, it has been famously shown that single layer perceptrons are only capable of learning linearly separable patterns [START_REF] Minsky | Perceptrons -Expanded Edition: An Introduction to Computational Geometry[END_REF], whereas multi-layer perceptrons can learn any non-linear function (provided enough neurons are available) [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF]. Single-layer perceptrons therefore possess lower learning abilities than multi-layer perceptrons: their architecture critically constrains what they can learn. When artificial evolution is used to design a plastic ANN, the topology of the networks is the result of the interactions between the fitness function, the encoding and the associated variation operators. As a consequence, the encoding and the fitness function have to be carefully crafted so that plastic neural networks are able to learn in as many situations as possible and, specifically, in situations that are not explicitly tested in the fitness function.

The most classic approach is to design a fitness function that tests each neural network in several test cases and rewards individuals that successfully adapt their behavior to each of them. To ensure that networks possess general learning abilities, it is then required to assess their abilities to learn in a new set of test cases, that is, test cases that have never been encountered by the evolutionary process [START_REF] Chalmers | The evolution of learning: An experiment in genetic connectionism[END_REF]. The success of this "episodic fitness" approach relies on the assumption that if enough test cases are used, then it should become easier for the evolutionary process to design a generic structure than a specialized one.

Unfortunately, even in simplistic and constrained toy problems, the reported experiments show that many test cases need to be included in the fitness function to obtain general learning abilities2 (e.g. 10 to 20 test cases in [START_REF] Chalmers | The evolution of learning: An experiment in genetic connectionism[END_REF]). For more complex problems, one can expect an exponential growth in the number of required test cases, because the number of possible test cases grows exponentially with the number of inputs/outputs. This approach is, therefore, unlikely to scale-up to life-like neuralnetworks. This is where developmental systems have a role to play. These systems evolve short descriptions of large structures by exploiting regularities observed in Nature, such as repetition of useful sub-parts, symmetries, and symmetries with variation [START_REF] Stanley | A hypercubebased encoding for evolving large-scale neural networks[END_REF][START_REF] Hornby | Measuring, enabling and comparing modularity, regularity and hierarchy in evolutionary design[END_REF]. They more easily describe regular structures than irregular ones, because the former can be described by a few general rules whereas the latter require describing either each element, or a list of exceptions to general rules. As a consequence, developmental systems bias the search space towards regular structures [START_REF] Clune | On the performance of indirect encoding across the continuum of regularity[END_REF]. We here propose that this bias towards regularity is critical to evolve plastic neural networks that can learn in a large variety of situations. Intuitively, this bias makes it more likely to obtain generic networks that apply the same learning rules to whole sets of inputs instead of networks that are finelytuned to only solve the test cases used in the fitness function. A direct consequence is that using developmental systems to evolve plastic neural networks should facilitate the evolution of plastic ANNs with general learning abilities.

Experimental setup

This hypothesis is tested using a simulated "Skinner box" (Figure 2), a classic experimental setup for operant conditioning in which a caged animal must learn to associate stimuli (e.g. lights) to actions (e.g. pushing a lever). If the animal executes the correct action, it is rewarded (e.g. by some food); if it chooses the wrong one, it is punished (e.g. by an electric shock). There is no delay in the reward, so there is no credit assignement problem [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. We consider only one-to-one associations so that, for each simple stimulus (each light), there is a different action to perform. Four stimuli and four actions are used; there are therefore 256 possible sets of stimulus/action (4 4 = 256; see Appendix S1 for the list of possible association sets). We formalize the stimulus/action associations using the concept of association sets: Neurons are placed in a 3D substrate (top). To know whether two neurons are connected and the synaptic weight of each connection, a Compositional Pattern Producing Network (CPPN) is queried using the 3D coordinates of the two neurons (Methods). This CPPN is evolved using a direct encoding. To know the parameters of each node (neuron type and threshold value), a second CPPN is queried with the 3D coordinates of the neuron (Methods).

Definition 1 (Association) An association is a pair (I, O) of input/output that leads to the maximum positive reward. In our system (n = 4: 4 inputs, 4 outputs), (1, B) is an association that means that the agent must push the B lever when light 1 is on. Definition 2 (Association set) An association set A = (I1, K1), • • • , (In, Kn) is a list of associations that covers all the n possible inputs. For instance, the list of associations {(1, B), (2, C), (3, D), (4, A)} is an association set in our system (n = 4: 4 inputs, 4 outputs). Several inputs can be associated to the same output. For instance, the association set {(1, A), (2, A), (3, A), (4, A)} is also valid in our system.

Definition 3 (Global training set)

The global training set, called G, is the the set of all the possible association sets of an experimental setup. In our system, there are 4 possible outputs and 4 possible inputs (n = 4), therefore the size of G is 256 (|G| = 4 4 = 256; the complete list of assotiation sets is available in Appendix S1). The ideal plastic network should be able to learn every association sets of G. E is included in G; it does not change during an experiment. Depending of the experiment, the size of E varies between 1 and 7. The elements of E have been chosen at random.

The fitness function (Methods

The fitness function is normalized by the size of E, so that it actually corresponds to the the number of successfully learned sets divided by |E|. After each evolution experiment, we assess the ability of the network with the best fitness score to learn every possible association set, that is, we evaluate the fitness function on the global training set. We call the success rate of this test the General Learning Abilities score (GLA score). This score reflects how well networks that are selected for their capacity to learn a few association sets are able to learn association sets that have not been encountered during evolution.

The evolved ANNs (Figure 2, B) have one input for each possible stimulus (i.e., 4 stimuli inputs), one input for positive rewards and one input for negative rewards. They have 4 outputs, each of them representing the probability of choosing each action. The final action is selected thanks to a "softmax" function that randomly selects an action according to a distribution that gives a higher probability to actions that corresponds to high output values distribution [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] (Methods). In effect, the neural network can activate any combination of the four available outputs and the softmax function makes sure that only one action is chosen at a time (Figure 2,B). Only one light (input) is activated at a time.

Plasticity is implemented in the neural networks using neuromodulated Hebbian plasticity [START_REF] Floreano | Neuroevolution: from architectures to learning[END_REF][START_REF] Abbott | Synaptic plasticity: taming the beast[END_REF][START_REF] Soltoggio | Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF] (Methods). In this model, neurons are of two kinds, "standard" and "modulatory"; the strength of connection between each pair of neurons is modified using a classic Hebbian rule that is activated only if the sum of inputs from modulatory neurons is above an evolved threshold.

For each association of E, the fitness function first presents the stimuli to the neural network for a few time-steps (Methods). Once the final output is computed by the softmax, it is copied to the input layer (feedback inputs). The reward input (positive or negative) is set at the same time. Such feedback loops are often present in computational models of cortexbasal ganglia-thalamus-cortex loops for action selection [START_REF] Houk | A model of how the basal ganglia generate and use neural signals that predict reinforcement[END_REF][START_REF] Frank | Anatomy of a decision: striatoorbitofrontal interactions in reinforcement learning, decision making, and reversal[END_REF][START_REF] Girard | Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection[END_REF] and are implicit in actor-critic models of reinforcement learning [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. The neural network is then simulated for a few more time-steps (Methods). It is expected that the evolutionary process will connect one or several modulatory neurons to the reward input and that the ANNs will exploit the copied outputs to strengthen/weaken the connections that correspond to the action that has actually been performed. Nonetheless, it must be emphasized that weight changes can occur at any time, including during the first step of the evaluation of the ANN. Only the topology and the synaptic weights of the ANNs, which are designed by evolution, determine when and how synaptic weights change.

The ANNs that solve this task may seem trivial at first sight. However, the evolutionary process needs to add at least one modulatory neuron (inputs cannot be modulatory in our system) and we never found any solution with less than two hidden neurons (one of them being modulatory). Essentially, the challenge raised by this task is to discover learning rules that allow the ANN to exploit a reward to strengthen and weaken the right connections. Typical solutions require three main "discoveries": (1) identifying and correctly connecting the reward inputs, (2) gating the reward with the softmax choice to modify only the connections corresponding to the chosen action, and (3) applying the resulting reinforcement to a link between the inputs and the output.

The topology and the parameters of evolved ANNs are encoded with three encodings [START_REF] Floreano | Neuroevolution: from architectures to learning[END_REF], with three different levels of expected regularity (Figure 3). The first encoding, called the map-based encoding [START_REF] Mouret | Importing the computational neuroscience toolbox into neuroevolution-application to basal ganglia[END_REF] (Methods), is inspired by computational neuroscience models in which ANNs are described as graph of single neurons and neural maps (spatially-organized identical neurons) that are connected with a few possible connection schemes (usually only one-to-one and one-to-all) [START_REF] Girard | Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection[END_REF][START_REF] Gurney | A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour[END_REF][START_REF] Rougier | Emergence of attention within a neural population[END_REF]. This encoding produces very regular neural networks because it has to treat each neuron in a map in the exact same way as the other neurons of the same map. The second encoding is a simplified version HyperNEAT [START_REF] Stanley | A hypercubebased encoding for evolving large-scale neural networks[END_REF], called HNN, for Hyper Neural Network (Methods). HyperNEAT-like encodings are developmental encodings in which morphogen gradients are described as feed-forward networks of mathematical functions that operate in a Cartesian space. This indirect approach allows them to encode large networks with Nature-like connection patterns (symmetry, symmetry with variations, repetition, etc.). The last encoding is a classic direct encoding in which evolution directly acts on the structure and the parameters of the ANN (Methods). This encoding has no bias to produce regular networks.

To understand the relationship between encodings, regularity and learning abilities, we have to assess the regularity of evolved ANNs. According to Lipson [START_REF] Lipson | Principles of modularity, regularity, and hierarchy for scalable systems[END_REF], regularity is the compressibility of the description of the structure. Regrettably, this value is not computable [START_REF] Li | An introduction to Kolmogorov complexity and its applications[END_REF] and, to our knowledge, there exists no well-recognized approximation for weighted, directed graphs. The few algorithms designed to compress the graph structure are greedy approximations that only work well for sparse, undirected labeled graphs [START_REF] Peshkin | Structure induction by lossless graph compression[END_REF][START_REF] Hayashida | Comparing biological networks via graph compression[END_REF]. We follow another method to estimate the regularity of networks: counting the number of symmetry axes [START_REF] Mowshowitz | Entropy and the complexity of graphs: I. an index of the relative complexity of a graph[END_REF][START_REF] Mowshowitz | Entropy and the complexity of graphs: Ii. the information content of digraphs and infinite graphs[END_REF][START_REF] Zenil | Graph automorphism and topological characterization of synthetic and natural complex networks by information content[END_REF]. A graph has an axis of symmetry when two groups of nodes can be swapped without modifying the graph, that is, when there is a repetitive, structural pattern. More axes of symmetry means a better compression because the two groups need to be described only once [START_REF] Mowshowitz | Entropy and the complexity of graphs: I. an index of the relative complexity of a graph[END_REF][START_REF] Mowshowitz | Entropy and the complexity of graphs: Ii. the information content of digraphs and infinite graphs[END_REF][START_REF] Zenil | Graph automorphism and topological characterization of synthetic and natural complex networks by information content[END_REF]. In graph theory, this kind of symmetry is called an automorphism and fast algorithms exist to count them [START_REF] Mckay | Practical graph isomorphism[END_REF][START_REF] Junttila | Engineering an efficient canonical labeling tool for large and sparse graphs[END_REF][START_REF] Katebi | Graph symmetry detection and canonical labeling: Differences and synergies[END_REF] (Methods).

Networks are evolved using the classic multi-objective evolutionary algorithm NSGA-II [START_REF] Deb | Multi-objective optimization[END_REF][START_REF] Deb | A fast and elitist multiobjective genetic algorithm: NSGA-II[END_REF]. Two objectives are optimized: the fitness of networks (Methods) and a behavioral novelty objective [START_REF] Mouret | Encouraging behavioral diversity in evolutionary robotics: An empirical study[END_REF][START_REF] Soltoggio | Novelty of behaviour as a basis for the neuro-evolution of operant reward learning[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF][START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF][START_REF] Mouret | Novelty-based multiobjectivization[END_REF], to mitigate premature convergence (Methods). These two objectives are optimized during a maximum of 4000 generations of 400 individuals. Experiment are stopped as soon as the best individual of the population reaches a perfect fitness value on the evolutionary training set. At the end of each experiment, the novelty objective is discarded and we consider that the best individual is the one with the best fitness value.

We perform 7 series of independent experiments by varying the size of the evolutionary learning set from 1 to 7 (i.e., |E = 1, • • • , 7|). For each series, the three investigated encodings are tested (direct encoding, map-based encoding and HNN encoding). Each experiment is replicated 30 times to obtain statistics. We therefore launch a total of 3 × 7 × 30 = 630 experiments, each one lasting between 1 and 8 hours on our computers (Intel Xeon E5520@2.27GHz) depending on the time required to converge and the size of the evolutionary training set. Because of this large computational time, we were not able to extend our experiments to harder problems, for instance with more inputs/outputs.

Results

For each encoding, we compute the GLA score of networks with a perfect fitness on the evolutionary training set and we plot it as a function of the size of the evolutionary training set.

The results show a clear difference in the GLA scores obtained with each encoding (Figure 4,A). With a direct encoding, the GLA score grows linearly with the size of the evolutionary training set, which is consistent with previous results [START_REF] Chalmers | The evolution of learning: An experiment in genetic connectionism[END_REF], and the GLA scores obtained with small values of |E| are statistically different from those obtained with larger values (e.g., 1 versus 7: p = 4 × 10 -4 ; 4 versus 7, p = 2 × 10 -3 , 3 versus 6, p = 2 × 10 -3 ; unless otherwise specified, the statistical test in this paper is the Mann-Whitney U-test). With the direct encoding, using a fitness that tests at least 6 associations sets (|E| > 5) is required to obtain networks with a GLA-score similar to the one reached with the map-based encoding with only 2 association tests (p = 0.8). The HNN encoding appears as a trade-off between the direct encoding and the map-based encoding: for each value of |E|, the GLA score obtained with HNN is consistently higher than the one obtained with the direct encoding, yet it is lower than the one reached with the map-based encoding (for 2, 3 or 4 association sets, HNN versus direct encoding, p < 0.03; for 1, 2, 3 or 5 association sets, HNN versus map-based encoding, p < 6 × 10 -3 ) 3 .

As expected, each encoding leads to different levels of regularity, and increasing the number of association sets used in the fitness function increases the regularity of evolved neural networks (Figure 4,B). All the networks evolved with the mapbased encoding are regular: they all have at least one symmetry axis. The HNN encoding also leads to many networks with at least one symmetry axis (from 80% to 100%), whereas the direct encoding leads to substantially fewer regular networks (from 20% to 70%, depending on |E|). These numbers vary with the size of E. With the HNN encoding, three association sets are needed to obtain 100% of regular networks; with the direct en- coding, the number of regular networks grows from 20%, when one association set is used during evolution (|E| = 1), to 60-70% when more than 6 association sets are used (|E| > 5).

To further understand this result, we plot the network with the best learning abilities for each encoding and each size of the evolutionary learning set (figure 6). We observe the same overall link between learning abilities and regularity as on figure 4, but some networks have good learning abilities with only a few automorphisms, like the network evolved with a direct encoding and 6 association sets (GLA score of 0.99, 2 automorphisms). This result is possible because nothing encourages a directly encoded network to duplicate the same sub-structure several times: it may be sometimes easier to either re-invent 4 times the same function but with slight changes, or to design an integrated solution that relies on only one complex structure. This particular network seems to use a centralized structure with only one modulatory neuron that modulates all the plastic connections of the network. Conversely, some regular networks have a low GLA score, such as the network evolved with HNN and one association set (GLA score of 0.70, 24 automorphisms). There is no paradox in this result: the regularities can be at the wrong place to lead to high-learning abilities.

Whatever the encoding and the size of E are, networks with the best learning abilities are those that are the most regular (Figure 5, A; this figure use the same data as Figure 4). Hence, among networks evolved with the direct encoding, those that have at least 2 automorphisms (one axis of symmetry) have a better GLA score than those that have no automorphism (p = 6 × 10 -3 ). Those with more than 3 automorphisms also have statistically better learning abilities than those with two automorphisms (p = 0.05) and than those without any symmetry axis (p = 5 × 10 -4 ). The same tendency is present with the HNN encoding: networks with at least two automorphisms (i.e., networks with at least one symmetry axis) have a higher GLA score than those that have no symmetry axis (one automorphism, p = 0.04); networks with more than 17 automorphisms have a higher GLA score than those with at least two automorphisms (p = 2 × 10 -3 ).

With the HNN encoding, 41% of networks have exactly 24 automorphisms but only 23% of them have 25 or more automorphisms (Figure 5, B, blue line). With the map-based encoding, a drop from 100% to 87% occurs at the same number of automorphisms (Figure 5, B, grey line). A network with 24 automorphisms is a network in which a sub-network is repeated 4 times (24 = 4!, Methods). This number is particular in our experiments because both HNN and the map-based encoding group neurons by 4, therefore the number of automorphisms is expected to be a multiple of 24: a different number means that at least one neuron of a group has a connectivity pattern that is different from the rest of the group. With HNN, this kind irregularity is possible but unlikely. With the map-based encoding, it is not possible, that is why all map-based networks have a number of automorphisms exactly equals to a multiple of 24 (for instance, on figure 6, all map-based networks have 24 or 576 = 24 × 24 automorphisms).

Conclusion and discussion

The experiments reported in this paper add weight to the hypothesis that using a developmental encoding improves the learning abilities of evolved, plastic neural networks. Complementary experiments reveal that this result is the consequence of the bias of developmental encodings towards regular structures [START_REF] Clune | On the performance of indirect encoding across the continuum of regularity[END_REF]: [START_REF] Pfeifer | How the Body Shapes the Way we Think[END_REF] encodings that tend to produce more regular networks yielded networks with better general learning abilities;

(2) in our experimental setup, whatever the encoding is, networks that are the more regular are statistically those that have the best learning abilities. This second point implies that an GLA score: 0.700195 #Automorphisms: 24

Direct encoding (DNN) CPPN-based (HNN) Map-based (EvoNeuro)

Fig. 6. Network with the best learning abilities, for each encoding and each size of the evolutionary learning set. network is the best (in term of learning abilities) of the 30 independent runs. Inhibitory connections are represented as green line and excitatory ones as red lines.

The width of the lines is proportional to the corresponding synaptic weight (for modulated connections, the line width is determined after one of the learning phases, for one of the possible association sets). Input neurons are in green, output neurons in red, modulatory neuron in gray and standard neurons in blue. "#automorphisms" means "number of automorphisms" (Methods). Nodes that are not connected (directly or indirectly) to at least one input and one output are not drawn.

indirect encoding that is not biased towards regular network should not lead to ANNs with high learning abilities; it also implies that a direct encoding combined with a helper objective that encourages regularity should lead to ANNs with good learning abilities (see [START_REF] Mouret | Encouraging behavioral diversity in evolutionary robotics: An empirical study[END_REF] and [START_REF] Clune | The evolutionary origins of modularity[END_REF] for examples of helper objectives with a direct encoding). Nonetheless, our experiments show that current generative encodings and neuro-modulated Hebbian plasticity make a promising combination to evolve large, plastic neural networks. Future work in this direction should investigate whether this combination holds its promises in other tasks such as learning in a maze [START_REF] Soltoggio | Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF] or visual processing [START_REF] Stanley | A hypercubebased encoding for evolving large-scale neural networks[END_REF]. According to our results, neural networks evolved with an encoding biased towards regularity could be more flexible than those evolved with an unbiased encoding: they are better at learning association sets that have never been encountered during their evolution. To achieve this flexibility, they have to possess connections that were not directly selected during evolution. In other words, their flexibility stems from "spandrels" [START_REF] Gould | The spandrels of San Marco and the panglossian paradigm: a critique of the adaptationist programme[END_REF]: they are the byproducts of the bias that make evolution more likely to duplicate a sub-structure than to design a specialized circuit.

These results are in opposition to the general tendency of neural networks to minimize connection costs [START_REF] Clune | The evolutionary origins of modularity[END_REF][START_REF] Chklovskii | Wiring optimization in cortical circuits[END_REF][START_REF] Cherniak | Global optimization of cerebral cortex layout[END_REF][START_REF] Chen | Wiring optimization can relate neuronal structure and function[END_REF] because they show that flexible behaviors require maintaining many "useless" connections. They indicate that a selective pressure for flexibility is likely to favor developmental procedures that would result in connections that do not procure any shortterm advantage. In a constant environment, these connections should disappear; but in a constantly changing environmentwhich puts more pressure on flexibility -, these connections appear critical. This view is consistent with the theory of "variability selection", which posits that flexibility is one of the primary selective pressure that shaped the brains of hominids [START_REF] Potts | Variability Selection in Hominid Evolution[END_REF][START_REF] Richerson | Evolution on a restless planet: Were environmental variability and environmental change major drivers of human evolution?[END_REF].

The conflict between flexibility and connection costs also echoes the debate about the modularity/non-modularity of the mammalian brain [START_REF] Meunier | Modular and Hierarchically Modular Organization of Brain Networks[END_REF][START_REF] Grossberg | The complementary brain: unifying brain dynamics and modularity[END_REF][START_REF] Bullmore | Complex brain networks: graph theoretical analysis of structural and functional systems[END_REF], since the minimization of connection costs has been linked with the evolution of modularity [START_REF] Clune | The evolutionary origins of modularity[END_REF][START_REF] Striedter | Principles of brain evolution[END_REF]. Our results thus suggest that the parts of the brain that heavily rely on synaptic plasticity to achieve flexible behaviors should be less modular than simpler, less plastic parts. To test this proposition, it is possible to launch computational experiments in which plastic neural networks are evolved with a selective pressure to minimize connection costs and different flexibility requirements.

Pushed to the extreme, the results of our experiments suggest that the best flexibility would be achieved with fully connected networks, since this would be the best possible regularity. In real brains, such a connectivity would be challenging for pure physical reasons [START_REF] Braitenberg | Brain size and number of neurons: an exercise in synthetic neuroanatomy[END_REF][START_REF] Chklovskii | Wiring optimization in cortical circuits[END_REF]: if each neuron of a mouse was connected to each other, its brain (about 10 millions neurons) would at least occupy 350 cubic meters [START_REF] Braitenberg | Brain size and number of neurons: an exercise in synthetic neuroanatomy[END_REF] (about the cranial volume of an Orangutan). Artificial brains do not have such limitations and can be designed as fully connected [START_REF] Hopfield | Neural networks and physical systems with emergent collective computational abilities[END_REF], but most neural networks used in machine learning are made of layers of neurons, with each layer fully connected to the next one [START_REF] Cybenko | Approximation by superpositions of a sigmoidal function[END_REF][START_REF] Haykin | Neural Networks: A Comprehensive Foundation[END_REF]. Layers are a very specific structure that prevents some flexibility (non-Markovian tasks cannot be learned), but they make learning easier, because feed-forward networks have no intrinsic dynamics (contrary to recurrent neural networks). These networks are still very regular and flexible. In image processing, convolutional neural networks are classic feed-forward neural networks in which many, well-chosen connections are removed and many synaptic weights are constrained to be equal [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF]. These networks are much easier to train than classic layered neural networks, but they cannot learn when the input data do not look like images.

These examples highlight a potential trade-off between flexibility and trainability, or, put differently, between learning abilities and learning efficiency: in many situations, it seems beneficial to trade some flexibility to make the system easier to train 4 . Our experiments considered a simple situation in which trainability was not a major concern because the input/output patterns are simple and low-dimensional. In more challenging tasks, the evolutionary process would probably have to find the best trade-off between trainability and flexibility, and therefore between regularity and specialization. Nonetheless, although convolutional networks are less regular than multi-layer perceptrons, they are still very regular and could be generated with a generative encoding. Generative encodings that aim at intermediate regularity might thus be one of the key to explore this trainability/flexibility trade-off.

Overall, the present paper shows that evolution, development and synaptic plasticity are three interleaved processes that are hard to study separately. While an extensive understanding of their interactions is probably out of reach with the current state of knowledge, studies that combine simple models of each of these processes shed light on how one of them -here development -can simplify another -here learning. Such studies appear helpful for both building a global vision of the evolution of intelligent lifeforms as well as harnessing evolution to create intelligent agents.

Methods

Plastic neuron model

Following [START_REF] Soltoggio | Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios[END_REF][START_REF] Soltoggio | Novelty of behaviour as a basis for the neuro-evolution of operant reward learning[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF], we distinguish two types of neurons: "standard neurons" and "modulatory neurons". Inputs of each neuron are divided into modulatory inputs Im and standard Is inputs. The output ai of a neuron i is then defined as follows:

ai = ϕ1 j∈Is wijaj + bi ( 1
)
where i is the identifier of a neuron, ai its output, bi its bias, ϕ1(x) = 1 1+exp(-λx) a sigmoid on [0, 1], wij the synaptic weight between neurons i and j. Each non-modulatory synaptic weight wij is modified with regards to the sum of modulatory inputs and a constant coefficient η (η = 0.04 in our experiments):

mi = ϕ2 j∈Im wijaj (2) ∆wij = η • mi • ai • aj (3) wij(t + δt) =        min(max(wij(t) + ∆wij, 0), 30) if wij(t) ≥ 0 min(max(wij(t) -∆wij, -30), -10 -5 ) if wij(t) < 0 where ϕ2 = ϕ1(x) = 2 1+exp(-λx) -1 is a sigmoid on [-1, 1]
(to allow positive and negative modifications of synaptic weights).

Fitness function and behavioral descriptors

The fitness function computes the number of associations that the network successfully learn, given an evolutionary learning set E.

For each association set of E (for instance, {(1, B), (2, C), (3, D), (4, A)}), this function first randomly initializes the modulated weights, that is, the network does not have to un-learn what was previously learned. The network is then allocated 90 learning episodes, each one executing the following steps:

• successively select one of the four associations of the association set (for instance, (1, B));

• set the stimuli inputs of the neural network according to the chosen association and set the other inputs (reward and feedback) to zero (for instance, for the association (1, B), the input will be [0, 0, 0, 0, 1, 0, 0, 0]);

• compute the output of the network by simulating it during 5 time-steps (5 time-steps is enough to allow a signal to travel from the inputs neurons to the output neurons);

• select an action using the four outputs of the neural network and a softmax function (see section "Output selection");

• set the reward inputs (i.e., positive reward if the output is correct, negative otherwise) and the feedback inputs (for instance, if the output is "C", a wrong answer, then the new input will be [0, 0, 1, 0, 1, 0, 0, 1]);

• simulate the network again for 5 time-steps (this is the step where the network is expected to reinforce connections; however, nothing prevents adaptation to occur during the previous activation);

• if the last learning episode for this association set is reached:

the positive reward that corresponds to each input of the set is added to the fitness (for instance, if the network's output was C for association (1, B), C for (2, C), D for (3, D) and A for (4, A), then 3 is added to the current value of the fitness);

the output of the network (before the softmax) for each input of the set is appended to the behavior descriptor (for instance, for the previously described outputs, we would append the vector [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0]);

In summary, the final fitness value corresponds to the average number of associations of E that have been successfully learned.

The final behavior descriptor is a vector that contains the final output of the network for each association of E. This fitness function is described in a more algorithmic on figure 7. follows:

Output selection (softmax)

At the ouptut of the neural networks, the action is selected thanks to a softmax function [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]:

P (i) = exp(βai) n j=1 exp(βaj) (4)
where P (i) is the probability of selecting output i, ai is the activity of output i and β is a constant. In effect, this distribution gives a higher probability to actions that corresponds to high output values. If values are close, then they will have similar chances to be selected; if values are very contrasted, then the softmax function is equivalent to a "max" function. Using this distribution instead of a simpler "max" function allows network to explore -which is required to learn -and encourages the contrast between output. This technique is commonly employed in reinforcement learning [START_REF] Sutton | Reinforcement learning: An introduction[END_REF].

Map-based encoding

Many computational neuroscience models (e.g. [START_REF] Girard | Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection[END_REF][START_REF] Gurney | A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour[END_REF][START_REF] Rougier | Emergence of attention within a neural population[END_REF]) are described as graph of neural maps (spatially-organized identical neurons) in which each connection is labeled by a set of parameters that represent the connection scheme [START_REF] Mouret | Importing the computational neuroscience toolbox into neuroevolution-application to basal ganglia[END_REF](Figure 3). This description of neural networks can be seen as a developmental encoding according to which networks of maps are developed to form a neural networks.

In our model, each edge is associated with three parameters: (1) connection type ("1 to 1" or "1 to all" with uniform synaptic weights); (2) synaptic weight (all connections between maps have the same strength) (3) inhibitory or excitatory (a Boolean value). Similarly, three parameters describe each map: (1) isolated neuron or map of neurons (a Boolean value); (2) inhibitory or excitatory (a Boolean value -the whole map will be inhibitory or excitatory); (3) parameters of the neuron (float number, threshold value). Each label is encoded with a real number in [0, 1], mutated with polynomial mutation in the same way as intrinsic parameters of neurons and synaptic weights in a direct encoding. The section "parameters" describes how these numbers are translated into Boolean values and parameters.

In the present study, all maps have the same size. Each graph of neural maps is developped into a full neural network by analyzing each node and each edge of the graph to create the corresponding neurons, maps and connections.

Labeled graphs are evolved using the direct encoding (Methods).

HNN encoding (minimal HyperNEAT)

HyperNEAT is a developmental encoding in which morphogen gradients are described as feed-forward networks of mathematical functions that operate in a Cartesian space [START_REF] Stanley | A hypercubebased encoding for evolving large-scale neural networks[END_REF], called Compositional Pattern Producing Networks (CPPNs). When evolving an ANN, HyperNEAT evolves CPPNs that are then queried to know each synaptic weight. This indirect approach allows HyperNEAT to evolve large networks with Nature-like connection patterns (symmetry, symmetry with variations, repetition, etc.).

In the present study, we use a simplified version of Hyper-NEAT in which the CPPNs are evolved using a simple direct encoding instead of the NEAT method 5 . We call this encoding HNN (Hyper Neural Network).

We place 9 input neurons, 5 hidden neurons and 4 output neurons in a 3D substrate (Figure 3) 6 . We describe each individual with two CPPNs: one connection-centred CPPN that returns whether a connection exists (LEO link in HyperNEAT [START_REF] Verbancsics | Constraining Connectivity to Encourage Modularity in HyperNEAT[END_REF]) and the synaptic weigth, and one node-centred CPPN that returns the intrinsic parameters of each neuron, that is, the threshold value used in the sigmoid and whether the neuron is "modulatory" or "standard". Three functions are available to the CPPNs: sin(x), 2/(1 + e -x ) -1 (sigmoid), e -x 2 (Gaussian) and min(max(x, -3), 3)/3 (linear).

Many other substrates can be used and some of them undoubtedly lead to faster convergence; it is also possible to use a single CPPN for both the connection and the node parameters. Nevertheless, the present work is centered on the consequences of using any developmental encoding when evolving plastic neural networks. The relative performance of each encoding is irrelevant.

Direct encoding (control experiment, labeled graph, CPPN)

We use a straighforward direct encoding, loosely inspired by NEAT [START_REF] Stanley | Evolving neural networks through augmenting topologies[END_REF], to encode both the labeled graph of the map-based encoding and CPPN for the HNN experiments. We also use it as a control experiment. In this case, the evolved graph is employed in a more classic fashion to directly define a neural network.

In this encoding, a neural network (or a CPPN) is described as a directed graph and five mutation operators are implemented:

• add a connection between two randomly chosen neurons (probability: p • change random weights using the polynomial mutation [START_REF] Deb | Multi-objective optimization[END_REF] (probability: pw for each connection);

• change the intrinsic parameter of a neuron (e.g. the activation function when evolving CPPNs) (probability: p b for each neuron);.

Cross-over is not employed. To initiate the evolutionary process, neural networks of the first generation are feed-forward networks without hidden layer, each one with randomly generated weights. This encoding has been previously employed in many papers, with similar parameters [START_REF] Mouret | Encouraging behavioral diversity in evolutionary robotics: An empirical study[END_REF][START_REF] Mouret | Novelty-based multiobjectivization[END_REF][START_REF] Mouret | Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity[END_REF][START_REF] Pinville | How to promote generalisation in evolutionary robotics: the progab approach[END_REF][START_REF] Ollion | With a little help from selection pressures: evolution of memory in robot controllers[END_REF].

Counting automorphisms

Figure 8 shows the number of automorphisms for a few networks.

Definition 5 (Automorphism) An automorphism of a graph G = (V, E) is a permutation σ of the vertex set V , such that the pair of vertices (u, v) forms an edge if and only if the pair (σ(u), σ(v)) also forms an edge. Put differently, an automorphism is a graph isomorphism from G to itself.

Each network has at least one automorphism, itself. The number of symmetry axes of a network therefore corresponds to the number of automorphisms minus one. Counting and enumerating automorphisms is a NP-complete problem, but there exist fast, exact algorithms that work for most graphs [START_REF] Mckay | Practical graph isomorphism[END_REF][START_REF] Junttila | Engineering an efficient canonical labeling tool for large and sparse graphs[END_REF][START_REF] Katebi | Graph symmetry detection and canonical labeling: Differences and synergies[END_REF]. In the present work, we use the Bliss library [START_REF] Junttila | Engineering an efficient canonical labeling tool for large and sparse graphs[END_REF].

We count the number of automorphisms of the developped neural network (and not those of the genotype). Modulatory neurons are labeled as "m" and other neurons as "n". The Bliss library does not handle labeled edges, but edge labels can easily be transformed into node labels as follows: first, synaptic weights are binned into four categories (large negative, small negative, small positive, large positive); second, each of them is associated to a unique label; last, on each connection, a node is added and labeled by the category of the corresponding synaptic weight. Bias of neurons are ignored.

Evolutionary algorithm

Networks are evolved using NSGA-II [START_REF] Deb | A fast and elitist multiobjective genetic algorithm: NSGA-II[END_REF]. To mitigate premature convergence, we add to the fitness objective a behavioral novelty objective [START_REF] Mouret | Encouraging behavioral diversity in evolutionary robotics: An empirical study[END_REF][START_REF] Soltoggio | Novelty of behaviour as a basis for the neuro-evolution of operant reward learning[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF][START_REF] Lehman | Abandoning objectives: Evolution through the search for novelty alone[END_REF][START_REF] Mouret | Novelty-based multiobjectivization[END_REF] that rewards individuals that do something that has not been done before. In effect, we transform the single-objective problem of maximimizing the fitness into a two-objective optimization problem:

where d b (x, j) denotes the distance between the behaviors of individuals x and j, Nx the set of the 10 closest individuals to x in the archive and the current population. At the end of each learning session (90 episodes), for each input pattern, the 4 outputs of the neural network are appended to a behavior descriptor (see the "Fitness function" section). The distance d b (x, j) is the Euclidean distance between the behavior descriptors of x and j.

Fig. 1 .

 1 Fig. 1. Main hypothesis. Using developmental encodings should facilitate the evolution of plastic ANNs with high learning abilities.

Fig. 2 .

 2 Fig.2. A. Concept of the "Skinner box". A caged animal must learn to associate stimuli (here lights) to actions (here pushing a lever). The experimenter selects a stimulus/action association, presents it to the animal, record the action, and gives the reward to the animal. The experimenter then chooses another association in the association set and starts the cycle again. The association set is learned once the animal associates the right action to each stimulus. B. Formalization of the Skinner box as a task for an artificial neural network. Each stimulus is an input of the neural network. Positive and negative rewards are two additional inputs. The output is selected according to a softmax function (Methods) and the result of the softmax is looped back to the input layer.

Fig. 3 .

 3 Fig. 3. A. Principle of the map-based, developmental encoding. The neural network is encoded as a labeled graph (left), which is developped to a graph of maps according to the labels (right). (Methods). B. Principle of the HNN encoding (minimal HyperNEAT).Neurons are placed in a 3D substrate (top). To know whether two neurons are connected and the synaptic weight of each connection, a Compositional Pattern Producing Network (CPPN) is queried using the 3D coordinates of the two neurons (Methods). This CPPN is evolved using a direct encoding. To know the parameters of each node (neuron type and threshold value), a second CPPN is queried with the 3D coordinates of the neuron (Methods).

Definition 4 (

 4 ) assesses the ability to learn a subset of the global training set, called the evolutionary training set: Evolutionary training set) The evolutionary training set, called E, is the set of the association sets used in the fitness function.

Fig. 4 .

 4 Fig. 4. Relationship between encodings, general learning abilities and the size of the evolutionary training set (|E|). A.Generative encodings yield plastic ANNs with better general learning abilities than those evolved with a direct encoding. Morever, increasing the size of E increases the general learning abilities. Each box extends from the lower to upper quartile values of the data, with a symbol at the median. Whiskers extend to the most extreme data point within 1.5×IQR, where IQR is the interquartile range. Flier points (outliers) are those past the end of the whiskers. X-values are shifted for the map-based encoding and the direct encoding in order to make the figure readable. B. Generative encodings yield more regular networks than a direct encoding, and increasing the size of E increases the regularity of evolved networks.

Fig. 5 .

 5 Fig.5. Relationship between regularity and general learning abilities. Data are from the same experiments as Figure4. The "minimum number of automorphims" means that if, for example, a network has 4 automorphisms, it is included in columns 1,2,3 and 4. X-values are shifted for the map-based encoding and the direct encoding in order to make the figure readable. A. The more automorphisms a network has, the more likely it is to have good general learning abilities (GLA score). Each box extends from the lower to upper quartile values of the data, with a symbol at the median. Whiskers extend to the most extreme data point within 1.5 × IQR, where IQR is the interquartile range. Flier points (outliers) are those past the end of the whiskers. B. 7% of networks evolved with a direct encoding have more than 3 autormorphisms. 72% of those evolved with HNN have more than 3 automorphisms. 86% of networks evolved with the map-based encoding have more than 200 autmorphisms; 100% of them have at least 10 automorphisms.

  ƒ

•

  move the target or the source of a randomly chosen connection (probability:p (c) c );• add a neuron by splitting an existing connection in two (the connection weight is kept on the two connections) (probability: p(n) a );• delete a randomly chosen neuron and all related connections (probability: p

= 1 8

 1 j∈Nx d b (x, j)

  Algorithmic view of the fitness function.

	B	0		. Unnormalized fitness . Behavior descriptor
	for n = 0 to n = K do A associations[n] 2 E Randomly initialize the modulated weights	. K assocation sets . Select an association set . Un-learn everything
		for = 0 to = 90 do for j = 0 to j = 4 do Activate network 5 times for A[j]	. 90 learning episodes . 4 assocations in each set . Signal travels from inputs to outputs
			Select an action (softmax) and collect rewards	. See section "output selection"
			ƒ Set the reward and the feedback inputs ƒ + positive reward
			Activate network 5 times times for A[j]	. The network is expected to reinforce connections
			if >= P 4 then	. Final test (last episode for A)
			f	f + positive reward / 4	. Add the positive reward to the fitness value
			B end if	B [ o tp ts	. Append the outputs to the behavioral descriptor
		end for
		end for
	end for
	fitness	f / K	. Final fitness value
				Fig. 7.
				1

November 2013 | Volume 8 | Issue 11 | e79138

This paper extends some of the preliminary results published in a conference paper[START_REF] Tonelli | On the relationships between synaptic plasticity and generative systems[END_REF], which first suggested the existence of a link between developmental systems and plastic ANNs.

It should be emphasized that many authors do not test whether the plastic ANNs they evolve can learn in test cases that have not been encountered during the evolutionary process. For instance,[START_REF] Soltoggio | Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios[END_REF][START_REF] Risi | Evolving plastic neural networks with novelty search[END_REF][START_REF] Risi | Enhancing es-hyperneat to evolve more complex regular neural networks[END_REF] don't assess how evolved neural networks can cope with an unknown situation; counter-examples are[START_REF] Urzelai | Evolution of adaptive synapses: Robots with fast adaptive behavior in new environments[END_REF] and[START_REF] Chalmers | The evolution of learning: An experiment in genetic connectionism[END_REF].PLOS ONE | www.plosone.org

PLOS ONE | www.plosone.org

With

association sets and the HNN encoding, there are not enough networks with a perfect fitness score to perform a statistical analysis.PLOS ONE | www.plosone.org

Flexibility must sometimes be avoided because any change in the behavior might kill the animal or break the robot. This situation would correspond to a very easy learning, because current values are deemed as perfect, and a very low flexibility.PLOS ONE | www.plosone.org

We chose this simplified version to make easier the reproduction of our results, to enable the use of multi-objective evolutionary algorithms and to focus our study on the developmental process.

We checked that we were able to design a simple CPPN that generates an ANN that solves the task with the chosen substrate.PLOS ONE | www.plosone.org

Acknowledgments

The authors thank J. Clune, S. Doncieux and B. Girard for helpful comments on this manuscript.

This work was funded by the Agence Nationale de la Recherche (http://www.agence-nationale-recherche.fr/) by the grants Creadapt (ANR-12-JS03-0009) and EvoNeuro (ANR-09-EMER-005). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Source code

The source code for all the experiments is available at: http: //pages.isir.upmc.fr/evorob_db

Each neuron is labeled by a tuple (y1, y2, y3):

• map/single neuron: map if y1 > 0.5 single neuron, otherwise

• modulatory/standard: modulatory if y2 < 0.4 standard, otherwise

• bias: y3

CPPN

• available functions: sin(x), 2/(1 + e -x ) -1 (sigmoid), e -x 2 (Gaussian), min(max(x, -3), 3)/3 (linear)

• maximum connection weight: 3

• minimum connection weight: -3

• other parameters: see direct encoding

HNN encoding

• threshold for the creation of a connection (LEO link): 0.3

• synaptic weights are scaled to [0, 30]

• modulatory vs standard: modularatory if output is below 0.4


Youness Trachi
email: youness.trachi@univ-brest.fr

Elhoussin Elbouchikhi
email: elbouchikhi@isen-bretagne.fr

Vincent Choqueuse
email: vincent.choqueuse@univ-brest.fr

Mohamed Benbouzid
email: mohamed.benbouzid@univ-brest.fr

Stator Current Analysis by Subspace Methods for Fault Detection in Induction Machines

Keywords: Induction machine, bearing fault detection, stator current analysis, subspace techniques, Root-MUSIC, ESPRIT, fault severity

This paper aims to develop a condition monitoring architecture for induction machines, with focus on bearing faults. The main objective of this paper is to identify fault signatures at an early stage by using high-resolution frequency estimation techniques. In particular, we present two subspace methods, which are Root-MUSIC and ESPRIT. Once the frequencies are determined, the amplitude estimation is obtained by using the Least Squares Estimator (LSE). Finally, the amplitude estimation is used to derive a fault severity criterion. The experimental results show that the proposed architecture has the ability to measure the faults severity.

I. INTRODUCTION

Recently, the induction machines have become the most popular machines used in industrial applications thanks to their advantages such as low cost, reliability, availability, and high robustness. Unfortunately, these machines can be subjected to various faults. These faults can be categorized into two main classes: electrical and mechanical faults [START_REF] Zhang | A survey of condition monitoring and protection methods for medium-voltage induction motors[END_REF]- [START_REF] Elbouchikhi | Induction machine fault detection enhancement using a stator current high resolution spectrum[END_REF]. The electrical faults include stator winding short circuit, broken rotor bar, and broken end-ring. Besides, the mechanical faults comprise rotor eccentricity, bearing faults, shaft misalignment, and load faults. in order to reduce operating and maintenance costs, it is desirable to develop and search for reliable techniques allowing fault detection at an early stage [START_REF] Toliyat | Electric Machines: Modeling, Condition Monitoring, and Fault Diagnosis[END_REF]. Many condition monitoring techniques for induction machines can be exploited to detect the presence of faults such as vibration monitoring, temperature monitoring, oil monitoring, etc. However, these techniques are expensive and require additional sensors to be fitted to the machines. A solution to this problem is to use the stator current analysis. In fact, these techniques allow to use additional sensors in induction machine drives. Moreover, the stator currents are usually available for other purposes such as protection and control. In addition of that, the use of stator current presents other advantages such as its accessibility, the ease of implementation, the information richness, and the ability to detect electrical and mechanical faults [START_REF] Benbouzid | What stator current processingbased technique to use for induction motor rotor faults diagnosis?[END_REF].

Induction machines failures have an impact over intrinsic parameters such as magnetomotive forces, self and mutual inductances and stator currents. Consequently, the motor currents signature analysis (MCSA) have been widely investigated by researchers and engineers [START_REF] Garcia-Perez | Startup current analysis of incipient broken rotor bar in induction motors using highresolution spectral analysis[END_REF]. Several induction machines studies using the stator current prove that faults in induction machines introduce additional frequency components in the stator current spectrum [START_REF] Benbouzid | What stator current processingbased technique to use for induction motor rotor faults diagnosis?[END_REF]. Furthermore, the majority of mechanical or electrical faults are manifested by amplitude and/or frequency modulation of the stator current signal [START_REF] Blodt | Models for bearing damage detection in induction motors using stator current monitoring[END_REF], [START_REF] Choqueuse | Diagnosis of three-phase electrical machines using multidimensional demodulation techniques[END_REF]. The challenge of condition monitoring in induction machine using stator currents is to extract fault frequencies. Among the proposed techniques in the literature for stator current analysis, we can find techniques based on the Fourier Transform (FT), Fast Fourier Transform (FFT) and periodogram for stationary environments and time-frequency, time-scale, and demodulation techniques for non-stationary environments [START_REF] Choqueuse | Diagnosis of three-phase electrical machines using multidimensional demodulation techniques[END_REF], [START_REF] Elbouchikhi | Current frequency spectral subtraction and its contribution to induction machines bearings condition monitoring[END_REF]. A general review of induction machines monitoring and faults diagnosis techniques can be found in [START_REF] Benbouzid | What stator current processingbased technique to use for induction motor rotor faults diagnosis?[END_REF], [START_REF] Nandi | Condition monitoring and fault diagnosis of electrical motors-a review[END_REF]. If an a priori signal model is assumed, parametric methods can be employed to improve the frequency resolution. These techniques are generally called high-resolution methods and include two sub-classes: linear prediction methods and subspace techniques. The linear prediction class contains algorithms such as autoregressive-moving-average (ARMA) methods. The subspace methods include MUSIC and ESPRIT approaches [START_REF] Elbouchikhi | Induction machine fault detection enhancement using a stator current high resolution spectrum[END_REF], [START_REF] Kim | High-resolution parameter estimation method to identify broken rotor bar faults in induction motors[END_REF]- [START_REF] Xu | Improvement of the Hilbert method via ESPRIT for detecting rotor fault in induction motors at low slip[END_REF].

In the present paper, we consider the bearing faults detection based on stator currents analysis. The bearing faults are due to several reasons such as vibrations, internal stress, power electronic devices, and inherent eccentricity. The bearing faults has received a great deal of attention [START_REF] Blodt | Models for bearing damage detection in induction motors using stator current monitoring[END_REF], [START_REF] Elbouchikhi | Current frequency spectral subtraction and its contribution to induction machines bearings condition monitoring[END_REF]. Two main contributions can be listed herein. The first one considers the introduction of additional frequency components due to the presence of the fault [START_REF] Elbouchikhi | Current frequency spectral subtraction and its contribution to induction machines bearings condition monitoring[END_REF], [START_REF] Schoen | Motor bearing damage detection using stator current monitoring[END_REF]. The second contribution demonstrates that bearing faults introduce both eccentricity and load oscillations which leads to amplitude and frequency modulation of the stator currents [START_REF] Blodt | Distinguishing load torque oscillations and eccentricity faults in induction motors using stator current Wigner distributions[END_REF], [START_REF] Blodt | Models for bearing damage detection in induction motors using stator current monitoring[END_REF]. This paper proposes to use subspace techniques in order to overcome the poor resolution of the FFT-based approaches [START_REF] Zhang | A survey of condition monitoring and protection methods for medium-voltage induction motors[END_REF], [START_REF] Elbouchikhi | Induction machine fault detection enhancement using a stator current high resolution spectrum[END_REF]. They are concerned with estimating unknown signal parameters from stator current measurements. These approaches require a priori knowledge about the signal to be processed. In fact, a stator model is used in order to develop a frequency estimators. Moreover, the proposed methods require to estimate the number of frequency components known in signal processing community as model order selection. This model order is determined using information criteria rules [START_REF] Wax | Detection of signals by information theoretic criteria[END_REF]. Furthermore, we present frequencies amplitude estimation by using the Least Squares Estimator (LSE). Finally a fault severity is proposed in order to measure the fault degree using the algorithm introduced in [START_REF] Elbouchikhi | Nonstationary spectral estimation for wind turbine induction generator faults detection[END_REF].

II. PROBLEM FORMULATION

In this section, we present bearing faults effects over the stator current. Moreover, the stator current model is proposed in order to develop a fault detection technique.

A. Bearing Faults

We consider the problem of rolling-element bearings. These faults are the most frequent faults in induction machines [START_REF] Blodt | Models for bearing damage detection in induction motors using stator current monitoring[END_REF]. The bearing consists mainly of outer raceway, inner raceway, balls, and the cage which assures equidistance between the balls. The bearing faults can be classified according to the affected element -Outer raceway defect, -Inner raceway defect, -Balls defect, -Cage defect.

For each type of bearing faults, a characteristic frequency f c can be associated [START_REF] Blodt | Models for bearing damage detection in induction motors using stator current monitoring[END_REF]. Their expressions for the four considered fault types are given by

f o = N b 2 f r 1 -D b Dc cos β f i = N b 2 f r 1 + D b Dc cos β f b = Dc D b f r 1 - D 2 b D 2 c cos 2 β f ca = 1 2 f r 1 -D b Dc cos β , (1)
where f o is the outer raceway fault frequency, f i the inner raceway fault frequency, f b the ball fault frequency, f ca the cage fault frequency, N b the number of balls, f r the mechanical rotor speed in Hertz, D b the ball diameter, D c the bearing pitch diameter, and β the contact angle of the balls on the races.

The characteristic bearing fault frequencies can be reflected on stator currents. The relationship between these frequencies and current frequency can be described by

f bf = |f s ± kf c | , (2)
where f s is the supply fundamental frequency, f c is the bearing fault characteristic frequency, and k ∈ N * .

B. Stator Current Model

Induction machine stator current in presence of bearing faults can be expressed by a model, characterized by fundamental component, harmonics and faults. In this type of modeling, the observed model can be seen as a sum of a deterministic signal and noise. In this case, the stator current can be described by the following model

x[n] = L-1 k=0 a k e j(2πf k × n Fs +φ k ) + b[n], (3)
where

-x[n] denotes the stator current samples, -b[n] ∼ N c (0, σ 2 )is a complex circular white Gaussian noise i.e.: E [b(n)] = E [b(n)b(n + τ )] = 0, E [b(n)b * (n + τ )] = σ 2 δ(τ ), (4)
where E[.], (.) * , and δ(.) correspond to the statistical expectation, the complex conjugate, and the Dirac delta, respectively,

-L represents the model order, -F s is the sampling frequency, -a k , f k , and φ k are the signal parameters (amplitude, frequency, and initial phase, respectively) of the k th component.

At time n = 0, 1, 2, 3, ... the observed stator current vector x(n), defined as

x(n) = [x(n) . . . x(n + M -1)]
T , can be expressed as

x(n) = A(f k )v + b(n), (5)
where -M is the length of the time window vector, -(.) T refers to transpose,

-x[n] = [x[n] . . . x[n + M -1]] T is a M × 1 column vector containing the stator current samples, -b(n) = [b[n] . . . b[n + M -1]]
T is a M × 1 column vector containing the noise samples, -v is a L × 1 column vector containing the amplitudes and phases of the frequencies. This vector is given by v = a 0 e jφ0 a 1 e jφ1 . . . a L-1 e jφ L-1 , (6)

-A(f k ) is a M × L Vandermonde matrix given by A(f k ) =        1 1 . . . 1 e j2πf0× 1 Fs e j2πf1× 1 Fs . . . e j2πf L-1 × 1 Fs . . . . . . . . . . . . e j2πf0× M -1 Fs e j2πf1× M -1 Fs . . . e j2πf L-1 × M -1 Fs        . (7)
The main objective of the remaining sections is to propose parameters estimation techniques which will allow to extract the fault characteristic frequencies and to measure the fault severity based on the stator current observations. The proposed architecture is described by Fig. 1.

III. SUBSPACE TECHNIQUES

Many signal processing applications use methods called high-resolution techniques for frequency estimation, such applications include electrical signal processing [START_REF] Kim | High-resolution parameter estimation method to identify broken rotor bar faults in induction motors[END_REF]- [START_REF] Xu | Improvement of the Hilbert method via ESPRIT for detecting rotor fault in induction motors at low slip[END_REF]. These techniques are mainly based on the eigendecomposition of the covariance matrix and knowledge of the model order. These methods are presented in the following subsections.

A. Covariance Matrix

Subspace techniques are based on the covariance eigenvalue decomposition. In fact, the covariance matrix is decomposed in two orthogonal subspaces: signal subspace and noise subspace. It can be written in the form of two covariances

R x = E x[n]x H [n] = R s + R n , (8)
where R s denotes the signal covariance matrix, R n is the noise covariance matrix, and (.) H refers to Hermitian matrix transpose.

The covariance matrix eigenvalues decomposition can be written as follows

R x = UΛU H = S G Λ s 0 0 Λ n S G H , (9)
where Λ s and Λ n are the diagonal matrix containing eigenvalues of signal and noise subspaces arranged in descending order, respectively (S and G are the associated orthonormal eigenvectors, respectively).

The orthogonality between signal and noise subspaces (S H S = I and G H G = I), can be explained by the following expressions P G S = 0,

P S G = 0, (10)
where P S = SS H and P G = GG H are the projection operators onto signal and noise subspaces, respectively.

In practice, the theoretical covariance matrix R x and its eigendecomposition are not known but can be estimated from observations as follows

R x (n) = 1 G G n=1 x(n)x H (n), (11)
where

x(n) = [x(n) . . . x(n + M -1)] T has length M . Since we have N observations of x[n], G = N -M + 1.
The purpose of eigendecomposition analysis is to define the signal and the noise subspaces. In practice, we estimate them by using eigenvectors and eigenvalues of the sample correlation matrix. In fact, the greatest eigenvalues L of the covariance matrix in [START_REF] Kim | High-resolution parameter estimation method to identify broken rotor bar faults in induction motors[END_REF] are associated with the signal subspace. The corresponding eigenvectors span the signal subspace whereas, the remaining eigenvectors span the noise subspace. It appears then that the knowledge of the model order is indispensable. That's why, the following subsection deal with model order estimation.

B. Model Order Estimation

The knowledge of the sinusoids number in the stator current samples is mandatory in order to perform stator current spectral analysis. The model order estimation is obtained by minimizing the Akaike Information Criterion (AIC) or the Minimum Description Length (MDL) described in [START_REF] Wax | Detection of signals by information theoretic criteria[END_REF]. In this approach, the model order is determined by the eigenvalue decomposition of the covariance matrix. Criteria are given by the following expressions

AIC(k) = -2 log     p i=k+1 λ 1 p-k i 1 p-k p i=k+1 λ i     (p-k)N + 2k(2p -k), (12a) M DL(k) = -log     p i=k+1 λ 1 p-k i 1 p-k p i=k+1 λ i     (p-k)N + 1 2 k(2p-k) log(N ), (12b)
where λ i and p denote the ordered eigenvalues and the eigenvectors number of the covariance matrix R x , respectively. The model order of signal is determined as the value of k ∈ {0, 1, ....., p -1}, that minimizes the AIC or MDL criterion.

C. Root-MUSIC

MUSIC (MUltiple SIgnal Classification or MUltiple SIgnal

Characterization) method is a technique in statistical estimation theory based on the use of subspace concept. This frequency estimation technique exploits the orthogonality between signal and noise subspaces [START_REF] Garcia-Perez | Startup current analysis of incipient broken rotor bar in induction motors using highresolution spectral analysis[END_REF], [START_REF] Schmidt | A signal subspace approach to multiple emitter location spectral estimation[END_REF], [START_REF] Stoica | Introduction to Spectral Analysis[END_REF]. After these subspaces are identified, a frequency estimation function is used to find the component frequencies by using eigenvectors of the noise subspace. This frequency estimation is obtained by finding the highest local maxima L of the following cost function

f k = argmax {f } 1 a H (f ) G 2 F , ( 13
)
where G is formed of the orthonormal eigenvectors spanning the noise subspace, . F denotes the Frobenius norm, and a H (f ) denotes the row vector given by

a H (f ) = 1, e j2πf
Fs , e

To avoid searching for the peaks of the cost function given by ( 13), Root-MUSIC seems to be a solution. In fact, Root-MUSIC method is a polynomial-rooting technique, that converts the pseudo-spectrum function in [START_REF] Xu | Improvement of the Hilbert method via ESPRIT for detecting rotor fault in induction motors at low slip[END_REF] into a polynomial function given by [START_REF] Wax | Detection of signals by information theoretic criteria[END_REF]. The L roots of Q(z) have amplitude < 1 and are close to the unit circle [START_REF] Stoica | Introduction to Spectral Analysis[END_REF]. In this case, we can use a polynomial representation as follows

Q(z) = a T ( 1 z ) G G H a(z), (15)
where z = e j2πf Fs and a T (z) = 1, z -1 , z -2 , ........., z M -1 .

Finally, finding z k roots of Q(z) is equivalent to find frequencies according to the following equation

f k = arg( z k ) 2π × F s . (16)

D. ESPRIT

ESPRIT (Estimation of Signal Parameters via Rotational Invariance Techniques) is a parametric method of signal processing based on the notion of subspaces [START_REF] Roy | ESPRIT-estimation of signal parameters via rotational invariance techniques[END_REF]. Indeed the separation between two subspaces in the ESPRIT technique is based on singular value decomposition of the covariance matrix R x . It exploits the invariance property by rotation of the signal subspace to estimate frequencies of the sinusoidal model [START_REF] Stoica | Introduction to Spectral Analysis[END_REF]. Furthermore, ESPRIT has significant performance compared to MUSIC in terms of cost calculation and data storage [START_REF] Roy | ESPRIT-estimation of signal parameters via rotational invariance techniques[END_REF]. Another advantage of ESPRIT, is the fact that, it does not need to find peaks of pseudo-spectrum function. In fact, due to the characteristics mentioned, ESPRIT is the first method recommended in applications of frequency estimation [START_REF] Stoica | Introduction to Spectral Analysis[END_REF].

The objective of ESPRIT method is to solve the following rotation equation

S = A(f k )T. ( 17
)
Let A 1 = I M -1 0 A and A 2 = 0 I M -1 A be unstaggered and staggered components of the matrix A, respectively.

So, we can demonstrate the following equality

A 2 = A 1 Ω, (18)
where Ω = diag e j2πf0 , e j2πf1 , ..., e j2πf L-1 contains the unknown parameters.

Let S 1 = I M -1 0 S and S 2 = 0 I M -1 S be unstaggered and staggered signal subspaces, respectively.

According to the previous equations, we can write

S 1 = A 1 T, S 2 = A 2 T. (19)
From equations ( 18) and ( 19), we obtain

S 2 = S 1 T -1 ΩT = S 1 Φ, (20)
where Φ = T -1 ΩT is the relation between the two subspaces rotations. Indeed, eigenvalues of Φ must be equal to diagonal elements of Ω and columns of T are eigenvectors of Φ.

Using the Least-squares method [START_REF] Manolakis | Statistical and Adaptive Signal Processing: Spectral Estimation[END_REF], Φ is given by

Φ = S H 1 S 1 -1 S H 1 S 2 . (21)
In practice, we can estimate frequencies of signal according to the following expression

f k = arg(v k ) 2π × F s , (22)
where v k are eigenvalues of Φ estimate given by

Φ = S H 1 S 1 -1 S H 1 S 2 . (23)
IV. PROPOSED FAULT DETECTION APPROACH This section presents the proposed fault detection approach. First, an amplitude estimator is proposed based on LSE. Then, a fault detection criterion is described. Finally, the proposed algorithm for bearing fault detection is described.

A. Amplitude Estimation

The amplitude estimation for each frequency component is realized by the Least Squares Estimator (LSE). In practice, the LSE is widely used due to its ease of implementation [START_REF] Stoica | Amplitude estimation of sinusoidal signals: survey, new results, and an application[END_REF]. This estimator is based on the minimization of the square deviation between observations and supposed model of the signal. The difference is given by the following expression

v = argmin {v} x(n) -A( f k )v 2 . ( 24
)
Once frequencies are estimated based on the proposed approaches in the above section, the corresponding solution of the previous cost function is given by

v = A H ( f k )A( f k ) -1 A H ( f k )x(n). (25)
Finally, the amplitudes estimation is given by

a k = | v k |, (26)
where v k the k th component of v and |.| denotes the complex modulus.

B. Fault Severity Criterion

In order to detect the fault severity, we propose a criterion based on the evaluation of frequency component amplitudes previously estimated. The fault severity criterion is expressed mathematically as follows

C = k∈θ1 a 2 k l∈θ2 a 2 l , (27)
where

-θ 1 corresponds to the integers k that belong to [0, L -1] for which |f k -nf s | > ∆f (n ∈ N),
-θ 2 corresponds to the integers l that belong to [0, L -1] for which |f l -nf s | < ∆f (n ∈ N),

-∆f = 10 -2 f s is the authorized variation of frequency values.

The fault severity criterion is inspired from the total harmonic distortion (THD) of a signal [START_REF] Fuchs | Power Quality in Power Systems and Electrical Machines[END_REF]. This criterion allows

V. EXPERIMENTAL RESULTS

The efficiency of the proposed approach is demonstrated on experimental stator currents issued from an induction machine with bearing faults. Several fault degrees are considered which shows the ability of the proposed criterion to track fault severity.

A. Experimental Setup Description

The above signal processing methods are tested on experimental data recorded from an induction machine with bearing fault. The machine under test is a 230/400 V, 0.75-kW, three phases induction motor with pole pairs number p = 1 and 2780 rpm rated speed. Bearing faults are obtained by simply drilling holes in the bearing. The stator currents acquisition is performed by a 24 bits acquisition card with 10 kHz sampling frequency. These stator currents have been processed offline to reveal the presence of the fault using the proposed approaches. For all the experiments, the stator fundamental frequency was almost equal to f s = 50 Hz.

B. Bearing Faults Detection

The stator currents for healthy and faulty induction motors are given by Fig. 3. Figure 4 gives the corresponding periodogram using Hanning window. This figure shows that the periodogram does not allow to clearly distinguish frequencies. Thus, the periodogram can be used as an estimator of the spectrum, but it suffers the poor frequency resolution. the readability and the resolution capacity of high-resolution techniques compare to the periodogram. Indeed these figures show that it is possible to detect bearing fault signatures by high-resolution techniques. Moreover, faults in the induction machine are manifested by increased amplitudes associated to fault frequencies.

Several experiments have been conducted with various fault degrees. The evolution of fault detection criterion is given by Fig. 7. This figure shows that the value of the fault severity criterion increases when the fault severity increases. This figure shows that it is possible to detect the fault severity in induction machine using the proposed criterion.

VI. CONCLUSION

In this paper, we have proposed a new condition monitoring architecture for induction machines based on subspace methods for frequency estimation and the least squares estimator for amplitude estimation. These techniques offer better frequency resolution than periodogram applications for short data signals with stator current analysis. The major advantage of the proposed architecture is the ability to extract automatically the fault signature as well as giving the fault severity. The usefulness of the proposed condition monitoring architecture have been demonstrated for bearing faults detection on an actual induction machine.

The proposed approaches can be easily implemented at a reasonable computational cost in real-world industry applications such as variable speed drives and renewable energy applications.

Fig. 1 .

 1 Fig. 1. Frequency estimation by using subspace techniques.

Fig. 2 .Fig. 3 .

 23 Fig. 2. Fault detection criterion computation.

Figures 5 Fig. 4 .Fig. 5 .

 545 Figures 5 and 6 display high-resolution spectrum for healthy and faulty induction motors. For each representation, we can observe a difference between the periodogram and the high-resolution methods. This difference is explained by

Fig. 6 .

 6 Fig.6. The frequency estimations for a healthy and faulty induction motors are obtained with the ESPRIT for L=20. The model order estimation is obtained thanks to two proposed information criteria (AIC and MDL).

  ValueFaullt Severity Using Root-Music for Frequency Estimation Faullt Severity Using ESPRIT for Frequency Estimation

Fig. 7 .

 7 Fig. 7. Fault severity criterion value for different fault degrees in induction motors.