Léon Masurel

Carlo Bianca

Annie Lemarchand

Thermostatted Kinetic Theory Approach To The Competition Between Cancer And Immune System Cells In An Inhomogeneous System

The competition between a cancer and the immune system are modelled at cell scale in the framework of thermostatted kinetic theory. Cell activation and learning are reproduced by the increase of cell activity during interactions. The fluctuations of system activity are controlled by a thermostat which reproduces the regulation of the learning process and memory loss through cell death. An algorithm, including spatial description and inspired from the direct simulation Monte Carlo (DSMC) method, is used to simulate stochastic trajectories for cell numbers and activities. We focus on the decisive role played by the thermostat. For inefficient thermalization, the divergence of the number of cancer cells is obtained in spite of favored production of immune system cells. Conversely, when the activity fluctuations are controlled, the development of cancer is contained even for weakened immune defenses. These results may be correlated to unexpected clinical observations in the case of different cancers, such as carcinoma, lymphoma, and melanoma.

INTRODUCTION

The competition between immune system cells and cancer cells can be compared to the interactions between immune system cells and bacteria. In both cases, the presence of foreign cells induces a cascade of responses from the immune system [1,2,3]. Different types of cells providing specific responses to aggression are involved in the defense mechanism [START_REF] Siegrist | Vaccines[END_REF][START_REF] Leo | Understanding Modern Vaccine[END_REF]. Dentritic cells, so-called because of the arms they develop, are able to ingest cancer cells and to isolate antigens. The presentation of antigens to T cells triggers their activation and proliferation [START_REF] Guermonprez | [END_REF]. In particular, killer T cells, able to destroy cancer cells, are formed. However, cancer cells may proliferate and develop the ability to blend into the surrounding tissue, misleading the immune system cells which limit their own production [7,8].

A model of competition between cancer cells and immune system cells should reflect these different interactions as well as the activation and learning processes. With the aim of building a minimal model, we consider a single type of immune system cells i, cancer cells c, and normal cells n. The learning process is reproduced by the increase of a so-called activity u carried by each cell [START_REF] Bianca | Nonlinear Anal[END_REF]10,11,12,13,14]. Thermostatted kinetic theory appears as a suitable framework to model cancer and immune system competition at the cell scale [START_REF] Cercignani | The mathematical theory of dilute gases[END_REF][START_REF] Wennberg | [END_REF]17].

In the next section, we describe the chosen model of cell interactions, write the corresponding kinetic equations, and explain how the direct simulation Monte Carlo (DSMC) method has been extended to the present context [START_REF] Bird | Molecular Gas Dynamics and the Direct Simulation of Gas Flows[END_REF][START_REF] Alexander | [END_REF]. The results are given in Section 3. We show that the model reproduces the three E's (elimination, equilibrium, escape) observed in immunotherapy, where an apparent elimination of the tumor precedes a long period of equilibrium, eventually followed by the escape of the cancer cells from the control of the immune system [20]. We point out the role played by the control of activity fluctuations in the control of cancer. Section 4 is devoted to conclusions. From left to right: The cancer cell 1 and the immune system cell 2 are randomly chosen in a spatial box, the angle between their velocities v 1 and v 2 is assumed smaller than θ u = π and the activity u 2 of the immune system cell is assumed larger that the activity u 1 of the cancer cell. Interaction is accepted according to the appropriate probability, the cancer cell 1 is transformed into an immune system cell 1 and the activity of the immune system cell 2 is increased by ǫ; cell positions are updated, which may result in a change of spatial box; velocity randomization is illustrated for cell 3, the new velocity v 3new is randomly chosen inside an angle of ±θ v /2 = π/6 around the direction of the former velocity v 3old . In addition, activities of all cells i are updated according to ∆u i = (E -αu i )∆t.

MODEL, KINETIC EQUATIONS, AND SIMULATION ALGORITHM

In order to more easily link an interpretable effect in immunotherapy to an ingredient in the model, or even to a control parameter, we have introduced a minimum number of steps in the mechanism [13,14]. Specifically, we consider only three processes which include interaction, activation, proliferation or death as follows:

             c(u) + n(u ′ ) k cn (u -u ′ )H(u -u ′ ) -→ c(u + ǫ) + c(u ′ ) S -→ n(u ′′ ) (1)
i(u) + c(u ′ ) k ic (u -u ′ )H(u -u ′ ) -→ i(u + ǫ) + i(u ′ ) (2) c(u) + i(u ′ ) k ci (u -u ′ )H(u -u ′ ) -→ c(u + ǫ) + c(u ′ ) (3)
where k cn , k ic and k ci are constants.

The process given in Eq. ( 1) includes the mutation of a normal cell into a cancer cell and the division of a cancer cell. A reservoir S is introduced to maintain the number of normal cells constant. The injected normal cell has an activity u ′′ , randomly chosen from the same distribution P(u) as the one chosen to generate the initial state. The constant k cn is chosen much smaller than k ic and k ci to account for the slower rate of normal cell mutations compared to the interaction rates between cancer cells and immune system cells. The process given in Eq. ( 2) includes the death of a cancer cell and the division of an immune system cell. Symmetrically, the process given in Eq. (3) includes the death of an immune system cell and the division of a cancer cell. In addition to limiting the number of interaction steps, simultaneously considering the division of immune system cells and the death of cancer cells (or vice-versa) offers the opportunity to add a criterion on the relative activity of an interacting pair. Due to the Heaviside step function H(uu ′ ), the autocatalytic formation of an immune system cell according to Eq. ( 2) occurs if the immune system cell has a larger activity than the cancer cell, whereas the autocatalytic formation of a cancer cell through Eq. ( 3) happens if the activity of the cancer cell is larger. The rate of each process is supposed to be proportional to the difference of activities of the interacting pair, which introduces a possible regulation: For example, despite their autocatalytic production through Eq. ( 2), a large number of immune system cells with a smaller activity than the cancer cells do not necessarily proliferate. Learning is reproduced by the small increase ǫ of the activity of the cell with the already larger activity before the interaction. With this rule, normal cells never gain activity during interactions, as desired.

In the model, the regulation of cell activity is performed using a thermostat mimicking the loss of information due the action of regulatory T cells and to cell death which suppresses cells with high activities as well. By analogy with mechanics, we introduce a field E and a friction coefficient α, such that [START_REF] Wennberg | [END_REF][START_REF] Bianca | Nonlinear Anal[END_REF]

du k dt = E -αu k (4)
where u k is the activity of cell k. Imposing that the second moment u 2 of the activity of the total number of cells remains constant leads to

α = u E u 2 ( 5
)
where u is the mean activity of the system. According to the thermostatted kinetic theory, the kinetic equations for the probability distributions f j of the three kinds of cells, j = c, i, n are given by:

(∂ t + v • ∇ x ) f j (t, x, v, u) + ∂ u (E -αu) f j = I j + V j (6)
where the interaction term I c for cancer cells obeys

I c = R + κ cn (u -ǫ, u ′ ) f c (t, x, v, u -ǫ) f n (t, x, v ′ , u ′ )du ′ dv ′ (7)
+ R + κ cn (u ′ , u) f c (t, x, v, u ′ ) f n (t, x, v, u)du ′ dv ′ - R + κ ic (u ′ , u) f c (t, x, v, u) f i (t, x, v, u ′ )du ′ dv ′ + R + κ ci (u -ǫ, u ′ ) f c (t, x, v, u -ǫ) f i (t, x, v, u ′ )du ′ dv ′ + R + κ ci (u ′ , u) f c (t, x, v, u ′ ) f i (t, x, v, u)du ′ with κ c j (u ′ , u) = k c j (u ′ -u)H(u ′ -u)H (θ u -| ∠(v, v ′ ) |) for j = n, i and κ ic (u ′ , u) = k ic (u ′ -u)H(u ′ -u)H (θ u -| ∠(v, v ′ ) |) with 0 < θ u ≤ π. The notation ∠(v, v ′
) refers to the angle between the vectors v and v ′ . The condition on the directions of the velocities of the interacting couple ensures that the two cells remain close during a sufficient time for activation and learning to take place. The interaction term I i for immune system cell is:

I i = R + κ ic (u -ǫ, u ′ ) f c (t, x, v, u ′ ) f i (t, x, v, u -ǫ)du ′ dv ′ (8) + R + κ ic (u ′ , u) f c (t, x, v, u) f i (t, x, v, u ′ )du ′ dv ′ - R + κ ci (u ′ , u) f c (t, x, v, u ′ ) f i (t, x, v, u)du ′ dv ′
For normal cells, the interaction term I n reads:

I n = - R + κ cn (u ′ , u) f c (t, x, v, u ′ ) f n (t, x, v, u)du ′ dv ′ (9)
+ P(u)

R + R + k cn (u ′ -u)H(u ′ -u ′′ )H θ u -| ∠(v, v ′ ) | f c (t, x, v, u ′ ) f n (t, x, v, u ′′ )du ′ du ′′ dv ′
All the cells have the same speed, which is conserved during the evolution. Only the directions of the cell velocities change. The turning operator V j which models the velocity-jump process is a stochastic term given by:

V j = D v T (v ′ , v) f j (t, x, v ′ , u) -T (v, v ′ ) f j (t, x, v, u) dv ′ (10)
where the turning kernel obeys

T (v, v ′ ) = p v H (θ v -| ∠(v, v ′ ) |)
, where p v is the probability that the velocity is turned clockwise or anticlockwise by an angle smaller than θ v /2 with 0 < θ v ≤ π.

The direct simulation Monte Carlo (DSMC), initially used as a direct simulation of the Boltzmann equations associated with a dilute gas [START_REF] Bird | Molecular Gas Dynamics and the Direct Simulation of Gas Flows[END_REF][START_REF] Alexander | [END_REF], has been adapted to simulate the kinetic equations given in Eq. ( 6) for j = c, i, n. The algorithm is illustrated by the scheme given in Fig. 1. During the time step ∆t, interactions are performed between cells belonging to a same spatial box of length ∆x according to the mechanism given in Eqs. (1-3), which leads to updating the natures j and the activities of the cells. Cell positions are updated according to their velocities. The directions of cell velocities are randomized following the turning operator given in Eq. ( 10); Cell velocity is shifted by a randomly chosen angle in the interval [-θ v , θ v ] with a probability p v . Finally, thermalization, i.e. the control of activity fluctuations, leads to activity update following Eqs. [START_REF] Siegrist | Vaccines[END_REF][START_REF] Leo | Understanding Modern Vaccine[END_REF]. FIGURE 3. Typical results at small field E = 3 × 10 -4 . Top: Number of immune system cells (dashed line) and cancer cells (solid line) in the system versus time. Bottom: Mean activity of immune system cells (dashed line) and cancer cells (solid line) versus time. Illustration of the three E's (elimination, equilibrium, escape) of immunotherapy. Except for the field value, the parameter values are given in the caption of Fig. 2.

RESULTS

The aim is to study the ability of the immune system to control a tumor initially occupying the entire simulated system. The immune system is a priori boosted and we impose that the constant k ic associated with the autocatalytic production of immune system cells through Eq. ( 2) is ten times larger than the constant k ci associated with the autocatalytic production of cancer cells through Eq. ( 3). The results are illustrated in Fig. 2 for an initially homogeneous condition with the same number of cancer cells, immune system cells and normal cells in each spatial box. The top subfigures show the spatial anticorrelations between the numbers of cancer cells and immune system cells: At a given time, the remaining clusters of immune cells are located where cancer cells are most depleted. The increase of the number of cancer cells and the destruction of the immune system cells shown in Fig. 2 are not intuitive for k ic ≫ k ci . The final proliferation of cancer cells is observed due to the advantage arising from their larger activities, leading to faster rates for autocatalytic production of cancer cells through the step given in Eq. ( 3).

The results shown in Fig. 2 are obtained for an intermediate value of the field E controlling the efficiency of the thermostat, i.e. the level of activity fluctuations. Depending on the field E, very different evolutions of cell numbers and activities are obtained. Figure 3 give the results for a smaller value of E than in Fig. 2 whereas Fig. 4 is associated with a larger value of E than in Fig. 2. In Fig. 3, for inefficient thermalization, the number of cancer cells rapidly drops and then remains in a quasi steady state before exploding. This typical behavior is clinically observed and is called the three E's of immunotherapy for elimination, equilibrium and escape of cancer from immune system surveillance [20]. We already reproduced this phenomenon in the framework of a homogeneous description of the system without spatial dependence. The results given in Fig. 3 show the ability of the present spatial description to account for the three E's, a phenomenon which limits the chances of success of immunotherapy. On the contrary, the results displayed in Fig. 4 for a large field E show a stabilization of the activities of cancer cells and immune system cells to the same steady value. All cells in the system eventually have the same activity. In these conditions, the rates of the processes given in Eqs. (1-3) vanish and the numbers of cancer cells and immune system cells converges to steady values, N end c and N end i , respectively. Hence, for a sufficient thermalization, cancer is controlled by the immune system. For k ic ≫ k ci , we obtain a larger steady value of the numbers of immune system cells N end i ≫ N end c . The result can be immediately extended to the case k ci > k ic , for which cancer is still controlled but for a higher steady value of the numbers of cancer cells N end c > N end i . The paradoxical coexistence of cancer cells and T cells remains a subject of debate in immunotherapy and the present work proposes a possible mechanism leading to it [START_REF] Marincola | Immunotherapy of Cancer[END_REF].

The crucial role played by the thermostat is pointed out in Fig. 5. The probability of observing a final state in which all the immune system cells have been destroyed or with nonvanishing steady numbers of both cancer cells and immune system cells sensitively varies with the field E controlling thermalization. A sharp transition between cancer escape and cancer control is observed for the intermediate value of the field E considered in Fig. 2. The model offers some hints to control cancer by immunotherapy: As a result of the simulations, the processes which tends to limit the fluctuations of cell activity are favorable. Consequently and contrary to intuition, the lifetime of the cells, including the one of the immune system cells, should not be too long to prevent the formation of highly educated cells with a high activity which would result in increasing the second moment of system activity and enable large fluctuations of the number of cancer cells, offering them a possibility to escape from the control of the immune system cells.

CONCLUSION

The interactions between a tumor and the immune system are described at the cell scale in the framework of thermostatted kinetic theory. We use a model of cell interactions having already given account of the three 3E's of immunoediting [20]: Elimination, equilibrium, and escape from immune system control have been reproduced in an homogeneous system [13,14]. In the present paper, we have introduced a spatial description of the phenomenon, taking into account cell positions and velocities. Cell interactions may modify cell type and cell activity, a quantity accounting for the level of learning of immune system cells exposed to antigens as well as the degree of invisibility reached by cancer cells. Interestingly, memory loss related to cell death and regulation of the immune system are reproduced by the effect of a "thermostat", which regulates activity fluctuations in an analogous way as an actual thermostat controls temperature fluctuations. An algorithm inspired by the direct simulation Monte Carlo method [START_REF] Bird | Molecular Gas Dynamics and the Direct Simulation of Gas Flows[END_REF][START_REF] Alexander | [END_REF] is used to numerically solve the kinetic equations for the probability densities of normal cells, cancer cells, and immune system cells. The simulations generate stochastic trajectories for the numbers of cells and the activity of FIGURE 5. Probability of occurrence of the two evolutions shown in Figs. 3 and4 versus field E associated with the thermostat. Dashed line and triangles: Vanishing number of immune system cells at the end of the simulations, N end i = 0. Solid line and squares: Nonvanishing numbers of both immune system cells and cancer cells at the end of the simulations, N end i N end c 0. Except for the field value, the parameter values are given in the caption of Fig. 2. The statistics is based on 20 runs for each field value. For E < 3.5 × 10 -4 , cancer escapes from immunosurveillance, for E > 3.5 × 10 -4 , cancer is controlled.

the system.

We focus on the effect of the thermostat, i.e. on the control of activity fluctuations using a field E. We conclude that thermalization is favorable to cancer surveillance. For values of the field E larger than a critical value, the cancer cells are not eliminated but their number converges toward a steady value. The cancer is controlled in a sufficiently dissipative system where the fluctuations of activity are regulated. Dissipation of activity occurs through cell death. Our results may explain the poor performance of immunotherapy in the case of some patients, for which mutations of cancer cells, all the more likely as cell life is long, may induce adaptive resistance to the treatment [START_REF] Zaretsky | [END_REF]. Similarly, although regulatory T cells are known to suppress T cell and hinder the immune response against cancer, their thermalizing role could explain why high levels of regulatory T cells may be associated with a positive prognosis, for example in the case of colorectal carcinoma and follicular lymphoma [23,24].

As a perspective, we intend to perform a sensitivity analysis of the different parameters of the model and in particular to study the effect of cell speed on the formation or destruction of clusters of cancer cells for different inhomogeneous initial conditions.

FIGURE 1 .

 1 FIGURE 1. Illustration of the algorithm for two spatial boxes: During a time step, cell natures, positions, velocities, and activities are updated. From left to right: The cancer cell 1 and the immune system cell 2 are randomly chosen in a spatial box, the angle between their velocities v 1 and v 2 is assumed smaller than θ u = π and the activity u 2 of the immune system cell is assumed larger that the activity u 1 of the cancer cell. Interaction is accepted according to the appropriate probability, the cancer cell 1 is transformed into an immune system cell 1 and the activity of the immune system cell 2 is increased by ǫ; cell positions are updated, which may result in a change of spatial box; velocity randomization is illustrated for cell 3, the new velocity v 3new is randomly chosen inside an angle of ±θ v /2 = π/6 around the direction of the former velocity v 3old . In addition, activities of all cells i are updated according to ∆u i = (E -αu i )∆t.

FIGURE 2 .

 2 FIGURE 2. Spatial distribution of immune system cells (top left) and cancer cells (top right) at time t = 5200 indicated by a disc on the evolution of cell numbers (bottom left) and cell activities (bottom right). Contrary to the number of immune system cells, the number of cancer cells initially drops and finally diverges. The mean activity of the cancer cells is always larger than the mean activity of the immune system cells, the line in between is the mean activity of the system. Parameter values: System size l x = l y = 20, ∆l = 1, initial number of cells in spatial box N n (t = 0) = 50, N i (t = 0) = N c (t = 0) = 20, mean µ = 0.5 and standard deviation σ = 0.2 of the initial Gaussian distribution P(u) of cell activity, time step ∆t = 1, rate constants k ic = 0.1, k ci = 0.01, k cn = 10 -5 , angle of interaction θ u = π, activity increase ǫ = 10 -3 , cell speed v = 0.0333, probability of velocity deflection p v = 0.1, angle of velocity deflection θ v = π/3, field associated with the thermostat E = 3.5 × 10 -4 .

FIGURE 4 .

 4 FIGURE 4. Same caption as in Fig. 3 for a large field E = 4 × 10 -4 . Illustration of cancer control.


Jack Hadfield

Georgia Chalvatzaki

Petros Koutras
email: pkoutras@cs.ntua.gr

Mehdi Khamassi
email: mehdi.khamassi@upmc.fr

Costas S Tzafestas

Petros Maragos
email: maragos@cs.ntua.gr

A Deep Learning Approach for Multi-View Engagement Estimation of Children in a Child-Robot Joint Attention Task

In this work we tackle the problem of child engagement estimation while children freely interact with a robot in a friendly, room-like environment. We propose a deep-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed regularly in the room, fuse the results and feed them to a deep neural network trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers, and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next generation social robots as an attention monitoring tool during various Child Robot Interaction (CRI) tasks both for Typically Developed (TD) children and children affected by autism (ASD).

I. INTRODUCTION

As robots become more integrated in modern societies, the cases of interacting with humans during daily life activities and tasks are increasing. Human-Robot Interaction (HRI) refers to the communication between robots and humans. This communication can be verbal or non-verbal, remote or proximal. A special case of HRI is Child-Robot Interaction (CRI) [START_REF] Gordon | Can children catch curiosity from a social robot?[END_REF]. Robots enter children's lives as companions, entertainers or even educators [START_REF] Belpaeme | Social robots for education: A review[END_REF]. Motivated by this need, in the context of BabyRobot EU project [START_REF]BabyRobot project[END_REF], we have developed and evaluated systems which employ multiple sensors and robots, for childrens' speech, gesture and action recognition during CRI scenarios [START_REF] Tsiami | Multi3: Multi-sensory perception system for multimodal child interaction with multiple robots[END_REF]- [START_REF] Efthymiou | Multi-view fusion for action recognition in child-robot interaction[END_REF], in a specially designed area, the "BabyRobot room". Children are very adaptive, quick learners with unique communication skills, able to easily convey or share complex information with little spoken language. A major challenge in the field of CRI, however, entails equipping robots with the ability to pick up on such information and adapt their behavior accordingly, in order to achieve a more fruitful interaction.

Robots assisting children is of particular importance in modern research, especially for mediating ASD therapy towards the development of their social skills, [START_REF] Othman | How could robots improve social skills in children with autism?[END_REF]. A review on social robots for education can be found in [START_REF] Belpaeme | Social robots for education: A review[END_REF]. Children affected by ASD can benefit from interacting with robots, since such a CRI may help them overcome the impediments posed by face-to-face interaction with humans. Moreover, it is important that the robot's behavior can adapt to the special needs of each specific child and maintain an identical behavior for as long as needed in the intervention process [START_REF] Huijnen | How to implement robots in interventions for children with autism? a co-creation study involving people with autism, parents and professionals[END_REF].

One key issue for social robots is the development of their ability to evaluate several aspects of interaction, such as user experience, feelings, perceptions and satisfactions [START_REF] Anzalone | Evaluating the engagement with social robots[END_REF]. Human engagement in HRI according to [START_REF] O'brien | What is user engagement? a conceptual framework for defining user engagement with technology[END_REF] "is a category of user experience characterized by attributes of challenge, positive affect, endurability, aesthetic and sensory appeal, attention, feedback, variety/novelty, interactivity, and perceived user control". Poggi in [START_REF] Poggi | Mind, Hands, Face and Body: A Goal and Belief View of Multimodal Communication[END_REF] adds that engagement is the level at which a participant attributes to the goal of being together with other participants within a social interaction and how much they continue this interaction. Given this rich notion of engagement, many studies have explored human-robot engagement [START_REF] Anzalone | Evaluating the engagement with social robots[END_REF]. Lemaignan et. al [START_REF] Lemaignan | From real-time attention assessment to with-me-ness in human-robot interaction[END_REF] explored the level of "with-me-ness", by measuring to what extent the human is with the robot during an interactive task, for assessing the engagement level. Human engagement has been modeled in a number of works, using solely gaze [START_REF] Boucher | I reach faster when i see you look: Gaze effects in humanhuman and humanrobot face-to-face cooperation[END_REF], speech and gaze [START_REF] Ivaldi | Towards engagement models that consider individual factors in hri: On the relation of extroversion and negative attitude towards robots to gaze and speech during a human-robot assembly task[END_REF], and human pose with respect to the robot from static positions [START_REF] Anzalone | Evaluating the engagement with social robots[END_REF], [START_REF] Foster | Automatically classifying user engagement for dynamic multi-party human-robot interaction[END_REF].

Engaging children in CRI tasks is of great importance. The social characteristics that robots should have when performing as tutors were examined in [START_REF] Zaga | The effect of a robot's social character on children's task engagement: Peer versus tutor[END_REF], [START_REF] Schodde | How to manage affective state in child-robot tutoring interactions?[END_REF]. Specific focus is given in estimating the engagement of children with ASD interacting with adults [START_REF] Chorianopoulou | Engagement detection for children with autism spectrum disorder[END_REF] or robots [START_REF] Tapus | Children with autism social engagement in interaction with nao, an imitative robot: A series of single case experiments[END_REF]. A study analyzing the engagement of children participating in a robot-assisted therapy can be found in [START_REF] Rudovic | Measuring engagement in robot-assisted autism therapy: A cross-cultural study[END_REF]. A method for the automatic classification of engagement with a dynamic Bayesian network using visual and acoustic cues and support vector machine classifiers is described in [START_REF] Feng | Engagement evaluation for autism intervention by robots based on dynamic bayesian network and expert elicitation[END_REF]. Another approach considers the facial expressions of children with ASD to evaluate their engagement [START_REF] Javed | Adaptive framework for emotional engagement in child-robot interactions for autism interventions[END_REF]. A robot-mediated joint attention intervention system using as input the child's gaze is presented in [START_REF] Zheng | Design, development, and evaluation of a noninvasive autonomous robot-mediated joint attention intervention system for young children with asd[END_REF]. A deep learning framework for estimating the child's affective states and engagement is presented in [START_REF] Rudovic | Personalized machine learning for robot perception of affect and engagement in autism therapy[END_REF] in a static setting in front of the robot. Previous work from our research team focused on using engagement estimates in a reinforcement learning setting to adapt robot's motion and expressivity [START_REF] Khamassi | Robot fast adaptation to changes in human engagement during simulated dynamic social interaction with active exploration in parameterized reinforcement learning[END_REF], [START_REF] Khamassi | An extended framework for robot learning during childrobot interaction with human engagement as reward signal[END_REF]. Hence, estimating children engagement level is important for adapting the behavior of a robot companion.

In this paper we present an deep learning approach for estimating the engagement of children in a CRI collaborative task aiming to establish joint attention between a child and a robot. In the specific scenario examined, the robot's aim is to elicit child's attention through an experiment consisting of a handover task, that tests the child's attentiveness and social coordination skills, with the hope that the child will correctly understand the robot-agent's intentions and ultimately take part in a successful collaboration. In this setting, children engagement refers to rich information about their movement in the room w.r.t. the robot agent. Our method employs the children full body pose estimates along with high-level computed features, such as gaze and arm extension, to capture a wide range of movements and actions relevant to the child's engagement.

Our method incorporates a multi-view estimation of the child's pose, where the child is inside a specially arranged room with a network of cameras (Fig. 1). An LSTM-based algorithm classifies the engagement of the child to the task using the child's pose as input and observations by experts as control targets. We experimentally validate our algorithms exploiting the RGB-D data from recordings of children who A major contribution of this work is the proposal and validation of an engagement estimation method generalizable to a multitude of CRI tasks. The setup allows the child to freely move and interact with the robot, without being limited to a stationary position in front of a robot, sensor or adult. This is in contrast to prior works which are restricted in such a sense. The data-driven nature of our approach also contributes to its generalizability, as no prior hypotheses are made regarding the actions most relevant to the task. Instead, the algorithm learns to distinguish the movements and stances most informative of the child's engagement. Finally, the multi-view fusion helps achieve a more robust and accurate classifier by confronting cases of body part occlusions and providing better pose estimations than if a single camera was used. We believe that such a pose-related framework can also apply to ASD children engagement estimation who present a variability of behaviors and motions in order to adapt the behavior of robot mediation accordingly.

II. METHOD

The main problem addressed in this work is to estimate the engagement level of children from visual cues. The problem is cast as one of multi-class classification, where each class corresponds to a different level of engagement. Specifically, we designate three distinct levels of engagement: the first (class 1) signifies that the child is disengaged, i.e. paying limited or no attention to the robot; the second (class 2) refers to a partial degree of engagement, where the child is attentive but not cooperative; the final level (class 3) means that the child is actively cooperating with the robot to complete the handover task. The task details are described in Sec. III-A. During the course of an interactive session, the engagement level varies. The goal is therefore to perform this classification across a number of fixed-length time segments during the session, rather than producing a single estimate for the entire interaction. In the remainder of this section we describe the proposed method to perform this classification.

A. Multi-View child pose estimation

Perhaps the most informative data for recognizing the engagement level in joint attention tasks is that of the child's pose. The problem of detecting human pose keypoints in images is a challenging one, due to occlusions and widely varying articulations and background conditions. Only recently has the problem been solved to a satisfactory degree, especially with the introduction of the Open Pose library [START_REF] Cao | Realtime multi-person 2d pose estimation using part affinity fields[END_REF]- [START_REF] Wei | Convolutional pose machines[END_REF] for 2D keypoint detection.

In [START_REF] Zimmermann | 3d human pose estimation in rgbd images for robotic task learning[END_REF], an end-to-end 3D pose estimator from RGB-D data is proposed. The Open Pose library is used to detect 2D keypoint score maps from the color image. These maps are then fed to a deep neural network, along with a voxel occupancy grid derived from the depth image. The network is trained to produce 18 3D keypoint estimates: two for each wrist, elbow, shoulder, hip, knee and ankle, one for the neck and five facial points, consisting of the ears, the eyes and the nose. We employ this system in our work, to estimate the child poses during their interaction with the robot.

When using multiple cameras, the keypoints can be extracted for each view and fused to produce the final estimates (Fig. 3). The first step to achieve this is to register the points of each camera reference frame to a single common frame. The registration parameters were found using the ICP algorithm [START_REF] Besl | A method for registration of 3-D shapes[END_REF], which provides the transformation that best fits the point cloud of one camera to that of another, given an initial transformation that we set manually.

After transforming the keypoint coordinates of all cameras to a common reference frame, the next step is to determine which keypoints are valid from each view. The pose estimation algorithm occasionally fails, either when some of the child's joints are hidden, or when the pose differs substantially from those used to train the algorithm. In such cases, the system produces noisy estimates or no detections at all for certain keypoints. Another problem is that the algorithm sometimes outputs multiple poses, when another person is in view or occasionally when the network is confused by some background artifact. To tackle such problems, we only average the points that are sufficiently close to those detected in the previous frame, i.e. at a maximum distance of 0.5m. If no such points exist for a certain joint, we mark the joint as missing in the current frame. Having fused the pose detections of multiple views, we interpolate the missing values using the previous estimates and then smooth the output using a simple low-pass filter. The points then undergo a final rotation, so that the coordinate axes co-align with the edges of the room.

B. Feature Extraction

The child's pose is particularly useful when examined in relation to the robot's position. Therefore, we would like to estimate the 3D keypoints within the robot's frame. Since the Nao robot isn't equipped with any localization sensors, we must estimate its position with respect to the world coordinates through other means. To this end, we detect the robot in the color stream of one of the cameras, and infer its 3D position via an inverse camera projection. The robot's localization is thus performed fully autonomously, alleviating the need to provide an initial position and monitor its trajectory withing the rooms.

We fine-tuned the YOLOv3 detection network [START_REF] Redmon | Yolov3: An incremental improvement[END_REF] to detect the robot's head on a set of 100 manually annotated images. Using this network, we then detected the robot position in all video frames. An example is shown in Fig. 4. Paired with the depth images, we converted the detections to 3D points. The robot detections also contained noise and missing values, and were subjected to a similar procedure as the keypoints, ie. missing value interpolation and smoothing. We also rejected erroneous detections if they lay outside a certain expected range, based on the limitations of the robot's movement. The body 3D keypoints are transformed to the detected robot frame each time instant, in order to capture the child's movements and transitions in relation to the robot.

Aside from the child's pose keypoints, we produce a number of high-level features that are expected to assist the classification process (Fig. 5). These include the angle between the child's gaze and the robot, the angle between the child's body facing and the robot and the distance of the child's hands from their respective shoulders. The gaze direction is calculated from the detected facial keypoints, by taking the ear-to-ear vector in the 2D plane and rotating it 90 deg. The body facing is calculated in a similar fashion from the shoulder keypoints. From the two resulting angles, we subtract the robot-to-child angle, which is calculated using the keypoint center of mass and the detected robot position. The high-level features are concatenated with the keypoint values relative to the robot, mentioned above, to form the input data to the classifier.

C. Engagement Estimation

A key observation worth noting is that the degree of engagement heavily depends on temporal information. One reason for this is that the child tends to display the same level of interest over the course of a few-second interval. More importantly, however, the child's movement and actions contain rich information which can be exploited. For example, if the child is constantly shifting their gaze, this is usually an indication of disengagement, whereas a steady focus signifies a higher level of interest. By choosing suitable machine learning algorithms capable of capitalizing on temporal data, we can expect a notable improvement over simply classifying each segment individually.

We use a deep NN to classify the engagement level over time, the architecture of which can be seen in Fig. 6a. The network consists of three fully connected (FC) layers, a Long Short-Term Memory (LSTM) network [START_REF] Hochreiter | Long short-term memory[END_REF] and a final FC layer, with a softmax function applied to the output to produce a probability score for each class. LSTM networks are a certain type of recurrent neural network that are known to be well-suited to dealing with time-varying data.

The network is fed a sequence of L inputs. We group the input features described earlier into segments of 5 frames, over which we compute the mean and standard deviation, thus further reducing noise and avoiding training on spurious data points, which can cause over-fitting. This gives us an input vector x t for each segment t ≤ L, with a dimension of

D = 2 • [3 • (18 keypoints) + 4 high-level features] = 116
The output y t (x 1 : t , W) is a function of the previous t inputs in the sequence and the weights W of the network, with a dimension equal to the number of classes (K = 3). The FC layers produce linear combinations of their inputs, operating on each sequence point individually. The softmax layer ensures that the elements of y t are positive and sum to one. The data are fed in batches of size N. The output dimensions of each layer are shown in the table in Fig. 6. The variable C is the size of the LSTM's hidden state. We set the output sizes of the first three FC layers to 2 • C after trial and error. The first three FC layers are each followed by a dropout layer, with probability 0.5, and a Rectified Linear Unit (ReLU).

III. EXPERIMENTAL ANALYSIS & RESULTS

A. Experimental setup 1) Setup: The experimental setup can be seen in Fig. 2. As shown, 4 Kinect devices are placed at various angles to capture multiple views of the room. Devices 1 and 2 capture the room from either side, device 3 from above and device 4 from the front. The Nao robot [START_REF] Gouaillier | Mechatronic design of NAO humanoid[END_REF] was chosen for the task because it is capable of a wide range of motion, and its human-like features make it suitable for child-robot interaction. One or more bricks were placed either on a table in front of the robot, or on the floor close by. The child is free to move around the room as they please.

2) Session Description: The experiments evolved as follows.

The robot approached one of the bricks and displayed the intention of picking up the brick, without being able to actually grasp it. With a series of motions it attempted to capture the child's attention and prompt the child to hand over the brick. These motions included pointing at the brick, opening and closing its hand, alternating its gaze between the child and the brick and a combination of head turning and hand movement. Example snapshots of such motions are shown in Fig. 7. If after a certain length of time the child failed to understand the robot's intent, the robot proceeded to ask the child verbally. Once the brick was successfully handed over, the robot thanked the child and in some cases looked for another brick to grasp.

3) Data Collection: We recorded a total of 25 sessions, resulting in approximately 85 min of recorded RGB-D data for the four cameras. The children were aged 6-10 years old, 15 male and 10 female. The videos were then given to experts for annotation, according to the scheme described earlier.

B. Experimental Validation

We evaluate the method described above trained on the recordings of the children. Since we only have 25 videos, rather than splitting the set into training and testing subsets, we carry out the evaluation via leave-one-out crossvalidation. 1) Implementation: We implemented the NN described in section II using the PyTorch library. The network was trained from scratch, with an initial learning rate of 0.1, momentum 0.5 and weight decay 10 -6 . We used early stopping on a random subset of the training data, with a patience level of 10 epochs. When the training converged, we dropped the learning rate by a factor of 10. We chose a training batch size of N = 16 and set the hidden state size to C = 560. The sequence length L was set to 30. These values were chosen after an extensive hyper-parameter search.

Since LSTM networks generally require a large amount of data to train successfully and avoid over-fitting, we employ a few methods of data augmentation. Namely, we add a small amount of Gaussian noise to the mean value of each segment and randomly choose the starting point of each sequence within a range of 2 seconds. We observed a further improvement when training the FC layers first, and then freezing their weights, adding the LSTM module and training the remaining network. This forces the initial layers to produce informative outputs with regard to each individual segment, which the LSTM can then utilize to extract meaningful temporal information. Additionally, since we observed occasional spikes in the network's gradients, we performed gradient clipping on the LSTM layer by capping the gradient norms at the value of 0.1.

The final classification is performed on 1-second segments, by process of a majority vote within the segment. At a frame rate of 30 fps, each second contains 6 smaller segments. This seemed a logical compromise between over-sampling the data points and segmenting the temporal stream too crudely to be of use.

The classes were notably imbalanced, with 281 segments belonging to class 1, 2578 to class 2 and 745 to class 3. Though we also experimented with under-sampling and oversampling, the best results were achieved using a weighted cross-entropy loss during training:

L = - 1 NL N ∑ j L ∑ t w c j,t log y c j,t (x j,1 : t , W) (1)
where y is the network output, c j denotes the class of the j-th sample in the minibatch and w is a vector containing the weights for each class. We set w = (9.16, 1.00, 3.42), based on the appearance frequencies of each class in the dataset.

2) Evaluation Metrics: Due to the large class imbalance, the standard accuracy measure is not very informative. Therefore, we use two other measures of performance. The first is the average F-Score across all three classes, which is high only when both the precision and recall of each class is high.

The second is the balanced accuracy of [START_REF] Brodersen | The balanced accuracy and its posterior distribution[END_REF], given by:

1 K K ∑ c=1 T P c T P c + FP c (2)
where T P c and FP c denote the true and false positives respectively of class c, and K = 3 is the number of classes.

3) Results: In Table I we compare the LSTM-based network against other popular classifiers, in particular a Support Vector Machine (SVM) and a Random Forest (RF). The SVM uses an RBF kernel with a regularization weight of C = 100 and a kernel coefficient of γ = 0.01. The RF consists of 10 trees with a maximum depth of 10. For fair comparison, the hyper-parameters of both classifiers were tuned via an equally extensive grid search as the LSTM network. We also include two baseline approaches. The first consists of the results if the majority class (class 1) is always predicted.

The second is another LSTM-based network trained solely on the gaze direction feature, which is a commonly used feature in literature [START_REF] Saran | Human gaze following for human-robot interaction[END_REF]. Notice that the proposed method outperforms all other classifiers, confirming our hypothesis that exploiting temporal relations in the input data can lead to better results. The SVM and the RF fail to capture temporal continuity and logical transitions between engagement states 1 . Both classifiers perform significantly better than simply predicting the majority class, however, meaning that even stationary pose information is partially descriptive of the engagement level. On the other hand, the network trained only on gaze direction performs rather poorly, since (a) the gaze estimate contains a lot of noise which the network can't deal with using redundant variables, and more importantly (b) it doesn't capture information such as arm extentions, walking and other movements.

In Table II we evaluate some other network architectures that we also tried. We experimented with the removal of the third FC layer (rows 2 and 4) and with the addition of a second LSTM layer (rows 3 and 4). As shown, the chosen architecture provides notably better results across all metrics. The additional LSTM layer causes over-fitting, rather than learning any deeper information in the data. The use of dropout allows a deeper network, with the addition of the third FC layer boosting the performance by a large margin.

In Table III we provide a comparison of different hyperparameter values. The optimal sequence length is small enough to allow the training set to be divided into enough sequences to avoid over-fitting, but large enough to capture long term dependencies in the data. A batch size of 16 also provides a compromise between finely sampling the training data in each iteration and avoiding local minima while training. In Fig. 8 we can see the ROC curves for each class separately along with the micro and macro-average ROC curves. The macro-average number of True/False positives is simply the average of each class. The micro-averages, on the other hand, are weighted by the number of samples in each class. Therefore, the micro-average curve is better suited due to the large class imbalance, and our network achieves an Area Under Curve of 0.83 regarding this ROC.

Finally, in order to examine which features are of greater importance to the model, we calculate the gradient of the loss function with respect to each input variable. A higher gradient means the network is more sensitive to changes of the input variable, indicating a higher importance. Due to 1 See supplementary video for examples.

the high correlation between inputs, the network focused on different variables each time it is trained. Therefore, we trained the model 100 times and averaged the gradients calculated in each run. The results are shown in Fig. 9. We can see that both the high-level features and the keypoint positions contribute to the algorithm's output, justifying our pose-based approach to the problem. While the child's gaze is the most informative variable, a number of other inputs also score similarly. The child's body facing can sometimes complement the gaze, while the 2D position conveys how close the child is to the robot. The limb positions allow the network to exploit temporal differences to detect walking and reaching movements. Also note that although the segment standard deviations are less important than the mean values, they are not entirely uninformative, as they contain information regarding abrupt movements that can be indicative of the child's reactions to the robot's actions.

As we see from the results above, the proposed deep network architecture can learn and accurately track the child's engagement based on their pose variation during the proposed freely interaction task. The developed system can be further improved with the presence of more annotated data and can easily be adapted to a range of CRI tasks that require body movement on the child's behalf. The whole engagement module can be integrated alongside with child's speech, action or emotion recognition modules in order to create next generation social robots that can detect and recognize the children's behavior.

IV. CONCLUSIONS & FUTURE WORK

In this work we proposed, by taking advantage of recent progress in deep learning, a method of child engagement estimation during child-robot collaboration without restricting their movement. The use of child pose data, in conjunction with an LSTM-based neural network, proved to be effective towards this goal. This is especially important considering the difficulty of the problem. Differences in child behavior and personality, a wide range of possible motions and actions and various technical challenges all contribute to this difficulty. The concept of engagement is not rigidly defined, thus making the task hard even for humans. Despite this, we achieve relatively high evaluation metrics across a dataset of 25 children. An important direction for future work will be to test and adapt the system to children affected by ASD. Naturally, this imposes a further challenge, as ASD children act very differently to children in typical development.

Fig. 1 :

 1 Fig. 1: Examples of three levels of engagement: (a) Limited attention (class 1), (b) Attention but no cooperation (class 2), (c) Active cooperation (class 3).

Fig. 2 :

 2 Fig. 2: Setup for recording joint attention experiments.

Fig. 3 :

 3 Fig. 3: Multi-view pose estimation overview.

Fig. 4 :

 4 Fig. 4: The detected pose is shown along with the detected bounding box surrounding the robot's head.

Fig. 5 :

 5 Fig. 5: Extracted features shown from an overhead view of the room. The detected keypoints are shown as black circles.

  (a) Network architecture: fully connected layers FC1, FC2, FC3, a single LSTM layer and a final FC4 layer coupled with a softmax function on the output. Layer outputs sizes: N is the batch size, L is the sequence length, C is the hidden states size and K is the number of classes.

Fig. 6 :

 6 Fig. 6: (a) Architecture of neural network used to classify child engagement. (b) Network layer details.

Fig. 7 :

 7 Fig. 7: Examples of the robot's attempts to elicit the child's attention.

Fig. 8 :

 8 Fig.8: ROC curves of network with optimal hyper-parameters.

TABLE I :

 I Performance results of different algorithms on the data. Results are averaged across folds of leave-one-out cross-validation.

	Method	Mean F-Score Accuracy Balanced Accuracy
	Majority class	27.90	71.97	33.33
	Gaze LSTM	32.78	71.58	36.10
	SVM	54.79	68.27	58.61
	RF	56.41	68.60	61.78
	3FC+LSTM	62.18	77.11	61.88

TABLE II :

 II Cross-validation results for different network architectures.

	Net Architecture	Mean F-Score Accuracy Balanced Accuracy
	3 • FC + LSTM		62.18	77.11	61.88
	2 • FC + LSTM		56.23	71.86	58.46
	3 • FC + 2 • LSTM	54.78	70.60	56.30
	2 • FC + 2 • LSTM	54.45	69.71	56.91
	Parameters	Mean F-Score Accuracy Balanced Accuracy
	N=8, L=30		58.75	74.41	58.40
	N=16, L=30		62.18	77.11	61.88
	N=32, L=30		55.36	69.34	58.68
	N=16, L=10		47.21	61.68	51.58
	N=16, L=60		48.25	57.32	56.86

TABLE III :

 III Results for different hyper-parameter values.

ACKNOWLEDGMENT

The authors would like to thank psychologists Asimenia Papoulidi and Christina Papailiou for their annotations, and our colleagues Niki Efthymiou and Panagiotis Filntisis for helping organize and record the experiments.

This research work has been supported the EU-funded Project BabyRobot (H2020, grant agreement no. 687831).


James Gillespie
email: gillespie-j10@ulster.ac.uk

Iñaki Rañó

Nazmul Siddique
email: nh.siddique@ulster.ac.uk

Jose Luis Santos
email: ja.santos@ulster.ac.uk

Mehdi Khamassi
email: mehdi.khamassi@upmc.fr

José Santos

Using Reinforcement Learning to Attenuate for Stochasticity in Robot Navigation Controllers

Keywords: Braitenberg Vehicle, Reinforcement Learning, Stochastic Environment

come   L'archive ouverte pluridisciplinaire

I. INTRODUCTION

Braitenberg Vehicles are a set of qualitative models of sensor-based steering behaviour in animals navigating towards, or away from, a stimulus [START_REF] Braitenberg | Vehicles. Experiments in synthetic psycology[END_REF]. They have been used for decades in different navigation tasks, and empirical works on real robots have shown that robots using these controllers can work in harsh and dynamic environments [START_REF] Lilienthal | Experimental analysis of smelling braitenberg vehicles[END_REF]. Because of their qualitative biological nature, these controllers have been typically implemented using different types of neural networks. Qualitative models of the closed-loop have been derived for the case of ideal (noiseless) sensors [START_REF] Rañó | Biologically inspired navigation primitives[END_REF], where the control mechanism is modelled as a generic function.

The assumption of deterministic state and perception had the advantage of greatly simplifying the analytic treatment of the closed-loop motion models. Moreover, it has been recently shown that Policy Gradient Reinforcement Learning can be used to find neural controllers that approximate functions leading to closed-loop stable behaviour of Braitenberg Vehicle 3a [START_REF] Gillespie | Reinforcement learning for bio-inspired target seeking[END_REF]. However, these controllers might not perform well in presence of sensor noise. Recently, a new model of Braitenberg Vehicle 3a has been derived for noisy sensors as a nonlinear Stochastic Differential Equation [START_REF] Rañó | A drift diffusion model of biological source seeking for mobile robots[END_REF], which allows us to simulate and obtain new theoretical results on the closedloop behaviour of this robot navigation controller. Like in the deterministic model, the navigation control function is modelled as a generic function of the sensor measurements, yet, in the stochastic case finding a controller to generate a stable closed-loop navigation behaviour is even more challenging. This paper uses Policy Search Reinforcement Learning to learn appropriate control functions, leading to stable closedloop behaviour, modelled as feedforward neural networks. Results show that the learnt neural controllers adapt to the stochasticity of the system trying to minimise the effects of noise. We rely on the stochastic model of Braitenberg vehicle 3a to analyse the resulting control functions approximated by the networks. Furthermore, we compare the performance of these controllers with a naïve linear controller, which has been shown to generate stable navigation in presence of sensor noise [START_REF] Rañó | A drift diffusion model of biological source seeking for mobile robots[END_REF].

Because of their flexibility, Braitenberg vehicles have been used for multiple robotic tasks like; finding sources of chemicals through mobile robots [START_REF] Lilienthal | Experimental analysis of smelling braitenberg vehicles[END_REF], navigating against a water current [START_REF] Salumäe | Against the flow: A braitenberg controller for a fish robot[END_REF], obstacle avoidance [START_REF] Bicho | The dynamic approach to autonomous robotics demonstrated on a low-level vehicle platform[END_REF], sound source localisation [START_REF] Shaikh | A Braitenberg lizard: Continuous phonotaxis with a lizard ear model[END_REF], and underwater navigation with electric sensors [START_REF] Lebastard | Underwater electro-navigation in the dark[END_REF]. All these examples used empirical tuning of the sensory-motor connection in the robots. However, as already stated some research works made use of learnt neural controllers to implement navigation behaviours. A recurrent neural network is used in [START_REF] Steingrube | Selforganized adaptation of a simple neural circuit enables complex robot behaviour[END_REF] using the qualitative principles of Braitenberg vehicles to control a legged robot. The navigation network is able to negotiate obstacles and interfaces the pre-processed sensory input with a Central Pattern Generator module that controls locomotion. Navigation towards a sound source (phonotaxis) is another existing application of Braitenberg vehicles implemented as neural controllers. In a series of works [START_REF] Webb | A Spiking Neuron Controller for Robot Phonotaxis[END_REF] [12] [START_REF] Reeve | New technologies for testing a model of cricket phonotaxis on an outdoor robot[END_REF] phonotaxis was implemented using spiking neural networks. The neural controllers replicate the motion of female crickets navigating towards the chirping males in wheeled robots. The robots were tested in both indoor and outdoor environments. Feedforward neural networks have been also used to learn, through evolutionary techniques, Braitenberg like controllers for sensor-based obstacle avoidance [START_REF] Mondada | Evolution of neural control structures: Some experiments on mobile robots[END_REF].

As we can see from the literature, neural controllers have been widely used to implement Braitenberg vehicles with desired closed loop dynamics of the robots. However, in the examples using spiking neural networks the neural controllers were hand tuned, and only in some cases were the controllers actually learnt through optimisation techniques, which are computationally costly. Reinforcement Learning (RL), meanwhile, can greatly reduce the time of learning compared to evolutionary techniques, enabling the learning agent the discovery of an optimal behaviour autonomously by trial-and-error according to a reward function [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF]. RL has been widely used in robotics because it provides solutions for hard to engineer and sophisticated problems of behaviour implementation that roboticists typically face. For example, Reinforcement Learning was used in several works to learn navigation strategies for mobile robots, both in simulated [START_REF] Weber | Robot docking with neural vision and reinforcement[END_REF] and on real robot [START_REF] Martinez-Marin | Fast reinforcement learning for vision-guided mobile robots[END_REF]- [START_REF] Zuo | A reinforcement learning based robotic navigation system[END_REF]. All of these works used Value Function Approaches to solve navigation problems, which are not well suited to deal with continuous state and action spaces. For instance, a change in the value function for some stateaction pair can alter the whole policy [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF], making these approaches prone to fail in highly stochastic problems. Recent research in RL focuses instead on Policy Search techniques which are better suited for continuous domains and stochastic environments.

Ng et al. [START_REF] Ng | Autonomous Inverted Helicopter Flight via Reinforcement Learning[END_REF] use a policy search method to learn a robust helicopter controller for autonomous inverted flight; a difficult problem as helicopters have high-dimensionality, are nonlinear, complex, and stochastic. The Pegasus policy search algorithm is used to learn a controller which can hover in place and execute a number of manoeuvres. In this work, a model of the helicopter is used for the learning. Although this speeds up the process of learning the controller, it can be dangerous because of modelling uncertainties, meaning the controller may not work when deployed on the real robot. State of the art work by [START_REF] Chebotar | Path integral guided policy search[END_REF] shows model-free policy search used to control a manipulator in a stochastic scenario: In a series of experiments, first a door position was varied in the x and y space in an environment and the robotic arm was able to consistently open the handle using monocular RGB images.

In a second experiment, the manipulator learns to pick up a bottle regardless of position in a 30cm × 40cm grid.

As we can see, multiple experimental works have shown that Braitenberg vehicles can perform target seeking behaviours with noisy sensors, yet none can ensure optimal or robust behaviour. Meanwhile, we have seen RL used to learn navigation controllers and robust controllers for stochastic systems. This work shows that RL can be used to learn a robust controller for a stochastic Braitenberg vehicle 3a. The learnt controller will be optimal with respect to the reward function, thus by carefully engineering the reward function for target seeking behaviour, we can take the first steps towards optimal controller design for a stochastic Braitenberg Vehicle 3a. The rest of the paper is organised as follows: Section II reviews the assumptions, the mathematical model of Braitenberg vehicle 3a as a stochastic differential equation, and how to use RL to learn the sensori-motor connection function. Section III presents results of the Stochastic Braitenberg Vehicle 3a with different amounts of noise on the sensors using a nonlinear connection function learnt using the Policy Gradient Reinforcement Learning algorithm. The learnt controllers are tested under noise and are compared against a hard-coded connection function. The paper concludes with a summary of the presented findings and some future work in Section IV.

II. A STOCHASTIC MODEL OF BRAITENBERG VEHICLE 3A

Before applying any Reinforcement Learning techniques, we will first briefly introduce the stochastic closed-loop model of Braitenberg vehicle 3a as drift-diffusion equations, already presented in [START_REF] Rañó | A drift diffusion model of biological source seeking for mobile robots[END_REF], which we will later use for our simulations. As seen in figure 1, vehicle 3a is a unicycle type robot with a wheelbase d and distance between the sensors δ. The two symmetrically arranged sensors have an inhibitory connection to the corresponding wheel motor on the same side of the vehicle as the sensor, and have a white noise signal following a Gaussian distribution in our sensor readings (ξ r , ξ l ), turning the standard model of vehicle 3a into a stochastic one. Since the sensori-motor connection is decreasing, illustrated using the '-' sign in the figure, the vehicle's wheels turn at a faster speed when low stimulus readings are recorded by the sensors, and as the sensed stimulus gets stronger closer to the origin, the wheel speeds slow down until they eventually stop moving when the vehicle has reached the source (which has the highest stimulus reading). This inhibitory connection also controls the steering of the vehicle. For example, if the stimulus source is to the left side of the vehicle, the intensity recorded by the left sensor will be higher than the intensity recorded by the right sensor. This results in the left wheel turning slower than the right one, hence the vehicle steers towards the stimulus source. [START_REF] Rañó | A steering taxis model and the qualitative analysis of its trajectories[END_REF] presents the mathematical model of this bio-inspired non-linear controller for unicycle type robots, although this work assumes noiseless sensors. [START_REF] Rañó | A drift diffusion model of biological source seeking for mobile robots[END_REF] however develops a stochastic closed-loop model of Braitenberg vehicle 3a using drift-diffusion equations which allows us to model sensor noise.

Let us assume in a domain D there is a non-negative smooth scalar stimulus S(x). The connection function between the wheels and sensors is a function of the stimulus, F (s), and since the connection for vehicle 3a should be decreasing, that means F (s) < 0 for all perceived values s of the stimulus. To model the sensor noise we define a random variable ξ with a variance smoothly dependent on the measured stimulus value as:

ξ = σ(s)dW t (1)
where σ(s) is the variance of the noise, describing how far from the true value of the measured stimulus the random perturbations can go, and W t is a Wiener process. The sensor noise has a zero mean Gaussian distribution at every point in time. Although the noise is different for each sensor, we will assume it has the same statistical properties. Under these assumptions, for a vehicle with the left and right sensors located at x l and x r respectively, the speed of the left and right wheels are:

v l = F (S(x l ) + σ(S(x l ))dW l t ) v r = F (S(x r ) + σ(S(x r ))dW r t ) (2)
where dW l t and dW r t are independent noise processes on the left and right sensors. Remembering that the distance between the sensors is δ, we can approximate v l and v r as a Taylor series around the middle point between the sensors x. We can use these approximations of the wheel velocities (v l and v r ) to obtain the forward speed (v) and turning rate (ω = θ) of the vehicle:

ω = - δ d ∇F (x) T êp - 1 d D 1 (x, θ)dW - t - 1 d D 2 (x, θ)dW + t v = F (x) + 1 2 D 1 (x, θ)dW + t + 1 2 D 2 (x, θ)dW - t ( 3
)
where d is the wheelbase of the vehicle, dW + t = dW l t + dW r t and dW - t = dW l t -dW r t are two stochastic processes combining the two sensors noise, and the diffusion terms D1(x, θ) and D2(x, θ) are:

D 1 (x, θ) = δ 2 4 F (x)σ(x)[∇S(x) • êp ] 2 + F (x)σ(x) D 2 (x, θ) = δ 2 ∇S(x) • êp [F (x)σ (x) + F (x)σ(x)] (4)
If we substitute equations (3) into the unicycle kinematic model we get the following closed-loop system of stochastic differential equations:

dx t = F (x t ) cos θ t dt + 1 2 D -(x t , θ t ) cos θ t dW r t + 1 2 D + (x t , θ t ) cos θ t dW l t dy t = F (x t ) sin θ t dt + 1 2 D -(x t , θ t ) sin θ t dW r t + 1 2 D + (x t , θ t ) sin θ t dW l t dθ t = - δ d ∇F (x t ).ê p (θ t )dt - 1 d D + (x t , θ t )dW l t + 1 d D -(x t , θ t )dW r t ( 5
)
where

D -(x t , θ t ) = D 1 (x t , θ t ) -D 2 (x t , θ t ) and D + (x t , θ t ) = D 1 (x t , θ t ) + D 2 (x t , θ t ).
Note we have eliminated the compound functions in the equations to simplify the notation, i.e. F (x t ) = F (S(x t )), F (x t ) = F (S(x t )), and so on. The diffusion terms in equations ( 4) rely on the noise variance σ(s), the connection function F (s), and each of their derivatives. They model non-additive noise since the terms multiplying the increments of the Wiener Processes dW l t and dW r t have a functional dependency on the state of the system, x t and θ t . Now that we have obtained the system of Stochastic Differential Equations needed to simulate our Stochastic Braitenberg Vehicle 3a, we can use Reinforcement Learning to learn the connection function, F (s), between the wheels and the noisy sensors.

A. The Reinforcement Learning Problem

The problem of learning the target seeking behaviour for a stochastic Braitenberg vehicle 3a given a stimulus function can be restated as finding a non-increasing function of the stimulus to compute the velocities of the vehicle's wheels. Given the stimulus S(x) defined in an environment D ⊆ 2 , and with some initial vehicle pose (x 0 , θ 0 ) ∈ D × S 1 the trajectory followed depends on the connection function F (s). Since the trajectory unfolds in time (x(t), θ(t)) but also depends on the initial pose and F (s) we will write (x(t, x 0 , θ 0 , F ), θ(t, x 0 , θ 0 , F )). We define a scalar reward function r(x, θ) for each pose of the vehicle to measure how good being at that state is. However, since the trajectory depends on the initial pose and the connecting function, we have to make the reward along a given trajectory a function of the initial pose and the connecting function too, that is r(x, θ) = r(t, x 0 , θ 0 , F ).

Since the problem we are trying to solve is a stochastic dynamical system (a partially observable Markov decision process), we need to choose a Reinforcement Learning methodology which best leans itself towards stochastic data. While this renders Value Function Approaches unsuitable, Policy Search Methods are highly effective when the policy is stochastic -the most common of which are Policy Gradient Methods. Policy Gradient Methods are a type of Reinforcement Learning technique which works by optimising parameterised policies w.r.t. the expected return using gradient ascent. We decided to use the Simultaneous Perturbation Stochastic Approximation Algorithm [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF] to estimate the gradient of the policy, and used Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], a state of the art algorithm for stochastic optimisation to tune the weights of the neural network that would be used to approximate the sensori-motor connection function F (s). This method would be used to learn a robust controller for our stochastic Braitenberg Vehicle 3a. We opted to use a Radial Basis Function Feed-Forward Neural Network. The final form of the RBF was as follows:

F (s) = N i=1 φ i (s)w i ( 6
)
where N is the number of basis functions, the radial basis functions φ i (s) were Gaussian kernel functions centred at fixed equidistant positions s i within the range of the stimulus (from s = 0 to s = S(0)), and w i is the weight of the i-th neuron. By denoting the weight vector of the RBF as Φ = (w i ) we can state the reinforcement learning problem as maximising the following total return:

R[Φ] = (x0,θ0)∈D×S 1 p(x 0 , θ 0 )dx 0 dθ 0 ∞ 0 r(t, x 0 , θ 0 , Φ)dt (7)
where p(x 0 , θ 0 ) is the probability of the initial conditions being x 0 , θ 0 and we need to integrate for all the initial conditions. In other words, we need to integrate for all the initial poses of the vehicle -the starting coordinates in the workspace x 0 and all orientations in the unit circle θ 0 ∈ S 1 -while also needing to integrate over the whole trajectory. It is impossible, however, to evaluate these integrals due to the total return function depending on the solution of the nonlinear dynamical system modelling the Braitenberg vehicle. Furthermore, in our case it is not feasible to integrate over the whole trajectory, so we changed the upper integration limit in equation ( 7) to a finite time t f for our simulations. To solve the problem of evaluating the total return we used the sampling trick -where we sampled the space using random initial poses of the vehicle -to estimate, through simulations, the value of the integral. By using the Simultaneous Perturbation Stochastic Approximation Algorithm (SPSA), we can estimate the expected return:

∆ Ri ≈ R(Φ i + ∆Φ i ) -R(Φ i -∆Φ i ) ( 8
)
where R is a function which generates an episode, returning the total return from equation ( 7), Φ i is the RBF weight vector, ∆Φ i is a d-dimensional vector of perturbations defined as

∆Φ i = C k • X ∼ ±1.Bernoulli( 1 2 ) and C k = C 0 /k γ ,
where k is the iteration and γ is a non-negative coefficient. From this, we can estimate the gradient of the total return using regression yielding:

∇ Φ R[Φ] = (∆Φ T ∆Φ) -1 ∆Φ T ∆R (9)
With the initial estimate of the gradient ∇ Φ R[Φ] obtained, we can use the Adam algorithm for Stochastic Gradient Descent on the return to tune the weight vector Φ of the RBF:

Φ k+1 = Φ k + α • mk /( vk + ) (10)
where mk and vk are bias-corrected first and second moment estimates of the gradient ∇ Φ R[Φ k ], tuned using the default settings from [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], α is the step size, and is a parameter whose value is 10 -8 . As we mentioned, the environment of the vehicle is the whole plane D = 2 , however we can simplify things in the domain of the integral in equation ( 7), i.e. the domain in which the initial conditions are selected for the sampling process by defining the environment as a square region around the origin

D = [X -3 , Y -3 ] × [X 3 , Y 3 ] where [X -3 , Y -3 ]
is the minimum starting coordinates and [X 3 , Y 3 ] is the maximum starting coordinates. To further simply the learning process we decided to select the initial angular directions of the vehicle to be pointing towards the source within a ±90 • range, i.e. θ 0 ∈ [θ t -π/2, θ t + π/2], where θ t = arctan y0 x0 -π.

III. SIMULATIONS

This section presents the results of learning a robust sensorimotor connection for a stochastic Braitenberg Vehicle 3a using the Reinforcement Learning methodology presented in Section II. In the experiments, the sensor noise variance is increased while the system tries to effectively attenuate for the stochastic environment. The results are compared against a naïve linear sensori-motor connection, previously used for Stochastic Braitenberg vehicle 3a control in [START_REF] Rañó | A drift diffusion model of biological source seeking for mobile robots[END_REF]. In the RL experiments, the RBF function approximator was initialised using 11 neurons, the perturbation size C 0 was initialised to 0.7, the maximum iterations was set to 500, and the number of randomly initialised episodes per iteration was 330. The parameters for the Adam optimiser followed the default settings in the documentation and were α = 0.001, β 1 = 0.9, and β 2 = 0.999. To simulate the Braitenberg Vehicle we integrate numerically the Stochastic Differential Equations presented in equations (3) using the Euler-Maruyama method with a fixed time step of h= 0.05 and a simulation time of t f = 8.

A. Defining the reward function

Instead of directly imposing conditions on the learning, RL uses an agent to preform trial and error interactions with its environment, learning via a reward signal which indicates whether the generated behaviour of the agent was desirable. However, defining a reward function which will generate the exact behaviour we want the agent to learn is often seen as a difficult to engineer problem. While there are multiple ways we could allocate reward to achieve the target seeking behaviour of a stochastic Braitenberg Vehicle 3a, we have to be careful that the reward function we design cannot be "gamed" by the agent, learning some function which maximises the return but does not produce the behaviour we want -a phenomenon known as reward hacking. To ensure we avoid reward hacking by the agent, we kept the reward function simple since it is noted that the more complicated the reward function, the higher the likelihood and severity of any hacking [START_REF] Amodei | Concrete problems in ai safety[END_REF]. Since our robot is biologically inspired, we wanted to base our reward function on how animals in their natural environment learn a navigation task i.e. based on the perceived strength of a stimulus using their receptor organs. The function we designed allocated reward directly corresponding to the strength of the perceived stimulus at the agents current location, and since the sensors are noisy the reward function would take the noisy observations as it's parameters. We summed these noisy readings recorded by the vehicle's two sensors and used this as the reward:

r = S l ξ l + S r ξ r (11)
where S l|r are the true sensor readings and, as mentioned previously, ξ l|r are white noise signals following a Gaussian distribution with a variance smoothly dependent on the measured stimulus value. To fulfil the criteria that the motion of the vehicle is never backwards, i.e. F : + ∪{0} → + ∪{0}, the reward function will only assign reward when the movement of the vehicle is in the forward direction.

B. Learning target seeking with a stochastic vehicle

Although our Braitenberg vehicle could be used to find the origin of any stimulus source, the most popular application is the phototaxis Braitenberg Vehicle i.e. using light sensors to find the source of a light stimulus. Given that a light source follows the inverse-square law, we can define the stimulus as a function of the position of the vehicle S(x) = g0 1+ηx T Ax where η = 0.25, g 0 = 4 and A is a 2 × 2 identity matrix defining the isotropic nature of the stimulus source. Since the goal of this work is target seeking, we want our vehicle to reach as close to the origin of the stimulus source as possible. Based on our definition of the light source, we know that in theory the limit of proximity to the source and thus the ideal stopping point of the vehicle is when the light sensors on the vehicle measure the maximum stimulus value, i.e. S(0) = g 0 , which is recorded when x = (0, 0). However, due to the stimulus being sampled by the sensors the vehicle can never reach this point. To calculate the real limit of proximity to the source we need to account for δ, the distance between the sensors, as the vehicles cannot get closer to the origin than this -thus the maximum "S" they can reach is S 1

√ 2 δ δ . Combining
δ with the stimulus function we get

S(x) = g 0 1 + ηx T + 1 √ 2 δ δ T Ax + 1 √ 2 δ δ (12)
where, in our case δ = 0.1 hence the largest stimulus value we can ever expect the vehicle to perceive is 3.99, instead of g 0 = 4.

For our first experiment, we want to check if we can effectively use Reinforcement Learning to train a neural controller when there is noise present in the learning process. We will use the Newton-Raphson method for obtaining the roots of functions to calculate the stimulus value at the equilibrium point of the learned functions. This will allow us to analyse how well our controllers preform as we increase the amount of noise on the sensors. Ten learning experiments were ran each time we increased the sensor noise variance (σ), from 0 to 0.1 in 0.005 increments. Although the noise is applied in the simulations used for the learning process, it should be emphasised that the results presented in this section are being evaluated in a noise-free environment so we can focus on the effectiveness of the algorithm at attenuating for noise. Fig. 2. Stimulus values at the RBF zero crossings as the variance of the sensor noise σ is increased from 0 -0.1 in 0.005 increments.

Figure 2 shows a plot of stimulus values at the vehicles equilibrium point (i.e. the RBF zero crossing) averaged over 10 trajectories, as a function of the sensor noise variance σ. The grey shaded area shows the maximum and minimum equilibrium points over the 10 trials, while the red line is the average. While S(δ) = 3.99 is the ideal RBF zero crossing, we can see from figure 2 that in the experiment where σ = 0 i.e. when there is no noise on the sensors, the equilibrium point is at S(0) = 3.87. We can therefore use this value as our new baseline to compare the effect sensor noise has on the learning process. As soon as we add noise to the sensors, the robot stops at an equilibrium point further from the source. Even at the lowest level of noise, σ = 0.005, the equilibrium point is S(0) = 3.65, a significant change from the no noise scenario. However, as the variance increases from σ = 0.005 to σ = 0.06, the equilibrium point remains largely unaffected displaying a mean of 3.6232 across the range with a standard deviation of 0.0304, showing that Reinforcement Learning is robust with noise present in the learning process. After this point, the vehicles equilibrium point starts increasing. The last experiment which shows an average equilibrium point below S(0) = 4 is when the sensor noise variance σ = 0.085, with variance σ = 0.09 experiments showing an average equilibrium point of S(0) = 4.0172.

Although we may think the Reinforcement Learning algorithm is no longer able to attenuate for the stochasticity in the system effectively once the sensor noise variance increases beyond σ = 0.085, later we will see that using the neural controller learnt even when σ = 0.1 is still very effective when it is used for stochastic vehicle control. Fig. 3. RBF network outputs defining the connection function Figure 3 shows a comparison between the trained RBF Neural Networks under no sensor noise (turquoise plot) and when the sensor noise variance σ is 0.08 (plum plot). Ten learnt networks are shown on each plot. These networks map the relationship between the wheel velocity and the strength of the stimulus. While the plots help visualise the equilibrium points presented in figure 2, what is interesting is that as the sensor noise variance increases, the standard deviation of the RBF networks remains the same i.e. the stability of the algorithm remains unaffected as noise is added to the system. All 10 trials in each case take positive values meaning the vehicles are moving forward, and have a negative slope which indicates the positive taxis behaviour of Braitenberg Vehicle 3a. The figures show that the slope of the RBF is much steeper when there is no noise in the system (turquoise plot) compared to when the RBF is trained under noise (plum plot), a behaviour expected as the neural network attenuates for noise. Notice that as the variance σ increases, as well as the steepness of the slope at the equilibrium point decreasing, the velocity of the vehicle at any of the sampled stimulus values is also smaller. This is because the weights of the network are tuned using the estimated gradient of the total return. In our simulations, if the angular direction of the vehicle was not pointing towards the source within a ±90 o range during any stage of the simulation, the simulation stopped. Therefore, once the noise in the system is increased, the heading angle is much more likely to deviate from this range during the simulation. This is confirmed by our results: in the deterministic case with no noise, there was a 60% success of full simulation, compared to 47% success of full simulation at σ = 0.005 (the lowest amount of sensor noise variance we applied to our system) and 15% success of full simulation when σ = 0.08. This leaves us with a smaller ∆R compared to the deterministic case, thus the weight updates are not as large and therefore not as large a velocity is learnt.

C. Testing the controller on a simulated stochastic vehicle

The previous section has shown that Reinforcement Learning can effectively learn a robust sensori-motor connection function for Braitenberg Vehicle 3a control, even under varying levels of sensor noise. However, we also need to test how effective the learnt neural controllers are at attenuating for stochasticity when the vehicle experiences sensor noise during its target seeking task, i.e. we need to ensure that the robot with noisy sensors can successfully preform target acquisition using the learnt connection functions. To give some comparison, we ran simulations of the stochastic Braitenberg vehicle under the same levels of noise using the learnt sensorimotor connection and a hard coded linear connection function, defined as F (s) = 4-s where s is the strength of the stimulus, so we can see if using RL improves Stochastic Braitenberg Vehicle performance. The plots in Figure 4 show an example of the time evolution of the mean of 4000 simulated trajectories using the naïve hard coded connection function (orange plot) and the RBF connection function learnt using our RL algorithm (blue plot).

Each simulation is ran for 20 seconds. The figure shows the results when when the sensor noise is following a normal distribution with a variance σ = 0.08. The red dashed line is our control experiment and represents the evolution obtained from the deterministic equations of the moments, i.e. how the vehicle performs in optimal conditions when the sensor noise variance is zero. In each simulation, we start the vehicle at x = (-2.8, 0), θ = 0, while the target is x = (0, 0), θ = 0. We can see from the orange plot which uses the hard coded connection function, that by introducing sensor noise there is a large effect on the trajectory of the vehicle. Not only does the x coordinate not reach close to zero (stops at x = -1.0129), the y coordinate and θ coordinate drift. Comparatively, the experiment which uses the RBF connection function performs much better when tested under the same level of sensor noise. Although the x coordinate does not reach zero (x = 0.0144), it still performs much better than the hard coded controller counterpart. Additionally, the y and θ coordinates, although slightly noisy over the entire evolution of the trajectory, do not drift. We can further analyse the effectiveness of using the RBF controller over a hard coded linear controller by evaluating the strength of the stimulus perceived by the robot at the end of the simulation, T = 20. Remembering that the maximum stimulus value that can be perceived is S(0) = 4, when we use the hard coded connection function, the robot stops moving at S(0) = 3.1666, while the connection function learned using RL stops at S(0) = 3.9992.

Table I documents the complete results of the above experiment for all tested variances in the sensor noise σ, from 0 to 0.1 in 0.01 increments. As above, each controller is evaluated 4000 times and the mean is taken. Then, using the coordinates of the vehicle when T = 20 we calculate the mean value of the perceived stimulus at the end of the target seeking task for each realisation of sensor noise. The results indicate that the hard coded naïve controller performs well up to σ = 0.05, however after this point the controller starts to significantly deteriorate. The RBF controller on the other hand maintains a consistent performance as σ is increased, even at σ = 0.1. Interestingly, compared to the first experiment (see figure 2), instead of overshooting the target and continuously roaming the environment when σ = 0.1, the vehicle now stops close to the source due to having noisy sensors.

As we have seen above, the neural controller preforms well and locates the stimulus source under different levels of sensor noise when the controller is evaluated over 4000 trajectories and the mean taken. However, this does not necessarily tell a fair story about the success rate of the controller in a real life navigation task since a real robot only takes one trajectory to attempt to locate the source of the target. Thus, in our final experiment, we seek to find out how the target seeking robot performs over individual trajectories when there is noise in the system. We tested both controllers under each level of noise as we increased the variance from 0 to 0.1 in 0.01 increments. We ran 50 simulations for each experiment and computed how many trajectories converged at the source of the stimulus. coordinates, x = (-2.8, 1.2), θ = 0, while the goal was at the origin, x = (0, 0). We classified a successful single trajectory as one which reached ±0.9 distance units from the source (i.e. the recorded stimulus value at the vehicles equilibrium point had to be at least 3.3) Table II documents the complete results of successful individual trajectories as the sensor noise variance is increased. Both controllers remain unaffected by the sensor noise in the initial experiments, each displaying a 100% success at target seeking until the variance reaches 0.03. At this point, the RBF controller manages 98% (one trajectory did not reach the source) while the naïve controller still achieves a 100% success. However, the experiments from σ = 0.04 until the maximum noise level, σ = 0.1, show that the naïve controller's performance is consistently diminishing until, at σ = 0.1, the controller can only converge to the stimulus 46% of the time. We compare these results to our neural controller which consistently is able to attenuate for the noise, and at σ = 0. Figure 5 illustrates the results: while Figure 5(a) shows the successful results of 50 trajectories using the naïve controller, Figure 5(b) shows the successful trajectories using the RBF controller, both when the variance of the sensor noise was σ = 0.1. Each simulation was initialised with the same starting coordinates of the vehicle, x = (-2.8, 1.2), θ = 0, marked on the figures using a blue triangle. The goal was x = (0, 0), marked with a red asterisk. The heatmap represents the stimulus strength at each point in the environment. From Figure 5(a), which uses the hand tuned controller, we can see that out of the 50 test trajectories, only 23 (46%) successfully converge at the stimulus source. We can also see that the trajectories are jagged as the naïve controller cannot attenuate for the noise in the closed-loop dynamical system. Meanwhile, Figure 5(b), which uses the neural controller, shows 44 (88%) successfully converged trajectories at the stimulus source. As well as almost doubling the successful number of trials, we can see from figure 5(b) that the trajectories are much more smooth as the robot successfully attenuates for the noise in its sensors and easily navigates to the source of the stimulus.

IV. CONCLUSIONS

This paper has presented the first implementation of Reinforcement Learning for a target seeking Stochastic Braitenberg Vehicle 3a. While previous empirical works use hand tuned parameters or Evolutionary strategies to optimise a fitness function on a real robot with potentially noisy sensors, this work uses Reinforcement Learning with a simulated robot to demonstrate that a Policy Gradient algorithm can be used with a RBF neural network to learn a robust connection function, even under white Gaussian noise. Our results show that the connection function can be learnt under varying amounts of noise without affecting the stability of the algorithm. Moreover, the connection function learnt using RL performs significantly better when tested under noise compared to the hard coded linear connection function, allowing the robot to reach much closer to the origin of the source before stopping. Finally, when simulated in a real life scenario, i.e. when we cannot take the average of the trajectories, the neural controller significantly outperforms the naïve controller. This is demonstrated in the results when measuring the successfully converged trajectories, where some levels of noise show a converged trajectory improvement of 42% using the learnt controller compared to the naïve hard coded controller.

Future work will focus on implementing the Reinforcement Learning algorithm onto a real robot for a target seeking problem. Eventually, this will enable real robots to be used more effectively in disaster zones where there is likely to be noise affecting the sensors, for example in nuclear disaster areas or earthquakes where there are large amounts of dust.

Fig. 1 .

 1 Fig. 1. A Stochastic Braitenberg Vehicle 3a.

Fig. 4 .

 4 Fig.4. The mean of 4000 trajectories evaluated using the naïve controller and RBF controller when σ = 0.08, compared against a noise-free trajectory.

  (a) Naïve controller preforming target seeking in real world scenario. 23/50 trajectories reach the source with jagged trajectories as the controller fails to attenuate for the sensor noise.(b) RBF controller preforming target seeking in real world scenario. 44/50 trajectories reach the source with very smooth trajectories as the controller successfully attenuates for the sensor noise.

Fig. 5 .

 5 Fig. 5. Simulated trajectories using each control mechanism, when sensor noise variance is σ = 0.1. Only successful trajectories are shown.

  Each vehicle stimulation started from the same

	Sigma Naïve Controller RBF Controller
	0	3.979	3.865
	0.01	3.980	3.631
	0.02	3.985	3.756
	0.03	3.991	3.716
	0.04	3.979	3.796
	0.05	3.991	3.788
	0.06	3.711	3.870
	0.07	3.647	3.992
	0.08	2.967	3.995
	0.09	2.821	3.885
	0.1	2.415	3.708

  1 shows a 88% successful convergence rate.

	Sigma Naïve Controller RBF Controller
	0	100%	100%
	0.01	100%	100%
	0.02	100%	100%
	0.03	100%	98%
	0.04	92%	96%
	0.05	90%	88%
	0.06	80%	84%
	0.07	68%	84%
	0.08	62%	90%
	0.09	54%	86%
	0.1	46%	88%

ACKNOWLEDGEMENTS

This work was partially supported by the Royal Society International Exchange Scheme under grant IE151293.