Matthieu Geist
email: matthieu.geist@supelec.fr

Olivier Pietquin
email: olivier.pietquin@supelec.fr

Parametric Value Function Approximation: a Unified View

Keywords: Reinforcement learning, value function approximation, survey

Reinforcement learning (RL) is a machine learning answer to the optimal control problem. It consists of learning an optimal control policy through interactions with the system to be controlled, the quality of this policy being quantified by the so-called value function. An important RL subtopic is to approximate this function when the system is too large for an exact representation. This survey reviews and unifies state of the art methods for parametric value function approximation by grouping them into three main categories: bootstrapping, residuals and projected fixed-point approaches. Related algorithms are derived by considering one of the associated cost functions and a specific way to minimize it, almost always a stochastic gradient descent or a recursive least-squares approach.

I. INTRODUCTION

Optimal control of stochastic dynamic systems is a trend of research with a long history. Several points of view can be adopted according to the information available on the system such as a model of the physics ruling the system (automation) or a stochastic model of its dynamic (dynamic programming). The machine learning response to this recurrent problem is the Reinforcement Learning (RL) paradigm [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF][2] [START_REF] Sigaud | Markov Decision Processes and Artificial Intelligence[END_REF].

The system to be controlled is usually modeled as a Markov Decision Process [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF] (MDP) {S, A, P, R, γ}. An MDP is made up of a set S of states (the different configurations of the system), a set A of actions (which cause a change of the system's state), a set P : s, a ∈ S × A → p(.|s, a) ∈ P(S) of Markovian transition probabilities (the probability to transit from one state to another under a given action), a bounded reward function R : s, a, s ∈ S ×A×S → r = R(s, a, s ) ∈ R associating a scalar to each transition and a discounting factor γ which decreases long-term rewards' influence. How the agent acts with the system is modeled by a so-called policy, π : s ∈ S → π(.|s) ∈ P(A), which associates to each state a probability distribution over actions. The quality of such a policy is quantified by a so-called value function, V π (s) = E[ ∞ i=0 γ i r i |s 0 = s, π], which associates to each state the expected cumulative discounted reward from starting in the considered state and then following the given policy. An optimal policy is one of those which maximize the associated value function for each state: π * ∈ argmax π V π .

Thanks to the Markovian property, value functions can be (more or less simply) computed using so-called Bellman equations. The value function of a given policy satisfies the (linear) Bellman evaluation equation and the optimal value function (which is linked to one of the optimal policies) satisfies the (nonlinear) Bellman optimality equation. Alternatively, these value functions can be seen as being the fixedpoints of associated Bellman operators (see Section II). These Bellman equations are very important for dynamic programming and reinforcement learning, as they allow computing the value function.

If the model (that is transition probabilities and the reward function) is known and if state and action spaces are small enough, the optimal policy can be computed using dynamic programming [START_REF] Bellman | Dynamic Programming[END_REF]. A first scheme, called policy iteration, consists in evaluating an initial policy (that is computing the associated value function using the linear Bellman evaluation equation) and then improving this policy, the new one being greedy respectively to the computed value function (it associates to each state the action which maximizes the expected cumulative reward obtained from starting in this state, applying this action and then following the initial policy). Evaluation and improvement are iterated until convergence (which occurs in a finite number of iterations). A second scheme, called value iteration, consists in computing directly the optimal value function (using the nonlinear Bellman optimality equation and an iterative scheme based on the fact that the value function is the unique fixed-point of the associated Bellman operator). The optimal policy is greedy respectively to the optimal value function. There is a third scheme, based on linear programming; however, it is not considered in this article.

RL aims at estimating the optimal policy without knowing the model and from interactions with the system. Value functions can no longer be computed exactly, they have to be estimated, which is the main scope of this paper. RL heavily relies on dynamic programming, in the sense that most of approaches are some sort of generalizations of value or policy iteration. A first problem is that computing a greedy policy (required for both schemes) from a value function requires the model to be known. The state-action value function (also named Q-function) alleviates this problem by providing an additional degree of freedom on the first action to be chosen:

Q π (s, a) = E[ ∞ i=0 γ i r i |s 0 = s, a 0 = a, π].
A greedy policy can thus be obtained by maximizing the Qfunction over actions, for any given state. The state-action value function generalizes the value function in the sense that

V π (s) = E a|s,π [Q π (s, a)].
Therefore, the rest of this paper focuses uniquely on the Q-function.

There are two main approaches to estimate an optimal policy. The first one, based on value iteration, consists in estimating directly the optimal state-action value function which is then used to derive an estimate of the optimal policy.

The second one, based on policy iteration, consists in mixing the estimation of the Q-function of the current policy (policy evaluation) with policy improvement in a generalized policy iteration scheme (generalized in the sense that evaluation and improvement processes interact, independently of the granularity and other details). This scheme presents many variations. Generally, the Q-function is not perfectly estimated when the improvement step occurs. Each change in the policy implies a change in the associated Q-function; therefore, the estimation process can be non-stationary. The policy can be derived from the estimated state-action value function (for example, using a Boltzmann distribution or angreedy policy): this implies an underlying dilemma between exploration and exploitation. The policy can also have its own representation, which leads to actor-critic architectures.

All these approaches share a common subproblem: estimating the (state-action) value function (of a given policy or the optimal one directly). The aim of this paper is to review the more classic related algorithms by adopting an unifying view (an extended and more detailed version being provided in [START_REF] Geist | A Brief Survey of Parametric Value Function Approximation[END_REF]). Before this, the underlying formalism is shortly presented.

II. PRELIMINARIES

Thanks to the Markovian property of transitions, the stateaction value function Q π of a given policy π satisfies the linear Bellman evaluation equation:

Q π (s, a) = E s ,a |s,a,π [R(s, a, s ) + γQ π (s , a )].

A Bellman evaluation operator related to the

Q-function, T π : Q ∈ R S×A → T π Q ∈ R S×A , can be defined: T π Q(s, a) = E s ,a |s,π [R(s, a, s ) + γQ(s , a )].
This operator is a contraction and Q π is its unique fixed-point:

Q π = T π Q π .
Using it is not practical, as it requires knowing the model. Therefore, a sampled Bellman evaluation is introduced. For a transition (s i , a i , s i+1 , a i+1 ), a i+1 being sampled according to π, and the associated reward r i , it is defined as:

T π Q(s i , a i ) = r i + γQ(s i+1 , a i+1 ).
The optimal state-action value function Q * satisfies the nonlinear Bellman optimality equation:

Q * (s, a) = E s |s,a [R(s, a, s ) + γ max a ∈A Q * (s , a )].
The associated Bellman optimality operator,

T * : Q ∈ R S×A → T * Q ∈ R S×A , is defined as: T * Q(s, a) = E s |s,a [R(s, a, s ) + γ max a ∈A Q(s , a )].
A more practical sampled Bellman optimality operator is also defined. Assume that a transition (s i , a i , s i+1 ) and associated reward r i are observed, it is given by:

T * Q(s i , a i ) = r i + γ max a∈A Q(s i+1 , a).
An important RL subtopic is to estimate the (state-action) value function of a given policy or directly the Q-function of the optimal policy from samples, that is observed trajectories of actual interactions. This article focuses on parametric approximation: the estimated state-action value function is of the form Qθ , where θ is the parameter vector; this estimate belongs to an hypothesis space H = { Qθ |θ ∈ R p } which specifies the architecture of the approximation. For example, if the state space is sufficiently small an exact tabular representation can be chosen for the value function. The estimate is thus of the form Qθ (s, a) = e T s,a θ with e s,a being a unitary

θ i = argmin ω∈R p i j=1 T Qξ (s j , a j ) -Qω (s j , a j ) 2 , (1)
and they differs on how ξ ∈ R p is instantiated. First, bootstrapping approaches (ξ = θ j-1 , see Equation (3) in Section III) consist in treating value function approximation as a supervised learning problem. As values are not directly observable, they are replaced by an estimate computed using a sampled Bellman operator (bootstrapping refers to replacing an unobserved value by an estimate). Second, residual approaches (ξ = ω, see Equation (4) in Section IV) consist in minimizing the square error between the (state-action) value function and its image through a Bellman operator. Practically, a sampled operator is used, which leads to biased estimates. Third, projected fixed-point approaches (ξ = θ i or ξ = θ i-1 depending on direct or iterated resolution, see Equations [START_REF] Bellman | Dynamic Programming[END_REF][START_REF] Geist | A Brief Survey of Parametric Value Function Approximation[END_REF] in Section V) minimize the squared error between the (state-action) value function and the projection of the image of this function under the (sampled) Bellman operator onto the hypothesis space (see Figure 1). Therefore, there are basically three cost function, which can be minimized either using a stochastic gradient descent or a recursive leastsquares approaches. All algorithms discussed in this survey enter in this categorization, as summarized in Table I. Moreover, we will show that most of them can be written in the form of a Widrow-Hoff update. Generally speaking, this means that parameters are updated proportionally to the prediction error made on the last observation. In the case of value function approximation, this prediction error takes the specific form of a so-called temporal difference error δ i , which is the difference between right and left sides of a Bellman equation when the sampled operator is considered:

δ i = T Qθi-1 (s i , a i ) -Qθi-1 (s i , a i ).
Therefore, most of presented algorithms can be written in the following form:

θ i = θ i-1 + K i δ i , (2)
where K i is an algorithm-dependent gain which indicates in which direction the parameters are corrected (the temporal difference error quantifying the magnitude of this correction). Notice that if this paper focuses on how to learn the (stateaction) value function from samples, it is not concerned with how these samples are generated. Otherwise speaking, the control problem is not addressed. Nevertheless, this does not mean that we restrict ourselves to the on-policy learning setting (consider the Bellman optimality operator). Extensions of the reviewed algorithms to eligibility traces (see Section VI for a definition) are only mentioned.

III. BOOTSTRAPPING APPROACHES

Bootstrapping approaches cast the (state-action) value function approximation into a supervised learning problem. The (state-action) value of interest is assumed to be observed, and corresponding theoretical cost functions are considered, given that the (state-action) value function of a given policy π is evaluated or that the optimal Q-function is directly estimated. The corresponding theoretical cost function is written generically as: J(θ) = Q-Qθ 2 . The corresponding empirical optimization problem for i observed state-action couples (s j , a j ) and associated unobserved Q-values q j is:

θ i = argmin ω∈R p i j=1 (q j -Qω (s j , a j )) 2 .
The principle of bootstrapping algorithms is to derive an online algorithm which minimizes the above cost function. Generally, such an algorithm can be written in the form of a Widrow-Hoff update which corrects the parameter vector in the direction provided by the algorithm-dependent gain K i and with a magnitude proportional to the prediction error:

θ i = θ i-1 + K i (q i - Qθi-1 (s i , a i )).
However, the Q-value q i is never observed. It is where the bootstrapping principle applies: the unobserved q i value is replaced by some estimate. Given that we are looking for a fixed-point of one of the Bellman operators, this estimate is provided by T Qθi-1 (s i , a i ):

θ i = θ i-1 + K i ( T Qθi-1 (s i , a i ) -Qθi-1 (s i , a i )).
Alternatively, this can be seen as solving the following optimization problem [START_REF] Choi | A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning[END_REF]:

θ i = argmin ω∈R p i j=1 T Qθj-1 (s j , a j ) -Qω (s j , a j ) 2 .
(

Practical algorithms are then derived given that this cost function is solved using a stochastic gradient descent or a recursive least-squares approach, and given what Bellman operator is considered.

A. Bootstrapped Stochastic Gradient Descent

Stochastic gradient descent consists in adjusting parameters by an amount proportional to an approximation of the gradient of the cost function, only evaluated on the last training example. Let α i be a learning rate satisfying the classical stochastic approximation criterion,

∞ i=1 α i = ∞ and ∞ i=1 α 2 i < ∞.
Applied to the bootstrapped cost-function (3), the stochastic gradient descent approach provides the following update rule [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] (assuming an initial parameter vector θ 0 ):

   K i = α i ∇ θi-1 Qθ (s i , a i ) θ i = θ i-1 + K i T Qθi-1 (s i , a i ) -Qθi-1 (s i , a i ) .
Here the gain is the gradient, and the prediction error δ i = T Qθi-1 (s i , a i )-Qθi-1 (s i , a i ) is the temporal difference error announced in Section II.

With the Bellman evaluation operator and if the Q-function is replaced by the value function, it is the TD algorithm with function approximation; with the Bellman evaluation operator, this provides the SARSA algorithm; with the Bellman optimality operator, this is the Q-learning algorithm [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Under some assumptions, notably a linear parameterization hypothesis, TD with function approximation can be shown to be convergent [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF]. However, this is no longer the case when it is combined with a nonlinear function approximator [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF]. Despite this, one of the important successes of reinforcement learning is based on TD with neural network-based function approximation [START_REF] Tesauro | Temporal Difference Learning and TD-Gammon[END_REF]. Convergence results holding for TD with function approximation apply rather directly to SARSA with function approximation. Under some strong assumptions, Qlearning with function approximation can be shown to be convergent too [START_REF] Melo | An analysis of reinforcement learning with function approximation[END_REF]. All these algorithms have been extended to eligibility traces [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. However, notice that extending the Q-learning algorithm to eligibility traces should be done with caution, because of its off-policy aspect. This problem of combining off-policy learning with eligibility traces is discussed further later.

B. Bootstrapped Recursive Least-Squares

The fixed-point Kalman Filter (FPKF) [START_REF] Choi | A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning[END_REF] also seeks at minimizing the empirical cost function [START_REF] Sigaud | Markov Decision Processes and Artificial Intelligence[END_REF]. However, the parameterization is assumed to be linear and a (recursive) least-squares approach is adopted instead of the stochastic gradient descent used for preceding algorithms. The considered hypothesis space is of the form

H = { Qθ : (s, a) ∈ S × A → φ(s, a) T θ ∈ R|θ ∈ R p },
where φ(s, a) is a feature vector (to be chosen beforehand). For a given stateaction couple (s j , a j ), φ(s j , a j ) is shortened as φ j . Thanks to linearity in parameters, cost function (3) is convex and has a unique minimum. This optimization problem can be solved analytically by zeroing the gradient respectively to ω: this is the principle of the least-squares method. Moreover, thanks to the Sherman-Morrison formula1 , this estimation process can be made recursive (therefore recursive least-squares). This provides the following algorithm (assuming an initial parameter vector θ 0 as well as the associated p × p matrix P 0 ):

       K i = Pi-1φi 1+φ T i Pi-1φi θ i = θ i-1 + K i T Qθi-1 (s i , a i ) -φ T i θ i-1 P i = P i-1 - Pi-1φiφ T i Pi-1 1+φ T i Pi-1φi
.

This update is still in the form of a Widrow-Hoff update, with the aforementioned temporal difference error and with a gain K i depending on the P i matrix. It can be shown to be convergent under some assumptions, for both sampled operators (evaluation and optimality) [START_REF] Choi | A Generalized Kalman Filter for Fixed Point Approximation and Efficient Temporal-Difference Learning[END_REF]. As far as we know, it has not been extended to eligibility traces.

IV. RESIDUAL APPROACHES Residual approaches aim at finding an approximation of the fixed-point of one of the Bellman operators by minimizing the distance between the (state-action) value function and its image through one of the Bellman operators. The associated theoretical cost function is:

J(θ) = Qθ -T Qθ 2 .
Practically, learning is done using samples and the Bellman operator is replaced by a sampled Bellman operator, the model (particularly transition probabilities) being unknown. The associated empirical cost function is therefore:

θ i = argmin ω∈R p i j=1 T Qω (s j , a j ) -Qω (s j , a j ) 2 (4)
A common drawback of all approaches aiming at minimizing this cost function is that they produce biased estimates of the (state-action) value function: minimizing the empirical cost function [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF] does not lead to minimize the theoretical cost function Qθ -T Qθ2 asymptotically. Basically, this is due to the fact that the expectation of a square is not the square of the expectation:

E[ Qθ -T Qθ 2 ] = Qθ -T Qθ 2 + Var( T Qθ ).
There is an unwanted variance term acting as a penalty factor which favorises smooth functions. If such penalties are commonly used for regularization, this one is harmful as it cannot be controlled [START_REF] Antos | Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path[END_REF]. All methods presented below can be modified so as to handle the problem; this will be shortly discussed. However, it is important to note that any algorithm aiming at minimizing this cost presents this bias problem.

A. Residual Stochastic Gradient Descent

The so-called residual algorithms (R-SGD for residual stochastic gradient descent) [START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF] minimize the empirical cost function (4) using a stochastic gradient descent. The corresponding update rule is therefore (assuming an initial θ 0 ):

K i = ∇ θi-1 Qθ (s i , a i ) -T Qθ (s i , a i ) θ i = θ i-1 + α i K i ( T Qθi-1 (s i , a i ) -Qθi-1 (s i , a i ))
.

Once again, we obtain a Widrow-Hoff update. A first problem arises when the sampled Bellman optimality operator is considered. In this case, the gradient of the max operator must be computed 2 : ∇ θi-1 (max a∈A Qθ (s i+1 , a)). Another problem is that these algorithms compute biased estimates of the (stateaction) value function, as explained above. In order to handle this problem, a double sampling scheme can be used [START_REF] Baird | Residual Algorithms: Reinforcement Learning with Function Approximation[END_REF]. Let us consider the Bellman evaluation operator. Two transitions are independently generated from the current state-action couple. One of them is used to compute the gain K i and the other one to compute the TD error δ i . These two transitions being sampled independently, taking the expectation of K i δ i leads to the use of the true (that is unsampled) Bellman operator, without variance term contrary to the use of the same transition in both gain and TD error. However, this suggests that transitions can be sampled on demand (for example using a simulator), which can be a strong assumption. As far as we know, this algorithm has not been extended to eligibility traces.

B. Residual Least-Squares

The Gaussian Process Temporal Differences (GPTD) framework [START_REF] Engel | Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning[END_REF] models the value function as a Gaussian process [START_REF] Rassmussen | Gaussian Processes for Machine Learning[END_REF]. A generative model linking rewards to values through the sampled Bellman evaluation operator and an additive noise is set, the Gaussian distribution of a state's value conditioned on past observed rewards is computed by performing Bayesian inference, and the value of this state is estimated as the mean of this Gaussian distribution. The associated variance quantifies the uncertainty of this estimate. Notice that the optimality operator cannot be considered in this framework because of a mandatory linearity assumption (linearity of the generative model linking). A problem is that the considered Gaussian process is actually a vector with as many components as the number of states encountered during learning. To alleviate this scaling problem, an online sparsification scheme is used [START_REF] Engel | The Kernel Recursive Least Squares Algorithm[END_REF], which actually constructs online a kernel-based linear parametric representation. If sparsification is done in a preprocessing step or if the representation is considered asymptotically (after an infinite number of interactions), the GPTD value function representation can be seen as a parametric one. Moreover, a parametric version of the GPTD framework exists [START_REF] Engel | Algorithms and Representations for Reinforcement Learning[END_REF]. It is the view adopted here: feature selection is an important topic, however this paper focuses on learning the parameters of a representation, not on learning the representation itself.

The (parametric) GPTD algorithm assumes a linear parameterization and the Bellman evaluation operator and minimizes cost function (4) using a classical recursive linear least-squares approach 3 . Using the same notations as before, Equation ( 4) can be rewritten as θ i = argmin ω∈R p i j=1 (r j + γφ T j+1 ω -φ T j ω) 2 . Let note ∆φ j = φ j -γφ j+1 . The unique parameter vector minimizing the above convex cost can be computed analytically and recursively (thanks to Sherman-Morrison), which provides the GPTD update (assuming initial θ 0 and P 0 ):

     K i = Pi-1∆φi 1+∆φ T i Pi-1∆φi θ i = θ i-1 + K i r i -∆φ T i θ i-1 P i = P i-1 - Pi-1∆φi∆φ T i Pi-1 Pn i +∆φ T i Pi-1∆φi
One can recognize the temporal difference error δ i = r i -∆φ T i θ i-1 and a gain K i , to be linked again to the discussed Widrow-Hoff update. Notice that P i is actually a variance matrix quantifying the uncertainty over current parameters estimation (it is the variance of the parameter vector conditioned on past i observed rewards). This is not clear from the proposed least-squares-based derivation, however it is direct by adopting a Bayesian (or even pure Kalmanian) perspective: under linear and Gaussian assumptions, recursive least-squares, Kalman filtering and Bayesian filtering provide all the same solution [START_REF] Chen | Bayesian Filtering : From Kalman Filters to Particle Filters, and Beyond[END_REF]. As all other residual methods, GPTD produces biased estimates of the value function when transitions are stochastic. Using a proper colored noise model (instead of the classical white noise assumption) leads to an unbiased estimate [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF]. The corresponding algorithm is named MC-GPTD. However, this noise also induces a memory effect which prevents from learning in an off-policy manner, much like eligibility traces does [START_REF] Precup | Eligibility Traces for Off-Policy Policy Evaluation[END_REF]. Notice also that this leads to minimize another cost function, linking states' estimates to Monte Carlo samples of the discounted return [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF].

The so-called Kalman Temporal Differences (KTD) framework [START_REF] Geist | Kalman Temporal Differences: the deterministic case[END_REF][21] generalizes the GPTD framework: it handles nonlinear parameterizations (e.g., multilayered perceptron [START_REF]Eligibility Traces through Colored Noises[END_REF]), it handles the Bellman optimality operator (which provide a sort of "Q-learning" extension of GPTD), and it allows learning in non-stationary conditions (the system to be controlled can be non-stationary, but also generalized policy iteration induces a non-stationary followed policy [START_REF] Phua | Tracking value function dynamics to improve reinforcement learning with piecewise linear function approximation[END_REF] [START_REF] Geist | Tracking in Reinforcement Learning[END_REF]). The idea behind the original derivation of KTD is to cast (state-action) value function approximation into the Kalman filtering paradigm [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF], as developed in [START_REF] Geist | Kalman Temporal Differences[END_REF]. However, unscented Kalman filtering can be seen as a statistically linearized recursive least-squares approach [START_REF] Geist | Statistically Linearized Recursive Least Squares[END_REF]. Therefore, KTD can be seen as a second-order algorithm minimizing cost function (4) using a derivative-free statistical linearization. The full derivation of KTD using this recursive leastsquares point of view follows ideas of [START_REF] Geist | Statistically Linearized Recursive Least Squares[END_REF] and is provided in [START_REF] Geist | A Brief Survey of Parametric Value Function Approximation[END_REF]. As other residual approaches, KTD suffers from the bias problem when system transitions are stochastic. In order to handle this issue, a new colored noise model, based on the idea of eligibility traces and generalizing the GPTD colored noise [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF], has been introduced [START_REF]Eligibility Traces through Colored Noises[END_REF]. This leads to the KTD(λ) family of algorithms; as far as we know, it is the only residual approach which makes use of eligibility traces (there is a GPTD(λ) algorithm [START_REF] Engel | Algorithms and Representations for Reinforcement Learning[END_REF], however it should be linked to projected fixed-point approaches, not to residual ones). Nevertheless, as mentioned before, this induces some memory effects which prevent from learning in an off-policy manner. Consequently, the sampled Bellman optimality operator can no longer be considered in this setting, because of its offpolicy aspect. Using a colored noise also leads to minimize a slightly different cost function [START_REF]Eligibility Traces through Colored Noises[END_REF]. Notice also that the available uncertainty information (the P i matrix) can be useful for the dilemma between exploration and exploitation [START_REF]Managing Uncertainty within Value Function Approximation in Reinforcement Learning[END_REF].

V. PROJECTED FIXED-POINT APPROACHES Projected fixed-point approaches seek at minimizing the distance between the estimated state-action value function Qθ and the projection ΠT Qθ (the projection being noted Π) of the image T Qθ of this function under a Bellman operator onto the hypothesis space H: J(θ) = Qθ -ΠT Qθ 2 with Πf = argmin f ∈H f -f 2 . This is illustrated in Figure 1. The state-action value function estimate Qθ lies in the hypothesis space H. Its image under a Bellman operator T Qθ does not necessarily lies on this hypothesis space. Residual approaches of Section IV try to minimize the distance between these two functions, that is line 1 in Figure 1, with the drawback that using a sampled Bellman operator leads to biased estimates, as discussed before. The function T Qθ can be projected onto the hypothesis space, this projection minimizing the distance between T Qθ and the hypothesis space (line 2 in Figure 1). Projected fixed-point methods aim at minimizing the distance between this projection and Qθ , represented by line 3 in Figure 1. The related empirical cost function is θ i = argmin θ∈R p i j=1 ( Qθ (s j , a j ) -Qω θ (s j , a j )) 2 , with ω θ = argmin ω∈R p i j=1 ( Qω (s j , a j ) -T Qθ (s j , a j )) 2 . Obviously, this cost is minimized for θ i = ω θi (admitting that this equation has a solution), which summarizes the two nested optimization problems:

θ i = argmin ω∈R p i j=1
Qω (s j , a j ) -T Qθi (s j , a j ) 2 [START_REF] Bellman | Dynamic Programming[END_REF] Notice that as θ i appears in both sides of this equation, this is not a pure quadratic cost function. Least-squares methods are presented here before stochastic-gradient-based approaches for historical reasons.

A. Least-Squares-based Approaches

The least-squares temporal differences (LSTD) algorithm [START_REF] Bradtke | Linear Least-Squares algorithms for temporal difference learning[END_REF] assumes a linear parameterization and the (sampled) Bellman evaluation operator in order to solve the above optimization problem. It has been originally introduced as the minimization of a residual cost function. Using a sampled Bellman operator can be interpreted as a correlation between the noise and inputs in the corresponding observation model (therefore the noise is not white, which is a mandatory assumption for least-squares). This correlation can be shown to cause a bias (in this case, the bias presented in Section IV). A classic method to cope with this problem are instrumental variables [START_REF] Söderström | Instrumental variable methods for system identification[END_REF], which were used to provide the first derivation of the LSTD algorithm. This point of view is historical. Later, it has been interpreted as a projected fixed-point minimization [START_REF] Lagoudakis | Least-squares policy iteration[END_REF], and it is the point of view adopted here.

LSTD assumes a linear parameterization as well as the sampled Bellman evaluation operator. Using the same notations as before, optimization problem (5) can be rewritten as:

θ i = argmin ω∈R p i j=1 (r j + γφ T j+1 θ i -φ T j ω) 2 .
Thanks to linearity in ω (linear parameterization assumption), this can be analytically solved. Moreover, the update can be done recursively thanks to the Sherman-Morrison formula. This provides the LSTD algorithm (assuming initial θ 0 and p × p M 0 matrix):

     K i = Mi-1φi 1+∆φ T i Mi-1φi θ i = θ i-1 + K i r i -(∆φ i ) T θ i-1 M i = M i-1 -K i M T i-1 (φ i -γφ i+1 ) T
Once again, K i is a gain and r i -(∆φ i ) T θ i-1 a temporal difference error, to be linked to a Widrow-Hoff update. This algorithm has been extended to eligibility traces [START_REF] Boyan | Technical Update: Least-Squares Temporal Difference Learning[END_REF] and can be shown to be convergent under general assumptions [START_REF] Yu | Convergence of Least Squares Temporal Difference Methods Under General Conditions[END_REF].

The statistically linearized LSTD (slLSTD) algorithm [START_REF] Geist | Statistically Linearized Least-Squares Temporal Differences[END_REF] generalizes LSTD: it does not assume a linear parameterization nor the Bellman evaluation operator. Notably, this allows using nonlinear parameterization such as multilayered perceptrons and it provides a "Q-learning extension" of the LSTD algorithm. How slLSTD generalizes LSTD is very close to how KTD generalizes GPTD: a statistical linearization is performed, which allows solving the related optimization problem analytically. See [START_REF] Geist | Statistically Linearized Least-Squares Temporal Differences[END_REF][6] for details.

B. Stochastic Gradient Descent-based Approaches

GTD2 and TDC algorithms [START_REF] Sutton | Fast gradient-descent methods for temporal-difference learning with linear function approximation[END_REF] aim at minimizing cost (5) while considering the Bellman evaluation operator, and they differs on the route taken to express the gradient followed to perform the stochastic gradient descent. Both methods rely on a linear parameterization, and are based on a reworked expression of the cost function. Let δ j (θ) = r j + γφ T j+1 θ -φ T j θ be the temporal difference error, Equation (5) can be rewritten as

θ i = argmin ω∈R p J i (ω) with J i (ω) = ( i j=1 φ j δ j (ω)) T ( i j=1 φ j φ T j ) -1 ( i j=1 φ j δ j (ω)
). This result is far from being direct, see [START_REF] Sutton | Fast gradient-descent methods for temporal-difference learning with linear function approximation[END_REF][6] for details. The negative gradient of this cost is given by: -

1 2 ∇ θ J i (θ) = ( i j=1 ∆φ j φ T j )w i with w i = ( i j=1 φ j φ T j ) -1 ( i j=1 δ j (θ)φ j ).
In order to avoid a bias problem, a second modifiable parameter vector ω ∈ R p is used to form a quasi-stationary estimate of the term

w i = ( i j=1 φ j φ T j ) -1 ( i j=1 δ j (θ)φ j ),
this being called the weight-doubling trick. Parameter vector θ is updated according to a stochastic gradient descent:

θ i = θ i-1 + α i ∆φ i φ T i ω i-1 .
There remains to find an update rule for ω i . In order to obtain a O(p) algorithm, it is estimated using a stochastic gradient descent too [START_REF] Sutton | Fast gradient-descent methods for temporal-difference learning with linear function approximation[END_REF]. One can remark that ω i is actually the solution of a linear least-squares optimization problem:

w i = argmin ω∈R p i j=1 φ T j ω -δ j (θ)
2 . This suggests the following update rule:

ω i = ω i-1 + β i φ i (δ i (θ i-1 ) - φ T i ω i-1 ).
Learning rates satisfy the classical stochastic approximation criterion. Moreover, they are chosen such that β i = ηα i with η > 0. The GTD2 algorithm is thus (assuming an initial θ 0 , ω 0 being set to zero):

θ i = θ i-1 + α i ∆φ i φ T i ω i-1 ω i = ω i-1 + β i φ i δ i (θ i-1 ) -φ T i ω i-1
Contrary to previous algorithms, the GTD update is not in the form of Equation ( 2). However, one can recognize from this update rule that w T φ(s, a) is actually a parametrization of the advantage function A π (s, a) = Q π (s, a) -V π (s) (the temporal difference error being a bootstrapped estimate of the advantage [START_REF] Bhatnagar | Incremental natural actor-critic algorithms[END_REF]). Under some assumptions, this algorithm can be shown to be convergent [START_REF] Sutton | Fast gradient-descent methods for temporal-difference learning with linear function approximation[END_REF].

By expressing the gradient in a slightly different way, another algorithm called TDC can be derived, the difference being how the θ parameter vector is updated. The gradient can be rewritten as:

-1 2 ∇ θ J i (θ) = ( i j=1 δ j (θ)φ j ) - γ( i j=1 φ j+1 φ T j )w i .
This gives rise to the following update for θ, ω being updated as before:

θ i = θ i-1 + α i φ i δ i (θ i-1 ) - α i γφ i+1 φ T i ω i-1 .
This algorithm is called TD with gradient correction because the first term, α i φ i δ i (θ i-1 ), is the same as for TD with function approximation (see Section III-A), and the second term, -α i γφ i+1 φ T i ω i-1 , acts as a correction. For TDC, learning rates α i and β i are chosen such as satisfying the classic stochastic approximation criterion, and such that lim i→∞ αi βi = 0. This means that θ i is updated on a slower time-scale. The idea behind this is that ω i should look stationary from the θ i point of view. The TDC algorithm can be summarized as follows (assuming an initial θ 0 , ω 0 being set to zero):

θ i = θ i-1 + α i φ i δ i (θ i-1 ) -α i γφ i+1 φ T i ω i-1 ω i = ω i-1 + β i φ i (δ i (θ i-1 ) -φ T i ω i-1 )
As GTD, TDC is not in the form of Equation ( 2), but ω is still a parameter vector to be linked to the advantage function. This algorithm can also be shown to be convergent under some assumptions, see [START_REF] Sutton | Fast gradient-descent methods for temporal-difference learning with linear function approximation[END_REF] again.

GTD2 and TDC algorithms have been extended to the case of a general nonlinear parameterization Qθ , as long as it is differentiable respectively to θ [START_REF] Maei | Convergent temporal-difference learning with arbitrary smooth function approximation[END_REF]. The underlying idea is to perform a projection onto the tangent plane of the hypothesis space. The TDC algorithm has also been extended to eligibility traces [START_REF] Maei | GQ(λ): a general gradient algorithm for temporal-differences prediction learning with eligibility traces[END_REF]. Moreover, this algorithm, called GQ(λ), allows off-policy learning, that is learning the value of one target policy while following another behaviorial policy. This new algorithm (for which some convergence guarantees can be provided) still minimizes the empirical cost function linked to [START_REF] Bellman | Dynamic Programming[END_REF]. However, instead of the T π Bellman operator considered so far, an eligibility-based T π λ operator is used (λ being the eligibility factor), this operator being defined as the expected λ-return. See [START_REF] Maei | GQ(λ): a general gradient algorithm for temporal-differences prediction learning with eligibility traces[END_REF] for more details. Using eligibility traces induces a memory effect which prevents from learning in an off-policy manner without caution, see [START_REF] Precup | Eligibility Traces for Off-Policy Policy Evaluation[END_REF] for example. To cope with this problem, [START_REF] Maei | GQ(λ): a general gradient algorithm for temporal-differences prediction learning with eligibility traces[END_REF] use ideas from the importance sampling field (as [START_REF] Precup | Eligibility Traces for Off-Policy Policy Evaluation[END_REF] actually). They present GQ(λ) as an extension of Q-learning, which can be misleading. Actually, they consider off-policy learning (with a known, fixed, target policy), but not the Bellman optimality operator. Nevertheless, the TDC algorithm has also been extended to this operator [START_REF] Maei | Toward Off-Policy Learning Control with Function Approximation[END_REF] (this new algorithm being called Greedy-GQ). To do so, the Bellman evaluation operator T π θ for a policy π θ , which depends on the currently estimated state-action value function (through parameters θ), is considered. Therefore, the learnt policy is non-stationary (it evolves with parameters' estimation). If π θ is greedy respectively to the learnt value function, then it is equivalent to considering the Bellman optimality operator. However, in this case, there are some non-differentiability problems (due to the max operator), and Fréchet subgradients are required [START_REF] Maei | Toward Off-Policy Learning Control with Function Approximation[END_REF]. They also consider the case where π θ is a stochastic Gibbs policy built upon the currently estimated value function. In this case, there are no differentiability problems. There is a convergence analysis for these algorithms [START_REF] Maei | Toward Off-Policy Learning Control with Function Approximation[END_REF].

C. Iterated-Solving-based Approaches

Methods presented so far in Section V aim at minimizing the distance between the state-action value function and the projection of the image of this function under a Bellman operator onto the hypothesis space: J(θ) = Qθ -ΠT Qθ 2 . This can be interpreted as trying to find a fixed-point of the operator ΠT , which is the composition of the projection operator Π and of one of the Bellman operator T . Assuming that this operator is a contraction (which is not always the case, it depends on the projection operator), there exists a unique fixed-point which can be found by iterating the application of the ΠT operator: Qθi = ΠT Qθi-1 . Methods presented in this section adopt this point of view to provide algorithms.

Fitted-Q is a batch algorithm (therefore, it cannot be linked to the Widrow-Hoff update). It assumes that a set of N transitions is available beforehand: {(s j , a j , r j , s j+1 , a j+1 )} 1≤j≤N . An initial Q-function Qθ0 is (more or less arbitrary) initialized, and estimates are refined by iterating the ΠT operator: Qθi = Π T Qθi-1 , ∀i > 0. A sampled Bellman operator is used, because as usual transition probabilities are not known. Most of time, fitted-Q suggests the sampled Bellman optimality operator, but this approach is of course still valid for the sampled Bellman evaluation operator. The Π operator indeed represents any supervised learning algorithm matching inputs (s j , a j ) to pseudooutputs T Qθi-1 (s j , a j ). Notice that the representation of the estimated state-action value function is not necessarily parametric (e.g., if a support vector machine is used as the supervised learning algorithm). The fitted-Q idea probably dates back to [START_REF] Samuel | Some studies in machine learning using the game of checkers[END_REF]. Its convergence properties have been analyzed in [START_REF] Gordon | Stable Function Approximation in Dynamic Programming[END_REF] through the contraction property of the ΠT operator. Its performance bounds in L p norm has been analyzed in [START_REF] Munos | Performance Bounds in Lp norm for Approximate Value Iteration[END_REF]. This is particularly judicious: if performance bounds of supervised learning algorithms are very often analyzed in L p norm, this is not the case for (approximate) dynamic programming which is most of the time analyzed in L ∞ norm. Fitted-Q has been considered with many different function approximators: kernel-based regression [START_REF] Ormoneit | Kernel-Based Reinforcement Learning[END_REF], neural networks [START_REF] Riedmiller | Neural Fitted Q Iteration -First Experiences with a Data Efficient Neural Reinforcement Learning Method[END_REF], tree-based methods [START_REF] Ernst | Tree-Based Batch Mode Reinforcement Learning[END_REF], etc.

The least-squares policy evaluation (LSPE) algorithm has been directly introduced using the concept of eligibility traces [START_REF] Nedić | Least Squares Policy Evaluation Algorithms with Linear Function Approximation[END_REF], but this aspect is left apart in this article. It can be roughly seen as a fitted-Q algorithm using a linear parameterization, the (sampled) Bellman evaluation operator and for which a new training sample is added to the training set at each iteration. Thanks to linearity (linear parameterization and evaluation operator), an efficient online algorithm can be obtained.

The LSPE algorithm solves the following optimization problem:

θ i = argmin ω∈R p i j=1 T Qθi-1 (s j , a j ) -Qω (s j , a j ) 2 . ( 6
)
Thanks to the linearity (linear parameterization and Bellman evaluation operator), and using the Sherman-Morrison formula, a recursive estimate can be provided (assuming initial θ 0 and B 0 , A 0 and b 0 being set to 0):

         B -1 i = B -1 i-1 - B -1 i-1 φiφ T i B -1 i-1 1+φ T i B -1 i-1 φi A i = A i-1 + φ i (∆φ i ) T b i = b i-1 + φ i r i θ i = θ i-1 + B -1 i (b i -A i θ i-1 )
Actually, the way LSPE is presented here differs from [START_REF] Nedić | Least Squares Policy Evaluation Algorithms with Linear Function Approximation[END_REF] in the sense that the B -1 i matrix is originally scaled with a learning rate. The algorithm presented here is the case where the learning rate is chosen constant and equal to one, which can be shown to be convergent [START_REF] Bertsekas | Learning and Approximate Dynamic Programming[END_REF]. This Widrow-Hoff update is not of the form of Eq. 2, but one can remark that the term b i -A i θ i-1 = i j=1 φ j δ j (θ i-1 ) implies the temporal differences errors of all transitions encountered so far. Notice that ideas behind LSPE have other applications [START_REF] Bertsekas | Projected Equation Methods for Approximate Solution of Large Linear Systems[END_REF] and can also be linked to variational inequalities [START_REF] Bertsekas | Projected Equations, Variational Inequalities, and Temporal Difference Methods[END_REF].

The LSPE algorithm has been extended to the Bellman optimality operator in the case of optimal stopping problems [START_REF] Yu | Q-Learning Algorithms for Optimal Stopping Based on Least Squares[END_REF], a restrictrive class of MDPs. Convergence of the resulting Q-OSP algorithm can still be ensure, however obtaining a computationally efficient recursive estimate requires some approximations.

VI. CONCLUSION This article has reviewed a large part of the state of the art in (state-action) value function approximation. Basically, it has been shown that all these approaches can be categorized in three main classes, given the considered cost function (related to bootstrapping, residual or projected fixed-point). In each of these groups, they can be categorized given that the cost function is minimized using a stochastic gradient descent or a recursive least-squares approach (except fitted-Q, which can be considered with any supervised learning algorithm). Projected fixed-point approaches can be divided into two approaches, given that the cost function is directly minimized or that the underlying possible fixed-point is searched for using an iterative scheme. All of this is summarized in Table I. A link between the Widrow-Hoff update and most of the reviewed methods has also be drawn through this article.

A point not discussed so far is the computational (and memory) complexity of the reviewed algorithms. There are basically two cases. For stochastic gradient descent-based algorithms, complexity is in O(p), and for recursive leastsquares-based approach, complexity is in O(p 2 ). Notice that for algorithms handling the Bellman optimality operator, these complexities have to be scaled by the number of actions (because of the max computation). Exceptions (to the two above cases) are fitted-Q (which complexity depends on the considered supervised learning scheme) and Q-OSP. Considering the sample efficiency of these algorithms, second order approaches are usually more efficient.

All algorithms for value function approximation have not been discussed here. Notably, most of the presented ones have been extended to eligibility traces. Basically, methods presented here are based on a one-step prediction (the reward plus the estimate of the value function in the transition state), whereas eligibility traces are based on a weighted average of multiple step predictions [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. Such an approach as a clear advantage for methods based on stochastic gradient descent, as it speeds up the learning. However, for least-squares-based approaches, this advantage is less clear, notably because these methods are generally much more sample efficient [START_REF] Boyan | Technical Update: Least-Squares Temporal Difference Learning[END_REF]. Moreover, eligibility traces present a (possibly severe) drawback: they induces a memory effect which prevents off-policy learning without caution. Off-policy learning is actually still possible, but it implies to add some importance sampling scheme [START_REF] Precup | Eligibility Traces for Off-Policy Policy Evaluation[END_REF]. By the way, using eligibility traces can be expressed as using a modified T λ Bellman operator (see [START_REF] Maei | GQ(λ): a general gradient algorithm for temporal-differences prediction learning with eligibility traces[END_REF] for example), and the work presented here can globally be extended to this case.

Other approaches extend algorithms presented here. For example, LSTD has been kernelized [START_REF] Xu | Kernel-Based Least Squares Policy Iteration for Reinforcement Learning[END_REF] as well as LSPE [START_REF] Jung | Kernelizing LSPE(λ)[END_REF]. A variation of LSTD has been proposed by [START_REF] Geramifard | Incremental Least-Squares Temporal Difference Learning[END_REF] who use the sparsity of used feature vectors to reduce the computational complexity. The LSTD algorithm has also been extended to L 2 regularization in reproducing kernel Hilbert spaces [START_REF] Farahmand | Regularized policy iteration[END_REF] as well as to L 1 regularization [START_REF] Kolter | Regularization and Feature Selection in Least-Squares Temporal Difference Learning[END_REF]. These are examples, among other.

Fig. 1 .

 1 Fig. 1. Projected fixed-point principle.

It allows inverting a rank-one perturbed matrix efficiently.

Actually, max a∈A Qθ (s i+1 , a) is generally non-differentiable respectively to θ. A solution could be to rely on Fréchet sub-gradients.

This derivation differs from[START_REF] Engel | Bayes Meets Bellman: The Gaussian Process Approach to Temporal Difference Learning[END_REF], its validity relies strongly on the link between Bayesian inference, Kalman filtering and recursive least-squares under Gaussian and linear assumptions[START_REF] Chen | Bayesian Filtering : From Kalman Filters to Particle Filters, and Beyond[END_REF].


Jeremy Fix
email: jeremy.fix@supelec.fr

Matthieu Geist

Olivier Pietquin

Hervé Frezza-Buet

Dynamic Neural Field Optimization using the Unscented Kalman Filter

Dynamic neural fields have been proposed as a continuous model of a neural tissue. When dynamic neural fields are used in practical applications, the tuning of their parameters is a challenging issue that most of the time relies on expert knowledge on the influence of each parameter. The methods that have been proposed so far for automatically tuning these parameters rely either on genetic algorithms or on gradient descent. The second category of methods requires to explicitly compute the gradient of a cost function which is not always possible or at least difficult and costly. Here we propose to use unscented Kalman filters, a derivative-free algorithm for parameter estimation, which reveals to efficiently optimize the parameters of a dynamic neural field.

I. INTRODUCTION

Dynamic neural fields have been proposed as a continuous model of a neural tissue. They are used in computational neuroscience to model biological functions of the primate brain but also as a computational paradigm to control artificial agents. In both contexts, tuning the parameters of a neural field is a challenging issue that most of the time relies on the knowledge of an expert and on trials and errors. The main difficulty behind tuning these parameters relies on the difficulty to obtain analytical results on the evolution of such a high-dimensional non-linear dynamical system. The first property observed on dynamic neural fields is the existence of stable localized patterns of activity [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF]. Dynamical patterns such as traveling waves have also been obtained. These two properties are understood analytically in a restricted context, for a specific transfer function. Other dynamical properties are more tricky to handle analytically, in particular competition and working memory. These two behaviors are important sub-functions of decision making, for instance in cognitive robotics [START_REF] Fix | A dynamic neural field approach to the covert and overt deployment of spatial attention[END_REF]- [START_REF] Vitay | A distributed model of spatial visual attention[END_REF]. The former endows an artificial agent with the ability to settle a competition between, for example, alternative motor programs that cannot be executed simultaneously. It may also allow targets of one motor action to compete. The latter allows an agent keeping a trace of a transiently presented information relevant for guiding future behaviors, or temporarily store the spatial position of a set of stimuli when several have to be investigated (see for example [START_REF] Fix | A distributed computational model of spatial memory anticipation during a visual search task[END_REF]).

Using dynamic neural fields to design cognitive architectures is a challenging problem, especially because the functions one wants to introduce in the system result from distributed computations within the fields. These interactions are themselves ruled by a set of parameters that have a non-linear influence on a neural field's behavior. Such a hard problem motivates finding optimization algorithms to automatically tune neural field parameters.

So far, two optimization techniques have been applied for automatically tuning the parameters of a neural field: genetic algorithms [START_REF] Quinton | Exploring and optimizing dynamic neural fields parameters using genetic algorithms[END_REF] and genetic algorithms combined with a gradient descent [START_REF] Igel | Optimization of dynamic neural fields[END_REF]. In [START_REF] Quinton | Exploring and optimizing dynamic neural fields parameters using genetic algorithms[END_REF], the authors apply genetic algorithms to tune the parameters of a two-dimensional neural field performing selection and dynamic tracking of a moving input stimulus within noise or surrounded by distractors. In [START_REF] Igel | Optimization of dynamic neural fields[END_REF], the authors combine a genetic algorithm with a gradient-descent based method. The later requires to compute locally the gradient of a cost function which may be difficult or at least costly to perform.

Kalman filters are a popular collection of algorithms in the optimal control and machine learning community for estimating a hidden state from noisy observations, knowing a model of the underlying process and of the observation apparatus. There are several algorithms that have been developed since the seminal work of Kalman [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF] on systems involving linear evolution and observation processes. For example the Extended Kalman Filter (EKF) and the Unscented Kalman filter (UKF) are efficient methods that can be used for three applications. They can be used for state estimation, i.e. estimating a hidden state observed through noisy measurements. They can also be used for parameter estimation, i.e. determining the parameters of a non-linear system given noisy observations. These two algorithms can finally be combined for performing so-called dual estimation, i.e. estimating both a hidden state and the parameters governing a non-linear system. The Extended Kalman Filter requires to explicitly compute the derivatives of the process' equations. In some applications (like the one we address in this article), analytically computing these derivatives can be difficult or even impossible in some cases. The Unscented Kalman Filter does not need to explicitly compute these derivatives. Neural networks are a particular case of non-linear processes and the techniques mentioned above can be applied to tune their parameters. The Kalman Filters reveal to be quite efficient compared to the more popular backpropagation algorithm [START_REF] Haykin | Kalman filtering and neural networks[END_REF]. The derivative-free aspect of UKF combined with its efficiency in optimizing the parameters of neural networks motivate the study presented in this article, where we seek to optimize the parameters of dynamic neural fields.

The Unscented Kalman Filter [START_REF] Julier | Unscented filtering and nonlinear estimation[END_REF] belongs to the family of Sigma-Point Kalman filter (SPKF) which can be used to estimate hidden variables producing observations through a non-linear function (therefore extending the original Kalman filtering framework [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF] applicable on linear functions). The challenge is to estimate the parameters of this nonlinear model only from partial and noisy observations. The Unscented Kalman Filter has already been applied on several machine learning problems. For example, in [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF], UKF is used for supervised learning and in [START_REF] Geist | Kalman temporal differences[END_REF], the authors use UKF in the context of reinforcement learning to approximate the value function given a reward scheme which provides the indirect noisy observations. The paper is organized as follow: the unscented Kalman filter algorithm is provided in section II-A, the dynamic neural field equation and parameters are described in section II-B and two example scenarios are studied in section III. Extensions of the presented framework and the implications of such a method in the context of dynamic neural fields applied to cognitive robotics are then discussed.

II. METHODS

A. Unscented Kalman Filter (UKF)

The Unscented Kalman filter (UKF) is an efficient derivative-free method for nonlinear filtering problems. In particular, it can be applied to estimate the parameters of a non-linear function. It relies on the unscented transform (UT) [START_REF] Julier | Unscented filtering and nonlinear estimation[END_REF] which allows to compute a local linear approximation of the non-linear function. To do so, the unscented transform introduces a set of so-called sigma-points from which statistics of interest are computed. We briefly sketch below the different steps involved in the UKF for parameter estimation but a more detailed explanation of this algorithm can be found in [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF].

Basically, the parameter estimation problem is stated as: given a non-linear function f θ parametrized by the vector θ, given a set of input-output samples (x i , y i ), find the parameter vector θ that best accounts for the mapping ∀i, y i = f θ (x i )+ v i in a least square sense, where v i is a white observation noise (zero-mean and independent). Therefore, at time step i, UKF seeks at minimizing the following cost function:

θ i = argmin θ ( i j=1 1 P vvj (y j -f θ (x j )) 2 ) (1)
where P vvj is the variance of the noise v j . This cost function is defined on the samples presented so far. The general purpose of UKF, like Kalman filtering, is to track the hidden state of some system given sequential noisy observations. Here, the state is the hidden parameter vector. The associated state-space formulation of the parameter estimation problem is given by:

θ i = θ i-1 + n i y i = f θi (x i ) + v i
Using the vocabulary of Kalman filtering, the first equation is the evolution equation: it specifies that the true parameter vector follows a random walk, the expectation of which corresponds to the optimal estimate of the parameters (in a least-square sense). The evolution noise n i is white (independent and centered) and of variance P nni (to be chosen by the practitioner). This allows handling nonstationary parameters (without harming the stationary case), but it also allows avoiding local minima of the cost-function (1) (its effects can roughly be understood as simulated annealing). The second equation is the observation equation: it links the parameters to the observations through the parametrized function f θ . The observation noise v i is also white and of variance P vvi (to be chosen by the practitioner, possibly on the basis of some physical considerations). UKF is an incremental algorithm that improves step by step the parameters. The idea is to obtain a Widrow-Hoff-like linear update rule of the form

θ i = θ i-1 + K i (y i -ȳi )
where ȳi is a prediction of what should be the next observation y i given the current estimate of the parameters and K i is the so-called Kalman gain. The UKF algorithm aims at incrementally compute the best gain K i , that is the one that minimizes the cost function [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF].

The algorithm we present below 1 and depicted schematically on figure 1 uses the Unscented Transform for clarity of presentation. However, it is the Scaled Unscented Transform [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF] that is used in practice for efficiency reasons. Both rely on the same ideas, the Unscented Transform being shorter to introduce, and slight variations lead to the scaled transform which is provided in more details in [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF]. In addition, the algorithm provided below is the vectorial algorithm while we used scalar notations previously for introducing it. The algorithm is initialized with random parameters θ 0 and a prior on the variance of these parameters P 0 (in practice, a diagonal matrix). At step i, it first updates the covariance of the parameters :

P i|i-1 = P i-1 + P nni
Then, a set of 2p + 1 so-called sigma-points θ j i , j ∈ [0..2p] (p is the number of parameters to estimate) is introduced :

θ 0 i = θ i-1 θ j i = θ i-1 + (p + κ)P i|i-1 j , 1 ≤ j ≤ p θ j i = θ i-1 - (p + κ)P i|i-1 j-p , p + 1 ≤ j ≤ 2p
1 In the description of the algorithm, a matrix is written with a capital bold letter (e.g. P i ), a vector with a bold letter (e.g. x i ) and a scalar with a normal letter (e.g. x i ). where (p + κ)P i|i-1 j is the j-th column of the Cholesky decomposition of the matrix (p + κ)P i|i-1 , and κ ≥ 0 a parameter of the algorithm which controls the spread of the sigma-points sampling. The image of each of the sigma-points through the non-linear function f θ is then computed, each sigma-point defining a specific parametrization:

y j i = f θ j i (x i ), ∀j
A set of weights, used to compute different statistics of interest is then defined as:

w 0 = κ p + κ and w j = 1 2(p + κ) , ∀j > 0
From the image of the sigma-points, the set of weights previously defined and the current sigma-points, one can compute the following statistics:

ȳi = j w j y j i P θyi = j w j (θ j i -θ i-1 )(y j i -ȳi ) T P yyi = j w j (y j i -ȳi )(y j i -ȳi ) T (2)
The last step is to update the Kalman gain K i , the parameters θ i as well as the covariance matrix of the parameters P i :

K i = P θyi • (P vvi + P yyi ) -1 θ i = θ i-1 + K i (y i -ȳi ) P i = P i|i-1 -K i (P vvi + P yyi )K T i B.

Dynamic neural fields

Dynamic neural fields have been introduced as a model of a continuous neural tissue [START_REF] Amari | Dynamics of pattern formation in lateral-inhibition type neural fields[END_REF], [START_REF] Wilson | A mathematical theory of the functional dynamics of cortical and thalamic nervous tissue[END_REF], [START_REF] Taylor | Neural bubble dynamics in two dimensions: foundations[END_REF]. The neural field equation [START_REF] Schoner | Dynamical systems approaches to cognition[END_REF] states that the evolution of the membrane potential at position x and time t, depends on an input at that position i(x, t), on a baseline h and on the lateral influence within the field characterized by a weight function w(x, y) and a transfer function f .

→ u (., t i ) (a) → u ({x i , t i }) (b)
Fig. 2. Two sampling methods are considered. In (a), a time is randomly chosen, and the activity of the whole field at that time is presented to the algorithm, i.e. the samples are (t i , u (., t i )). In (b), a set of time and space positions are randomly chosen. Therefore, the samples at time i are

({x j i , t j i }, {u (x j i , t j i )}) τ ∂u dt (x, t) = -u(x, t) + y w(x, y)f (u(y, t))dy + i(x, t) + h (3)
To be properly defined, the previous equation also requires an initial condition u 0 = {u(x, 0), ∀x}. In order to cast the optimization of the parameters of a neural field into a Kalman filtering problem, one needs to parametrize the neural field equation and specify how it is sampled. To parametrize the neural field, we first discretize equation (3) both in space and time using an Euler scheme. The following equations are therefore specific for the Euler scheme but could be easily adjusted to other schemes. The discrete neural field equation therefore reads:

u(x, t + ∆t) = (1 - ∆t τ )u(x, t) + ∆t τ y w(x, y)f (u(y, t)) + ∆t τ (i(x, t) + h) (4)
We will consider in the following a specific transfer function and a specific weight function. The transfer function is defined as a sigmoidal function parametrized by (a, b, x 0 ):

f (x) = a 1 + exp(b(x -x 0 )) (5)
The weight function is defined as a difference of gaussians parametrized by (A + , σ + , A -, σ -):

w(x, y) = A + exp - (x -y) 2 2σ 2 + + A -exp - (x -y) 2 2σ 2 - ( 6
) This symmetric weight function is suitable for the neural field scenarios we consider in the following. However, we may have easily considered other weight functions such as the asymmetrical one used in [START_REF] Zhang | Representation of spatial orientation by the intrinsic dynamics of the head-direction cell ensemble: a theory[END_REF] in order to obtain traveling patterns of activity. Given the previous parametrization, our problem is to optimize 9 parameters: θ = (τ, h, a, b, x 0 , A + , σ + , A -, σ -). Now, we need to specify the samples that are presented to the algorithm (inputs x i ) and the constraints that specify the optimization problem (outputs y i ).

The optimization problem is defined by providing both the input feeding the neural field and a rough estimate of the desired mean firing rate (called desired output in the following). We use the mean firing rate f (u(x, t)) instead of the membrane potential u(x, t) since it is bounded and easier to specify a desired firing rate than a desired membrane potential. This constraint is just introduced for practical reasons but could be dropped if necessary. A given input and desired output define a neural field scenario. Two example scenarios are shown on figures 3a,b and 4a,b. These figures illustrate space/time representations of two scenarios: a competition scenario and a working memory scenario. Given these scenarios, a sample (the pairs (x i , y i ) we introduced in section II-A) can be defined in several ways. We consider in this article two sampling methods, illustrated on figure 2. The first sampling method selects randomly a time and the whole neural field activity at that time is presented to the algorithm. The second sampling method selects randomly pairs of times and positions and presents the neural field activity at these times and positions. The performance of the algorithm with these two sampling methods will be compared in the Results section and others will be discussed in conclusion. The first sampling method is probably the most natural of the two for sampling the state of a dynamical system but the second makes the algorithm faster to converge. Given we used an Euler scheme for discretizing in time equation (3), we consider only temporal samples that are multiple of the time-step ∆t.

To state the optimization of neural fields' parameters as a Kalman filtering problem, we must define the evolution and observation equations. The evolution equation states that the parameters follow a random walk, with a white independent noise n i . The observation equation links the parameters with the observations. To define this equation, we need to introduce an additional notation. Let's denote u(., t) the activity of the whole neural field at a given time t and denote g θ the parametric function linking u(., t + ∆t) with u(., t): u(., t + ∆t) = g θ (u(., t))

g θ (u) = (1 - ∆t τ )u + ∆t τ W • f (u) + ∆t τ (i + h) (7)
where W denotes the weight matrix, f (u) denotes the vector whose components correspond to the components of u on which f is applied. The state of the neural field at a certain time t, multiple of ∆t, is defined as the repeated composition of g θ applied on the initial condition u 0 : u(., j∆t) = g θ (g θ (...g θ (u 0 )...)) = g j θ (u 0 ) We can now define the so-called state-space description of the system, for a certain input x i = (k, j∆t), k ∈ [0, n[:

θ i = θ i-1 + n i y i = g j θi (u 0 )(k) + v i
The previous state-space description is given for a pair of spatial/temporal position and should be slightly adjusted to correspond to the previous definition of a sample. We do not specify it further to avoid unreadable equations. One should note that the previous definition of the Kalman filtering problem is not dependent on the specific choice of the weight function and transfer function. Given that UKF is a derivativefree algorithm, we do not even need to use derivable functions. This is in contrast to gradient-descent-based methods such as in [START_REF] Igel | Optimization of dynamic neural fields[END_REF] which require to compute derivatives. The simulations of the neural field and the UKF algorithm were written in C++/GSL.

III. RESULTS

We applied the UKF algorithm on two classical scenarios of dynamic neural fields: competition and working memory. In the first scenario (fig. 3), several regions of a 1D neural field are locally excited and we want to find the parameters of the neural field settling a competition between these excited regions so that the field's activities converge to the representation of only one of the excited regions. Figures 3a,b illustrate the input feeding the neural field as well as the desired firing rate. The input consists in two locally excited regions with relatively close magnitudes (1.0 and 0.75), mixed with a random noise. As an output, we want the neural field to produce a competition between these two regions leading to the selection of the stimulus with the highest amplitude. Intuitively, this competition relies on lateral inhibition where both regions are inhibiting each other. A proper range and amplitude for the lateral inhibition term has to be found in order to get this competition. Figure 3c . The parameters of the scaled UKF (see [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF] section 3.2.2 for the meaning of these parameters) were : α = 0.3, β = 2.0, κ = 0, P 0 = 0.1.I, Pnn i = 0.0, Pvv i = 0.1.I. The membrane potentials ranging from 0.0 to 1.0 are represented with a graded color from white to black. A video of the optimization algorithm is available at http://jeremy.fix.free.fr/demo.php?demo=CCMB2011 .

the algorithm has found a proper set of parameters allowing this behavior. To compare the two sampling methods and to get an idea of the time it takes for the algorithm to converge, we repeated 1000 times the optimization procedure. It took on average respectively 260 steps and 52 steps to converge (to reach a RMS error of 0.1) for the two sampling methods. The algorithm reached a local minima 13% of the trials for the sampling in time, and 3.5% of the trials for the sampling in time and space. The evolution of the mean RMS, through the iteration of the algorithm, is shown on figure 5a. In addition, some local minima reached by the algorithm with their respective RMS are shown. Sampling randomly in both time and space clearly leads to a faster convergence.

A more difficult scenario is proposed on figure 4. A work-ing memory should be able to maintain the representation of an information despite the presence of distractors, and this as soon as some signals triggered the entrance of this information into the working memory. The input and desired output shown on figures 4a,b illustrate this behavior. There, the neural field is fed with two locally excited regions of initially weak amplitude. A transient increase of the amplitude of the input stimuli, reflecting some kind of selection or gating for the entrance in working memory, triggers the emergence of the stimulus in working memory. Removing the gating signal leads to decrease the amplitude of the selected stimulus back to its initial and weak value. Despite this decrease in amplitude, the stimulus should stay in the working memory. There are mainly three critical parameters allowing to get this 4. Working memory scenario. (a) Space/time representation of the input feeding the neural field. The two localized excitations are gaussians with an initial amplitude of 0.3, transiently increased up to 1.0, and a standard deviation of 2.0. A uniform random noise of amplitude 0.1 is added to the input. The transient increase of the stimuli's amplitude reflects some kind of selection, allowing a target to enter the working memory. (b) The desired firing rate specifies a working memory behavior: when the amplitude of the input reaches a threshold, the stimulus enters the working memory and stays there until the input is completely suppressed. (c) Obtained neural field after 1000 iterations and a RMS error of 0.077. The algorithm converged to the following parameters: a = 0.92, b = -1.93, x 0 = 1.12, A + = 1.21, σ + = 2.11, A -= 0.34, σ -= 5.28, τ = 0.5, h = -0.5. The neural field, involving 40 neurons, is simulated synchronously during 60 seconds with a time step ∆t = 0.1s.. The parameters of the scaled UKF (see [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF] section 3.2.2 for the meaning of these parameters) were : α = 0.3, β = 2.0, κ = 0, P 0 = 0.1.I, Pnn i = 0.0, Pvv i = 0.1.I. The firing rates ranging from 0.0 to 1.0 are represented with a graded color from white to black. A video of the optimization algorithm is available at http://jeremy.fix.free.fr/demo.php?demo=CCMB2011 . 3 and4. For each figures, the curves show the evolution of the RMS through the iteration of UKF for the two sampling methods shown on figure 2 : sampling the whole state of the neural field at a random time or sampling randomly both time and space positions. The filled regions represent the standard deviation. The mean and standard deviation of the RMS are computed for the trials that did not converge to a local minima, defined by a threshold on the RMS. This threshold is set to 0.1 for the competition scenario and 0.12 for the working memory scenario. For both scenarios, we also show illustrations for some local minima with their respective RMS. a) The algorithm converged in a local minima 13% of the trials for the sampling in time, and 3.5% of the trials for the sampling in time and space. b) The algorithm converged to a local minima 42% of the trials for the sampling in time and 32% of the trials for the sampling in time and space.

behavior: the baseline, the lateral excitation and the lateral inhibition. The baseline allows to set up the threshold above which a stimulus will enter the working memory. The lateral excitation allows to maintain a stimulus in working memory when the input's amplitude is set back to its initial value. This lateral excitation compensates the decrease of input drive. Usually, this lateral excitation is much stronger than in the competition scenario. Given the strength of this recurrent term, it may propagate over the whole field and saturate it completely. Lateral inhibition prevents this spread by somehow constraining the size of the excited regions within the field. Given the strength of the lateral excitation, the neural field may also become insensitive to the input somehow becoming saturated as soon as the input reaches a threshold. One may need that a stimulus gets removed from working memory when it disappears from the input, a behavior that cannot be guaranteed if lateral excitation is too strong. Therefore, a proper balance of lateral excitation/inhibition must be found. This additional constraint is added to the scenario by requiring the firing rate of the neural field to go back to the resting level as soon as the input stimuli disappear, which we see for the last time steps on figures 4a,b . Figure 4c shows the neural field obtained after 1000 iterations of the algorithm. Repeating the optimization procedure on 1000 trials, it took on average respectively 256 steps and 130 steps to converge (to reach a RMS error of 0.12) for the two sampling methods. The algorithm got trapped in a local minimum respectively 42% and 32% of the trials. The stopping criterion used in the simulations relied on the RMS error between the desired and obtained neural field activities. The bound at which the algorithm is stopped is defined qualitatively since it depends on the level of noise in the input, that remains in the neural field activities, but also on the level of noise introduced by defining a desired neural field activity. This desired activity profile is only a rough estimation of the state the neural field can reach. Therefore, a proper bound for the RMS error can only be tuned qualitatively. When setting this bound in the working memory scenario, we usually observed that a RMS error of 0.12 corresponds to a neural field activity that was relatively close to the desired profile (at least, all the trials that ended with such a RMS corresponded to a satisfactory solution from a visual inspection). When it was larger, it usually corresponds to a neural field which maintains only one of the two stimuli as shown on figure 5b. This illustration also shows the evolution of the RMS during the iteration of the algorithm. Again, we observe a tendency for sampling in space and time to converge more fastly and more reliably to the solution.

IV. DISCUSSION

The Unscented Kalman Filter is an efficient statistical learning method that can be used for parameter estimation. We have shown here that it can be used efficiently for optimizing the parameters of a dynamic neural field, given a desired behavior (an input and a desired firing rate) has been specified and regardless the complexity of the neural field equation. In particular, non-linear or even non-derivable functions can be introduced in the equation. This derivativefree optimization algorithm is particularly efficient and makes it a good candidate to automatically tune the parameters of a neural field, which is a critical issue when one wants to apply dynamic neural fields to control artificial agents.

The major advantage of UKF compared to the previous works [START_REF] Quinton | Exploring and optimizing dynamic neural fields parameters using genetic algorithms[END_REF], [START_REF] Igel | Optimization of dynamic neural fields[END_REF] is its derivative-free aspect which does not restrict to the usage of derivable functions (e.g. a Heaviside function is also a classical transfer function). In terms of complexity, the genetic algorithms require to compute the fitness over the whole time horizon of all the individuals, i.e. an order of O(N I n T n 2

x ) computations, where N I is the number of individuals, n T the number of time steps and n x the number of space positions. They also require to update the parameters but given that there are much less parameters than the number of space and time positions (i.e. p n x and p n T ), this step has a low impact on the overall complexity. The gradient-descent-based method used in [START_REF] Igel | Optimization of dynamic neural fields[END_REF] requires an order of O(p n T n 2

x ) computations. UKF involves different operations such as matrix inversion, Cholesky decomposition, etc. . However, for the same reason as before (i.e. p n x and p n T ), the complexity of UKF is only dependent on 2p + 1 evaluations of a neural field (when computing the image of the sigma-points). Therefore, a single UKF update requires an order of O(p n T n 2

x ) computations. The complexity of UKF and a gradient-descent are similar but one should expect that UKF converges faster since UKF belongs to the family of Sigma-Point Kalman Filters (SPKF) which are an online form of a modified Gauss-Newton method which is a variant of natural gradient descent (see [START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF], ch. 4.5). More than just estimating the complexity of a single step, one would also need to perform benchmarks of the three algorithms to determine the number of steps they require to converge in order to provide a fair comparison. In addition, we discuss below some improvements of the proposed method to make UKF even faster to converge.

Two improvements of the presented method are currently investigated. The first one is to propose a more efficient sampling method for speeding up the convergence of the algorithm. In particular, one can use the information on the covariance of the output (equation 2) to bias the sampling of the inputs. Instead of the random sampling used so far, we could use a probabilistic draw biased by the covariance of the output of the non-linear function. It would speed up the convergence by presenting the samples where the uncertainty on the predictions (of the observations) is higher. This uncertainty depends both on the uncertainty on the value of the parameters and also on the sensitivity of the non-linear function respect to the parameters. A second improvement is also based on the way the neural field is sampled. In particular, instead of evaluating the whole neural field until a certain time, one could use transitions from, say state u(t) to u(t + 1) given that u(t) is provided by the desired output. Given the desired output is only a rough estimation of the state that the dynamic neural field can reach, or said differently that the desired output is a noisy observation of a reachable output starting from a given initial condition, one can introduce additional equations in the state-space formulation to consider that the neural field state is also part of the evolution equation. These improvements would allow to scale up more easily the algorithm to deal with two or higher dimensional neural fields which are more common in practical applications.

ACKNOWLEDGMENT

This work has been supported by the Région Lorraine.

Fig. 1 .

 1 Fig. 1. Signal flow diagram for UKF parameter estimation. Based on a prior estimate of the parameter's covariance P 0 , of the parameters θ and on a new sample (x i , y i ), UKF for parameter estimation updates the covariance matrix as well as the parameters minimizing the cost function defined by equation (1).

Fig. 3 .

 3 Fig. 3. Competition scenario. (a) Space/time representation of the input exciting the neural field. The two localized excitations are gaussians with an amplitude of 1.0 and 0.75, a standard deviation of 4.0, mixed with a uniform random noise of a maximal amplitude of 0.1 (b) The desired membrane potential seeks to produce competition between the two excitation regions, the one with the highest amplitude being the winner. (c) Obtained neural field after 233 iterations of the algorithm and a RMS error of 0.05. The algorithm converged to the following parameters: a = 0.93, b = -4.99, x 0 = 0.59, A + = 1.53, σ + = 31.68, A -= 1.50, σ -= 47.74, τ = 0.56, h = 0.02. The neural field, involving 40 neurons, is simulated synchronously during 40 seconds with a time-step ∆t = 0.1s.. The parameters of the scaled UKF (see[START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF] section 3.2.2 for the meaning of these parameters) were : α = 0.3, β = 2.0, κ = 0, P 0 = 0.1.I, Pnn i = 0.0, Pvv i = 0.1.I. The membrane potentials ranging from 0.0 to 1.0 are represented with a graded color from white to black. A video of the optimization algorithm is available at http://jeremy.fix.free.fr/demo.php?demo=CCMB2011 .

  Fig.4. Working memory scenario. (a) Space/time representation of the input feeding the neural field. The two localized excitations are gaussians with an initial amplitude of 0.3, transiently increased up to 1.0, and a standard deviation of 2.0. A uniform random noise of amplitude 0.1 is added to the input. The transient increase of the stimuli's amplitude reflects some kind of selection, allowing a target to enter the working memory. (b) The desired firing rate specifies a working memory behavior: when the amplitude of the input reaches a threshold, the stimulus enters the working memory and stays there until the input is completely suppressed. (c) Obtained neural field after 1000 iterations and a RMS error of 0.077. The algorithm converged to the following parameters: a = 0.92, b = -1.93, x 0 = 1.12, A + = 1.21, σ + = 2.11, A -= 0.34, σ -= 5.28, τ = 0.5, h = -0.5. The neural field, involving 40 neurons, is simulated synchronously during 60 seconds with a time step ∆t = 0.1s.. The parameters of the scaled UKF (see[START_REF] Van Der Merwe | Sigma-point kalman filters for probabilistic inference in dynamic state-space models[END_REF] section 3.2.2 for the meaning of these parameters) were : α = 0.3, β = 2.0, κ = 0, P 0 = 0.1.I, Pnn i = 0.0, Pvv i = 0.1.I. The firing rates ranging from 0.0 to 1.0 are represented with a graded color from white to black. A video of the optimization algorithm is available at http://jeremy.fix.free.fr/demo.php?demo=CCMB2011 .

Fig. 5 .

 5 Fig.5. Mean RMS error curves for the scenarios shown on figure3 and 4. For each figures, the curves show the evolution of the RMS through the iteration of UKF for the two sampling methods shown on figure2: sampling the whole state of the neural field at a random time or sampling randomly both time and space positions. The filled regions represent the standard deviation. The mean and standard deviation of the RMS are computed for the trials that did not converge to a local minima, defined by a threshold on the RMS. This threshold is set to 0.1 for the competition scenario and 0.12 for the working memory scenario. For both scenarios, we also show illustrations for some local minima with their respective RMS. a) The algorithm converged in a local minima 13% of the trials for the sampling in time, and 3.5% of the trials for the sampling in time and space. b) The algorithm converged to a local minima 42% of the trials for the sampling in time and 32% of the trials for the sampling in time and space.