Caroline ; Éléonore

Keywords: Deep learning, neural networks, Stochastic control, Markov Decision Process, nonlinear PDEs, QVIs, optimal stopping problem BSDEs, RBSDEs, McKean-Vlasov control, performance iteration, value iteration, hybrid iteration, global regression, local regression, regress-later, quantization, limit order book, pure-jump controlled process, algorithmic-trading, market-making, high-dimension Réseaux de neurones, contrôle stochastique, MDPs, EDPs, IQVs, Temps d'arrêt optimaux, EDPRs, EDPRs réfléchies, contrôle de type McKean-Vlasov, régression globale, regress-later, regression locale, quantification, carnet d'ordres, trading algorithmique, market-making, grandedimension McKean-Vlasov control, polynomial class, quantization, regress-later, control randomization

Ces quelques lignes sont pour moi l'occasion d'exprimer ma gratitude envers tous ceux qui m'ont accompagné dans cette thèse. J'espère aussi qu'elles divertiront les plus courageux qui sont venus me soutenir soutenir: notamment les malheureux qui n'ont pas révisé le cours de contrôle stochastique et les EDPs avant de venir, et ceux dans le fond qui trouvent le temps long mais qui n'osent pas sortir le journal.

Je tiens avant tout à remercier mes deux directeurs de thèse: Frédéric Abergel et Huyên Pham, avec qui se fut enrichissant de travailler avec vous pendant ces 3+1 dernières années. J'ai énormément appris à vos côtés. Frédéric, c'était une vraie chance de te connaître. C'était un plaisir d'aller te voir de temps-en-temps à l'École Centrale pour te parler des grands malheurs que je rencontrais dans mes premières années de recherche, et de constater que j'avais toujours ton soutien à la fin des rendez-vous: je repartais souvent requinqué, avec le sourire, et l'impression naïve mais éphémère que tous les problèmes étaient enfin réglés. Huyên, merci beaucoup pour votre présence tout au long de cette thèse. Je vous remercie pour votre disponibilité: vous avez toujours pris le temps de répondre à mes questions, même quand je toquais à votre porte après 19h alors que vous pensiez sûrement pouvoir travailler tranquillement à cette heure là. Je vous suis très reconnaissant pour les projets passionnants sur lesquelles vous m'avez proposé de travailler; et sur lesquels je me suis beaucoup épanoui pendant ma thèse.

Je remercie sincèrement mes rapporteurs: Romuald Élie et John Schoenmakers, pour avoir accepté de relire attentivement mon manuscrit. Je suis honoré de la présence dans mon jury de Jean-François Chassagneux, Emmanuel Gobet, Charles-Albert Lehalle, Gilles Pagès et de mes deux rapporteurs. Vos travaux et nos discussions m'ont beaucoup marqué.

Merci aux membres du LPSM avec qui j'ai partagé de grands moments. Je remercie en particulier les doctorants de mon bureau à p7, le bureau Serpentard, dans lequel le choixpeau arguably tend à placer le top du top des doctorants. Il y a d'abord les anciens: Clément, Adrien et 婷婷, avec qui j'ai eu le plaisir de partager ma première année de thèse; David Krief, qui m'a supporté 3 ans; 俊 超; 易洋, qui ne me refuse jamais une partie de Go, surtout quand je pose le Goban au milieu de son bureau et que je rapproche les pierres; 厚志, au labo 7j/7; Johann qui, je l'espère, a fini par se remettre de son séjour au Limmathof; Enzo, qui a très peur de ne pas voir son nom apparaître dans cette liste; William, toujours en costume; Mamadou; Shanqiu; Remy; et Médéric. Je tiens aussi à remercier les doctorants que le choixpeau n'a pas jugé judicieux de mettre chez Serpentard. Dans le bureau Serdaigle: merci à Cyril; Lucas, "quand est-ce que tu retournes à New York?"; Laure, que je suspecte encore beaucoup d'aimer secrètement la finance; Assaf, dont l'absence à ma soutenance de thèse est pardonnée parce ça le tuerait de se lever avant 11h; Guillaume, et sa balle de tennis; Sothea; et Hiroshi. Dans le bureau Gryffondor, je remercie Luca, que j'ai eu le plaisir de bien connaître à Zurich; Fabio 4kyu, pour nos conversations quotidiennes sur la course à pieds; Vu Lan; Bogdan, qui aime bien le silence; Clément, qui ne supporte pas le silence; Yann, qui est un excellent câlin provider; Benjamin, dont une regression avec régularisation LASSO montrerait facilement que ses passages v fréquents en chaussettes expliquent à eux seuls la propreté du sol dans le labo; Mi-Song; Arturo; 晓 利, la plus discrète. Dans le bureau Pouffesoufle, il y a Ziad, qui a passé les 6 premiers mois de sa thèse à jurer que jamais, jamais au grand jamais, il ne s'abaisserait à faire du numérique, et qui passe maintenant ses journées à débugger ses milliers de lignes de C++; Marc; S. (qui préfère rester anonyme); Barbara; Clément; ainsi que Georg. Je remercie Nathalie et Valérie et Amina qui ont fait un travail remarquable pour m'accompagner dans toutes mes démarches administratives, et sur qui j'ai pu compter même quand je faisais n'importe quoi avec les ordres de mission.

Merci aux doctorants de p6 que je croisais quelques fois au groupes de travail, en particulier: 雅 婷; et Thibaut, avec qui j'ai passé de très bons moments dans les restaurants d'altitude à Métabief. Je remercie les membres de la Chair of quantitative finance à CentraleSupelec; en particulier XiaoFei, avec qui c'était un plaisir de consacrer les vendredis midi à la préparation des TDs, autour de bons repas chinois ou japonais. J'en profite enfin pour remercier les participants du CEMRACS2017, notamment les membres de mon projet: Isaque, Alessandro, et Mathieu.

Au cours de ma scolarité, j'ai rencontré beaucoup de professeurs qui ont su me transmettre la passion pour les maths. Je remercie en particulier Sébatien Chevalier, mon professeur au lycée, qui a été le premier à m'enseigner la rigueur mathématique; ainsi que René Martel, mon professeur en mathématiques spéciales, pour son enseignement divin, sa justesse et sa gentillesse. Merci à Pierre, bluffant en maths et encore plus en kite-surf; Yohan, sur qui l'on compte pour développer l'AI de demain; et Quentin, qui fait toujours preuve d'une grande patience lorsqu'on le fait tourner en rond pendant des heures à la recherche du meilleur restaurant.

Chapter 1

Introduction

The thesis is divided into three parts that can be read independently. In the first part, we propose algorithms based on quantization and regression to solve Markov Decision Processes (MDPs). We then present two applications: a problem of market-making and a McKean-Vlasov-type stochastic control problem1 . In the second part of the thesis, we present a class of numerical schemes based on neural-networks to solve MDPs. Consistence results are derived, which show the convergence of our new algorithms. We then test them on several interesting examples found in the literature. In the third part of the thesis, we present neural-network-based numerical schemes to solve PDEs, QVIs, BSDEs and RBSDEs. Consistence results of our estimates are derived, and tests are performed which illustrate the great behaviors of our algorithms compared to other recent algorithms available in the literature. After reviewing literature about Markov decision processes (MDPs) that are usually solved by regression methods in practice, we proposed a new algorithm based on local regression and quantization, which we refer to as Qknn in this thesis.

Contents

Presentation of the MDPs

Let us consider a stochastic control problem with finite horizon and discrete time, i.e. we fix a finite horizon N ∈ N \ {0}, and consider the (controlled process) X α = (X α n ) N n=0 , taking values in X ⊂ R d , with d ∈ N being the dimension of the problem, and s.t. it admits the following dynamics:

X α 0 = x 0 ∈ R d , X α n+1 = F n (X α n , α n , ε n+1
), for n = 0, . . . , N -1, where:

• (ε n ) N
n=1 is a sequence of i.i.d. r.v., which takes values in a Borelian set (E, B(E)), and defined on a probability space (Ω, F, P). We denote by F = (F n ) N n=0 the σ-algebra generated by (ε n ) N n=1 (F 0 is the trivial σ-algebra).

• The control α is a F-measurable process taking values in A ⊂ R q .

• For n = 0, . . . , N -1, the F n are measurable functions from R d × R q ×E to R d .

Let β ≥ 0 be a discount factor, f : R d × R q → R be the instantaneous cost, and g : R d → R be the terminal cost. We define J(α), the cost functional associated to the control α, as follows:

J(α) = E N -1 n=0 β n f (X α n , α n ) + g(X α N ) . (1.1.1)
The control α is admissible if J(α) is well-defined, and we denote by C the set of the admissible controls. The control problem is to find

V 0 (x 0 ) := inf α∈C J(α), (1.1.2)
and characterize (at least) one optimal control α * ∈ C, which minimizes the cost functional, i.e. V 0 (x 0 ) =: J(α * ).

In this thesis, we refer to the control problems with finite horizon and discrete time as Markov Decision Processes with finite horizon, or shortly as MDPs.

The first difficulty for solving (1.1.2) is that the infimum is taken on the set of the admissible processes C, which can very often be of infinite dimension in practice. Before simplifying (1.1.2) using dynamic programming principle, let us first introduce some basic notations: we denote by {P a (x, dx ), a ∈ A, x ∈ X } the family of transition probabilities associated to the controlled Markov chain (1.1.1.1), i.e. P a (x, dx ) = P [F (x, a, ε 1 ) ∈ dx ] ,

1.1. Quantization and regression methods for MDPs. Applications to market-making and

McKean-Vlasov-type control problem with partial information and moreover, we define P a ϕ(x) = ϕ(x )P a (x, dx ) = E ϕ F (x, a, ε 1 ) , for all measurable function ϕ : X → R. With these notations, for all functions ϕ on X , and for all control α ∈ C, we have:

E[ϕ(X α n+1 )|F n ] = P αn ϕ(X α n ), ∀ n ∈ N.
By programming dynamic principle, V n (.), the value function at time n, for n = 0, . . . , N , is characterized as solution to the following backward equation2

     V N (x) = g(x), ∀x ∈ X , Q n (x, a) = f (x, a) + βP a V n+1 (x), ∀x ∈ X , a ∈ A, V n (x) = inf a∈A Q n (x, a)
, for n = N -1, . . . , 0,

(1.1.3) and moreover, α * n , the optimal feedback control at time n, is characterized as follows:

α * n (x) ∈ argmin a∈A Q n (x, a). (1.1.4)
The community of the Reinforcement Learning (RL) usually refers to the function Q n as the "Q-value function" or the "state-action function", and V n as the value function at time n.

Remark 1.1.1. A very general class of continuous/discrete with timeinfinite/finite horizon control problems can be reduced to (or approximated by) the scheme (1.1.3). For example, let us consider the following continous-time control problem with infinite horizon:

V (x) := sup α∈C E ∞ 0 e -λt f (X α t )dt ,
with λ > 0, and where we assume that the controlled process X α , for a control α follows the dynamic below: dX α t = b(t, X t , α t )dt + σ(t, X t , α t )dW t , where the drift b and the volatility σ are regular enough. Let us first discretize in time the problem: let us take a step size h < < 1, and discretize the time as follows:

V (x) ≈ V h (x) := sup α∈C E ∞ n=0 he -λnh f (X α nh ) ,
where the dynamic of the controlled Markov chain (X α nh ) n∈N reads:

X α 0 = X 0 X α h(n+1) = b(nh, X α nh , α nh )h + σ(nh, X nh , α nh ) √ hε n+1 ,
with (ε n ) n∈N a sequence of i.i.d. r.v. following the reduced Gaussian law N (0, 1).

We recognize a MDP with infinite horizon. Under standards assumptions3 , one can show that the value function V h is the unique fixed point of the following operator

T : C 0 (R) -→ C 0 (R) v -→ x -→ sup a∈A r(x) + e -λh E [v(X a h )|X 0 = x] ,
where we denote r(x) := hf (x). Using a Picard fixed point theorem argument, we can approximate the fixed point of T by considering the N -th iterate of T , starting from the zero function. Hence V 0 , defined in the scheme (2.1.5) below, converges toward the fixed point of T when N goes to infinity.

V N = 0 V n (x) = sup a∈A r(x) + e -λh E [V n+1 (X a h )|X 0 = x] , for all state x
We finally recognize the backward scheme (1.1.3), with β = e -λh , g = 0, f = r.

Remark 1.1.2. There exist more RL algorithms such as the Temporal Difference Learning and the Deep Q-learning, to solve control problem with infinite horizon, for which the value function is fixed point of an operator. We refer to [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] for an exhaustive introductions to them, [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] for a concrete application of Deep Q-learning to learn how to play ATARI at a better level than the best human players, and [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], [START_REF] Silver | Mastering the game of Go without human knowledge[END_REF] for a concrete application of Monte Carlo tree search on the game of Go, to learn how to manage with a large control space, and manage to play better than professional Go players.

The problem (1.1.3) is theoretically easier to solve than (1.1.2) since the infimum is now taken over A which is a subset of the finite-dimensional space R q , and moreover we have the nice characterization (1.1.4) of the optimal control that is associated to (1.1.3). Unfortunately, in most of the concrete applications, we still meet two difficulties when using the scheme (1.1.3):

PB1 the conditional expectation P a , that appears at each time step to compute Q n in (1.1.3), may not have any closed-form formula and need to be approximated.

PB2 the infimum which appears in the definition of V n in (1.1.3) may not admit any closed-form formula, and the approximation of the latter4 may be time-consuming when the control space is large.

Algorithms available in the literature to solve the Bellman equation

In the first part of the thesis, we decided to investigate the quantization and regression based methods to solve the Bellman equation (1.1.3).

Regress-Now on a basis of functions

The regress-now method to solve MDPs consists in the three steps procedure:

(i) Randomize the control, i.e. simulating the control as an exogenous process, so that the latter becomes a component of the state variable (ii) Regress the value function at time n + 1 on a basis functions at time n to estimate the Q-value at time n.

(iii) Optimize the Q-value at time n to estimate the value function at time n.

We refer to [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF], [START_REF] Carmona | Valuation of energy storage: an optimal switching approach[END_REF] and the recent paper [START_REF] Ludkovski | Simulation methods for stochastic storage problems: a statistical learning perspective[END_REF] for more details on the regress-now method to solve control problems. One of the main difficulty with the regress-now method is the choice of the basis functions: when the dimension of the control problem is low, usual basis functions (such as the polynomial family) works well; but when the dimension is getting high, it becomes more difficult or almost impossible to find good basis functions to regress the value functions on without any over-fitting or under-fitting. An idea, developped in [START_REF] Belomestny | Optimal stopping via reinforced regression[END_REF], consists in adapting the basis functions B n on which we regress V n+1 , the value function at time n+1, by adding the functions that looks like V n+1 in B n . Doing so, the estimate of the conditional expectation is better in the case where the MDP admits a "continuity" of the value function w.r.t. time, i.e. the MDP is a reduction of a time-continuous control problem on which the value function is continuous w.r.t. the time component.

Regress-Later

The regress-later method consists in the following three steps procedure:

1.1. Quantization and regression methods for MDPs. Applications to market-making and

McKean-Vlasov-type control problem with partial information (i) interpolate the value function at time n+1 on basis functions (ii) regress the interpolation 5 , and estimate the Q-value at time n.

(iii) Optimize the Q-value at time n to get an estimate of the value function at time n

We refer to [START_REF] Balata | Regress-Later Monte Carlo for optimal control of Markov processes[END_REF] for a detailed presentation of the regress-later method applied to solve appliquée MDPs. Note that this method is particularly efficient in the (not so rare) case where there are some closed-form formula for the computation of the conditional expectation of the basis functions. In this case, the regression step is almost instantaneous, and the agent does not have any error coming from the regression step to look after, i.e. make sure that the error does not blow up. In the case where there is no closed-form formula for the computation of the conditional expectations of the chosen basis functions, one can proceed to an approximation of the latter using e.g. quantization-based methods.

Regress-later is more recent than regress-now, and regularly outperforms the latter. We refer to [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF] for a comparison of regress-later and regress-now algorithms on a concrete example of micro-grid management, in which one can see that regress-later seems to give better estimates of the optimal strategy.

Motivation

It is surprising to see that most of the algorithms available in the literature for control problems are based on global regression methods. When dimension is high (more than 10), global regression is probably the only tool that numerically proved to work well; but when dimension is low, one should also consider the local regression methods to build algorithms. We refer to [START_REF] Belomestny | Regression methods for stochastic control problems and their convergence analysis[END_REF] for some details on the local regression for MDPs. Also, quantization methods appears to be very well-suited to solve control problems, as shown in [START_REF] Pagès | An optimal Markovian quantization algorithm for multi-dimensional stochastic control problems[END_REF].

In the first part of the thesis, we want to mix the local regression and quantization techniques; which seems to be adapted to some control problems where the dimension is not too high6 and where the noise can be quantized.

Contributions (Chapter 4)

We propose the Qknn algorithm, based on Quantization of the exogenous noise ε n , n = 0, . . . , N -1 and local regression (knearest neighbors algorithm) to approximate the conditional expectations. As we can see in the pseudo-code of Qknn, written in Algorithm 1.1: at each time step n, the agent chooses a finite family of points called "training points" on which to estimate the value function at time n. The computation of the conditional expectation is then done by quantizing the exogenous noise, i.e. approximating the noise r.v. ε n+1 by a discrete r.v. εn+1 which can take a finite number of state e 1 , . . . , e with probability p 1 , . . . , p . εn+1 shall be chosen such that the L 2 norm of the difference between ε n+1 and εn+1 is small enough7 .

We refer to Section 3.4.2 page 76 in Chapter 3 for a detailed introduction of Qknn algorithm. In this chapter, we also derive a consistence result for Qknn (see Theorem 3.4.1 page 78). We present below a simplified version of the latter: Theorem 1.1.1. Take K = M 2+d points for the quantification of the exogenous noise ε n , n = 1, . . . , N . Under some regularity assumptions, there exists a constant C > 0 such that for all n = 0, ..., N -1,

V Q n (X n ) -V n (X n ) 2 ≤ C N k=n+1 ε proj k + ε Q k + O 1 M 1/d ,
when M → +∞, and where ε Q k := εk -ε k 2 is a quantization error, and ε proj n := sup a∈A Proj n+1 (F (X n , a, εn )) -F (X n , a, εn ) 2 Chapter 1. Introduction Algorithm 1.1 Qknn Algorithm Input:

• Γ k , k = 0, . . . , N : grids of training points in R d ,

• Γ = {e 1 , . . . , e L } , (p ) 1≤ ≤L : the L-optimal grid for the quantization of the i.i.d. exogenous noise (ε i ) N -1 i=0 , and its associated weights.

1: Assign N : V Q ∆t (t N , z) = g(z), ∀z ∈ Γ N . 2: for n = N -1, . . . , 0 do 3:

Estimate the value function at time n:

V Q ∆t (t n , z) = max a∈A f (z, a) ∆t + L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t (z, a, e )
, ∀z ∈ Γ n .

4: end for Output:

α(t n , z) z∈Γn,0≤n≤N -1 : estimate of the optimal strategy; -V Q ∆t (0, z) z∈Γ0 : estimate of the value function;

a projection error when decision a is taken at time n.

Theorem 1.1.1 states that the L 2 norm of the difference between the Qknn-estimate of the value function and the value function is bounded by the quantization and the projection errors made at each time step n = 0, . . . , N -1. These two errors cancel themselves respectively when the number of points taken for the quantization and the number of points chosen for the grids go to infinity.

Application of the MDPs to market-making (Chapter 3)

In Chapter 3, we are interested in solving a market-making problem. We solve it theoretically by rewriting it as a MDP, and numerically using Qknn. We introduce below the market-making problem, and present our theoretical and numerical contributions.

Presentation of the market-making control problem

We consider a financial instrument which can be traded in a financial market through an order book, which is an electronic list of buy and sell orders available at the different price levels. All the participants in the market observe the order book and can at any time:

• buy or sell the instrument at the best price (we refer to this order as "market order")

• propose to buy or sell the instrument at certain prices (we refer to this order as "limit order")

• cancel their limit orders before the latter are executed (we refer to this order as "cancel order") A market-maker is a participant in the market that can emit some limit orders in the order book, and cancel her orders. The main job of the market-maker is to liquefy the market by emitting buy and sell limit orders at different prices. A strategy followed by the market-maker, which is referred to as market-making strategy, consists in emitting or canceling buy and sell limit orders, while maximizing a certain quantity such as the wealth of the market-maker.

Remark 1.1.3. One must differentiate limit and market orders. A participant proposes to buy or sell an instrument at a certain price when emitting limit orders, whereas she buys or sells it at the best price when emitting a market order. Unlike the market orders, there is no transaction fees for limit orders, since they liquidate the market.

1.1. Quantization and regression methods for MDPs. Applications to market-making and

McKean-Vlasov-type control problem with partial information

Model et mathematical representation of the problem

Let us take K ≥ 1. We represent the order book by its K first limits on the ask and bid sides, as proposed in [START_REF] Abergel | Limit Order Books[END_REF]. We denote pa t the best-ask at time t, which is the cheapest price a participant is willing to sell an instrument at time t, and by pb t the best-bid at time t, which is the most expensive price a participant is willing to buy an instrument at time t. The order book is then represented as the couple a t , b t = a 1 t , ..., a K t , b 1 t , . . . , b K t where

• a i t is the number of sell limit orders i ticks away from pb t .

• b i t is the number of buy limit orders i ticks away from pa t .

The order book can receive market, limit or cancel orders at any time, which modify the value of (a t , b t ). We model the arrivals of these events as point processes.

We consider a market-maker that observes the order book, and can choose at any time to send limit or cancel orders in the order book. We denote by A the set of all the admissible market-making strategies, which are the F-predictable processes, where F is the filtration generated by the order arrivals processes, which represent the positions of the market-maker on the order book, i.e. where are the limit orders of the market maker in the order book. In Chapter 3, we denote by Z the "controlled order book", which is the controlled process that gathers up a t , b t = a 1 t , ..., a K t , b 1 t , . . . , b K t and the positions of the market-maker. Z is a process that lies on a discrete space that we call E, since the prices the participants can buy or sell the instrument are organized by levels.

The goal of the market-maker is to find the optimal strategy that maximizes the following quantity

V (t, z) = sup α∈A E α t,z T t f α s , Z s ds + g Z T , (t, z) ∈ [0, T ] × E,
where g is the terminal reward (for example the terminal wealth of the market-maker plus a penalization term of the terminal inventory); and where f is the instantaneous reward (for example the risk aversion of the market-maker).

Contributions

Theoretical contributions:

In the Section 3.3.2 of Chapter 3, we show8 that the time-continuous market-making control problem (where the controlled process is Z and the value function is V ) can be reformulated as a MDP. From the reformulation, we can derive9 a characterization of V as the unique fixed-point of the operator T defined as follows:

T : C 0 ([0, T ] × E) -→ C 0 ([0, T ] × E) v -→ (t, x) → sup a∈Az r(t, z, a) + E [v(T n+1 , Z n+1 )|T n = t, Z n = z, a] ,
where r is the instantaneous reward defined as: r(t, z, a) := -f z, a e -λ(z)(T -t) (T -t)1 t>T + f z, a 1 -e -λ(z)(T -t) λ(z) + e -λ(z)(T -t) g(z)1 t≤T , with λ being the intensity of the order arrivals in the order book. More precisely, we show in Section 3.3 of Chapter 3 the following theorem:

Theorem 1.1.2. T a unique fixed point v, and we have v = V .

Numerical contributions:

Chapter 1. Introduction

Figure 1.1 -Histogram of the terminal wealth of the market-maker when the latter follows a naive strategy (light blue), and the Qknn-estimated optimal strategy (dark blue). Notice that the wealth is better when following the Qknn-estimated optimal strategy, by reducing the losses.

Remark 1.1.4. The state space of the market-making is discrete, sparse, large and lies on a space of dimension 12. The regress-now and regress-later methods presented above do not seem to be adapted for this problem since it is hard to choose good basis functions. Local regression that does not rely on basis functions looks to be a wiser choice.

In Section 3.4.4 of Chapter 3, we present the numerical results when using the Qknn algorithm on the market-making problem. Qknn seems to be efficient to estimate the value function. In particular, we observe in Figure 1.1 that the estimate of the optimal strategy does better than a naive marketmaking strategy, which should be seen as a good benchmark. In Figure 1.1, we plot the histogram of the terminal wealth of the market-maker, when the latter follows the Qknn-estimated optimal strategy and the naive strategy: observe that the losses are reduced when using the optimal strategy. Other plots that illustrate the good behavior of Qknn in this problem are available in Section 3.4.4 page 82.

Application of the MDPs to McKean-Vlasov type control problems (Chapter 4)

In Chapter 4, we propose a theoretical method based on Markovian embedding to reduce McKean-Vlasov control problems with partial information to standard finite-dimensional control problems. Numerical results are presented for different McKean-Vlasov problems, where regress-later, regressnow, Qknn and finite difference-based methods are tested and compared. We introduce below the McKean-Vlasov problem with partial information framework, and present our theoretical and numerical contributions.

Presentation of McKean-Vlasov type control problems

We consider a McKean-Vlasov type control problem (MKV) with partial information and common noise: given two independent Brownian motions B and W 0 , defined on a probability space (Ω, F, P), let us consider a stochastic process X which dynamic reads: dX s = b(X s , P W 0 Xs , α s ) ds + σ(X s , P W 0 Xs , α s ) dB s + σ 0 (X s , P W 0 Xs , α s ) dW 0 s , X 0 = x 0 ∈ R n , where P W 0 Xs is the distribution of X s conditioned by F 0 s where F 0 = (F 0 t ) t is the filtration generated by W 0 , and where the control α is a F 0 -progressive process which takes its values on a Polish space A. The measurability condition on the control means that the agent partially observe X, i.e. she only observes the common noise.

Question 1: How can we reduce the MKV control problems to standard control problems?

Answer to Question 1:

Let us make the following two assumptions:

Assumptions on the dependence of the coefficients w.r.t. x. We consider that the drift and the volatility in X are linear w.r.t. the state variable x, i.e. Assumptions about the dependence of the coefficients w.r.t. µ: We assume in the following that the coefficients depend on µ only through its p first moments, i.e.,        b 0 (µ, a) = b0 (μ 1 , μ2 , . . . , μp , a), b 1 (µ, a) = b1 (μ 1 , μ2 , . . . , μp , a) ϑ 0 (µ, a) = θ0 (μ 1 , μ2 , . . . , μp , a), ϑ 1 (µ, a) = θ1 (μ 1 , μ2 , . . . , μp , a) γ 0 (µ, a) = γ0 (μ 1 , μ2 , . . . , μp , a), γ 1 (µ, a) = γ1 (μ 1 , μ2 , . . . , μp , a) f k (µ, a) = fk (μ 1 , μ2 , . . . , μp , a), g k (µ) = ḡk (μ 1 , μ2 , . . . , μp ), k = 0, . . . , p, where, for µ ∈ P p (R), we denote μk =

x k µ(dx), k = 1, . . . , p.

Let Y be the following process Table 1.1 -Estimates of the value function for the systemic risk problem for different parameters; using regress-later (RLMC),regress-now (CR), Qknn, and a finite difference based algorithm. The most accurate algorithm is the one providing the smallest values: Qknn is never beaten! For the sake of simplicity in notations, we assume n = 1, thus obtaining the following dynamic for Y by Itô formula:

Y (k) t = E[X k t |W 0 ], k =
dY (k) t = B k (Y (1) t , Y (2) t , . . . , Y (p) t , α t )dt + Σ k (Y (1) t , Y (2) t , . . . , Y (p) t , α t )dW 0 t , Y (k) 0 = x k 0 , k = 1, . . . , p,
where we denote y 0 = 1, y -1 = 0, B k (y 1 , y 2 , . . . , y p , a) = k b0 (y 1 , . . . , y p , a)y k-1 + k b1 (y 1 , . . . , y p , a)y k + k(k -1) 2 ( θ0 (y 1 , . . . , y p , a)) 2 y k-2 + k(k -1) 2 ( θ1 (y 1 , . . . , y p , a)) 2 y k +k(k -1) θ0 (y 1 , . . . , y p , a) θ1 (y 1 , . . . , y p , a)y k-1 + k(k -1) 2 (γ 0 (y 1 , . . . , y p , a)) 2 y k-2 + k(k -1) 2 (γ 1 (y 1 , . . . , y p , a)) 2 y k +k(k -1)γ 0 (y 1 , . . . , y p , a)γ 1 (y 1 , . . . , y p , a)y k-1 , k = 1, . . . , p, Σ k (y 1 , y 2 , . . . , y p , a) = k (γ 0 (y 1 , . . . , y p , a)y k-1 + γ1 (y 1 , . . . , y p , a)y k ) , k = 1, . . . , p, and the cost functional Y reads:

J(α) = E T 0 f (Y (1) t , Y (2)
t , . . . , Y

t , α t )dt + ḡ(Y

T , Y

T , . . . , Y

T ) .

The McKean-Vlasov type controlled problem with partial information is then reduced to a finitedimensional control problem where the controlled process is now (Y (1) , Y (2) , . . . , Y (p) ), which is an answer to Question 1.

Numerical applications to solve MKV controlled problem We refer to Section 4.4 of Chapter 4 for examples of McKean-Vlasov controlled problem with partial information which are numerically solved using Qknn, regress-now, regress-later, and methods based on finite differences. The numerical results illustrate the accuracy of Qknn.

We consider a portfolio liquidation problem, a portfolio selection problem (see Section 4.4.1.2 page 125); and a systematic risk problem (see Section 4.4.2 page 125). The numerical results are available respectively in 4.1, 4.2 et 4.3 page 128, 131 et 134. To illustrate the efficiency of Qknn, we decided to present here Table 1.1 which is a part of Table 4.3 that can be found in Section 4.4.2.2. In Table 1.1, we tested different algorithms using different parameters to solve the systemic risk problem. The algorithm which gives the smallest result is the most accurate one, and as we can see, Qknn always provides the best estimates.

Neural networks for MDPs

The framework in the second part of this thesis is the same as the one of the first part. We present briefly the framework below, and refer to Section 2.1.1.1 page 30 for more details.

Neural networks for MDPs

Fixing the horizon N , we want to get an estimate of the value function

V n (x) := inf α∈C E N -1 k=n f (X α k , α k ) + g(X α N ) X n = x ,
associated to the MDP X α = (X α n ) N n=0 , which dynamic reads

X α 0 = x 0 ∈ R d , X α n+1 = F n (X α n , α n , ε n+1
), for n = 0, . . . , N -1, where (ε n ) N n=1 is a sequence of i.i.d. noise generating a filtration F = (F n ) N n=0 ; and where C is the set of admissible controls. By dynamic programming principle, V n , the value function at time n, for n = 0, . . . , N , is characterized as solution to the following backward equation 10 :

     V N (x) = g(x), ∀x ∈ X , Q n (x, a) = f (x, a) + P a V n+1 (x), ∀(x, a) ∈ X × A, V n (x) = inf a∈A Q n (x, a)
, for n = N -1, . . . , 0.

(1.2.1)

In the second part of this thesis, we propose to solve MDPs using neural networks. Neural networks being well-suited for high-dimension, we expect our new schemes to be efficient for high-dimensional control problems (i.e. d > > 1). In the next section, we introduce neural networks and gradient descent-based methods, on which our algorithms rely to learn functions of interest.

Representation of functions by neural networks & training

Let L ≥ 1. We define a neural network with L layers as the parameterized function Φ

x ∈ X -→ Φ(z; θ) ∈ R o , with o ∈ N, and θ ∈ Θ ⊂ R p standing for the parameters of Φ, such that Φ is the composition of L "simple" parameterized functions 11 . More precisely, Φ = Φ L , where Φ L is defined by induction as follows:

Φ 1 (x) = σ(A 1 x + b 1 ) Φ n+1 = σ(A n+1 Φ n + b n ) for 1 ≤ n ≤ L -1,
where σ is a monotonic function 12 ; where A 1 , . . . , A L are matrices, and b 1 , . . . , b L vectors, whose dimensions are such that the matrix multiplications and the sums of vectors are well-defined; and where, for all q-dimensional vector y, we used the notation σ(y) = (σ(y 1 ), . . . , σ(y q )) . The matrices A 1 , . . . , A L are referred to as the weights of the neural networks, the b 1 , . . . , b L as their bias, and the weights and bias are the set of parameters for the neural network that we denote by θ above. Generally speaking, after representing a desired function by a neural network Φ, we want to learn the optimal parameters such that Φ approximates the latter. In the second part of this thesis, we need to approximate functions that can minimize objective functions. We will introduce the gradient descent-based algorithms that are well-designed to minimize such objective functions in the next section.

Gradient-descent algorithms

Let us consider the following situation where one has to optimize w.r.t. θ an expectation of the form J (θ) = E[G(X; θ)], with X ∼ µ, (1.2.2)

10 Scheme (1.2.1) is often referred to as the Bellman equation in the literature of the stochastic control.

11 These simple functions are called "layers" in the literature of deep learning. Chapter 1. Introduction where µ is a general probability measure, and where G is a differentiable function such that derivative and expectation symbols are interchangeable. As shown in Example 1.2.1, this situation is met when we train the neural networks to learn the optimal control and the value function of any control problem.

Example 1.2.1. Let us propose a representation of the optimal control at time n by a neural network A(.; β n ) and the value function at time n by a neural network Φ(.; θ n ):

Relying on scheme (1.2.1), the optimal parameter β * n that approximates the optimal control n is such that:

β * n = argmin β E V n+1 (X β n+1 ) + f (X n , A(X n ; β)) ,
and the optimal parameter θ * n whose Φ(.; θ * ) neural network approximates V n , the value function at time n, is such that:

θ * n = argmin θ E V n+1 (X β * n n+1 ) + f (X n , A(X n ; β * n )) -Φ(X n ; θ) 2 ,
where, we denoted by X β n+1 the state at time n + 1 when the feedback control A(.; β) is taken at time n. These two problems are of the same form as (1.2.2).

Gradient-descent

The naive method to optimize J is to apply a deterministic gradient-descenttype algorithm, for which a basic pseudo-code is written in Algorithm 1.2 Algorithm 1.2 Gradient-descent algorithm Input: -Learning parameter η;

-Threshold ε < < 1; -Training set {X m , 1 ≤ m ≤ M } with M > > 1; -θ ∈ R q ; 1: while | ∂θ J (θ)| > ε do 2: Compute ∂θ J (θ) := Ê[∂ θ V (X; θ)],
where Ê is the expectation under the empirical measure 1 M M m=1 δ Xm ;

3:

∂θ J (θ) is an estimate of the derivative of J at point θ.

4:

Re-assign θ := θ -η ∂θ J (θ); 5: end while Output: optimal parameter θ * ; Estimate ∂θ J (θ) gives the direction to take to optimize θ by reducing the gradient, but its computation is heavy and time-consuming, which is why Algorithm 1.2 is intractable. In order to alleviate the computational time, one should consider a more stochastic estimate of the derivative, as proposed in the stochastic gradient descent (SGD) algorithms. We wrote a pseudo-code of simple SGD-based method in Algorithm 1.3. The SGD is much faster since the stochastic estimate of the derivative ∂θ J (θ) is computed almost instantaneously 13 .

The stochastic gradient descent algorithms are designed to optimize J , and the success of neural networks to learn functions of interest arguably comes from them. The most recent algorithms used to optimize J are based on SGD. Adagrad, Adadelta, RMSprop and Adam, are recent algorithms that numerically proved to improve the SGD, mainly by choosing wisely the learning rate in an adaptive way.

Remark 1.2.1. In this thesis, we always used Adam algorithm, natively implemented in TensorFlow, to train neural networks.

Neural networks for MDPs

Algorithm 1.3 Stochastic Gradient Descent (SGD) Input: -Learning rate η; -Threshold ε < < 1; -Number of iterations M > > 1; -Initial parameter θ ∈ R q ;

1: for m = 0, . . . , M do 2: Draw X ∼ µ; 3:

Compute

∂θ J (θ) := ∂ θ V (X; θ); ∂θ J (θ) is an estimate of the derivative of J at point θ.

4:

Re-assign θ := θ -η ∂θ J (θ); 5: end for Output: optimal parameter θ * ;

Neural networks for MDPs

In this section, we propose a short review of the RL literature to solve (1.1.2), relying on neural networks to learn the optimal controls and the value functions at time n = 0, . . . , N -1.

There are two main families of algorithms to solve (1.1.1) using neural network. The first one, which we refer to as the "direct methods", contains the algorithms that only represent the optimal control by neural networks, and highly relies on the recent improvement made to learn optimal parameters on very deep neural networks, i.e. a neural networks with a large number of layers, (see Section 1.2.1). The second family, which we refer to as "Actor-critic" in the sequel, contains all the algorithms that represent the optimal control and the Q-value function by neural networks; and rely on the scheme (1.1.3) to estimate the value function.

Direct methods

The main idea for the algorithms in this family is to represent the controls α n at time n, for n = 0, . . . , N -1; and then regard J(α) as a parameterized expectation (with lot of parameters!) that remains to be optimized, as shown in Section 1.2.1. To be more precise, for all n = 0, . . . , N -1, we represent the control α n of the process (X α n ) at time n, by a neural network

α n = A(.; β n ), (1.2.3)
where β n ∈ R q is the parameter. We then plug these representations in (2.1.1), so that the control problem (1.1.2) reads:

V 0 (x 0 ) = inf β0,...,βn∈R q E N -1 n=0 f (X n , A n (X n , β n )) + g(X N ) , (1.2.4)
where the dynamic of (X n ) is controlled using the neural network representation of the controls (1.2.3). Note that the process should be written (X β n ), but we omit the superscript β for the sake of simplicity. The r.h.s. of (1.2.4) is a parameterized expectation that can be solved, as shown in Section 1.2.1.

Comments on the direct methods

There is no representation of the value function by neural network, which implies that there is no approximation and estimation errors to look after coming from the estimation of the value functions at time n = N -1, . . . , 0.

The method is very unstable, especially when N is getting large and/or the optimal control is a complex function. Indeed: if we represent each optimal control at time n using neural networks Chapter 1. Introduction with 2 layers, then J is a large neural network with 2 × N layers that is hard to train. Deep neural networks remain very difficult to train, even for the most recent SGD-based algorithms; but tricks such as highway and resnet architectures are available to deal with them.

Actor-critic

In the second family of algorithms, we represent the Q-value and the control by neural networks, and we rely on (1.1.3) to derive estimate of the value functions at time n = N -1, . . . , 0. For all n = 0, . . . , N -1, we represent the control by a neural network as follows

α n = A(X n ; β n ),
and represent the Q-value by a neural network as follows

Q(X n , α n ) = Q(X n , A(X n ; β n ); θ n ).
The learning of the optimal parameters β * n and θ * n , for n = 0, . . . , N -1 is performed backward, as shown in the pseudo-code Algorithm 1. 4. One can see that the optimal parameter for the Q-value is learnt at the same time as the optimal parameter for the optimal control, i.e. one cannot learn one without learning the other!14 Algorithm 1. 4 Actor-critic Input: -Learning rate η; -Threshold ε > 0; -Initial parameter θ n , β n ∈ R q , for n = 0, . . . , N -1;

1: Assign V N = g; 2: for n = N -1, . . . , 0 do Minimize w.r.t. θ n the quantity:

E (f (X n , A(X n ; β n )) + Q(X n+1 , A(X n+1 ; β * n+1 ); θ * n+1 ) -Q(X n , A(X n ; β n ); θ n )) 2 ,
and re-assign θ n as its argmin ;

5:

Re-assign ∆θ n as the difference between the new and the old value of θ n ;

6:

Minimize w.r.t. β n the quantity:

E f (X n , A(X n ; β n )) + Q(X n+1 , A(X n+1 ; β * n+1 ); θ n ) ,
and re-assign β n as its argmin ;

7:

Re-assign ∆β n as the difference between the new and the old value of β n ;

8:

end while 9: end for Output: optimal parameters (θ * n ) N -1 n=0 , (β * n ) N -1 n=0 for the approximation of the Q-value and the optimal control at time n = 0, . . . , N -1;

Comments on the Actor-critic methods

The training of the neural networks for the optimal controls and the value functions at time n = 0, . . . , N -1 is done backward; and moreover we train the parameters of the optimal control and the value function at time n by using the trained one at time n + 1.

If the MDP comes from a time-continuous control problem, where the continuity of the value function and the optimal control w.r.t. the time holds, then one can improve the quality of the 1.2. Neural networks for MDPs training by using the pre-training trick15 : i.e. initialize the parameters of the control and the Q-value at time n with the optimal ones that are already computed at time n + 1, and reduce the learning rate η so that the training will lightly and perfectly optimize the parameters.

Relying on an approximation of the Q-value to estimate the value function implies that one has to deal with estimation and approximation errors for the Q-value approximation at time n = N -1, . . . , 0. One shall make sure that this error does not blow up when running the backward procedure of the algorithms.

Contributions (Chapters 5 and 6)

Motivations: As we just saw, there exist already several algorithms to solve control problems. As every algorithms, they have drawbacks and we propose to fix some, in particular we want to improve the fact that the training procedure to learn the optimal parameters of the control and the Q-value must be done at the same time for the Actor-critic algorithms; and that the dynamic programming principle is ignored in the direct methods.

Question 2a: How can we improve the direct methods by relying on the dynamic programming principle? Question 2b: How can we improve the Actor-critic methods by decoupling the training of the optimal parameters for the optimal control and the Q-value at time n = N -1, . . . , 0?

We propose new algorithms in the second part of this thesis: some belong to the direct methods family and answer to Question 2a, and others to the actor-critic family, and answer to Question 2b.

Algorithms to answer Question 2a

We propose NNcontPI and ClassifPI algorithms to answer to Question 1a. We shortly present algorithm NNcontPI here, and refer to page 189 of Chapter 6 for the pseudo-code of ClassifPI algorithm. As it can be seen in Algorithm 1.5, the main idea behind NNcontPI is to represent the optimal control at time n by a neural network A(.; β n ), and then learn the optimal parameter β * n in a backward way, relying on the dynamic programming principle and "training" measures (µ n ) N -1 n=0 : assume that the optimal parameter β * k is already learnt at time k = n + 1, . . . , N -1, we have

βn ∈ argmin β∈R q E f X n , A(X n ; β) + N -1 k=n+1 f X β k , âk X β k + g X β N
where X n is drawn under the training measure µ n and where X β k N k=n+1 is defined by induction:

   X n ∼ µ n , X β n+1 = F X n , A X n ; β , ε n+1 , X β k+1 = F X β k , âk X β k ; β , ε k+1 , for k = n + 1, . . . , N -1.

Comments on NNcontPI

The training of the neural networks is done in a backward way; and the optimal control at time n is learnt based on the optimal parameters that have already been learnt at times n+1, . . . , N -1.

A nice consequence of this feature is that one can use the pretraining trick to highly improve the time and quality of the training.

The learning of the optimal parameter β n at time n relies on the training measures µ k , k = n, . . . , N -1, chosen by the agent. In the case where the agent knows where to drive the process in order to maximize her utility function, then she can choose the training measures to be such as the density is high where the process is likely to go when optimally driven. In the case where the agent does not know where the optimally driven process is likely to go, she can rely on some Chapter 1. Introduction bootstrapping methods, as explained in Section 4.3.3. The choice of the training measure might be a difficult task (especially when the dimension of the control space is high) when the agent has no idea of where to drive the process.

The training of the neural network for the control at time n, using NNcontPI and ClassifPI, requires the re-simulation of the trajectories from time n to time N , as it can be seen in Algorithm 1.5; which implies that these methods are very volatile, especially when N is large, or when the rewards f and g are complex functions.

Algorithm 1.5 NNContPI

Input: Training measures (µ n ) N -1 n=0 ; 1: for n = N -1, . . . , 0 do 2: Compute βn ∈ argmin β∈R q E f X n , A(X n ; β) + N -1 k=n+1 f X β k , âk X β k + g X β N
where X n ∼ µ n and where X β k N k=n+1 is defined by induction:

X β n+1 = F X n , A X n ; β , ε n+1 X β k+1 = F X β k , âk X β k ; β , ε k+1 , for k = n + 1, . . . , N -1.
for m = 1, . . . , M ;

3:

Assign ân = A(.; βn ); ân is an estimate of the optimal control at time n 4: end for Output: (â n ) N -1 n=0 , estimates of the optimal controls at time n, for n = 0, . . . , N -1;

We present in Chapter 5 consistence results for our algorithm NNcontPI, w.r.t. the size of the training and the considered neural network space.

Theorem 1.2.1 (Convergence of NNContPI). Assume that there exist some feedback optimal controls (a opt k ) N -1 k=n , and denote by V n the value functions at time n = 0, . . . , N . Let X n ∼ µ. We then have, as M → ∞16 :

E V M n (X n ) -V n (X n ) = O ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| ,
Moreover, we have when M → ∞17 :

E M V M n (X n ) -V n (X n ) = O P ρ N -n-1 M γ N -n-1 M η N -n-2 M log(M ) M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| ,
where E M is the expectation conditioned by the training set used to estimate the optimal controls (â M k ) n≤k≤N -1 .

Neural networks for MDPs

Since V M n ≥ V n always holds, Theorem 1.2.1 states that the value function at time n, computed using the estimates of the optimal controls at times k = n, . . . , N -1, is close to the value function at time n; and the mean of the difference between these two quantities goes to 0 when the size of the training set goes to infinity and the space of considered neural networks gets large. This result is interesting in particular because it takes into account the size of the training set used to run the SGD in the training procedure.

Algorithmes to answer Question 2b

We propose the algorithms Hybrid-Now, Hybrid-Later and Classif-Hybrid to answer to Question 2b. We refer to Algorithms 6.4 and 6.6, respectively, pages 192 and 209 for the pseudo-codes of Hybrid-Later and Classif-Hybrid, and choose to only present here the Hybrid-Now algorithm, for which the pseudo-code is written in Algorithm 1.6. In the Hybrid-Now algorithm, we represent the optimal control and the value function18 at time n by two (independent) neural networks. As we can see from Algorithm 1.6, the main idea in Hybrid-Now for the training is to learn first the optimal control at time n by minimizing the cost functional; and then rely on a martingale principle for the optimally driven process to learn the value function at time n. Just like the NNcontPI algorithm, Hybrid-Now is based on training measures (µ n ) N -1 n=0 that have to be chosen by the agent, such that the density of the distribution µ n should by high in the region where the optimally driven process is likely to go.

Algorithm 1.6 Hybrid-Now

Input: training distributions (µ n ) N -1 n=0 ; 1: Set VN = g; 2: for n = N -1, . . . , 0 do 3: Compute: βn ∈ argmin β∈R q E f X n , A(X n ; β) + Vn+1 (X β n+1 )
where X n ∼ µ, and X β n+1 = F X n , A(X n ; β) , ε n+1 );

4:

Assign ân = A(.; βn ); ân is an estimate of the optimal control at time n;

5:

Compute θn ∈ argmin

θ∈R p E (f (X n , ân (X n )) + Vn+1 (X βn n+1 ) -Φ(X n ; θ) 2 ; 6:
Assign Vn = Φ(.; θn );

7:
Vn is an estimate of the value function at time n 8: end for Output:

-Estimate of the optimal control (â n ) N -1 n=0 ; -Estimate of the value function ( Vn ) N -1 n=0 ;

Comments on Hybrid-Now

The training of the neural networks to approximate the optimal control and the value function at time n is uncoupled in Algorithm 1.6.

Chapter 1. Introduction

The training of the neural networks at time n is very fast since it only requires the simulation of the controlled process from time n to time n + 119 .

The approximations of the value function, which are used to build the estimates of the optimal controls, implies that additional estimation and approximation errors appear, and need to be looked after, i.e. one should make sure that these errors do not make the algorithm diverge when running the backward procedure.

In Chapter 5, we study the consistence of the algorithms Hybrid-Now and Hybrid-Later, w.r.t. the size of the training set and the neural networks. We decided to only mention below the Theorem 1.2.2 page 158, where we derived the convergence of Hybrid-Now algorithm, .

Theorem 1.2.2 (Consistence of Hybrid-Now). Assume that there exist optimal feedback control, i.e. (a opt k ) N -1 k=n , and denote by V n the value function at time n = 0, . . . , N , and let X n ∼ µ. Then, we have, as M → +∞:

E M | V M n (X n ) -V n (X n )| = O P γ 4 M K M log(M ) M 1 2(N -n) + γ 4 M ρ 2 M η 2 M log(M ) M 1 4(N -n) + sup n≤k≤N inf Φ∈V M E M |Φ(X k ) -V k (X k )| 2 1 2(N -n) + sup n≤k≤N inf A∈A M E |A(X k ) -a opt k (X k )| 1 2(N -n)
, where E M is the expectation conditioned by the training set used to build the estimates (â M k ) n≤k≤N -1 .

Theorem 1.2.2 states that the mean of the difference between the estimate of the value function at time n using HybridNow and the value function at time n cancels when the size of the training set goes to infinity and the space of the neural networks get larger and larger. Once again, an interesting part about the consistence result is that it takes into account the size of the training set.

Numerical Contributions (Chapter 6) Our algorithms are well-designed (or easily customizable) to solve several kind of control problems. This is what we tried to illustrate in Section 6.3 of Chapter 6. In this section, we studied the following different kinds of control problems:

• SemiLinearPDE: a control problem where the state space and the control space are of dimension 100.

• Option Hedging: a control problem where the controlled process is discrete.

• Valuation of Energy Storage: a control problem where the control space is finite, and the state space has continuous and discrete components.

• Smart Grid Management: a control problem where the state space is continuous, and the control admits a continuous and discrete component.

In particular Classif-Hybrid provided excellent results in the Smart Grid Management control problem, introduced in Section 6.3.4 page 204, which we decide to briefly present here. In the Smart Grid Management problem, we consider an agent who have a battery of charge C that can be charged using the received solar energy P , and who has to deliver the energy D to her clients. The agent can choose to activate a generator to deliver herself some energy to charge the battery or meet the demand of her clients at any time. She can also choose to deactivate the battery, however switching cost has to be paid every time when she changes the mode of the generator (switching cost forces her not to switch 1.3. Neural networks for PDEs and BSDEs too many times because it might deteriorate the generator). The agent is penalized when she does not meet the demand D of her clients, and needs to pay for the energy delivered by the generator. In Figure 1.2 below, which comes from Figures 6.11 page 212 and 6.10 page 211, we can see the optimal quantity of energy to get from the generator w.r.t. the residual demand R = D -P ; estimated using Classif-Hybrid and Qknn20 algorithms. In Figure 1.2, the case m = 1 refers to the situation where the generator is already activated right before time 1 and where deactivation would be penalized; the case where m = 0 refers to the situation where the generator is deactivated right before time 1, and where activation at time 1 would be penalized. We can see that the optimal decisions estimated by the two algorithms are very close and smooth. Moreover, the estimated optimal decisions look natural: when the residual demand is high and the battery is empty, i.e. R is large and is C small, it is optimal to activate the generator and buy some energy to meet the demand of the clients; on the other hand, if the residual demand is low and the battery is full: the agent does not need to activate the generator and can meet the demand by using the battery.

When comparing the value function, see Table 6.4 page 211, we can see that Classif-Hybrid does actually better than Qknn!

Perspectives

The quality of the estimates provided by our new algorithms highly depends on the quality of the training measures. The bootstrapping methods can help to build better and better estimates by relying on "adaptive" training measures, but they are not very efficient when the dimension of the problem is high.

It would be interesting not to rely on training measures, or at least to find adaptive methods to build training measures that work well in high-dimension. Notice that it is extremely difficult to have general method to get good training measures. In the community of the RL, this problem is referred to as exploration problem, and research in this field is very active.

Neural networks for PDEs and BSDEs

In the third part of this thesis, we propose and study a family of algorithms to solve PDEs, QVIs, BSDEs, and RBSDEs. We first present the framework, recall the usual and recent methods that are already available in the literature, and finally present our new algorithms. Consistency of our algorithms is proved, and the algorithms are tested on several numerical examples from dimension 1 to 40.

General framework

Let us fix T > 0. Let µ and σ be two functions such that µ :

[0, T ] × R d → R d , and σ : [0, T ] × R d → M d (R), where M d (R)
is the space of the d × d real matrices. µ and σ are assumed to be smooth enough so that there is existence and uniqueness of a solution X to the following SDE:

X t = x 0 + t 0 µ(s, X s ) ds + t 0 σ(s, X s ) dW s , 0 ≤ t ≤ T,
where W is a d-dimensional Brownian motionon a probability space (Ω, F, P) equipped with a filtration F = (F t ) 0≤t≤T which statisfies the usual assumptions.

BSDEs and PDEs

The first problem we tackle in the third part of the thesis is to find a couple of F-adapted processes (Y, Z), which take values in R × R d and such that the following holds:

Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s , 0 ≤ t ≤ T.
(1.3.1) (a) Estimate of the optimal quantity of energy to get from the generator at time n = 1 w.r.t. the couple of state variables (R, C), in the case where m = 0 or m = 1, using the Qknn algorithm. The region below the blue line is the one where it is optimal to turn off (or keep off) the generator. The region above the blue line is the one where it is optimal to deliver some energy from the generator.

Neural networks for PDEs and BSDEs

Equation (1.3.1) is a Backward Stochastic Differential Equation (BSDE) since the terminal condition g is imposed, and the couple (Y, Z) is called solution to the BSDE. It is well-known that (1.3.1) admits a unique solution under some regularity conditions on f and g (see [START_REF] Pardoux | Adapted solution of a backward stochastic differential equation[END_REF]) and that, moreover, we can write Y and Z as follows:

Y t = u(t, X t ), Z t = σ D x u(t, X t ), 0 ≤ t ≤ T,
where u is solution to the following parabolic non-linear partial differential equation:

∂ t u + Lu + f (., ., u, σ D x u) = 0, on [0, T ) × R d , u(T, .) = g, on R d , (1.3.2)
and where L is the infinitesimal generator of X , defined as follows

Lu := 1 2 Tr σσ D 2 x u + µ.D x u.
Hence, finding the solution to the BSDE (1.3.1) is connected to solving the PDE (1.3.2).

RBSDEs and QVIs

The second problem that we tackle is to find the triplet (Y, Z, K) of F-adapted processes solution to the following equation:

Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s + K T -K t , Y t ≥ g(X t ), 0 ≤ t ≤ T, (1.3.3)
where K is an non-decreasing process such that T 0 Y t -g(X t ) dK t = 0.

(1.3.3) is a classical representation of optimal stopping time problems, and appears in particular in the problems of pricing of American options. The Y component of the solution to (1.3.3) reads Y t = u(t, X t ), where it is well-known that u is solution to the Quasi-Variational Inequality (QVI):

min -∂ t u -Lu -f (t, x, u, σ D x u), u -g = 0, t ∈ [0, T ), x ∈ R d , u(T, x) = g(x), x ∈ R d , (1.3.4)
as shown e.g. in [START_REF] Karoui | Reflected Solutions of Backward SDEs, and related obstacle problems for PDEs[END_REF].

The second problem is equivalent to find the solution to the QVI (1.3.4).

Numerical methods available in the literature

When X cannot be simulated, the first step to estimate the solution of BSDEs or RBSDEs is to discretize the process on a time-grid:

π = {t 0 = 0 < t 1 < . . . < t N = T }, with modulus |π| = max i=0,...,N -1 ∆t i , where ∆t i := t i+1 -t i .
The Euler discretization X = X π of X then reads:

X ti+1 = X ti + µ(t i , X ti )∆t i + σ(t i , X ti )∆W ti , i = 0, . . . , N -1, X 0 = x 0 ,
where we denote ∆W ti := W ti+1 -W ti . To alleviate the notations, we denote by X the Euler discretization of X , and omit the dependence of the latter to π. The approximation of (1.3.2) then follows from the Euler discretization of the forward representation (1.3.1):

u(t i+1 , X ti+1 ) ≈ F (t i , X ti , u(t i , X ti ), σ (t i , X ti )D x u(t i , X ti ), ∆t i , ∆W ti ),
where we define:

F (t, x, y, z, h, ∆) := y -f (t, x, y, z)h + z ∆. (1.3.5)
Chapter 1. Introduction

Numerical methods for PDEs and BSDEs

We review in this section the some important algorithms already available in the literature to solve (1.3.1).

Finite-Difference Method

The PDE (1.3.2) can be solved using finite difference method by discretizing the derivative operators on the grids (resp. sparse-grids) when dimension is low (resp. medium). The convergence of finite difference methods is proved under some conditions on the time and space discretization size steps.

Comments on the finite difference methods:

The finite difference-based algorithms impose conditions on the boundaries that are rarely provided in the statement of the control problems (1.3.2). In pratice, one should add some artificial boundary conditions to apply these methods. (see the Benchmark in Section 4.4.2.2 page 132 for an example of application of a finite difference based algorithm to solve the HJB of a control problem, which is a non-linear PDE)

The complexity of these methods is O d+1 , which explodes when dimension gets high! The method is actually adapted for low or medium dimensions.

Regression on basis functions

As presented in [START_REF] Lemor | Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations[END_REF], an idea to estimate the solutions to BSDEs is to use Monte-Carlo regression on basis functions, with the following backward scheme:

   Y t N = g(X t N ) Z ti = 1 ∆ti E Y ti+1 ∆W ti |F ti , Y ti = E Y ti+1 |F ti + f (t i , X ti , Y ti , Z ti )∆t i , for i = N -1, . . . , 0.
At time t i , Z ti is approximated by regressing Y ti+1 ∆W ti at time n on basis functions. Then, Y ti , defined in (2.3.6) as as fixed-point of the contracting (for h small enough) operator L

2 (σ(X ti )) → L 2 (σ(X ti )): Y ti → E i Y ti+1 + hf (t i , X ti , Y ti , Z ti )
, is approximated using Picard iteration technics. The convergence of this algorithm is studied in [START_REF] Lemor | Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations[END_REF].

Comments on methods based on regression on basis functions:

Methods based on regression on basis functions are very fast when dimension is no more than medium (≤ 10) since closed form formulas exist to compute the estimates of the conditional expectations.

Methods based on regression on basis functions are very efficient to solve BSDEs with medium dimensions, but suffer from the curse of dimensionality: when dimension is high, overfitting or underfitting problems appear when regressing quantities on basis functions.

The two methods that we just presented suffer from the curse of dimensionality. Recent methods are available in the literature to solve BSDEs in high-dimension using deep learning.

DGM As introduced in [SS18]

, DGM21 consists of representing the solution to (1.3.2), denoted U(t, x; θ), by neural networks. Let us consider the following (training!) measures: ν T whose support is [0, T ], ν Ω whose support is Ω, ν ∂Ω whose support is ∂Ω; and let us define the following objective function J:

J(θ) = ∂ t U(., .; θ) + LU(., .; θ) + f (t, x, U(., .; θ), σ D x U(., .; θ) [0,T ]×Ω,νΩ + U(T, .; θ) -g(.) ∂Ω,ν ∂Ω ,
where . [0,T ]×Ω,νΩ stands for the canonical norm of

L 2 ([0, T ] × Ω, ν T × ν Ω ), and . [0,T ]×∂Ω,ν ∂Ω stands for the canonical norm of L 2 ([0, T ] × ∂Ω, ν T × ν ∂Ω ).
The main idea in the DGM algorithm for the training of the neural networks is to find the optimal parameters for the neurla network to minimize J by SGD method. When the space of neural networks 1.3. Neural networks for PDEs and BSDEs goes large, the best neural network U(., .; θ * ) almost cancels the objective function, which implies that U(., .; θ * ) is solution to the PDE and meets the boundary conditions, hence is close to the solution u of (1.3.2).

Comments on DGM:

We remind here that the computation of the derivatives of neural networks w.r.t. θ is not time-consuming when relying on reverse back-propapagation.

The fact that the algorithm relies on training measures that are chosen by the agent can be positive when the agent really want to have precise estimates of the solution on some regions, and does not mind about being precise in other regions. Indeed, in this case: the agent can choose training measures whose densities are high at the interesting regions, and low in the other ones.

NNZ NNZ (Neural Network approximation of Z) is introduced in [EHJ17] and the consistency is studied in [START_REF] Han | Convergence of the deep BSDE method for coupled FBSDEs[END_REF]. This algorithm is our main benchmark to solve high-dimensional BSDEs in Chapter 7. The main idea is to approximate Z ti for t i , i = 0, . . . , N -1 using neural networks; and rely on the following formal equalities:

u(t n , X tn ) ≈ u(t n-1 ; X tn-1 ) -f (t n-1 , X tn-1 , u(t n-1 , X tn-1 ), Z(t n-1 , X tn-1 ))(t n -t n-1 ) + Z(t n , X tn ) T (W tn+1 -W tn ),
to represent, by induction, for n = 0, . . . , N , the function u(t n , .) by a neural network U n (., θ) built from the composition of the neural networks representations of Z ti , with i = 0, . . . , n -1. The objective function for NNZ is the function J defined as follows:

J(θ) := E (U N (X N , θ) -g(X N )) 2 .
and the idea is that the optimal controls Z ti , for i = 0, N -1 will be learnt by training U N (X N , θ) such that it cancels the objective function J.

Comments on NNZ:

All the neural networks representing the Z ti are trained at the same time, which implies that the algorithm converges quickly.

There is no approximation and estimation errors coming from the approximation of u. The price to pay for this feature is the time-consuming re-simulations of the process from time k = 0, . . . , N to train the neural networks.

Since the training of the neural networks is done all at the same time, NNZ can actually be seen as an algorithm that performs a training of large neural network with n Z × N layers, where n Z is the number of layers chosen to represent each Z ti . When N is large, the neural network to train is very deep, and the training of such a neural network is still very difficult today. Hence, the risk about NNZ is that the SGD actually converges to a poor local minima, or does not converge at all. The two situations were met in Chapter 7.

Contributions (Chapter 7)

Motivation Because of the curse of dimensionality, solving (1.3.1) and (1.3.3) in high-dimension has always been a difficult problem to tackle for practitioners. Likewise, it is tough to find solution to (1.3.2) and (1.3.4) in high-dimension. The usual methods based on regression on basis functions or finite difference can provide good estimates in low or medium dimension. However, either their complexities explode when dimension gets high, or they suffer from overfitting and underfitting. Methods based on neural networks, such as NNZ and DGM algorithms, seem to be the only ones providing good estimates in high dimension. However, NNZ appears to provide poor estimates of the solution to the BSDE (or even does not converge at all in certain cases) when the number of time steps N is large or when the functions to learn are too complex (see Section 7. As said above: Question 3a and 3a(bis) are related, so are Question 3b and 3b(bis). We introduce in Chapter 7 two new algorithms, that we named DBDP1 and DBDP2, to answer Question 2a; and one new algorithm, named RDBDP to answer Question 2b. Consistence results are derived for all the algorithms, and they are tested and compared to NNZ on several examples. We briefly present the three algorithms in the next section, and highlight the important consistence results that have been derived.

Presentation of DBDP1 and consistency result

The main idea in DBDP1 algorithm is to approximate, for i = N -1, . . . , 0, the couple (u(t i , .), Z(t i , .)) by neural networks:

u(t i , .) = U i (.; θ) Z(t i , .) = Z i (.; θ),
where we denote by θ the parameters of the two neural networks U i and Z i . We learn the optimal parameters θ * i , i = 0, . . . , N -1, which is the optimal one for U i and Z i to be close to u(t i , .) and Z ti , in a backward way, relying, as written in the pseudo-code in Algorithm 1.7, on the following equality:

U i+1 (X ti+1 ; θ * i+1 ) ≈ F (t i , X ti , U i (X ti ; θ * i ), Z i (X ti ; θ * i ), ∆t i , ∆W ti ),
where θ * i+1 is the optimal parameter of U i+1 and Z i+1 for the approximation of u(t i+1 , .) and Z ti , and is assumed to be already computed at time t i+1 , and where F was introduced in (1.3.5).

Algorithm 1.7 DBDP1 Algorithm Input:

1: Initialize U (1) N = g 2: for i = N -1, . . . , 0 do 3: Compute: θ * i ∈ argmin θ∈R Nm E U (1) i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 ; 4: Assign U (1) i = U i (.; θ * i ); 5: Assign Z (1) i = Z i (.; θ * i ); 6: end for Output: -( U (1) i ) N i=0 : Estimates of Y ti for i = 0, . . . , N ; -( Z (1) i ) N i=0 : Estimates of Z ti for i = 0, . . . , N ;
The convergence of DBDP1 is studied in Chapter 7 (see Theorem 7

E ( U (1) , Z (1) ), (Y, Z) ≤ C E g(X T ) -g(X T ) 2 + |π| + ε Z (π) + N -1 i=0 N ε N ,v i + ε N ,z i ,
where:

E ( U (1) , Z (1) ), (Y, Z) := max i=0,...,N -1 E Y ti -U (1) i (X ti ) 2 + E N -1 i=0 ti+1 ti Z t -Z (1) i (X ti ) 2 dt
should be seen as a L 2 -error between the estimates ( U

(1) i ) N i=0 and ( Z (1)
i ) N i=0 provided by DBDP1 and the real solutions u(t i , .) and Z, for i = 0, . . . , N , and where:

• ε N ,v i := inf ξ E vi (X ti ) -U i (X ti ; ξ) 2 , ε N ,z i := inf η E ẑi (X ti ) -Z i (X ti ; η)
2 should be seen as approximation errors coming from the approximation of functions using neural networks.

•

ε Z (π) := E N -1 i=0 ti+1 ti |Z t -Zti | 2 dt , with Zti := 1 ∆t i E i ti+1 ti Z t dt is the L 2 -regularity de Z 22
Theorem 1.3.1 states that the L 2 -error made by DBDP1 algorithm when estimating Y and Z is bounded by the sum of:

• a time-discretization of [0, T ] error,

• an Euler-discretization of X error,

• the L 2 -regularity of Z,

• the quantity

N -1 i=0 N ε N ,v i + ε N ,z i
that should be seen23 as the sum of approximation errors that are made by approximating Y and Z in a backward way.

By standard arguments and under standard assumptions, we can show that these errors all vanish when the time-discretization step size goes to zero, and the space of considered neural networks gets larger.

Presentation of DBDP2, and consistency result

One may have noticed in the pseudo-code of DBDP1 that we ignored the well-known result: Z = σ (t i , .)D x u(t i , .), by representing Z by the neural networks Z i (.; θ). Indeed, since Z = σ (t i , .)D x u(t i , .), one can represent Z by the derivative of U i (.; θ). This is the main underlying idea of DBDP2.

Let us consider a neural network U to represent u. We then have the following representation for the couple (Y, Z):

u(t i , .) = U i (.; θ) Z(t i , .) = σ (t i , .)D x U i (.; θ), Chapter 1. Introduction
where θ is the parameter associated to U. We learn the optimal parameter θ * i , such that U N is close to u(t i , .), and σ (t i , .)D x Ûi (.; θ) close to Z(t i , .), in a backward way, relying, as we can see in the pseudo-code of DBDP2 written in Algorithm 1.8, on the following equality:

U i+1 (X ti+1 ; θ * i+1 ) = F (t i , X ti , U i (X ti ; θ * i ), σ (t i , X ti )D x U i (X ti ; θ * i ), ∆t i , ∆W ti ).
Algorithm 1.8 DBDP2 Algorithm Input:

1: Initialize U (1) N = g 2: for i = N -1, . . . , 0 do 3: Compute: θ * i ∈ argmin θ∈R Nm E U (2) i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), σ (t i , X ti ) Dx U i (X ti ; θ), ∆t i , ∆W ti ) 2 ; 4: Assign U (1) i = U i (.; θ * i ); 5: Assign Z (1) i = σ (t i , .)D x U i (.; θ * i ); 6: end for Output: -( U (1) i ) N i=0 : Estimates of Y ti for i = 0, . . . , N ; -( Z (1) i ) N i=0 : Estimates of Z ti for i = 0, . . . , N ;
Remark 1.3.1. The complexity to compute ∇U i (.; θ) and U i (.; θ) is roughly the same, when using reverse back-propagation, as natively implemented in Tensorflow. The computation of ∇U i (.; θ) when running DBDP2 is then not time-consuming.

The convergence of DBDP2 is studied in Chapter 7 (see Theorem 7.4.2 page 230), and we present here the simplified version of the result: Theorem 1.3.2. (Consistency of DBDP2) Under usual regularity assumptions, there exists a constant C > 0, independent of π such that24

E ( U (2) , Z (2) ), (Y, Z) ≤ C E g(X T ) -g(X T ) 2 + γ 6 m N + ε Z (π) + N N -1 i=0 ε N ,m i .
Theorem 1.3.1 states that the L 2 -norm of the error of estimation of u and Z by DBDP2 algorithm is bounded by the sum of

• a time-discretization of [0, T ] error,
• an Euler-discretization of X error,

• the quantity γ 6 m N which appears because we had to control the size of the space of neural network, • the L 2 -regularity of Z,

• the quantity

N -1 i=0 N ε N ,v i + ε N ,z i
that should be seen25 as the sum of approximation errors that are made by approximating Y and Z in a backward way.

All these errors vanish when the time-discretization step goes to zero, and the size of the neural network gets large.

Neural networks for PDEs and BSDEs

Presentation of RDBDP, and consistency result

The main idea of the RDBDP algorithm is to approximate the pair (u(t i , .), Z(t i , .)), for i = N -1, . . . , 0, by neural networks in the following way:

u(t i , .) = U i (.; θ) Z(t i , .) = Z i (.; θ),
where we denote by θ the parameters of the two neural networks; and rely on the scheme described in Algorithm 1.9 to learn u(t i , .) et Z(t i , .), for i = N -1, . . . , 0.

Algorithm 1.9 RDBDP Algorithm Input:

1: Initialize U (1) N = g 2: for i = N -1, . . . , 0 do 3: Compute: θ * i ∈ argmin θ∈R Nm Li (θ),
where

Li (θ) := E U i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 4:
Assign U i = max U i (.; θ * i ), g];

5:

Assign Z i = Z i (.; θ * i ); 6: end for Output: -( U (1) i ) N i=0 : Estimates of Y ti for i = 0, . . . , N ; -( Z (1) i ) N i=0 : Estimates of Z ti for i = 0, . . . , N ;
The convergence of RDBDP is studied in Chapter 7 (see Theorem 7.4.4 page 235), and we present here the simplified version of the result: Theorem 1.3.3. (Consistency of RDBDP) Under usual assumptions on the regularity of the coefficients of the RBSDE, there exists a constant C > 0, independent of π, such that

E ( U , Z ), (Y, Z) ≤ C ε(π) + N -1 i=0 N ε N ,ṽ i + ε N ,z i , where ε(π) = O(|π| 1 2 ) if g is smooth, and ε(π) = O(|π|) if g and σ are smooth.

Numerical results

Numerical tests of our new algorithms DBDP1, DBDP2 and RDBDP are presented in Section 7.5 of Chapter 7. Our algorithms provided some very accurate results. In particular DBDP1 and DBDP2 appeared to be more stable than NNZ, which was taken as a benchmark in all the considered examples. Our algorithms converge as well as NNZ when the latter converges, and converge in some cases where NNZ either converges to poor estimates or diverges. We noticed that NNZ does not converge when the functions u and Z to learn are too complex, or N is too large. We refer to Section 7.5.1 page 237 for examples on which DBDP1, DBDP2 and NNZ work well; and Section 7.5.2 page 241 for examples where DBDP1 et DBDP2 work well, but NNZ diverges. Finally we did not observe any situations where NNZ converges whereas any of our algorithms fail to converge.

The RDBDP algorithm also appeared to work well in the considered application: we consider a problem of pricing an American option from dimension 1 to 40. Part of the results are available in Table 1.2, and the whole results are available in Table 7.11 page 245. For the payoff that we considered, the American put pricing problem can actually always be reduced to a problem of dimension 1, and

Perspectives

The results of convergence proved for our new algorithms are interesting, but it would be more natural to have the quantity

N -1 i=0 ε N ,ṽ i instead of N -1 i=0 N ε N ,ṽ i
in the bounds proposed in Theorems 1.3.1, 1.3.2, and 1.3.2 (Note that the N disappeared). Indeed, the quantity

N -1 i=0 ε N ,ṽ i
is exactly the error made by approximating some functions using neural networks at each time step, and the N that we had to add in our results is just an artefact. Adding the N to get consistency results was the best we could do.

Chapter 2

Introduction (in French)

La thèse se divise en trois parties que l'on peut lire indépendamment. Dans la première partie, nous proposons des algorithmes basés sur des méthodes de quantification et de régression pour résoudre numériquement les processus de décisions markoviennes (MDPs). Puis nous présentons deux applications : la première porte sur un problème de market-making, et la deuxième sur des problèmes de contrôle stochastique de type McKean-Vlasov1 . Dans la seconde partie de cette thèse, nous présentons une famille de schémas numériques basée sur les réseaux de neurones pour résoudre les MDPs. Nous étudions d'abord la convergence théorique de nos nouveaux schémas en fonction de la taille du training set et de l'espace des réseaux de neurones considéré ; puis les testons sur des exemples variés issus de la littérature. Dans la troisième partie de cette thèse, nous présentons une classe de schémas numériques, à base de réseaux de neurones, pour résoudre numériquement les EDPs, IQVs, EDSRs et EDSRs réfléchies. Nous avons obtenu des résultats théoriques de convergence des schémas en fonction de l'erreur d'approximation des réseaux de neurones, et avons illustré leurs performances sur de nombreux tests numériques. Après avoir introduit les problèmes de décisions Markoviennes et rappelé quels sont les algorithmes qui existent aujourd'hui dans la littérature pour résoudre numériquement les MDPs, nous présentons un nouvel algorithme basé sur la regression locale et la quantification, qui est proposé et étudié dans la première partie de cette thèse.

Présentation des MDPs

Un problème de contrôle stochastique à temps discret et horizon fini est la donnée d'un horizon fini

N ∈ N \ {0}, et d'un processus (dit contrôlé) X α = (X α n ) N n=0
, à valeurs dans X ⊂ R d , avec d ∈ N la dimension du problème de contrôle, qui admet la dynamique suivante :

X α 0 = x 0 ∈ R d , X α n+1 = F n (X α n , α n , ε n+1 ), pour n = 0, . . . , N -1, où : • (ε n ) N
n=1 est une suite de v.a. supposées i.i.d., à valeurs dans un espace de Borel (E, B(E)), et définie sur un espace de probabilité (Ω, F, P) muni de la tribu

F = (F n ) N n=0 engendrée par le bruit (ε n ) N n=1 (F 0 est la tribu triviale),
• le contrôle α est un processus F-mesurable à valeur dans A ⊂ R q ,

• pour n = 0, . . . , N -1, les F n sont des fonctions mesurables de

R d × R q ×E dans R d .
Étant donné un facteur d'actualisation β ≥ 0, une fonction de coût instantané f : R d × R q → R, et une fonction de coût terminal g : R d → R, on définit J(α), la fonctionnelle de coût associée au processus de décision α, de la manière suivante :

J(α) = E N -1 n=0 β n f (X α n , α n ) + g(X α N ) . (2.1.1)
On dit qu'un contrôle α est admissible si la fonction J(α) est bien définie, et l'on note dans la suite C l'ensemble des contrôles admissibles. Le problème de contrôle se formule de la manière suivante : d'abord trouver

V 0 (x 0 ) := inf α∈C J(α), (2.1.2)
puis caractériser (au moins) une stratégie optimale α * ∈ C, définie comme étant une stratégie qui minimise la fonctionnelle de coût, i.e. V 0 (x 0 ) =: J(α * ).

Dans cette thèse on parlera de MDP à horizon fini, ou plus simplement de MDP, pour désigner les problèmes de contrôle à horizon fini et temps discret.

Observons qu'une première difficulté dans la résolution de (2.1.2) est que l'infimum est pris sur l'espace de dimension infinie des processus admissibles C. En utilisant un principe de programmation dynamique, on peut réécrire (2.1.2) de manière plus simple. Pour se faire, introduisons d'abord les notations standards suivantes : on note par {P a (x, dx ), a ∈ A, x ∈ X }, la famille des probabilités de transition associée à la chaîne de Markov contrôlée (2.1.1.1), i.e.

P a (x, dx ) = P [F (x, a, ε 1 ) ∈ dx ] ,
2.1. Quantification et régression pour les MDPs. Application au market-making et aux problèmes de contrôle de type McKean-Vlasov avec information partielle et de plus on note

P a ϕ(x) = ϕ(x )P a (x, dx ) = E ϕ F (x, a, ε 1 ) ,
pour toute fonction mesurable ϕ : X → R. Avec ces notations, pour toute fonction ϕ sur X , et pour tout contrôle α ∈ C, on a :

E[ϕ(X α n+1 )|F n ] = P αn ϕ(X α n ), ∀ n ∈ N.
Par un principe de programmation dynamique, la fonction valeur à l'instant n, pour n = 0, . . . , N , que l'on note V n , est caractérisée comme solution du schéma rétrograde suivante2 :

     V N (x) = g(x), ∀x ∈ X , Q n (x, a) = f (x, a) + βP a V n+1 (x), ∀x ∈ X , a ∈ A, V n (x) = inf a∈A Q n (x, a), pour n = N -1, . . . , 0,
(2.1.3) et de plus, le contrôle feedback optimal au temps n, noté α * n , est caractérisé de la manière suivante :

α * n (x) ∈ argmin a∈A Q n (x, a). (2.1.4)
La fonction Q n est communément appelé la "Q-value" ou encore "state-action function" dans la communauté de l'apprentissage par renforcement (RL), et V n est la fonction valeur au temps n.

Remark 2.1.1. Le schéma (2.1.3) est très général dans le sens où il peut approximer la solution de problème à horizon fini ou infini, à temps continu ou discret. Par exemple, considérons le problème à horizon infini et temps continu suivant :

V (x) := sup α∈C E ∞ 0 e -λt f (X α t )dt ,
avec λ > 0, et où l'on suppose que le processus contrôlé X α par un processus admissible α admet la dynamique suivante : dX α t = b(t, X t , α t )dt + σ(t, X t , α t )dW t , avec b et σ assez régulières. Faisons une discrétisation en temps : en notant h < < 1 le pas de discrétisation en temps, on obtient l'approximation suivante :

V (x) ≈ V h (x) := sup α∈C E ∞ n=0 he -λnh f (X α nh ) ,
où la dynamique de la chaîne de Markov contrôlée (X α nh ) n∈N s'écrit :

X α 0 = X 0 X α h(n+1) = b(nh, X α nh , α nh )h + σ(nh, X nh , α nh ) √ hε n+1 , avec (ε n ) n∈N une suite i.i.d. de loi normale centrée réduite N (0, 1).
On reconnaît un problème à temps discret et horizon infini. Sous des hypothèses standards3 , on montre qu'alors V h est l'unique point fixe de l'opérateur

T : C 0 (R) -→ C 0 (R) v -→ x -→ sup a∈A r(x) + e -λh E [v(X a h )|X 0 = x] ,
où l'on a posé r(x) := hf (x). Le théorème du point fixe de Picard montre que l'on peut approximer le point fixe de T en itérant un nombre de fois N assez grand l'opérateur T en partant de la fonction Chapter 2. Introduction (in French)

nulle. En utilisant cet argument, on montre que V 0 , défini dans le schéma (2.1.5) ci-dessous, converge vers le point fixe de T quand N tend vers l'infini.

V N = 0 V n (x) = sup a∈A r n (x) + e -λh E [V n+1 (X a h )|X 0 = x] , for all state x (2.1.5)
On reconnaît alors le schéma rétrograde (2.1.3), avec β = e -λh , g = 0, f = r n .

Remark Le problème (2.1.3) est en théorie beaucoup plus simple à résoudre que (2.1.2) puisque l'infimum est maintenant pris sur A qui est un sous-ensemble R q , de dimension finie, et nous avons de plus la caractérisation pratique (2.1.4) du contrôle optimal qui accompagne (2.1.3). Pour autant, dans la plupart des cas, l'équation de Bellman (2.1.3) pose encore deux problèmes : PB1 l'espérance conditionnelle P a qui apparaît à chaque pas de temps dans l'expression de Q n de

(2.1.3) n'admet généralement pas d'expression explicite.

PB2 l'infimum dans l'expression de V n de (2.1.3) n'admet généralement pas de formule fermée, et une approximation de cette dernière peut se révéler très coûteuse en temps de calculs (resp. impossible), dans le cas où le cardinal de l'espace d'états est très grand (resp. infini).

2.1.1.2 Algorithmes issus de la littérature pour les équations de Bellman (Chapitre 4)

Pour résoudre l'équation de Bellman, nous nous sommes intéressés dans la première partie de la thèse à des méthodes basées sur la régression et la quantification.

Regress-Now sur une base de fonctions La méthode regress-now sur une base de fonctions consiste à d'abord randomiser le contrôle, i.e. simuler le contrôle à partie d'un processus aléatoire exogène, puis à régresser la fonction valeur au temps n + 1 sur la base de fonctions au temps n pour estimer la Q valeur au temps n. Il suffit alors d'optimiser la Q valeur au temps n pour obtenir une estimée de la fonction valeur à la date n. Nous référons à [KLP14], [START_REF] Carmona | Valuation of energy storage: an optimal switching approach[END_REF] et aux travaux plus récents dans [START_REF] Ludkovski | Simulation methods for stochastic storage problems: a statistical learning perspective[END_REF] pour des méthodes basées sur la méthode regress-now pour résoudre numériquement des problèmes de contrôle. L'une des principale difficulté lorsque l'on cherche à appliquer un algorithme du type regress-now est le choix de la base de fonctions : plus la dimension du problème de contrôle est grande, moins il est facile de trouver une bonne base de fonctions pour régresser la fonction valeur en évitant le sous ou sur-apprentissage. Pour pallier à ce problème, une idée développée dans [START_REF] Belomestny | Optimal stopping via reinforced regression[END_REF] consiste à adapter la base de fonctions B n sur laquelle régresser V n+1 , la fonction valeur au temps n + 1, au temps n, en ajoutant des fonctions du type V n+1 dans B n . En faisant ainsi, l'estimation de l'espérance conditionnelle est meilleure dans le cas courant où la MDP est issue d'un problème de contrôle en temps continu dans lequel on a la continuité de la fonction valeur par rapport au temps.

Regress-Later La méthode regress-later consiste à d'abord interpoler la fonction valeur au temps n + 1 sur une base de fonctions, puis à régresser l'interpolation

Motivation

Il est surprenant de constater qu'aujourd'hui, beaucoup des algorithmes issus de la littérature sont basés sur des méthodes de régression Monte Carlo, i.e. opèrent une régression globale. En moyenne dimension, il est clair que c'est la seule méthode qui marche numériquement. Mais en petite dimension, d'autres méthodes de régression, de type locales, sont disponibles dans la littérature de statistique, et adaptables aux problèmes de contrôle comme détaillé par exemple dans [START_REF] Belomestny | Regression methods for stochastic control problems and their convergence analysis[END_REF]. Aussi, les méthodes de quantification semblent adaptées pour résoudre les MDPs, comme montré dans [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF]. Dans la première partie de cette thèse, nous voulons mixer les méthodes de régression locale avec les méthodes de quantification, dans le but de pouvoir résoudre des MDPs en moyenne dimension.

L'algorithme que nous proposons repose sur une méthode de régression locale, et semble adaptée au problème de market-making présenté à la Section 2.1.2.

Contributions

Nous proposons pour résoudre le problème de market-making, un algorithme baptisé Qknn, basé sur des techniques de quantification de variables aléatoires, pour approximer les espérances conditionnelles, et de régression locale (i.e. projection sur les plus proches voisins). La régression locale en utilisant les plus proches voisins a déjà été proposée pour résoudre des problèmes de contrôle, comme présenté par exemple dans [START_REF] Belomestny | Regression methods for stochastic control problems and their convergence analysis[END_REF]. Comme on peut le voir dans le pseudo-code de Qknn, écrit dans Algorithme 2.1 : à chaque pas de temps n, l'agent choisit une famille finie de points appelé "training points" sur lesquels estimer la fonction valeur au temps n. Le calcul de l'espérance conditionnelle est fait en quantifiant le bruit exogène, i.e. en approximant la variable exogène de bruit ε n+1 par une variable aléatoire εn+1 qui prend un nombre d'états fini e 1 , . . . , e avec probabilité p 1 , . . . , p ; choisie de telle manière que la norme L 2 de la différence entre ε n+1 et εn+1 soit aussi petite que possible 5 .

Nous référons au Chapitre 3 pour une introduction plus développée de Qknn et de ses variantes. Dans ce même chapitre, nous présentons aussi le résultat de convergence obtenu pour Qknn (voir Théorème 3.4.1 page 78), dont nous proposons ci-dessous une version simplifiée : Theorem 2.1.1. En prenant K = M 2+d points pour la quantification du bruit exogène ε n , n = 1, . . . , N , et sous des hypothèses de régularité des coefficients, il existe une constante C > 0 telle que pour n = 0, ..., N -1,

V Q n (X n ) -V n (X n ) 2 ≤ C N k=n+1 ε proj k + ε Q k + O 1 M 1/d , quand M → +∞, où ε Q k := εk -ε k 2 est l'erreur de quantification et ε proj n := sup a∈A Proj n+1 (F (X n , a, εn )) -F (X n , a, εn ) 2
est l'erreur de projection quand la décision a est prise au temps n.

Le théorème 2.1.1 stipule que la norme L 2 de la différence entre l'estimation par Qknn de la fonction valeur et la fonction valeur est bornée par une quantité proportionnelle à la somme des erreurs de projection et des erreurs de quantification faites à chaque pas de temps. Ces deux types erreurs tendent vers 0 lorsque, respectivement, le nombre de point des grilles de projections et de quantification tendent vers l'infini.

Chapter 2. Introduction (in French) Algorithm 2.1 Algorithme Qknn Entrée :

• Γ k , k = 0, . . . , N : grille de training points R d ,

• Γ = {e 1 , . . . , e L } , (p ) 1≤ ≤L : une L-grille optimale pour la quantification du bruit exogène ε, et ses poids associé.

1: Poser N : V Q ∆t (t N , z) = g(z), ∀z ∈ Γ N . 2: for n = N -1, . . . , 0 do 3:
Estimer la fonction valeur à l'instant n : Dans le Chapitre 3, on note Z ce que l'on a appelé le "carnet d'ordres contrôlé", et qui est le processus contrôlé qui comprend l'ensemble les ordres limites de tous les participants ainsi que les ordres limites du market-maker. Z est un processus ponctuel qui vit dans un espace d'états discret que l'on appelle E lorsque l'on considère que les ordres limites ne sont pas divisibles.

V Q ∆t (t n , z) = max a∈A f (z, a) ∆t + L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t (z,
Le but du market-maker est de suivre une stratégie de market-making, dite optimale, qui maximise la quantité suivante :

V (t, z) = sup α∈A E α t,z T t f α s , Z s ds + g Z T , (t, z) ∈ [0, T ] × E,
où g est la récompense terminale (par exemple la richesse à la date terminale T > 0 du market-maker plus une pénalisation de l'inventaire terminal) ; et où f est la récompense instantanée (par exemple l'aversion au risque du market-maker).

Contributions

Contributions théoriques : Dans la Section 3.3.2 du Chapitre 3, nous montrons6 que le problème de market-making en temps continu (le processus contrôlé Z et la fonction valeur V ) peut être réécrit comme un Processus de Décisions Markoviennes (MDP). Cette réécriture permet alors 7 de caractériser V , la fonction valeur du problème de market-making en temps continu, comme point fixe de l'opérateur T défini ainsi :

T : C 0 ([0, T ] × E) -→ C 0 ([0, T ] × E) v -→ (t, x) → sup a∈Az r(t, z, a) + E [v(T n+1 , Z n+1 )|T n = t, Z n = z, a] ,
où r est la récompense définie ci-dessous : Dans la section 3.4.4 du Chapitre 3, nous présentons les résultats numérique de la résolution du problème de market-making en utilisant l'algorithme Qknn. Ce dernier semble adapté à ce problème, et en particulier, on observe qu'il fait mieux qu'une stratégie naïve de market-making, qui doit être vue comme un très bon benchmark. Nous illustrons dès maintenant le bon comportement de Qknn en proposant la figure 2.1, extraite du Chapitre 3, dans laquelle on a tracé l'histogramme de la richesse terminale du market-maker (i.e. son P&L), lorsque celui-ci suit soit la stratégie optimale estimée par Qknn, soit une bonne stratégie naïve. On observe que Qknn améliore la richesse clairement la richesse du market-maker, en minimisant les pertes.

r(t, z, a) := -f z, a e -λ(z)(T -t) (T -t)1 t>T + f z, a 1 -e -λ(z)(T -t) λ(z) + e -λ(z)(T -t) g(z)

Application des

dX s = b(X s , P W 0 Xs , α s ) ds + σ(X s , P W 0 Xs , α s ) dB s + σ 0 (X s , P W 0 Xs , α s ) dW 0 s , X 0 = x 0 ∈ R n , où P W 0

Xs

est la distribution de X s conditionnée à F 0 s où F 0 = (F 0 t ) t est la filtration générée par W 0 , et où le contrôle α est un processus F 0 -progressif à valeur sur un espace polonais A. La condition de mesurabilité sur le contrôle signifie que l'agent n'observe que partiellement le processus, i.e. il observe le bruit commun.

Nous faisons les hypothèses Lipschitz standards sur les coefficients b(x, µ, a), σ(x, µ, a), σ 0 (x, µ, a) par rapport à (x, µ) dans R n ×P 2 (R n ), uniformement dans a ∈ A, où P 2 (R n ) est l'espace des mesures de probabilité dans (R n , B(R n )) avec un second moment fini, et muni de la distance de Wasserstein W 2 . Cela implique que le MKV contrôlé défini dans (4.1.1) est bien défini. La fonctionnelle de coût, à horizon fini T associée au contrôle α est donnée par :

J(α) = E T 0 f (X t , P W 0 X t , α t )dt + g(X T , P W 0 X T
) , et l'objectif est de maximiser cette dernière sur un ensemble de contrôles admissibles A : Nous faisons deux types d'hypothèses sur les coefficients du modèle : la première est sur la dépendance en x, et l'autre sur la dépendance en µ. Hypothèses sur la dépendance en x. On considère que le drift et les volatilités dans la dynamique de X sont linéaires par rapport à la variable d'état x, i.e. Soit Y le processus suivant :

V 0 = sup α∈A J(α). 2
Y (k) t = E[X k t |W 0 ], k = 1, . . . , p.
En supposant pour simplifier que n = 1 et en utilisant la formule d'Itô et sous les hypothèses ci-dessus, on obtient les formules suivantes sur la dynamique de Y dY

(k) t = B k (Y (1) t , Y (2)
+ k(k -1) 2 (γ 0 (y 1 , . . . , y p , a)) 2 y k-2 + k(k -1) 2 (γ 1 (y 1 , . . . , y p , a)) 2 y k +k(k -1)γ 0 (y 1 , . . . , y p , a)γ 1 (y 1 , . . . , y p , a)y k-1 , k = 1, . . . , p,
Σ k (y 1 , y 2 , . . . , y p , a) = k (γ 0 (y 1 , . . . , y p , a)y k-1 + γ1 (y 1 , . . . , y p , a)y k ) , k = 1, . . . , p, et la fonctionnelle de coût associée à Y s'écrit :

J(α) = E T 0 f (Y (1) t , Y (2)
t , . . . , Y

t , α t )dt + ḡ(Y

T , Y

T , . . . , Y

T ) .

Le problème de type McKean-Vlasov est ainsi réduit à un problème fini-dimensionnel où le processus contrôlé est maintenant (Y (1) , Y (2) , . . . , Y (p) ), ce qui répond à la Question 3.

Applications numériques à des problèmes de contrôle MKV avec informations partielles

Nous référons à la Section 4.4 du Chapitre 4 pour des exemples de problèmes de contrôle de type McKean-Vlasov et d'informations partielles que nous avons résolus en utilisant différents algorithmes comme Qknn, regress-now, regress-later, ainsi que des méthodes de différence finie. Ce travail a permis notamment d'illustrer le haut potentiel de Qknn8 pour résoudre des problèmes de contrôle en petite dimension.

Nous avons considéré un problème de liquidation de portefeuille, un problème de sélection de portefeuille (voir section 4. 4

Réseaux de neurones pour les MDPs

Le cadre dans lequel on se place dans la seconde partie de la thèse est le même que celui de la première partie. Nous référons à la Section 2.1.

V n (x) := inf α∈C E N -1 k=n f (X α k , α k ) + g(X α N ) X n = x ,
où l'horizon N est fixé, où le processus contrôlé X α = (X α n ) N n=0 admet la dynamique suivante

X α 0 = x 0 ∈ R d , X α n+1 = F n (X α n , α n , ε n+1 ), pour n = 0, . . . , N -1, avec (ε n ) N n=1 le bruit i.i.d. qui génère la filtration F = (F n ) N n=0
; et où C est l'ensemble des contrôles admissibles. Par un principe de programmation dynamique, la fonction valeur à l'instant n, pour n = 0, . . . , N , notée V n , est caractérisée comme solution du schéma rétrograde suivant :

     V N (x) = g(x), ∀x ∈ X , Q n (x, a) = f (x, a) + P a V n+1 (x), ∀x ∈ X , a ∈ A, V n (x) = inf a∈A Q n (x, a), pour n = N -1, . . . , 0,
(2.2.1)

Nous proposons dans la seconde partie de la thèse de résoudre le problème de contrôle en utilisant des méthodes à base de réseaux de neurones. Les réseaux de neurones étant les meilleurs pour apprendre des fonctions en grande dimension, on s'attend à ce que ces méthodes soient très adaptées à ce cadre, i.e. à des dimensions d > > 1. Dans la section qui suit, nous introduisons les réseaux de neurones considérés, ainsi que la méthode de descente de gradient, fondamentale pour entraîner ces derniers.

Représentation de fonctions par réseaux de neurones & apprentissage

Soit L ≥ 1. On définit un perceptron à L couches comme la fonction paramétrée Φ :

x ∈ X -→ Φ(z; θ) ∈ R o ,
qui est la composée de fonctions simples paramétrées9 , avec o ∈ N, et où θ ∈ Θ ⊂ R p est appelé le paramètre du perceptron Φ. Formellement, Φ = Φ L , où Φ L est définie par récurrence : Chapter 2. Introduction (in French)

Φ 1 (x) = σ(A 1 x + b 1 ) Φ n+1 = σ(A n+1 Φ n + b n ) for 1 ≤ n ≤ L -
et somme de vecteurs soient bien définis ; et où pour tout vecteur y q-dimensionnel, on a utilisé la notation σ(y) = (σ(y 1 ), . . . , σ(y q )) . Les matrices A 1 , . . . , A L sont appelés les poids du réseaux de neurones et les b 1 , . . . , b L sont les biais. L'ensemble des poids et des biais forment les paramètres, que l'on note θ, du perceptron à L couches.

Les réseaux de neurones considérés dans cette thèse sont les perceptrons multicouches. De manière générale : après avoir représenté la fonction d'intérêt par un réseau de neurones, nous devons apprendre les paramètres du réseau pour que ce dernier approxime au mieux la fonction d'intérêt. Dans la deuxième partie de cette thèse, l'approximation de la fonction d'intérêt passe par la minimisation d'une fonction objective définie sous forme d'une espérance. Nous introduisons dans la prochaine section les méthodes de descente de gradient stochastique (SGD) qui ont été créées pour ce type de tâches. Ces dernières sont très performantes et contribuent pour beaucoup au succès des réseaux de neurones en grande dimension.

Les méthodes d'optimisation d'espérance à paramètres

Considérons le problème qui consiste à minimiser par rapport à un paramètre θ l'espérance à paramètre suivante :

J (θ) = E[G(X; θ)], avec X ∼ µ, ( 2
β * n = argmin β E V n+1 (X β n+1 ) + f (X n , A(X n ; β)) ,
et le paramètre optimal θ * n dont le réseau de neurone Φ représente le mieux V n , la fonction valeur au temps n, vérifie :

θ * n = argmin θ E V n+1 (X β * n n+1 ) + f (X n , A(X n ; β * n )) -Φ(X n ; θ) 2 .
La notation X β n+1 dans les deux égalités ci-dessus signifie que le contrôle feedback A(X n ; β est choisi au temps n. Ces deux problèmes sont bien des cas particuliers de (2.2.2). Descente de gradient Une méthode standard et naturelle pour minimiser J est d'appliquer une méthode déterministe de descente de gradient dont l'algorithme 2.2 en est un pseudo-code rudimentaire.

Le calcul de ∂θ J (θ), qui donne la direction pour la descente de gradient, est très lourd en temps de calculs (notez que l'estimateur du gradient s'exprime comme somme de M termes !), ce qui rend l'algorithme 2.2 inutilisable. Pour alléger le temps de calculs, il est mis en place des algorithmes de descente de gradient encore plus stochastique, dont un pseudo-code est écrit dans Algorithme 2.3. Ce type d'algorithmes est beaucoup plus rapide, puisque l'estimateur stochastique ∂θ J (θ) qui donne une direction (stochastique) pour la descente de gradient est beaucoup très rapide à calculer (notez que l'estimateur du gradient s'exprime comme la somme d'un unique terme !).

Les algorithmes qui sont aujourd'hui le plus utilisés sont basés sur la descente de gradient stochastique. Citons les méthodes Adagrad, Adadelta, RMSprop et Adam, qui sont des améliorations (dans de nombreux cas pratique) bien connues dans la littérature du Machine-Learning, pour apprendre efficacement les paramètres optimaux, en jouant principalement sur un choix adapté du paramètre d'apprentissage η.

Réseaux de neurones pour les MDPs

Algorithm 2.2 Descente de gradient Input :

-paramètre d'apprentissage η ; -un seuil ε < < 1 ; -Un training set {X m , 1 ≤ m ≤ M } avec M > > 1 ; -θ ∈ R q ; 1: while | ∂θ J (θ)| > ε do 2: Calculer ∂θ J (θ) := Ê[∂ θ V (X; θ)],
où Ê est l'espérance sous la mesure empirique 1 M M m=1 δ Xm ;

3:

∂θ J (θ) est un estimateur de la dérivée de J au point θ.

4:

Re-assigner θ := θ -η ∂θ J (θ) ; 5: end while Output : paramètre optimal θ ; Algorithm 2.3 Descente de gradient stochastique Input : -paramètre d'apprentissage η ; -un seuil ε < < 1 ; -un nombre d'itérations M > > 1 ; -un paramètre initial θ ∈ R q ; 1: for m = 0, . . . , M do 2:

Tirer X ∼ µ ; 3: Calculer ∂θ J (θ) := ∂ θ V (X; θ);
∂θ J (θ) est un estimateur de la dérivée de J au point θ.

4:

Re-assigner θ := θ -η ∂θ J (θ) ; 5: end for Output : paramètre optimal θ ; Chapter 2. Introduction (in French) Remark 2.2.1. Dans cette thèse, nous nous sommes exclusivement servis de l'algorithme Adam, nativement implémenté dans Tensorflow, pour optimiser les espérances à paramètre.

La descente de gradient stochastique est la méthode d'optimisation de l'espérance à paramètre J à qui les réseaux de neurones doivent certainement une grande partie de leur succès.

Revue de littérature sur les réseaux de neurones pour les MDPs

Nous passons en revue quelques méthodes numériques bien connues principalement dans la littérature de l'apprentissage par renforcement (RL) pour résoudre (2.1.2) en représentant les fonctions d'intérêt, comme les contrôles optimaux ou les fonctions valeur au temps n = 0, . . . , N -1, par des fonctions paramétrées.

Il existe deux grandes familles d'algorithmes pour résoudre (2.1.1) avec des réseaux de neurones. La première, que l'on appelle "méthodes directes", contient les algorithmes qui ne représentent que le contrôle optimal par un réseau de neurones, et reposent essentiellement sur les progrès récents qui ont été fait dans l'apprentissage des paramètres optimaux pour optimiser les "espérances à (très nombreux !) paramètres" (voir section 2.2.1). La deuxième famille, que l'on appelle "Acteur-critique", contient les algorithmes qui représentent le contrôle optimal ainsi que la Q-value par des réseaux de neurones, et qui reposent essentiellement sur la représentation (2.1.3) de la fonction valeur.

Méthodes directes

L'idée pour cette première famille d'algorithmes est de représenter le contrôle α n au temps n par un réseau de neurones, pour tout n = 0, . . . , N -1 ; puis de voir que J(α) s'écrit alors comme une espérance paramétrique, qu'il suffit d'optimiser. Plus précisément, pour tout n = 0, . . . , N -1, représentons le contrôle feedback α n par un réseau de neurones comme ci-dessous :

α n = A(X n ; β n ),
(2.2.3) où β n ∈ R q est le paramètre, et réinjectons les représentations des contrôles dans (2.1.1). Le problème de contrôle (2.1.2) devient alors :

V 0 (x 0 ) = inf β0,...,βn∈R q E N -1 n=0 f (X n , A n (X n , β n )) + g(X N ) , ( 2
Q(X n , α n ) = Q(X n , A(X n ; β n ); θ n ).
L'apprentissage des paramètres optimaux β * n et θ * n , pour n = 0, . . . , N -1 se fait de manière rétrograde comme décrit dans Algorithme 2.4. Il est à noter que l'on apprend les paramètres optimaux du contrôle optimal et de la Q-value en même temps dans ce type d'algorithme, i.e. on ne peut pas apprendre l'un puis l'autre !11 Algorithm 2.4 Acteur-critique Input :

-paramètre d'apprentissage η ; -un seuil ε > 0 ; -initialiser au hasard θ n , β n ∈ R q , pour n = 0, . . . , N -1 ;

1: Poser V N = g ; 2: for n = N -1, . . . , 0 do 3: while max(∆θ n , ∆β n ) > ε do 4:
Minimiser par rapport à θ n la quantité suivante :

E (f (X n , A(X n ; β n )) + Q(X n+1 , A(X n+1 ; β * n+1 ); θ * n+1 ) -Q(X n , A(X n ; β n ); θ n )) 2 ,
et réassigner θ n comme son argmin ;

5:

Réassigner ∆θ n comme la différence de la valeur actualisée de θ n et l'ancienne ;

6:

Minimiser par rapport à β n la quantité suivante : Nous proposons deux familles de nouveaux algorithmes dans cette thèse. La première famille regroupe ceux basés sur l'approximation du contrôle uniquement par réseaux de neurones, et reposent sur le principe de programmation dynamique pour alléger l'apprentissage des contrôles, répondant ainsi à la question 1a. La deuxième famille repose sur l'approximation du contrôle optimal et de la fonction valeur par réseaux de neurones et répondent à la question 1b.

E f (X n , A(X n ; β n )) + Q(X n+1 , A(X n+1 ; β * n+1 ); θ n ) ,
Algorithmes qui répondent à la Question 2a Nous proposons les algorithmes NNcontPI et ClassifPI pour répondre à la question 2a. Nous présentons succinctement dès maintenant NNcontPI, et référons à la page 189 du Chapitre 6 pour la présentation du pseudo-code de ClassifPI. L'idée dans NNcontPI, dont on propose un pseudo-code dans Algorithme 2.5, est de représenter le contrôle optimal à la date n par un réseau de neurones A(.; β n ) et d'apprendre le paramètre optimal β * n de manière rétrograde, en reposant sur le principe de programmation dynamique et sur la notion de mesures de training (µ n ) N -1 n=0 : supposons les paramètres optimaux β * k déjà appris pour k = n + 1, . . . , N -1. Alors il vient :

βn ∈ argmin β∈R q E f X n , A(X n ; β) + N -1 k=n+1 f X β k , âk X β k + g X β N
où X n est tiré selon une mesure de "training" µ n et où X β k N k=n+1 est définie par induction :

   X n ∼ µ n , X β n+1 = F X n , A X n ; β , ε n+1 , X β k+1 = F X β k , âk X β k ; β , ε k+1 , for k = n + 1, . . . ,
) N -1 n=0 ; 1: for n = N -1, . . . , 0 do 2: Compute βn ∈ argmin β∈R q E f X n , A(X n ; β) + N -1 k=n+1 f X β k , âk X β k + g X β N où X n ∼ µ n et où X β k N
k=n+1 est définie par induction :

X β n+1 = F X n , A X n ; β , ε n+1 X β k+1 = F X β k , âk X β k ; β , ε k+1 , for k = n + 1, . . . , N -1.
pour m = 1, . . . , M ;

3:

Pose ân = A(.; βn ) ; ân est un estimateur du contrôle optimal à la date n 4: end for Sortie : (â n ) N -1 n=0 , estimateur des stratégies optimales aux instants n, pour n = 0, . . . , N -1 ;

Nous présentons dans le Chapitre 5 des résultats de convergence de l'algorithme NNcontPI en fonction de la taille de l'espace des réseaux de neurones ainsi que de la taille du training set.

Theorem 2.2.1 (Convergence de NNContPI). Supposons qu'il existe des contrôles optimaux feedbacks (a opt k ) N -1 k=n et notons V n les fonctions valeurs aux temps n = 0, . . . , N . Soit X n ∼ µ. On a alors, lorsque M → ∞16 :

E V M n (X n ) -V n (X n ) = O ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| ,
De plus, on a, quand M → ∞17 : Algorithmes qui répondent à la question 2b Nous proposons les algorithmes Hybrid-Now, Hybrid-Later et Classif-Hybrid pour répondre à la question 2b. Nous référons aux Algorithmes 6.4 et 6.6 respectivement aux pages 192 et 209 pour un pseudo-code de Hybrid-Later et Classif-Hybrid, et choisissons de ne présenter ici que l'algorithme Hybrid-Now, dont le pseudo-code est disponible dans Algorithme 2.6. Dans l'algorithme Hybrid-Now, nous représentons le contrôle et la fonction valeur au temps n par deux réseaux de neurones. Comme on peut le voir dans Algorithme 2.6, l'idée dans Hybrid-Now est d'abord d'apprendre le paramètre optimal du contrôle au temps n ; puis de reposer sur un principe martingale associé au processus contrôlé de manière optimale pour en déduire le paramètre optimal pour la représentation de la fonction valeur au temps n. Comme l'algorithme NNcontPI présenté précédemment, Hybrid-Now se base sur la donnée de mesures de training (µ n ) N -1 n=0 choisies par l'agent, où la densité de µ n doit être prise grande dans les régions optimale où l'agent cherche à effectuer un apprentissage précis du contrôle optimal.

E M V M n (X n ) -V n (X n ) = O P ρ N -n-1 M γ N -n-1 M η N -n-2 M log(M ) M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| , où E M est l'
Algorithm 2.6 Hybrid-Now

Entrée : distributions de training (µ n ) N -1 n=0 ; 1: Set VN = g ; 2: for n = N -1, . . . , 0 do 3: Calculer : βn ∈ argmin β∈R q E f X n , A(X n ; β) + Vn+1 (X β n+1 ) où X n ∼ µ, et X β n+1 = F X n , A(X n ; β) , ε n+1 ) ; 4:
Assigner ân = A(.; βn ) ; ân est un estimateur de la stratégie optimale au temps n ;

5:

Calculer θn ∈ argmin

θ∈R p E (f (X n , ân (X n )) + Vn+1 (X βn n+1 ) -Φ(X n ; θ) 2 ; 6:
Assigner Vn = Φ(.; θn ) ;

7:

Vn est un estimateur de la fonction valeur au temps n 8: end for Sortie :

-estimateurs de la stratégie optimale : (â n ) N -1 n=0 ; -estimateurs de la fonction valeur : ( Vn )

Réseaux de neurones pour les MDPs

Nous proposons, dans le Chapitre 5, une étude de la convergence de Hybrid-Now et Hybrid-Later, en fonction de la taille du training set et de l'espace de réseaux de neurones considéré. Nous ne mettons en avant ici seulement que le théorème 2.2.2 suivant obtenu pour la convergence de l'algorithme Hybrid-Now, que l'on retrouve page 158.

Theorem 2.2.2 (Convergence de Hybrid-Now). Supposons qu'il existe des contrôles optimaux qui soient feedback, i.e. (a opt k ) N -1 k=n , et notons par V n les fonctions valeurs du problème au temps n = 0, . . . , N . Soit X n ∼ µ. Alors, on a quand M → +∞ :

E M | V M n (X n ) -V n (X n )| = O P γ 4 M K M log(M ) M 1 2(N -n) + γ 4 M ρ 2 M η 2 M log(M ) M 1 4(N -n) + sup n≤k≤N inf Φ∈V M E M |Φ(X k ) -V k (X k )| 2 1 2(N -n) + sup n≤k≤N inf A∈A M E |A(X k ) -a opt k (X k )| 1 2(N -n)
, où E M est l'espérance conditionnée au training set qui a permis de construire les estimateurs âM k , pour k = n, . . . , N -1.

Le théorème 2.2.2 stipule que la différence entre la fonction valeur et l'approximation de cette dernière, calculée en utilisant les estimations du contrôle optimal par l'algorithme Hybrid-Now, tend en moyenne vers 0 lorsque la taille du training set et l'espace des réseaux de neurones tendent vers l'infini. Encore une fois, le résultat est intéressant parce qu'il prend en compte la taille du training set utilisé pour la SGD. Nos algorithmes sont adaptés, ou facilement adaptables, à de nombreux types de problèmes de contrôle. C'est ce que nous avons tenté d'illustrer dans la section 6.3 du Chapitre 6, où nous avons étudié les exemples suivants :

• SemiLinearPDE, où l'espace d'états et de contrôle sont continus et de dimension 100.

• Option Hedging, où le processus contrôlé est discret.

Perspectives

La performance de nos algorithmes dépend fortement de la qualité de nos mesure de training. Nous avons pensé aux méthodes de type bootstrapping pour construire de manière itérative de bonnes mesures de training, mais ces dernières ne marchent pas toujours en dimension élevée. L'idée principale à retenir est que, comme tel, si l'on ne connaît pas de bonne mesure de training en grande dimension, alors cela se ressentira dans la qualité des estimations des stratégies optimales par nos algorithmes. Il serait intéressant de trouver des méthodes qui marchent en grande dimension pour trouver de bonnes mesures de training. Notons que cette recherche de bonne mesure de training fait en fait partie d'un domaine extrêmement actif dans la communauté de l'apprentissage par renforcement. Cette dernière parle de ce problème comme du problème d'exploration ; et de nombreux papiers sortent ces derniers temps à propos de la bonne manière de faire de l'exploration, i.e. trouver une bonne mesure de training.

Réseaux de neurones pour les EDPs et les EDSRs

Dans la troisième partie de cette thèse, nous proposons et étudions une famille de schémas numériques pour la résolution des EDPs, IQVs, EDSRs et des EDSRs réfléchies. Nous commençons par présenter le cadre, puis rappelons les méthodes numériques qui existent déjà pour résoudre ces problèmes, ainsi que leurs limites que l'on a voulu repousser en proposant nos nouveaux algorithmes. La convergence de nos nouveaux algorithmes est prouvée, et ces derniers sont testés numériquement avec succès sur différents exemples de dimension 1 à 40.

Cadre général

Fixons T > 0. Soient µ et σ deux fonctions telles que µ : [0, T ] × R d → R d , σ : [0, T ] × R d → M d (R), où M d (R) est l'
espace des matrices réelles de taille d × d. µ et σ sont supposées assez régulières pour qu'il y ait existence et unicité d'une solution X à l'équation différentielle stochastique suivante :

X t = x 0 + t 0 µ(s, X s ) ds + t 0 σ(s, X s ) dW s , 0 ≤ t ≤ T,
où W est un mouvement Brownien d-dimensionnel sur un espace de probabilité (Ω, F, P) équipé de la filtration F = (F t ) 0≤t≤T qui satisfait les conditions usuelles.

Problème EDSR et EDP

Le premier problème auquel on s'intéresse est celui qui consiste à trouver la paire (Y, Z) de processus F-adaptés et à valeurs dans R × R d qui soit solution de l'équation 2.3. Réseaux de neurones pour les EPDs et les EDSRs

Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s , 0 ≤ t ≤ T, (2.
Y t = u(t, X t ), Z t = σ (t, X t )D x u(t, X t ) 0 ≤ t ≤ T,
où u est solution de l'équation différentielle parabolique aux dérivées partielles non linéaire suivante :

∂ t u + Lu + f (., ., u, σ D x u) = 0, on [0, T ) × R d , u(T, .) = g, on R d , (2.3.2)
avec L le générateur infinitésimal de X , défini ainsi

Lu := 1 2 Tr σσ D 2 x u + µ.D x u.
Résoudre l'EDSR (2.3.1) est donc équivalent à résoudre l'EDP (2.3.2).

Problème RBSDE Le deuxième problème auquel on s'intéresse dans la troisième partie de cette thèse est celui de trouver le triplet (Y, Z, K) solution de l'équation suivante :

Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s + K T -K t , Y t ≥ g(X t ), 0 ≤ t ≤ T, (2.3.3) où K est un processus croissant vérifiant : T 0 Y t -g(X t ) dK t = 0.
(2.3.3) est communément appelée équation différentielle stochastique rétrograde réfléchie (EDSR réfléchie). C'est une représentation classique de problèmes de temps d'arrêt optimal, et apparaît en particulier dans les problèmes de pricing et de couverture d'options américaines. La solution à (2.3.3) s'écrit Y t = u(t, X t ), où il est bien connu que u est caractérisée comme solution de l'inégalité quasivariationnelle (IQV) suivante :

min -∂ t u -Lu -f (t, x, u, σ D x u), u -g = 0, t ∈ [0, T ), x ∈ R d , u(T, x) = g(x), x ∈ R d , (2.3.4)
comme montré par exemple dans [START_REF] Karoui | Reflected Solutions of Backward SDEs, and related obstacle problems for PDEs[END_REF].

La résolution de l'EDSR réfléchie (2.3.3) est donc équivalente à la résolution de l'IQV (2.3.4).

Méthodes numériques existantes dans la littérature

Quand X n'est pas simulable, la première étape pour obtenir des résultats numériques est de discrétiser le processus sur la grille de temps :

π = {t 0 = 0 < t 1 < . . . < t N = T }, de module |π| = max i=0,...,N -1 ∆t i , ∆t i := t i+1 -t i . La discrétisation d'Euler X = X π de X est alors : X ti+1 = X ti + µ(t i , X ti )∆t i + σ(t i , X ti )∆W ti , i = 0, . . . , N -1, X 0 = x 0 , 2
u(t i+1 , X ti+1 ) ≈ F (t i , X ti , u(t i , X ti ), σ (t i , X ti )D x u(t i , X ti ), ∆t i , ∆W ti ),
où l'on a posé :

F (t, x, y, z, h, ∆) := y -f (t, x, y, z)h + z ∆.
(2.3.5)

Méthodes numériques pour les EDSRs et les EDPs

Nous répertorions ici aux algorithmes qui existent déjà dans la littérature pour résoudre (2.3.1).

Différence finie L'EDP (2.3.2) peut se résoudre par méthode de différence finie en discrétisant les opérateurs de dérivation sur des grilles (resp. sparse-grids) en petite (resp. moyenne) dimension. La convergence de ce type d'algorithme est garantie sous certaines conditions entre les coefficients (notamment le pas de discrétisation en temps et celui en espace).

Commentaires sur les méthodes de différence finie :

Les méthodes de différence finie imposent des conditions aux bords qui ne sont pas données dans (2.3.2). En pratique, il faut donc ajouter des conditions artificielles pour appliquer ce type de méthodes. (voir le Benchmark dans la Section 4.4.2.2 page 132 pour un exemple d'application d'une méthode de différence finie pour résoudre une EDP non-linéaire de type HJB)

La complexité des méthodes basées sur les différences finies est en O d+1 et explose avec la dimension ! Cette méthode est en fait adaptée aux problèmes de petites dimensions (≤ 3), voir un peu plus si l'on utilise des méthodes sparses.

Régression sur une base de fonctions Une famille d'algorithmes est présentée dans [START_REF] Lemor | Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations[END_REF] pour résoudre les EDSRs. L'idée principale est d'estimer les solutions aux dates t i , i = 0, . . . , N de manière rétrograde, en utilisant le schéma suivant :

   Y t N = g(X t N ) Z ti = 1 ∆ti E Y ti+1 ∆W ti |F ti , Y ti = E Y ti+1 |F ti + f (t i , X ti , Y ti , Z ti )∆t i , for i = N -1, . . . , 0.
(2.3.6) A la date t i , Z ti est approximé en régressant l'espérance conditionnelle sur une base de fonction ; et

Y ti , défini dans (2.3.6) comme le point fixe de l'opérateur contractant L 2 (σ(X ti )) → L 2 (σ(X ti )) : Y ti → E i Y ti+1 + hf (t i , X ti , Y ti , Z ti )
, est approximé par itérations de Picard.La convergence des algorithmes est étudié dans ce même papier.

Commentaires sur les méthodes basées sur la regression sur une base de fonctions :

Ces méthodes sont très rapides et efficaces en petite et moyenne dimension car elles reposent sur des formules exactes et fermées pour les étapes de régression.

Ces méthodes sont inutilisables pour des problèmes en dimension supérieure à 10, car au-delà, il quasiment impossible de trouver combien et quelles fonctions de base prendre, pour assurer une approximation fidèle des fonctions inconnues tout en évitant le sur-apprentissage (overfitting) et/ou sous-apprentissage (underfitting).

Les deux méthodes que l'on vient de présenter souffrent de la dimension, et sont en particulier inapplicables en grande dimension. Certaines méthodes récentes, qui reposent sur les réseaux de neurones, sont plus adaptées à la grande dimension.

Chapter 2. Introduction (in French)

DGM La méthode DGM (pour Deep Galerkin Method) a été introduite dans [START_REF] Sirignano | DGM: A deep learning algorithm for solving partial differential equations[END_REF], et consiste à approximer par réseaux de neurones U(t, x; θ) la solution de (2.3.2). On considère ν T une mesure à support sur [0, T ], ν Ω une mesure à support sur Ω, ν ∂Ω une mesure à support sur ∂Ω, et l'on définit la fonction objective J ainsi :

J(θ) = ∂ t U(., .; θ) + LU(., .; θ) + f (t, x, U(., .; θ), σ D x U(., .; θ) [0,T ]×Ω,νΩ + U(T, .; θ) -g(.) ∂Ω,ν ∂Ω , où l'on note . [0,T ]×Ω,νΩ la norme canonique sur L 2 ([0, T ] × Ω, ν T × ν Ω ), et . [0,T ]×∂Ω,ν ∂Ω celle sur L 2 ([0, T ] × ∂Ω, ν T × ν ∂Ω ).
L'algorithme DGM consiste à déterminer, par descente de gradient stochastique, le θ optimal, noté θ * , qui minimise la fonction objective. L'algorithme converge car la fonction objective s'approche de zéro lorsque la classe de réseaux de neurones grandit ; ce qui implique que U(., .; θ * ) approxime de mieux en mieux la solution u de (2.3.2).

Commentaires sur DGM :

Nous rappelons que le calcul de la dérivée du réseau de neurones n'est pas cher en temps de calculs quand elle est calculée en utilisant la rétro-propagation inverse.

Le fait que DGM repose sur un choix des mesures de training peut être positif lorsque l'agent sait dans quelle région il recherche des estimées précises : il lui suffit de prendre une densité élevée dans les zones qu'il veut couvrir en priorité.

NNZ L'algorithme NNZ (Neural Network approximation for Z), introduit dans [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] et dont la convergence est étudiée dans [START_REF] Han | Convergence of the deep BSDE method for coupled FBSDEs[END_REF], constitue notre principal benchmark pour résoudre les EDSRs en grande dimension. Il repose sur la représentation Z ti par réseau de neurones de Z ti à chaque pas de discrétisation en temps t i , i = 0, . . . , N -1, ainsi que sur l'approximation suivante

u(t n , X tn ) ≈ u(t n-1 ; X tn-1 ) -f (t n-1 , X tn-1 , u(t n-1 , X tn-1 ), Z(t n-1 , X tn-1 ))(t n -t n-1 ) + Z(t n , X tn ) T (W tn+1 -W tn ),
pour représenter par induction, pour tout n = 0, . . . , N , la fonction u(t n , .) par un réseau de neurones U n (., θ) défini comme composé des réseaux de neurones qui représentent les Z ti avec i = 0, . . . , n -1. La fonction objectif de NNZ est la fonction J définie ainsi :

J(θ) := E (U N (X N , θ) -g(X N )) 2 .
et l'idée est d'entraîner U N (X N , θ) pour qu'il annule J.

Commentaires sur l'algorithme NNZ

Tous les réseaux de neurones qui représentent les Z ti sont entraînés en même temps, ce qui rend l'algorithme très rapide lorsque N est petit.

Il n'y a pas d'erreur d'approximation des fonction u qui se diffuse de manière rétrograde. Mais le prix à payer est la resimulation des processus de k = 0 à N pour apprendre les paramètres optimaux des représentations des contrôles optimaux α * n , n = 0, . . . , N -1.

L'apprentissage de tous les réseaux de neurones qui représentent les Z ti étant global, NNZ requiert l'entraînement d'un réseau de neurones très profond, avec n Z × N couches, où n Z est le nombre de couches choisies pour apprendre chaque Z ti . L'entraînement de réseaux de neurones très profonds est encore un problème très difficile à résoudre, même en utilisant les derniers algorithmes de descente de gradient. En fait, il est très facile d'observer des cas où la SGD n'arrive pas à minimiser les fonctions objectives : l'algorithme peut converger par descente de gradient vers des minimums locaux qui sont loin d'être optimaux, ou ne pas converger pas du tout. Voir Chapitre 7 pour des exemples où il se produit l'une ou l'autre des deux situations.

Réseaux de neurones pour les EPDs et les EDSRs

Contributions (Chapitre 7)

Motivation A cause de la malédiction de la dimension, la résolution numérique de (2.3.1) et (2.3.3) a toujours été difficile en grande dimension. De même, il est difficile de résoudre les EDPs (2.3.2) et IQVs (2.3.4) en grande dimension. Les méthodes usuelles basées sur la régression sur une base de fonctions, ou de différence finie, marchent très bien en petite et moyenne dimension uniquement. Les méthodes basées sur les réseaux de neurones comme NNZ et DGM sont des algorithmes bien adaptés à la grande dimension. Mais NNZ semble devoir se limiter à la résolution de BSDEs simples où le nombre de pas de discrétisation de la BSDE n'est pas trop élevé.

Question 3a :

Quel schéma stable proposer pour résoudre les EDSRs en grande dimension ? Question 3a(bis) :

Quel schéma stable proposer pour résoudre les EDPs en grande dimension ?

Question 3b : Quel schéma stable proposer pour résoudre les EDSRs réfléchies en grande dimension ? Question 3b(bis) : Quel schéma stable proposer pour résoudre les EDPs avec frontière libre en grande dimension ?

Remark 2.3.1. Comme dit précédemment la Question 3a est équivalente à la Question 3a(bis). De même, la Question 3b est équivalente à la Question 3b(bis).

Nous introduisons dans le Chapitre 7 de cette thèse deux algorithmes que l'on a baptisés DBDP1 et DBDP2, et qui répondent à la Question 2a ; ainsi qu'un algorithme, appelé RDBDP qui répond à la question 2b. Une étude théorique de la convergence de nos nouveaux algorithmes est menée, et de nombreux tests numériques sont disponibles pour illustrer les performances de nos algorithmes. Nous présentons ci-dessous succinctement les trois algorithmes et donnons les trois résultats de convergence importants établis dans la troisième partie de la thèse.

Présentation de DBDP1 et résultat de convergence associé

L'idée principale de l'algorithme DBDP1 est d'approximer, pour tout i = N -1, . . . , 0, la paire (u(t i , .), Z(t i , .)) par des réseaux de neurones :

u(t i , .) = U i (.; θ) Z(t i , .) = Z i (.; θ),
où l'on note par θ l'ensemble des paramètres des deux réseaux de neurones considérés. Nous apprenons θ * i , le paramètre optimal de U i et Z i pour l'approximation de u(t i , .) et Z ti , de manière rétrograde et en reposant, comme écrit dans Algorithme 2.7 ci-dessous, sur l'approximation suivante :

U i+1 (X ti+1 ; θ * i+1 ) ≈ F (t i , X ti , U i (X ti ; θ * i ), Z i (X ti ; θ * i ), ∆t i , ∆W ti ),
où l'on a noté θ * i+1 le paramètre optimal de U i+1 et Z i+1 pour l'approximation de u(t i+1 , .) et Z ti+1 , et où la fonction F a été introduite dans (2.3.5).

La convergence de DBDP1 est étudiée dans le Chapitre 7 (voir théorème 7.4.1 page 225). Nous présentons ci-dessous une version simplifiée du résultat de convergence.

Theorem 2.3.1. (Consistance de DBDP1) Sous des hypothèses classiques de régularité des coefficients, il existe une constante C > 0, indépendante de la grille de discrétisation en temps π, telle que

E ( U (1) , Z (1) ), (Y, Z) ≤ C E g(X T ) -g(X T ) 2 + |π| + ε Z (π) + N -1 i=0 N ε N ,v i + ε N ,z i , Chapter 2. Introduction (in French)
Algorithm 2.7 Algorithme DBDP1 Entrée :

1: Initialiser U (1) N = g 2: for i = N -1, . . . , 0 do 3:
Calculer :

θ * i ∈ argmin θ E U (1) i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 ; 4: Poser U (1) i = U i (.; θ * i ) ; 5: Poser Z (1) i = Z i (.; θ * i ) ; 6: end for Sortie : -( U (1) i ) N i=0 : Estimations des Y ti pour i = 0, . . . , N ; -( Z (1) i ) N i=0 : Estimations des Z ti pour i = 0, . . . , N ; où E ( U (1) , Z (1) ), (Y, Z) := max i=0,...,N -1 E Y ti -U (1) i (X ti ) 2 + E N -1 i=0 ti+1 ti Z t -Z (1) i (X ti ) 2 dt est l'erreur L 2 entre l'estimée par DBDP1 de (Y, Z) et le couple (Y, Z) lui-même, où ε N ,v i := inf ξ E vi (X ti ) -U i (X ti ; ξ) 2 , ε N ,z i := inf η E ẑi (X ti ) -Z i (X ti ; η)
2 sont des erreurs d'approximation par réseaux de neurones qui s'annulent lorsque l'espace des réseaux de neurones considéré grandit ; et où

ε Z (π) := E N -1 i=0 ti+1 ti |Z t -Zti | 2 dt , avec Zti := 1 ∆t i E i ti+1 ti Z t dt est la L 2 -régularité de Z.
Le théorème 2.3.1 stipule que l'erreur d'estimation en norme L 2 de Y et Z par l'algorithme DBDP1 est bornée par la somme de :

• l'erreur de discrétisation en temps de [0, T ], i.e. |π|,

• l'erreur de discrétisation d'Euler du processus X ,

• la régularité L 2 de Z, • la quantité N -1 i=0 N ε N ,v i + ε N ,z i
qu'il faut voir comme la somme d'erreurs d'approximation par réseaux de neurones de certaines fonctions.

On montre par des arguments standards et sous des hypothèses standards, que toutes ces erreurs tendent vers 0 lorsque la discrétisation en temps s'affine et que la taille de l'espace des réseaux de neurones grandit.

Présentation de DBDP2, et résultat de convergence associé

Remarquons que dans DBDP1, nous ignorons le résultat bien connu suivant : Z = σ (t i , .)D x u(t i , .), ce qui revient à ignorer que le réseau de neurones Z i (.; θ) s'exprime très simplement en fonction la dérivée de U i (.; θ). Nous présentons dans cette section l'algorithme DBDP2, qui est basé sur cette remarque.

Réseaux de neurones pour les EPDs et les EDSRs

Considérons donc ici qu'un unique réseau de neurones U pour apprendre u, et représentons (Y, Z) :

Y (t i , .) = U i (.; θ) Z(t i , .) = σ (t i , .)D x U i (.; θ),
où cette fois-ci, θ est le paramètre de réseaux de neurones qui approxime u. Nous apprenons θ * i , le paramètre optimal pour l'approximation de u(t i , .) et Z(t i , .), de manière rétrograde, en reposant, comme écrit dans Algorithme 2.8 ci-dessous, sur l'approximation suivante :

U i+1 (X ti+1 ; θ * i+1 ) = F (t i , X ti , U i (X ti ; θ * i ), σ (t i , X ti )D x U i (X ti ; θ * i ), ∆t i , ∆W ti ).
Algorithm 2.8 Algorithme DBDP2 Entrée :

1: Initialiser U (1) N = g 2: for i = N -1, . . . , 0 do 3:
Calculer :

θ * i ∈ argmin θ E U (2) i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), σ (t i , X ti ) Dx U i (X ti ; θ), ∆t i , ∆W ti ) 2 ; 4: Poser U (1) i = U i (.; θ * i ) ; 5: Poser Z (1) i = σ (t i , .)D x Ûi (.; θ * i ) ; 6: end for Sortie : -( U (1) i ) N i=0 : Estimations des Y ti pour i = 0, . . . , N ; -( Z (1) i ) N i=0 : Estimations des Z ti pour i = 0, . . .
E ( U (2) , Z (2) ), (Y, Z) ≤ C E g(X T ) -g(X T ) 2 + γ 6 m N + ε Z (π) + N N -1 i=0 ε N ,m i .
Le théorème 2.3.1 stipule que l'erreur d'estimation en norme L 2 de u et Z par l'algorithme DBDP2 est bornée par la somme de :

• l'erreur de discrétisation en temps de [0, T ], i.e. |π|,

• l'erreur de discrétisation d'Euler du processus X ,

• une erreur supplémentaire γ 6 m N , due aux contraintes que l'on doit exercer sur la taille de l'espace de réseaux de neurones,

Chapter 2. Introduction (in French) • la régularité L 2 de Z, • la quantité N -1 i=0 N ε N ,v i + ε N ,z i
qu'il faut voir comme la somme d'erreurs d'approximation par réseaux de neurones de certaines fonctions.

Ces erreurs tendent vers 0 lorsque la discrétisation en temps s'affine et que la taille de l'espace des réseaux de neurones grandit, ce qui implique la convergence de DBDP2.

Présentation de RDBDP et résultat de consistance

L'idée principale de l'algorithme RDBDP est d'approximer, pour tout i = N -1, . . . , 0, la paire (u(t i , .), Z(t i , .)) par des réseaux de neurones :

u(t i , .) = U i (.; θ) Z(t i , .) = Z i (.; θ),
où l'on note par θ l'ensemble des paramètres des deux réseaux de neurones considérés ; puis de reposer sur le schéma décrit dans Algorithme 2.9 ci-dessous, pour approximer les fonctions inconnues u(t i , .) et Z(t i , .). Algorithm 2.9 Algorithme RDBDP Entrée :

1: Initialiser U (1) N = g 2: for i = N -1, . . . , 0 do 3: Calculer : θ * i ∈ argmin θ Li (θ), où Li (θ) := E U i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 4:
Poser U i = max U i (.; θ * i ), g];

5:

Poser Z i = Ẑi (.; θ * i ) ; 6: end for Sortie :

-( U (1) i ) N i=0 : Estimations des Y ti pour i = 0, . . . , N ; -( Z (1) i ) N i=0 :
E ( U , Z ), (Y, Z) ≤ C ε(π) + N -1 i=0 N ε N ,ṽ i + ε N ,z i , où ε(π) = O(|π| 1
2 ) sous des hypothèses de régularité de g, ou ε(π) = O(|π|) sous des hypothèses de régularité de g et σ.

Le Théorème 2.3.3 stipule que l'erreur d'estimation de (Y, Z) par RDBDP tend vers 0 lorsque le pas de discrétisation en temps diminue et que l'espace des réseaux de neurones grandit.

Réseaux de neurones pour les EPDs et les EDSRs

Résultats numériques obtenus

Dans la section 7.5 du Chapitre 7 figurent de nombreux tests numériques qui mettent en avant les performances de nos algorithmes DBDP1, DBDP2 et RDBDP. Les résultats sont très satisfaisants ! En particulier nos algorithmes DBDP1 et DBDP2 sont beaucoup plus stables que NNZ, qui consistue notre principal Benchmark pour résoudre des EDSRs en grande dimension. Nos algorithmes convergent aussi bien que NNZ lorsque ce dernier converge, et convergent dans des exemples numériques dans lesquels NNZ soit diverge, soit converge vers une valeur qui n'est pas satisfaisante. En fait, NNZ a beaucoup de difficulté à converger lorsque le problème EDSR n'est pas simple, i.e. les fonctions u et Z à approximer sont complexes ou que N est grand. Nous référons à la section 7.5.1 page 237 pour des exemples simples sur lesquels DBDP1, DBDP2 et NNZ marchent bien, et la section 7.5.2 page 241 pour des exemples plus compliqués dans lesquels DBDP1 et DBDP2 marchent bien, mais NNZ diverge ! Enfin, nous n'avons constaté aucun exemple dans lequel NNZ fasse vraiment mieux que nos algorithmes, ce qui n'est pas étonnant lorsque l'on compare la structure des algorithmes.

RDBDP apparaît aussi comme un algorithme très performant dans l'application de pricing d'option américaine en grande dimension que l'on a considérée. Les résultats l'algorithme RDBDP sont aussi satisfaisants, et nous proposons dans Tableau 2.2 des résultats d'estimations par l'algorithme RDBDP de prix d'une option américaine pour différentes valeur de la dimension allant de 1 à 40. Pour le payoff considéré, il est possible de réduire de problème de pricing de dimension d ≥ 1 à un problème de dimension 1. Les valeurs de référence sont calculées en réécrivant réduisant ainsi à 1 la dimension du problème, et en utilisant une méthode standard par arbres pour le calcul de prix d'option américaine. Ce tableau est extrait du tableau 7.11 page 245.

Perspectives

Le résultats de convergence proposés sont très satisfaisants mais l'on peut espérer faire plus naturel dans le terme associé à la somme des erreurs d'approximation par réseaux de neurones de la fonction u et Z. Par exemple, dans les théorèmes 2.3.1, 2. Chapter 3

Algorithmic trading in a microstructural limit order book model1

Abstract We propose a microstructural modeling framework for studying optimal market making policies in a FIFO (first in first out) limit order book (order book). In this context, the limit orders, market orders, and cancel orders arrivals in the order book are modeled as Cox point processes with intensities that only depend on the state of the order book. These are high-dimensional models which are realistic from a micro-structure point of view and have been recently developed in the literature.

In this context, we consider a market maker who stands ready to buy and sell stock on a regular and continuous basis at a publicly quoted price, and identifies the strategies that maximize her P&L penalized by her inventory.

We apply the theory of Markov Decision Processes and dynamic programming method to characterize analytically the solutions to our optimal market making problem. The second part of the paper deals with the numerical aspect of the high-dimensional trading problem. We use a control randomization method combined with quantization method to compute the optimal strategies. Several computational tests are performed on simulated data to illustrate the efficiency of the computed optimal strategy. In particular, we simulated an order book with constant/ symmetric/ asymmetrical/ state dependent intensities, and compared the computed optimal strategy with naive strategies.

Keywords:

Limit

Introduction

Most of the markets use a limit order book (order book) mechanism to facilitate trade. Any market participant can interact with the order book by posting either market orders or limit orders. In such type of markets, the market makers play a fundamental role by providing liquidity to other market participants, typically to impatient agents who are willing to cross the bid-ask spread. The profit made by a market making strategy comes from the alternation of buy and sell orders.

From the mathematical modeling point of view, the market making problem corresponds to the choice of an optimal strategy for the placement of orders in the order book. Such a strategy should maximize the expected utility function of the wealth of the market maker up to a penalization of her inventory. In the recent litterature, several works focused on the problem of market making through stochastic control methods. The seminal paper by Avellaneda and Stoikov [START_REF] Avellaneda | High-frequency trading in a limit order book[END_REF] inspired by the work of Ho and Stoll [START_REF] Ho | Optimal dealer pricing under transactions and return uncertainty[END_REF] proposes a framework for trading in an order driven market. They modeled a reference price for the stock as a Wiener process, and the arrival of a buy or sell liquidity-consuming order at a distance δ from the reference price is described by a point process with an intensity in an exponential form decreasing with δ. They characterized the optimal market making strategies that maximize an exponential utility function of terminal wealth. Since this paper, other authors have worked on related market making problems. Gueant, Lehalle, and Fernandez-Tapia [START_REF] Guéant | Dealing with the Inventory Risk[END_REF] generalized the market making problem of [START_REF] Avellaneda | High-frequency trading in a limit order book[END_REF] by dealing with the inventory risk. Cartea and Jaimungal [CJ13] also designed algorithms that manage inventory risk. Fodra and Pham [START_REF] Fodra | Semi Markov model for market microstructure[END_REF] and [START_REF] Fodra | High Frequency trading and asymptotics for small risk aversion in a Markov renewal model[END_REF] considered a model designed to be a good compromise between accuracy and tractability, where the stock price is driven by a Markov Renewal Process, and solved the market making problem. Guilbaud and Pham [START_REF] Guilbaud | Optimal High Frequency Trading with limit and market orders[END_REF] also considered a model for the mid-price, modeled the spread as a discrete Markov chain that jumps according to a stochastic clock, and studied the performance of the market making strategy both theoretically and numerically. Cartea and Jaimungal [START_REF] Cartea | Modeling Asset Prices for Algorithmic and High Frequency Trading[END_REF] employed a hidden Markov model to examine the intra-day changes of dynamics of the order book. Very recently, Cartea, Penalva, and Jaimungal [START_REF] Cartea | Algorithmic and High-frequency trading[END_REF] and Gueant [START_REF] Guéant | The Financial Mathematics of Market Liquidity: From optimal execution to market making[END_REF] published monographs in which they developped models for algorithmic trading in different contexts. Abergel and El Aoud [START_REF] Aoud | A stochastic control approach for options market making[END_REF] extended the framework of Avellaneda and Stoikov to the options market making. A common feature of all these works is that a model for the price or/and the spread is considered, and the order book is then built from these quantities. This approach leads to models that predict well the long-term behavior of the order book. The reason for this choice is that it is generally easier to solve the market making problem when the controlled process is low-dimensional. Yet, some recent works have introduced accurate and sophisticated micro-structural order book models. These models reproduce accurately the short-term behavior of the market data. The focus is on conditional probabilities of events, given the state of the order book and the positions of the market maker. Abergel, Anane, Chakraborti, Jedidi, Muni Toke [START_REF] Abergel | Limit Order Books[END_REF] proposed models of order book where the arrivals of orders in the order book are driven by Poisson processes or Hawkes processes. Stoikov, Talreja, and Cont [CST07] also modeled the orders arrivals with Poisson processes. Lehalle, Rosenbaum and Huang [START_REF] Huang | Simulating and analyzing order book data: The queue-reactive model[END_REF] proposed a queue-reactive 3.2. Model setup model for the order book. In this model the arrivals of orders are driven by Cox point processes with intensities that only depend on the state of the order book (they are not time dependent). Other tractable dynamic models of order-driven market are available (see e.g. Stoikov, Talreja, and Cont [START_REF] Cont | A stochastic model for order book dynamics[END_REF], Rosu [START_REF] Rosu | A dynamic model of the limit order book[END_REF], Cartea, Jaimungal, Ricci [START_REF] Cartea | Buy Low Sell High: a High Frequency Trading Perspective[END_REF]).

In this paper we adopt the micro-structural model of order book in [START_REF] Abergel | Limit Order Books[END_REF], and solve the associated trading problem. The problem is formulated in the general framework of Piecewise Deterministic Markov Decision Process (PDMDP), see Bauerle and Rieder [START_REF] Bäuerle | Markov Decision Processes with Applications to Finance[END_REF]. Given the model of order book, the PDMDP formulation is natural. Indeed, between two jumps, the order book remains constant, so one can see the modeled order book as a point process where the time becomes a component of the state space. As for the control, the market maker fixes her strategy as a deterministic function of the time right after each jump time. We prove that the value function of the market making problem is equal to the value function of an associated non-finite horizon Markov decision process (MDP). This provides a characterization of the value function in terms of a fixed point dynamic programming equation. Jacquier and Liu in [START_REF] Jacquier | Optimal Liquidation in a level-I Limit order book for large tick stocks[END_REF] recently followed a similar idea to solve an optimal liquidation problem, while Baradel et al. [START_REF] Baradel | Optimal inventory management and order book modeling[END_REF] and Lehalle et al. [START_REF] Lehalle | Optimal liquidity-based trading tactics[END_REF] also tackled this problem of reward functional maximization in a micro-structure model of order book framework.

The second part of the paper deals with the numerical simulation of the value functions. The computation is challenging because the micro-structural model used to model the order book leads to a high-dimensional pure jump controlled process, so evaluating the value function is computationally intensive. We rely on control randomization and Markovian quantization methods to compute the value functions. Markovian quantization has been proved to be very efficient for solving control problems associated with high-dimensional Markov processes. We first quantize the jump times and then quantize the state space of the order book. See Pages, Pham, Printemps [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF] for a general description of quantization applied to controlled processes. The projections are time-consuming in the algorithm. Fast approximate nearest neighbors algorithms have been implemented to make it quicker (see [START_REF] Muja | Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration[END_REF]). We borrow the values of intensities of the arrivals of orders for the order book simulations from Huang et al. [START_REF] Huang | Simulating and analyzing order book data: The queue-reactive model[END_REF] in order to test our optimal trading strategies.

The paper is organized as follows. The model setup is introduced in Section 3.2: we present the micro-structural model for the order book, and show how the market maker interacts with the market. In Section 3.3, we prove the existence and provide a characterization of the value function and optimal trading strategies. In Section 3.4, we introduce a quantization-based algorithm to numerically solve a general class of discrete-time control problem with finite horizon, and then apply it on our trading problem. We then present some results of numerical tests on simulated order book. Section 3.5 presents an extension of our model when order arrivals are driven by Hawkes processes, and finally the appendix collects some results used in the paper .

Model setup

Order book representation

We consider a model of the order book inspired by the one introduced in chapter 6 of [START_REF] Abergel | Limit Order Books[END_REF].

Fix K ≥ 0. An order book is supposed to be fully described by K limits on the bid side and K limits on the ask side. Denote by pa t the best ask at time t, which is the cheapest price a participant in the market is willing to sell a stock at time t, and by pb t the best bid at time t, which is the highest price a participant in the market is willing to buy a stock at time t. We use the pair of vectors

a t , b t = a 1 t , ..., a K t , b 1 t , . . . , b K t
• a i t is the number of shares available i ticks away from pb t , • -b i t is the number of shares available i ticks away from pa t , to describe the order book. The vector a t and b t describe the ask and the bid sides at time t. The Chapter 3. Algorithmic trading in a LOB model quantities a i t , 1 ≤ i ≤ K, live in the discrete space qN where q ∈ R * is the minimum order size on each specific market (lot size). The quantities b i t , 1 ≤ i ≤ K, live in the discrete space -qN. By convention, the a i are non-negative, and the b i are non-positive for 0 ≤ i ≤ K. The tick size represents the smallest intervall between price levels. In the sequel we assume that the orders arrivals have the same size q = 1, and set the tick size to = 1 for simplicity.

Constant boundary conditions are imposed outside the moving frame of size 2K in order to guarantee that both sides of the LOB are never empty: we assume that all the limits up to the K-th ones are equal to a ∞ in the ask side, and equal to b ∞ in the bid side.

We shall assume some conditions on the structure of the orders arrivals in the order book.

(Harrivals) The orders arrivals from general market participants (market orders, limit orders and cancel orders) occur according to Markov jump processes which intensities only depends on the state of the order book. Moreover, we assume that the all the intensities are at most linear w.r.t. the couple a, b and are constant between two events. Under (Harrivals), let us define

• λ M + the intensity of the buy-to-market orders flow M + t ,
• λ M -the intensity of the sell-to-market orders flow M - t ,

• λ L + i , i ∈ {1, ...K}, the intensity of the sell orders flow L + i at the i th limit of the ask side,

• λ L - i , i ∈ {1, ...K}, the intensity of the buy orders flow L - i at the i th limit of the bid side,

• λ C + i , i ∈ {1, ...K}, the intensity of the cancel orders flow C + i at the i th limit of the ask side,

• λ C - i , i ∈ {1, ...K}, the intensity of the cancel orders flow C - i at the i th limit of the bid side, and let λ L , λ C , λ M be such that

K i=0 λ L ± i (z) ≤ λ L |a| + |b| , K i=0 λ C ± i (z) ≤ λ C |a| + |b| , λ M -(z) + λ M + (z) ≤ λ M |a| + |b| ,
for all state (a, b) of the LOB. We remind that λ L , λ C , λ M are well-defined under assumption (Harrivals).

Remark 3.2.1. The linear conditions on the intensities are required to prove that the control problem is well-posed.

Remark 3.2.2. We generalize the structure of the orders arrivals in section 3.5 by modeling them as Hawkes processes with exponential kernel.

We provide in figure 3.1 a graphical representation of an LOB that may help to get more familiar with the introduced notations.

Bid side pb

Bounding conditions pa+2

Bounding conditions New limit order Cancellation of limit order buy-to-market order

Figure 3.1 -Order book dynamics: in this example, K = 3, q = 1, a ∞ = 4, b ∞ = -4, a = (8, 6, 5), b = (-7, -5, -6).
The spread is equal to 1. At any time, the order book can receive limit orders, market orders or cancel orders.

Market maker strategies

We assume that the market is governed by a FIFO (First In First Out) rule, which means that each limit of the order book is a queue where the first order in the queue is the first one to be executed. We consider a market maker who stands ready to send buy and sell limit orders on a regular and continuous basis at quoted prices. A usual assumption in stochastic control in order to characterize value function as solution of HJB equation is to constrain the control space to be compact. In this spirit, we shall make the following assumption on the market maker's decisions.

(Hcontrol) Assume that at any time, the total number of limit orders placed by the marker maker does not exceed a fixed (possibly large) integer M .

Controls and strategies of the market maker

The market maker can choose at any time to keep, cancel or take positions in the order book (as long as she does not hold more than M positions in the order book). Her positions are fully described by the following M -dimensional vectors ra t , rb t , na t , nb t where ra (resp. rb) records the limits in which the market maker's sell (resp. buy) orders are located; and na (resp. nb) records the ranks in the queues of each market maker's sell (resp. buy) orders. In order to guarantee that the strategy of the market maker is predictable w.r.t. the natural filtration generated by the orders arrivals processes, we shall make the following assumption.

(Harrivals2) The intensities do not depend on the control. Moreover, the market maker does not cross the spread.

We discuss in Appendix how to control the intensity, by transferring the control on the probability measure, see Section 3.A To simplify the theoritical analysis, we also make the following assumption: (Harrivals3) Assume that the market maker does not change her strategy between two orders arrivals of the order book.

Chapter 3. Algorithmic trading in a LOB model

In other words, the market maker makes a decision right after one of the order arrivals processes L ± , C ± , M ± jumps, and keep it until the next the jump of an order arrival.

Note that assumption (Harrivals3) is mild if the order book jumps frequently, since the market maker can change her decisions frequently in such a case. We provide in figure 3.2 a graphical representation of the controlled LOB. Notice that the market maker interacts with the order book by placing orders at some limits. The latter have ranks that evolve after each orders arrivals.

Denote by (T n ) n∈N the sequence of jump times of the order book. We denote by A the set of the admissible strategies, defined as the predictable processes ra t , rb t t≤T such that:

• for all n ∈ N, ra t , rb t ∈ {0, ..., K} M × {0, ..., K} M are constant on T n , T n+1 • ra * , rb * ≥ a0
where, for every vector a: a * = min 0≤i≤K {a i s.t. a i = -1}; and: a0 = argmin 0≤i≤K a i s.t. a i > 0 . The control is the double vector of the positions of the M market maker's orders in the order book. By convention, we set: ra i (t) = -1 if the ith market maker's order is not placed in the order book.

Price Volume & Rank in the queues

Buy Orders of the market maker

Sell Orders of the market maker New market maker sell order Figure 3.2 -Example of market maker's placements and decisions she might make. In this example: her positions are ra = (0, 1, -1, ...), rb = (0, 2, -1...). The ranks vectors associated are na = (2, 1, -1 . . . ) and nb = (4, 2, -1, ...). After each order arrival, she can send new limit orders, cancel some positions, or just keep the latter unchanged.

Controlled order book

We describe the controlled order book by the following state process Z:

Z t := X t , Y t , a t , b t , na t , nb t , pa t , pb t , ra t , rb t ,
where, at time t:

• X t is the cash held by the market maker on a zero interest account.

Existence and characterization of the optimal strategy

• Y t is the inventory of the market maker, i.e. it is the (signed) number of shares held by the market maker.

• pa t is the ask price, i.e. the cheapest price a general market participant is willing to sell stock.

• pb t is the bid price, i.e. the highest price a general market participant is willing to buy stock.

• a t = (a 1 (t), . . . , a K (t)) (resp. b t = (b 1 (t), . . . , b K (t)
)) describes the ask (resp. bid) side: i ∈ {1, . . . , K}, a i (t) is the sum of all the general market participants' sell orders which are i ticks away from the bid (resp. ask) price.

• ra t (resp. ra t ) describes the market maker's orders in the ask (resp. bid) side: for i ∈ {1, ..., M }, ra t (i) is the number of ticks between the i-th market maker's sell (resp. bid) order and the bid (resp. ask) price. By convention, we set ra t (i) = -1 (resp. rb t (i) = -1) if the i-th sell (resp. buy) order of the market maker is not placed in the order book. As a result ra t (i), rb t (i) ∈ {0, . . . , K} ∪ {-1}.

• na t (resp. nb t ) describes the ranks of the market maker's orders in the ask (resp. bid) side. For i ∈ {1, ..., M }, na t (i) ∈ -1, ..., |a| + M (resp. nb t (i) ∈ -1, ..., |b| + M ) is the rank of the i-th sell (resp. buy) orders of the market maker in the queue. By convention, we assume that na t (i) = -1 (resp. nb t (i) = -1) if the i-th sell (resp. buy) order of the market maker is not placed in the order book.

The dynamics of (Z t ) has been computed in the case where the set of admissible strategies is restricted to those where the market maker only makes orders at the two best limits in the bid and ask sides. We present the computations in Section 3.B in the Appendix, for the case where the market maker can only send limit orders at the best-bid and best-ask. We only present the numerical results, in Section 3.4.4, in the case where the market maker can send limit orders at the two best limits in the ask and bid sides.

Existence and characterization of the optimal strategy

Definition and well-posedness of the value function

We denote by V the value function for the market-making problem, defined as follows:

V (t, z) = sup α∈A E α t,z T t f α s , Z s ds + g Z T , (t, z) ∈ [0, T ] × E, (3.3.1)
where:

• A is the set of the admissible strategies, defined in Section 3.2.2.1.

• f and g are respectively the running and terminal reward functions. A usual definition for g is the market maker's wealth function, possibly with an inventory penalization, i.e. g : z →

x + L(y) -ηy 2 where L 2 returns the amount earned from the immediate liquidation of the inventory; where η is the penalization parameter of the latter; and where we remind that y stands for the (signed) market maker's inventory.

2 L is defined as follows:

L(z) =    -1 k=0 a k (pa + k ) + (y -a 0 -... -a -1 )(pa +  ) if y < 0 --1 k=0 b k (pb -k ) + (y + b 0 + ... + b -1 )(pb - ) if y > 0 0 if y = 0,
for all state z = x, y, a, b, na, nb, pa, pb, ra, rb of order book, where:

 = min j j i=0 a i > -y if y < 0 min j j i=0 |b i | > y if y > 0.
• E α t,z stands for the expectation conditioned by Z t = z and when strategy α = (α s ) t≤s<T is followed on [t, T ].

We shall assume conditions on the rewards to insure the well-posedness of the market-making problem.

(Hrewards) The expected running reward is uniformly upper-bounded w.r.t. the strategies in A, i.e.

sup α∈A E α t,z T t f + (Z s , α s ) ds < +∞
holds. The terminal reward g(Z T ) is a.s. no more than linear with respect to the number of events up to time T , denoted by N T in the sequel, i.e. there exists a constant c 1 > 0 such as g(Z T ) ≤ c 1 N T , a.s.. Remark 3.3.1. Under Assumption (Hcontrols), Assumption (Hrewards) holds when g is defined as the wealth of the market maker plus an inventory penalization. In particular, we have g(Z T ) ≤ N T M , where M is the maximal number of orders that can be sent by the market maker, which holds a.s. since the best profit the market maker can make is when her buy (resp. sell) limit orders are all executed, and then the price keeps going to the right (resp. left) direction. Hence the second condition of (Hrewards) holds with c 1 = M .

The following Lemma 3.3.1 tackles the well-posedness of the control problem.

Lemma 3.3.1. Under (Hrewards) and (Hcontrols), the value function is well-defined, i.e.

sup α∈A E α t,z g(Z T ) + T t r α s , Z s ds < +∞,
where, as defined previously, E α t,z [.] stands for the expectation conditioned by the event {Z t = z}, assuming that strategy α ∈ A is followed in [t, T ].

Proof. Denote by (N t ) t the sum of all the arrivals of orders up to time t. Under (Hrewards), we can bound E α t,z T t f α s , Z s ds + g(Z T ) , the reward functional at time t associated to a strategy α ∈ A, as follows:

E α t,z T t f (α s , Z s ) ds + g(Z T ) ≤ sup α E α t,z [g(Z T )] + sup α E α T t f + (Z s , α s ) ds ≤ c 1 sup α E α t,0 [N T ] + sup α E α t,z T t f + (Z s , α s ) ds , (3.3.2)
where once again, for all general process M and all m ∈ E, E α t,m [M T ] stands for the expectation of M T conditioned by M t = m and assuming that the market maker follows strategy α ∈ A in [t, T ]. Let us show that the first term in the r.h.s. of (3.3.2) is bounded. On one hand, we have:

E α t,0 [N T ] ≤ λ ∞ T 0 E (|a| t + |b| t ) dt, (3.3.3) where λ ∞ := λ L + λ C + λ M is
E α t,0 [N T ]
≤ c 4 e c3T wit c 3 and c 4 > 0 that do not depends on α, which proves that the first term in the r.h.s. of (3.3.2) is bounded. Also, its second term in the r.h.s. of (3.3.2) is bounded under (Hrewards). Hence, the reward functional is bounded uniformly in α, which proves that the value function of the considered market-making problem is well-defined.

Markov Decision Process formulation of the market-making problem

In this section, we aim first at reformulating the market-making problem as a Markov Decision Process (MDP), and secondly deriving a characterization of the value function as solution of a Bellman equation.

We consider the Markov Decision Process (MDP) characterized by the following information

[0, T ] × E state space , A z market maker control , λ intensity of the jump , Q transitions kernel
, r reward such that:

• E := R ×N × N K × N K × N M × N M × N M × N M × R × R
is the state space of (Z t ). For z ∈ E, z = x, y, a, b, na, nb, ra, rb, pa, pb where: x is the cash held by the market maker, y her inventory; na (resp. nb) is the M -dimensional vector of the ranks of the market maker's sell (resp. buy) orders in the queues ; ra (resp. rb) is the M -dimensional vector of the number of ticks the M market maker's sell (resp. buy) orders are from the bid (resp. ask) price; pa (resp. pb) is the ask-price (resp. bid-price).

• for every state z ∈ E, denote by A z the space of the admissible controls which is the set of all the actions the market maker can take when the order book is at state z.

A z = ra, rb ∈ {0, ..., K} M × {0, ..., K} M rb * , ra * ≥ a0
, where we define c * = min i {c i |c i = -1} and c0 = argmin

0≤i≤K {c i > 0} for c ∈ N M . The control
is the vectors of positions of the market maker's orders. The condition for the control to be admissible comes from the assumption that the market maker is not allowed to cross the spread.

• Given a market-making strategy α, the stochastic evolution is given by a marked point process (T n , Z n ) where (T n ) is the increasing sequence of jump times of the controlled order book with intensity λ(Z n-1 ). Just after the jump at time T n , the process can jump again, due to the decision of the market maker. Then it remains constant on ]T n , T n+1 [ since the market maker does not change her strategy between two jumps.

We denote by φ a (z) ∈ E the state of the order book at time t such that T n < t < T n+1 , given that Z Tn = z and given that the strategy a has been chosen by the market maker at time T n .

• In the sequel, we denote

[0, T ] × E C := t, z, a ∈ E × {0, . . . , K} 2 M t ∈ [0, T ], z ∈ E, a ∈ A z , and E C := z, a ∈ E × {0, . . . , K} 2 M z ∈ E, a ∈ A z . Q is the stochastic kernel from E C
to E that describes the distribution of the jump goals, i.e., Q B|z, u is the probability that the order book jumps in the set B given that it was at state z ∈ E right before the jump, and the control action u ∈ A z has been chosen right after the jump time.

An admissible policy α = (α t ) is entirely characterized by decision functions

f n : [0, T ] × E → A such that 3 α t = f n (T n , Z n ) for t ∈ T n , T n+1
By abuse of notation, we denote in the sequel by α the sequence of controls (f n ) ∞ n=0 . The intensity of the controlled process (Z t ) is:

λ(z) := λ M + (z) + λ M - (z) + 1≤j≤K λ L + j (z) + 1≤j≤K λ L - j (z) + 1≤j≤K λ C + j (z) + 1≤j≤K λ C - j (z)
It does not depend on the strategy α chosen by the market maker since we assumed that the general participants does not "see" the market maker's orders in the order book. The intensity of the order book process only depends on the vectors a and b.

The transition kernel of the controlled order book, given a state z, is given by:

Q z |z, u =          λ M + (z) λ(z) if z = e M + (φ u (z)) . . . λ C + (z) λ(z) if z = e C + K (φ u (z)),
where φ u (z) is the new state of the controlled order book when decision u as been taken and when the order book was at state z before the decision; e M + (z) is the new state of the order book right after it received a buy market order, given that it was at state z before the jump; and e C ± i (z) is the new state of the order book right after it received a cancel order from a general market participant on its i th ask/bid limit, given that it was at state z.

Let us fix an admissible policy

α = (f n ) ∞ n=0 ∈ A and take t ∈ [0, T ].
Then, for all Borelian B in E, it holds:

P T n+1 -T n ≤ t, Z n+1 ∈ B|T 0 , Z 0 , ...T n , Z n = λ(Z n ) t 0 e -λ(Zn)s Q B|Z Tn , α Tn ds = λ(Z n ) t 0 e -λ(Zn)s Q B|Z Tn , f n (Z n ) ds,
so that the stochastic kernel Q of the MDP is defined as follows:

Q B × C|t, z, α := λ(z) T -t 0 e -λ(z)s 1 B (t + s)Q C|φ α (z), α ds + e -λ(z)(T -t) 1 T ∈B,z∈C ,
for all Borelian sets B ⊂ R + and C ⊂ E, for all (t, z) ∈ [0, T ] × E, and for all α ∈ A.

We denote by (T n , Z n ) n∈N the corresponding state of the controlled Markov chain. It remains to define the value function of this reformulated control problem.

Let r be the running reward function r : [0, T ] × E C → R defined as:

r(t, z, a) := -c z, a e -λ(z)(T -t) (T -t)1 t>T + c z, a 1 λ(z) - e -λ(z)(T -t) λ(z) + e -λ(z)(T -t) g(z)1 t≤T , (3.3.5)
and let us define the cumulated reward functional associated to the discrete-time Markov Decision Model for an admissible policy (f n ) ∞ n=0 as:

V ∞,(fn) (t, z) = E (fn) t,z ∞ n=0 r T n , Z n , f n (T n , Z n ) .
The value function associated to (T n , Z n ) n∈N is then defined as the supremum of the cumulated reward functional over all the admissible controls in A, i.e.

V ∞ (t, z) = sup (fn) ∞ n=0 ∈A V ∞,α (t, z), (t, z) ∈ [0, T ] × E, (3.3.6)
Notice that we used the same notation for admissible controls of the MDP and those of the continuoustime control problem.

Proposition 3.3.1. The value function of the MDP defined by (3.3.6) coincides with (3.3.1), i.e. we have for all (t, z) ∈ E :

V ∞ (t, z) = V (t, z).
(3.3.7)

Existence and characterization of the optimal strategy

Proof.

Step 1: We show that for all α = (f n ) ∈ A and all (t, z)

∈ E V α (t, z) = V (fn) ∞ (t, z). (3.3.8)
Let us first denote by H n := (T 0 , Z 0 , ..., T n , Z n ). Notice then that for all admissible strategy α:

V α (t, z) = E α t,z ∞ n=0 1 T >Tn+1 T n+1 -T n c Z n , α n + 1 [Tn≤T <Tn+1) g(Z T ) -ηY T 2 + (T -T n )c Z n , α n = ∞ n=0 E (fn) t,z r T n , Z n , f n (T n , Z n ) , (3.3.9)
where we conditioned by H n between the first and the second line. We recognize V (fn) ∞ in the r.h.s. of (3.3.9), so that the proof of (3.3.8) is completed.

It remains to take the supremum over all the admissible strategies A in (3.3.8) to get (3.3.7).

From Proposition 3.3.1, we deduce that the value function of the market-making problem is the same as the value function V ∞ of the discrete-time MDP. We now aim at solving the MDP control problem. To proceed, we first define the maximal reward mapping for the non finite horizon MDP:

(T v)(t, z) := sup a∈Az r(t, z, a) + v(t , z )Q(t , z |t, φ a (z), a) = sup a∈Az r(t, z, a) + λ(z) T -t 0 e -λ(z)s v(t + s, z )Q dz |φ a (z), a ds , (3.3.10)
where we recall that:

• φ α (z) is the new state of the order book when the market maker follows the strategy α and the order book is at state z before the decision is taken.

• λ(z) is the intensity of the order book process given that the order book is at state z.

We shall tighten assumption (Hrewards) in order to guarantee existence and uniqueness of a solution to (3.3.1), as well as characterizing the latter.

(HrewardsBis): The running and terminal rewards are at most quadratic w.r.t. the state variable, uniformly w.r.t. the control variable, i.e.

1. The running reward f is such that |c| is uniformly bounded by a quadratic in z function, i.e. there exists c 5 > 0 such that:

∀(z, a) ∈ E × A, |f (z, a)| ≤ c 5 (1 + |z| 2 ).
2. The terminal reward g has no more than a quadratic growth, i.e. there exists c 6 > 0 such that:

∀z ∈ E, |g(z)| ≤ c 6 (1 + |z| 2 ).
Remark 3.3.2. Assumption (HrewardsBis) holds in the case where g is the terminal wealth of the market maker plus a penalization of her inventory, and where with no running reward, i.e. f = 0.

The main result of this section is the following theorem that gives existence and uniqueness of a solution to (3.3.1), and moreover characterizes the latter as fixed point of the maximal reward operator defined in (3.3.10).

Chapter 3. Algorithmic trading in a LOB model Theorem 3.3.1. T admits a unique fixed point v which coincides with the value function of the MDP. Moreover we have:

v = V ∞ = V.
Denote by f * the maximizer of the operator T . Then f * , f * , ... is an optimal stationary (in the MDP sense) policy.

Remark 3.3.3. Theorem 3.3.1 states that the optimal strategy is stationary in the MDP formulation of the problem, but of course, it is not stationary for the original time-continuous trading problem with finite horizon (3.3.1), since the time component is not a state variable anymore in the original formulation. Actually, given n ∈ N and the state of order book z at that time, the optimal decision to take at time T n is given by f * T n , z .

We devoted the next section to the proof of Theorem 3.3.1.

Proof of Theorem 3.3.1

Remind first that we defined in the previous section

E C := z, a ∈ E×{0, . . . , K} 2 M z ∈ E, a ∈ A z and [0, T ] × E C := t, z, a ∈ [0, T ] × E × {0, . . . , K} 2 M t ∈ [0, T ], z ∈ E, a ∈ A z .
Let us define the bounding functions:

Definition 3.3.1. A measurable function b : E → R + is called a bounding function for the controlled process (Z t ) if there exists positive constants c c , c g , c Q , c φ such that: 1. |f (z, a)| ≤ c c b(z) for all (z, a) ∈ E C . 2. |g(z)| ≤ c g b(z) for all z in E. 3. b(z )Q (dz |z, a) ≤ c Q b(z) for all (z, a) ∈ E C . 4. b(φ α t (z)) ≤ c φ b(z) for all (t, z, α) ∈ [0, T ] × E C .
Proposition 3.3.2. Let b be such that :

∀z ∈ E, b(z) := 1 + |z| 2 .
Then, b is a bounding function for the controlled process (Z t ), under Assumption (HrewardsBis).

Proof. Let us check that b defined in 3.3.2 satisfies the four assertions of definition 3.3.1.

• Assertion 1 and 2 of definition 3.3.1 holds under (HrewardsBis).

• First notice that ra, rb are bounded by √ M K (where we recall that K is the number of limits in each side of the order book, and M is the biggest number of limit orders that the market maker is allowed to send in the market). Secondly, pa ∈ B(pa, K), pb ∈ B(pb, K), where B(x, r) is the ball centered in x with radius r > 0, because of the limit conditions that we imposed in our LOB model. And last, we can see that |a | ≤ |a| + a ∞ K. These three bounds are linear w.r.t. z so that assertion 3 holds.

• φ α (z) = z α only differs from z by its na, nb, and ra, rb components.

Existence and characterization of the optimal strategy

Let us define

Λ := (4K + 2)sup λ M ± |a| + |b| , λ L ± |a| + |b| , λ C ± |a(z)| + |b(z)| .
Note that Λ is well-defined under (Harrivals).

|r(t, z, a)| ≤ c g b(t, z), b(s, z )Q(ds, dz |t, z, a) ≤ c φ c Q e C(T -t) 1 1 + γ 0 b(t, z), with C = γ 0 ΛK(4K + 2) |a| ∞ + |b| ∞ .
Proof. Let z = x , y , a , b , na , nb , ra , rb be the state of the order book after an exogenous jump occursn given that it was at state z before the jump. Since

|a | ≤ |a| + a ∞ K and |b | ≤ |b| + b ∞
, where a ∞ and b ∞ are defined as the border conditions of the order book, we have:

γ(z ) ≤ γ(z) + C, (3.3.11) with C = γ 0 ΛK(4K + 2)(a ∞ + b ∞ ). Then, we get: b(s, z )Q(ds, dz |t, φ α (z), α) = λ(z) T -t 0 e -λ(z)s b(t + s, z )Q dz |φ α s (z), α ds = λ(z) T -t 0 e -λ(z)s b(z )e γ(z )(T -(t+s)) Q dz |φ α s (z), α ds ≤ λ(z) T -t 0 e -λ(z)s b(z )e (γ(z)+C)(T -(t+s)) Q dz |φ α s (z), α ds ≤ λ(z) T -t 0 e -λ(z)s e (γ(z)+C)(T -(t+s)) b(z )Q dz |φ α s (z), α ds ≤ λ(z) T -t 0 e -λ(z)s e (γ(z)+C)(T -(t+s)) c Q c φ b(z)ds ≤ λ(z)c Q c φ λ(z) + γ(z) + C e (γ(z)+C)(T -t) 1 -e -(T -t)(λ(z)+γ(z)+C) b(z) ≤ c Q c φ λ(z) λ(z) + γ(z) + C e C(T -t) 1 -e -(T -t)(λ(z)+γ(z)+C) b(t, z),
where we applied (3.3.11) at the thrid line. It remains to notice that

λ(z) λ(z) + γ(z) + C = λ(z) λ(z) 1 + γ 0 + γ 0 Λ(|a| + |b|) -λ(z) ≥0 ≤ 1 1 + γ 0 ,
we complete the proof of the Proposition.

Denote by . b the weighted supremum norm such that for all measurable function v : E → R,

v b := sup (t,z)∈E |v(t, z)| b(t, z) ,
Chapter 3. Algorithmic trading in a LOB model and define the set:

B b := v : E → R|v is measurable and v b < ∞ .
Moreover let us define

α b := sup (t,z,α)∈E ×R b(s, z )Q(ds, dz |t, φ α (z), α) b(t, z) .
From the preceding estimations we can bound α b as follows:

α b ≤ c Q c φ 1 1 + γ 0 e CT ,
So that, by taking:

γ 0 = c Q c φ e CT , we get: α b < 1.
In the sequel, we then assume w.l.o.g. that α b < 1. Recall that the maximal reward mapping for the MDP has been defined as:

T v : (t, z) → sup a∈Az r(t, z, a) + λ(z) T -t 0 e -λ(z)s v(t + s, z )Q dz |φ a (z), a ds
It is straightforward to see that:

T v -T w b ≤ α b v -w b , (3.3.12)
which implies that T is contracting, since α b < 1.

Let M be the set of all the continuous function in

B b . Since b is continuous, (M, . b ) is a Banach space. T sends M to M. Indeed, for all continuous function v in B b , (t, z, a) → r(t, z, a)+λ(z) T -t 0 e -λ(z)s v(t+ s, z )Q dz |φ a (z), a ds is continuous on [0, T ] × E C . A z is
finite, so we get the continuity of the application:

T v : (t, z) → sup a∈Az r(t, z, a) + λ(z) T -t 0 e -λ(z)s v(t + s, z )Q dz |φ a (z), a ds .
Proposition 3.3.4. There exists a maximizer for T , i.e. let v ∈ M, then there exists a Borelian function f : [0, T ] × E → A such that for all (t, z) ∈ E :

T v t, z, f t, z = sup a∈A r(t, z, a) + λ(z) T -t 0 e -λ(z)s v(t + s, z )Q dz |φ a (z), a ds Proof. D * (t, z) = a ∈ A T a v(t, z) = T v(t, z) is finite, so it is compact. So (t, z) → D * (t, z) is a compact-valued mapping. Since the application (t, z, a) → T a (t, z) -T (t, z) is continuous, we get that D * = (t, z, a) ∈ E C T a v(t, z) = T v(t, z) is borelian.
Applying the measurable selection theorem yields to the existence of the maximizer. (see [START_REF] Bäuerle | Markov Decision Processes with Applications to Finance[END_REF] p.352) Lemma 3.3.2. The following holds:

sup α∈A E α t,z ∞ k=n |r(T k , Z k )| ≤ α n b 1 -α b b(t, z),
and in particular, we have:

lim n→∞ sup α∈A E α t,z ∞ k=n r t k , Z k = 0.
Proof. By conditioning we get

E α t,z r(T k , Z k ) ≤ c g α b k b(t, z) for k ∈ N,
and for all α ∈ A. It remains to sum this inequality to complete the proof of Lemma 3.3.2.

Numerical Algorithm

We can now prove Theorem 3.3.1.

Proof. We divided the proof of Theorem 3.3.1 into four steps.

Step 1: Inequality (3.3.12) and Proposition 3.3.3 imply that T is a stable and contracting operator defined on the Banach space M. Banach's fixed point theorem states that T admits a fixed point, i.e. there exists a function v ∈ M such that v = T v, and moreover we have v = lim n→∞ T n 0. Notice that T N 0 coincides with v 0 defined recursively by the following Bellman equation:

v N = 0 v n = T v n+1 for n = N -1, ..., 0. (3.3.13)
The solution of the Bellman equation is always larger than the value function of the MDP associated (see e.g. Theorem 2.3.7 p.22 in [START_REF] Bäuerle | Markov Decision Processes with Applications to Finance[END_REF]). Then we have:

T n 0 ≥ sup (f k ) E (f k ) n n-1 k=0 r(t k , X k ) =: J n
, where J n is the value function of the MDP with finite horizon n and terminal reward 0, associated to (3.3.13). Moreover, by Lemma 7.1.4 p.197 in [START_REF] Bäuerle | Markov Decision Processes with Applications to Finance[END_REF], we know that J n n converges as n → ∞ to a limit that we denote by J. Passing at the limit in the previous inequality we get:

lim n→∞ T n 0 ≥ J, i.e. v ≥ J. (3.3.14)
Step 2: Let us fix a strategy α ∈ A, and take n ∈ N. We denote

J n (α) := E (α k ) 0 n-1 k=0 r(t k , X k )
, the reward functional associated to the control α on the discrete finite time horizon {0, . . . , n}. By definition, we have J n (α) ≤ J n . We get by letting n → ∞: lim n→+∞ J n (α) =: J ∞ (α) ≤ J. Taking the supremum over all the admissible strategies α finally leads to:

V ∞ ≤ J.
(3.3.15)

Step 3: Let us denote by f a maximizer of T associated to v, which exists, as stated in Proposition 3.3.4. v is the fixed point of

T so that v = T n f (v), for n ∈ N. Moreover v ≤ δ where δ := sup α∈A E ∞ k=0 r + (Z k , α k ) , so that T n f (v) ≤ T n f 0 + T n o δ, where T n o δ = sup α E α n ∞ k=n r + (t k , Z k ) . Lemma 3.3.2 implies that T n o δ → 0 as n → ∞.
V ∞ ≤ J ≤ v ≤ J f ≤ V ∞ . (3.3.18)
All the inequalities in (3.3.18) are then equalities, which completes the proof of Theorem 3.3.1.

Numerical Algorithm

In this section, we first introduce an algorithm to numerically solve a general class of discrete-time control problem with finite horizon, and then apply it on the trading problem (3.3.1).

Framework

Let us consider a general discrete-time stochastic control problem over a finite horizon N ∈ N \ {0}.

The dynamics of the controlled state process

Z α = (Z α n ) n valued in R d is given by Z α n+1 = F (Z α n , α n , ε n+1 ), n = 0, . . . , N -1, Z α 0 = z ∈ R d ,
with (ε n ) n is a sequence of i.i.d. random variables valued in some Borel space (E, B(E)), and defined on some probability space (Ω, F, P) equipped with the filtration F = (F n ) n generated by the noise

(ε n ) n (F 0 is the trivial σ-algebra), the control α = (α n ) n is an F-adapted process valued in A ⊂ R q , and F is a measurable function from R d × R q ×E into R d .
Given a running cost function f defined on R d × R q , a terminal cost function g defined on R d , the cost functional associated to a control process α is

J(α) = E N -1 n=0 f (Z α n , α n ) + g(Z α N ) .
The set A of admissible control is the set of control processes α satisfying some integrability conditions ensuring that the cost functional J(α) is well-defined and finite. The control problem, also called Markov decision process (MDP), is formulated as

V 0 (x 0 ) := sup α∈A J(α),
and the goal is to find an optimal control α * ∈ A, i.e., attaining the optimal value: V 0 (z) = J(α * ).

Notice that problem (3.4.1)-(3.4.2) may also be viewed as the time discretization of a continuous time stochastic control problem, in which case, F is typically the Euler scheme for a controlled diffusion process.

Problem (3.4.2) is tackled by the dynamic programming approach. For n = N, . . . , 0, the value function V n at time n is characterized as solution of the following backward (Bellman) equation:

V N (z) = g(z) V n (z) = sup a∈A f (z, a) + E a n,z [V n+1 (Z n+1 )] , z ∈ R d , (3.4.3)
Moreover, when the supremum is attained in the DP formula at any time n by a * n (z), we get an optimal control in feedback form given by: α * = (a * n (Z * n )) n where Z * = Z α * is the Markov process defined by

Z * n+1 = F (Z * n , a * n (Z * n ), ε n+1 ), n = 0, . . . , N -1, Z * 0 = z.
There are two usual ways that have been studied in the literature, to solve numerically (3.4.3): some methods make use of quantization to discretize to state space and approximate the conditional expectations by cubature methods; another way is to rely on MC regress-now or Later methods to regress the value functions V n+1 at time n for n = 0, . . . , N -1 on basis functions or neural networks. See e.g. [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF] for the regress-now and [START_REF] Balata | Regress-Later Monte Carlo for optimal control of Markov processes[END_REF] for the regress-Later methods for algorithms using basis functions, and e.g. [START_REF] Huré | Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis[END_REF] for regression on neural networks based on regress-now or regresslater techniques.

Presentation and rate of convergence of the Qknn algorithm

In this section, we present an algorithm based on k-nn estimates for local non-parametric regression of the value function, and optimal quantization to quantize the exogenous noise, in order to numerically solve (3.4.3).

Let us first introduce some ingredients of the quantization approximation:

• We denote by ε a K-quantizer of the E-valued random variable ε n+1 ∼ ε 1 , that is a discrete random variable on a grid Γ = {e 1 , . . . , e K } ⊂ E K defined by

ε = Proj Γ (ε 1 ) := K =1 e l 1 ε1∈Ci(Γ) ,
where C 1 (Γ), . . ., C K (Γ) are Voronoi tesselations of Γ, i.e., Borel partitions of the Euclidian space (E, |.|) satisfying

C (Γ) ⊂ e ∈ E : |e -e | = min j=1,...,K |e -e j | .
The discrete law of ε is then characterized by

p := P[ε = e ] = P[ε 1 ∈ C (Γ)], = 1, . . . , K.
The grid points (e ) which minimize the L 2 -quantization error ε 1 -ε 2 lead to the so-called optimal L-quantizer, and can be obtained by a stochastic gradient descent method, known as Kohonen algorithm or competitive learning vector quantization (CLVQ) algorithm, which also provides as a byproduct an estimation of the associated weights (p ). We refer to [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF] for a description of the algorithm, and mention that for the normal distribution, the optimal grids and the weights of the Voronoi tesselations are precomputed on the website http://www. quantize.maths-fi.com

• Recalling the dynamics (3.4.1), the conditional expectation operator is equal to

P âM n (z) W (x) = E W (Z âM n n+1 )|Z n = x = E W (F (z, âM n (z), ε 1 )) , z ∈,
that we shall approximate analytically by quantization via:

P âM n (z) W (z) := E W (F (z, âM n (z), ε)) = K =1 p W (F (z, âM n (z), e )).
Let us secondly introduce the notion of training distribution that will be used to build the estimators of value functions at time n, for n = 0, . . . , N -1. Let us consider a measure µ on the state space E. We refer to it in the sequel as the training measure. Let us take a large integer M , and for n = 0, . . . , N , introduce Γ n = Z The proposed algorithm reads as:

       V Q N (z) = g(z), for z ∈ Γ N , Qn (z, a) = K =1 p f (z, a) + V Q n+1 Proj n+1 F z, e , a , V Q n (z) = sup a∈A Qn (z, a), for z ∈ Γ n , n = 0, . . . , N -1. (3.4.4)
where, for n = 0, . . . , N , Proj n (z) stands for the closest neighbor of z ∈ E in the grid Γ n , i.e. the operator z → Proj n (z) is actually the euclidean projection on the grid Γ n .

Remark 3.4.1. We could have generalized the operator Proj n by considering z

∈ E → ẑ = 1 k k j=1 w j Z (j)
n , with the weight w j such as

w j (z) = z -Z (j) n k i=1 z -Z (i) n
, and where Z (j) n stands for the j th nearest neighbors of z in Γ n , for j = 1, . . . , k. This generalization brings continuity to the estimates.

In the sequel, we refer to (3.4.4) as the Qknn algorithm.

Chapter 3. Algorithmic trading in a LOB model

We shall make the following assumption on the transition probability of (Z n ) 0≤n≤N , to guarantee the convergence of the Qknn algorithm.

(Htrans) Assume that the transition probability P(Z n+1 ∈ A Z n = z, a) conditioned by Z n = z when control a is followed at time n admits a density r w.r.t. the training measure µ, which is uniformly bounded and lipschitz w.r.t. the state variable z, i.e. there exists r ∞ > 0 such that for all z ∈ E and control u taken at time n:

|r(y; n, x, a)| ≤ r ∞ and |r(y; n, x, a) -r(y; n, x , a)| ≤ [r] L |x -x |
and r is defined as follows: Denote by Supp(µ) the support of µ. We shall assume smoothness conditions on µ and F to provide a bound on the projection error.

P(Z n+1 ∈ O Z n = z, u) = O r(y; n, x, a)dµ(y).
(Hµ) We assume Supp(µ) to be bounded, and denote by µ ∞ the smallest real such that Supp(µ) ⊂ B (0, µ ∞ ). Moreover, we assume x ∈ E → µ B(x, η) to be Lipschitz, uniformly w.r.t. η, and we denote by [µ] L its Lipschitz constant.

(HF) For x ∈ E and a ∈ A, assume F to be L 1 -Lipschitz w.r.t. the noise component ε, i.e., there exists [F ] L > 0 such that for all x ∈ E and a ∈ A, for all r.v. ε and ε , we have:

E [|F (x, a, ε) -F (x, a, ε )|] ≤ [F ] L E [|ε -ε |]
We now state the main result of this section whose proof is postponed in Appendix 3.C. Theorem 3.4.1. Take K = M 2+d points for the optimal quantization of the exogenous noise ε n , n = 1, . . . , N . There exist constants [ V Q n ] L > 0, that only depends on the Lipschitz coefficients of f , g and F , such that, under (Htrans), it holds for n = 0, ..., N -1, as M → +∞:

V Q n (X n ) -V n (X n ) 2 ≤ N k=n+1 r N -k ∞ V Q k L ε proj k + [F ] L ε Q k + O 1 M 1/d , (3.4.5)
where ε Q k := εk -ε k 2 stands for the quantization error, and

ε proj n := sup a∈A Proj n+1 (F (X n , a, εn )) -F (X n , a, εn ) 2
stands for the projection error, when decision a is taken at time n.

Remark 3.4.2. The constants [ V Q n ] L > 0 are defined in (3.C.8).
From Theorem 3.4.1, we can deduce consistency and provide a rate of convergence for the estimator V Q n , n = 0, . . . , N -1, under some rather tough yet usual compactness conditions on the state space.

Corollary 3.4.1. Under (Hµ) and (HF), the Qknn-estimator V Q n is consistent for n = 0, . . . , N -1, when taking M d+1 points for the quantization; and moreover, we have for n = 0, ..., N -1, as M → +∞:

V Q n (X n ) -V n (X n ) 2 ≤ O 1 M 1/d .
Proof. We postpone the proof of Theorem 3.4.1 to Appendix 3.C.

Numerical Algorithm

Qknn agorithm applied to the order book control problem (3.3.1)

We recall the expression of the controlled order book, as described in section 3:

Z t = X t , Y t , a t , b t , na t , nb t , pa t , pb t , ra t , rb t .
In section 3.3.3, we proved that the value function V is characterized as the unique solution of the Bellman equation (3.3.10). In this section, some implementation details on the Qknn algorithm are presented in order to numerically solve the market-making problem.

Training set design

Inspired by [START_REF] Fiorin | Product Markovian quantization of a diffusion process with applications to finance[END_REF], we use product-quantization method and randomization techniques to build the training set Γ n on which we project (T n , Z n ) that lies on [0, T ] × E, where T n and Z n stands for the n th jump of Z and the state of Z at time t n , i.e. Z n = Z Tn , for n ≥ 0. This basic idea of Control Randomization consists in replacing in the dynamics of Z the endogenous control by an exogenous control (I Tn ) n≥0 , as introduced in [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF]. In order to alleviate the notations, we denote by I n the control taken at time T n , for n ≥ 0.

Initialization. Set: Γ E 0 = {z} and Γ T 0 = {0}. Randomize the control, using e.g. uniform distribution on A at each time step, and then simulate D randomized processes to generate

(T k n , Z k n ) N,D n=0,k=1 . For all n = 1, . . . , N , set Γ T n = {T k n , 1 ≤ k ≤ D},
which stands for the grid associated to the quantization of the n th jump time T n , and set Γ E n = {Z k n , 1 ≤ k ≤ D} which stands for the grid associated to the quantization of the state Z n of Z at time T n .

Remark 3.4.3. The way we chose our training sets is often referred to as an exploration strategy in the reinforcement learning literature. Of course, if one has ideas or good guess of where to optimally drive the controlled process, she shouldn't follow an exploration-type strategy to build the training set, but should rather use the guess to build it, which is referred to as the exploitation strategy in the reinforcement learning and the stochastic bandits literature. We refer to [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF] for several other applications of the exploration strategy to build training sets.

Let F and G be the Borelian functions such that Z n = F Z n-1 , d n , I n and T n = G T n-1 , n , I n , where n ∼ E(1) stands for the temporal noise, and d n is the state noise, for n ≥ 0.

Let us fix N ≥ 1 and consider T n , Z n N n=0 , the dimension-wise projection of T n , Z n N n=0 on the grids Γ T n × Γ E n , n = 0, . . . , N , i.e. T 0 = 0, Z 0 = z, and    T n = Proj G T n-1 , n , I n , Γ T n , Z n = Proj F Z n-1 , d n , I n , Γ E n , for n = 1, . . . , N.
T n , Z n , I n n∈{0,N } is a Markov chain, and its probability transition matrix at time n = 1, ..., N reads:

pij k (a) = P tk = t j k , Z k = z j k tk-1 = t i k-1 , Z k-1 = z i k-1 , I k = a = βij k pi k-1 , i = 1, ..., N k-1 , j = 1, ..., N k , a ∈ A
where:

pi k-1 = P tk-1 = t i k-1 , Z k-1 = z i k-1 = P F tk-2 , Z k-2 , k-1 , d k-1 ∈ C i (Γ k-1 × Γ E k-1 ) if k ≥ 2 1 if k = 1 βij k = P tk-1 = t i k-1 , Z k-1 = z i k-1 , tk = t j k , Z k = z j k = P F k tk-2 , Z k-2 , k-1 , d k-1 ∈ C i (Γ k-1 × Γ E k-1 ); F k tk-1 , Z k-1 , k , d k ∈ C i (Γ k × Γ E k ) if k ≥ 2 1 if k = 1
Chapter 3. Algorithmic trading in a LOB model and where, for all i, 0 ≤ i ≤ D, for all k ∈ N, we denoted by

C i (Γ k × Γ E k ) the Voronoï cell associated to the point (T i k , z i k ). Define then T Q n , Z Q n N n=0 as temporal noise-quantized version of T n , Z n , I n N n=0
. Note that we do not need to quantize the spacial noise since this noise already takes a finite number of states. Let εn be the quantized process associated to n . The process

T Q n , Z Q n N n=0 is then defined as follows: Z Q 0 = z, T Q 0 = 0 and ∀1 ≤ n ≤ N :    T Q n = Proj G tn-1 , εn , I n , Γ T n , Z Q n = Proj F Z n-1 , d n , I n , Γ E n . Denote by V Q,(N,D) n N n=0
the solution of the Bellman equation associated to

T Q n , Z Q n N n=0 : (B Q N,D ) :      V Q,(N,D) N = 0 V Q,(N,D) n (t, z) = r(t, z, a) + sup a∈A E a t,z V Q,(N,D) n+1 T Q n+1 , Z Q n+1
, for n = 0, . . . , N, where E a t,z [.] stands for the expectation conditioned by the events

T Q n = t,Z Q n = z and when decision I n = a is taken at time t.
We wrote the pseudo-code of the Qknn algorithm to compute (B

Q N,D ) in Algorithm 3.1.
We discuss in Remark 3.4.4 the reasons why we can apply Theorem 3.4.1. Remark 3.4.4. When the number of jumps of the LOB N ≥ 1 is fixed, the set of all the states that can take the controlled order book by jumping less than N times, denoted by K in the sequel, is finite. Hence, the reward function r, defined in (3.3.5), is bounded and Lipschitz on K.

The following proposition states that V Q,(N,D) n , built from the combination of time-discretization, k-nearest neighbors and optimal quantization methods, is a consistent estimator of the value function at time T n , for n = 0, . . . , N -1. It provides a rate of convergence for the Qknn-estimations of the value functions. Proposition 3.4.1. The estimators of the value functions provided by Qknn algorithm are consistent. Moreover, it holds as M → +∞:

V Q,(N,M ) n T n , Z n -V n T n , Z n M,2 = O α N + 1 M 2/d , for n = 0, . . . , N -1,
where we denote by . M,2 the L 2 (µ) norm conditioned by the training sets that have been used to build the estimator V Q,(N,M ) n+1

.

Proof. Splitting the error of time cutting and quantization, we get:

V n T n , Z n -V (N,M ) n T n , Z n M,2 ≤ V n T n , Z n -V (N ) n T n , Z n M,2 + V (N ) n T n , Z n -V (N,M ) n T n , Z n M,2 . (3.4.8)
Step 1: Applying Lemma 3.3.2, we get the following bound on the first term in the r.h.s. of (3.4.8):

V n T n , Z n -V (N ) n T n , Z n M,2 ≤ α N 1 -α b ∞ , (3.4.9) 3.4. Numerical Algorithm Algorithm 3.1 Generic Qknn Algorithm Inputs:
-N : number of time steps z: state in E at time T 0 = 0 -Γ ε = {e 1 , . . . , e L } and (p ) L =1 : the grid and the weights for the optimal quantization of (ε n ) N n=1 . -Γ n and Γ E n the grids for the projection of respectively the time and the state components at time n, for n = 0, . . . , N .

1: for i = N -1, . . . , 0 do 2:
Compute the approximated Qknn-value at time n:

Qn (z, a) = r (T n , z, a) + L =1 p V Q n+1 Proj G(z, e , a), Γ T n+1 , Proj F (z, e , a), Γ E n+1 , for (z, a) ∈ Γ n × A z ; 3:
Compute the optimal control at time n

Ân (z) ∈ argmin a∈Az Qn (z, a), for z ∈ Γ n ,
where the argmin is easy to compute since A z is finite for all z ∈ E;

4:

Estimate analytically by quantization the value function:

V Q n (z) = Qn z, Ân (z) , ∀z ∈ Γ n ; 5: end for Output: -( V Q 0 ): Estimate of V (0, z); Chapter 3. Algorithmic trading in a LOB model K , N → ∞ V (N ) V = V ∞ V (N,K) N → ∞ K → ∞ Figure 3.
3 -Numerical resolution of the algorithmic control problem. We first bound the number of events and then quantize the state space.

where b ∞ stands for the supremum of b over [0, T ] × E.

Step 2: Note that the assumptions of Theorem 3.4.1 are met as noticed in Remark 3.4.4, so that the latter provides the following bound for the second term in the r.h.s. of (3.4.8): We provide a diagram in figure 3.3 to summarize the two main steps in the estimation of the value function of the market-making problem defined in (3.3.1).

V (N ) n T n , Z n -V Q,(N,M ) n T n , Z n M,2 = M →∞ O 1 M 2/d . ( 3

Numerical results

In this section, we propose several settings to test the efficiency of Qknn on simulated order books. We take no running reward, i.e. f = 0, and take the wealth of the market maker as terminal reward, i.e. g(z) = x. The intensities are taken constant in some tests, and state dependent on other tests. The values of the intensities are similar to the ones in [START_REF] Huang | Simulating and analyzing order book data: The queue-reactive model[END_REF]. Although the intensities are assumed uncontrolled in section 3.3 for predictability reasons, the latter are controlled processes in this section, i.e. the intensities of the order arrivals depends on the orders in the order book from all the participant plus the ones of the market maker. The optimal trading strategies have been computed among two different classes of strategies: in section 3.4.4.1, we tested the algorithm to approximate the optimal strategy among those where the market maker is only allowed to place orders only at the best bid and the best ask. The dynamics of the controlled order book for such a class of controls are available in Section 3.B in the Appendix. In Section 3.4.4.2, we computed the optimal trading strategy among the class of the strategies where the market maker allows herself to place orders on the two best limits on each side of the order book. Note that the second class of controls is more general than the first one.

The search of the k nearest neighbors, that arise when estimating the conditional expectations using the Qknn algorithm, is very time-consuming; especially in the considered market-making problem which is of dimension more than 10. The efficiency of Qknn then highly depends on the algorithm used to find the k nearest neighbors in high-dimension. Qknn algorithm has been implemented using the Fast Library for Approximate Nearest Neighbors algorithm (FLANN), introduced in [ML09] and already available in libraries in C++, Python, Julia and many other languages. This algorithm is 3.4. Numerical Algorithm based on tree methods. Note that recent algorithms based on graph also proved to perform well, and can also be used.

Case 1:

The market maker only place orders at the best ask and best bid.

Denote by A1lim the class of controls where the placements of orders in allowed on the best ask and best bid exclusively. We implement the Qknn algorithm to compute the optimal strategy among those in A1lim. We then compared the optimal strategy with a naive strategy which consists in always placing one order at the best bid and one order at the best ask. The naive strategy is called 11 in the plots, and can be seen as a benchmark. The naive strategy is a good benchmark when the model for the intensities of order arrivals is symmetrical, i.e. the intensities for the bid and the ask sides are the same. Indeed, in this case, the market maker can expect to earn the spread in average.

Numerical results:

In Figure 3.4, we take constant intensities to model the limit and market orders arrivals, and linear intensity to model the cancel orders. In this setting, as we can see in the figure, the strategy computed using Qknn algorithm performs as well as the naive strategy. Note that, obviously, the market maker has to take enough points for the state quantization in order for Qknn algorithm to perform well. In figure 3.5, we plotted the P&L of the market maker when the latter compute the optimal strategy using only 6000 points for the state space discretization, and for such a low number of points for the grid, Qknn algorithm performs poorly.

In Figure 3.6, we plotted the empirical histogram of the P&L of the market maker using the Qknn-estimated optimal strategy, computed with grids of size N = 1000, 10000, 100000, 1000000 for the state space discretization; and the empirical histogram of the P&L of the market maker using the naive strategy. One can see that the larger the size of the grids are, the better the Qknn-estimation of the optimal strategy is. We plot in Figure 3.7 the results of simulations run taking a short terminal time T=1, and intensities that depend on the size of the queues. In this setting, notice that the naive strategy does not perform well anymore, but the Qknn algorithm still does well, when the market maker takes enough points for state space discretization.

In figure 3.7, we plot the P&L of the market maker following the Qknn strategy and the naive strategy, and we took the same parameters as in figure 3.6 to run the simulations expect from the terminal time that we set as T=10. As expected 4 , the expected wealth of the market maker is larger when terminal time is larger and when the latter follows the Qknn-estimated optimal strategy. Note that the expectation of the latter remains the same when she follows the naive strategy.

Case 2: the market maker place orders on the first two limits of the Orders Book

We extend the class of admissible controls to the ones where the market maker places order on the first two limits on the bid and ask sides of the order book. Denote by A2lim the latter. We run simulations to test the Qknn algorithm on A2lim. In figure 3.8 and figure 3.9, we plot the empirical distributions of the P&L when the market maker follows the three different strategies:

• Qknn-estimated optimal strategy among those in A2lim (PLOpt2lim).

• Qknn-estimated optimal strategy among those in A1lim (PLOpt1lim).

• naive strategy, i.e. always place orders on the best bid and best ask queues (PL11). i depend on the state of the order book. Figure 3.6a shows the P&L of the market maker when following the Qknn-estimated optimal strategy computed with 1000 points for the state space discretization. Figure 3.6b shows the P&L when following the Qknn-estimated optimal strategy computed with 9000 points for the state space discretization. Figure 3.6c shows the P&L when following the Qknn-estimated optimal strategy computed with 100000 points for the state space discretization. Figure 3.6d shows the P&L when following the Qknn-estimated optimal strategy computed with 1000000 points for the state space discretization. The reader can see that the market maker increases her expected terminal wealth by taking more and more points for the state space discretization. Also, the naive strategy is beaten when the intensities are state dependent. Figure 3.8 -P&L of the market maker who follows optimal strategies and the naive strategy (PL11). Short Terminal Time. asymmetrical intensities for the market order arrivals: the intensity for the buying market order process is taken higher than the one for the selling market order process. The wealth of the market maker is greater when she places orders on the two first limits of each sides of the order book, rather than when she places orders only on the best limits at the bid and ask sides. Figure 3.9 -P&L when following the optimal strategy or the naive strategy (PL11). Long Terminal Time. Symmetrical intensities for the arrival of market orders. 400000 points for the quantization. Notice that the Qknn strategy computed on the extended class of controls, i.e. order placements on the two first limits (StratOpt2lim), performs as well as the one computed on the original class of controls, i.e. order placements on the best-bid and best-ask (StratOpt1lim).

Note that the P&L of the market maker is always better when the class of admissible controls is extended, see figure 3.8, but in some models of order books, the extended set of controls doest not improve the P&L, i.e. sup α∈A2lim

V α = sup α∈A1lim V α .

Model extension to Hawkes Processes

We consider in this section a market maker who aims at maximizing a function of her terminal wealth, penalizing her inventory at terminal time T in the case where the orders arrivals are driven by Hawkes processes.

Let us first present the model with Hawkes processes for the LOB.

Model for the LOB:

We assume that the order book receives limit, cancel, and market orders. We denote by L + (resp. L -) the limit order arrivals process the ask (resp. bid) side; by C + (resp. C -) the cancel order on the ask (resp. bid) side; and by M + (resp. M -) the buy (resp. sell) market order arrivals Notice that the strategies computed by Qknn algorithm when taking A2lim performs as well as the one computed on the two best limits of the order book exclusively. Then, in this setting, placing orders only at the best-ask and best-bid seems to be the the optimal strategy. processes. In this section, the limit orders arrivals are assumed to follow Hawkes processes dynamics, and moreover we assume the kernel to be exponential. The order arrivals are then modeled by a (4K+2)-variate Hawkes process (N t ) with a vector of exogenous intensities λ 0 and exponential kernel φ, i.e. φ ij (t) = α ij β ij e βij t 1 t≥0 . Note that in the presented model, the following holds:

(Hλ) λ is assumed to be independent of the control.

Denoting by D = 4K + 2 the dimension of (N t ), the m th component of the intensity λ of N t writes, under (Hλ):

λ m t = λ m 0 + D j=1 α mj t 0 e -βmj (t-s) dN j s , for m = 1, . . . , D,
or equivalently:

dλ m t = D j=1 α mj -β mj λ m t -λ m 0 dt + α mj dN j t , for m = 1, . . . , D,
with given initial conditions: λ m 0 ∈ R * + for m = 1, . . . , D. It is well-known that for this choice of intensity, the couple becomes (N t , λ t ) t≥0 Markovian. See e.g. Lemma 6 in [START_REF] Massoulié | Stability results for a general class of interacting point processes dynamics, and applications[END_REF] for a proof of this result.

We can now rewrite the control problem (3.3.1) in the particular case where the order book is driven by Hawkes processes, there is no running reward, i.e. f = 0, and where the terminal reward G stands for the terminal wealth of the market maker penalized by her inventory. We then consider the following problem in this section:

V (t, λ, z) := sup α∈A E α t,z,λ G Z T , (3.5.1)
where G(z) denotes the wealth of the market maker when the controlled order book is at state z, plus a term of penalization of her inventory; and where A is the set of the admissible controls, i.e. the predictable decisions taken by the market maker until a terminal time T > 0.

We now present the main result of this section.

Chapter 3. Algorithmic trading in a LOB model Theorem 3.5.1. V is characterized as the unique solution of the following HJB equation:

                   f (T, z, λ) = G(z), for z ∈ E 0 = ∂f ∂t (t, z, λ) - D m=1   D j=1 β mj λ m -λ m 0 ∂f ∂λ m (t, z, λ) +λ m sup a∈Az f t, e a m (z), λ + α m -f t, z, λ ,
for 0 ≤ t < T, and (t, z, λ) ∈ R + ×E × R * + . (3.5.2) Moreover, V admits the following representation

V (t, z, λ) = sup α∈A ∞ n=0 E α t,z,λ 1 Tn≤T G Z α Tn exp -|λ 0 |(T -T n ) + D m=1 λ m Tn -λ m 0 D j=1 β mj e -D j=1 βmj (T -Tn) -1 , (3.5.3)
where, for n ≥ 0, T n stands for the n th jump time of Z after time t, and (Z α Tn ) ∞ n=0 is seen as a MDP controlled by α ∈ A; and where E α t,z,λ [.] stands for the expectation conditioned by Z t = z, λ t = λ when the control α is followed.

Remark 3.5.1. V is characterized in (3.5.3) as the value function associated to an MDP with infinite horizon, for which the instantaneous reward writes:

r(t, z, λ) = 1 t≤T G (z) exp -|λ 0 | 1 (T -t) + D m=1 λ m -λ m 0 D j=1 β mj e -D j=1 βmj (T -t) -1 , where |.| 1 denotes the L 1 R D norm.
Proof: (of Theorem 3.5.1)

Step 1: Let us check that (3.5.3) holds, where V is defined as solution of (3.5.1).

We want to show that (3.5.10) is the expression of the maximal reward operator associated to the PDMDP (3.3.9) that we will define later. First notice that (λ t , Z t ) t is a PDMDP, since (λ t , Z t ) t is deterministic between two jumping times. We then aim at rewriting the expression of the value function defined in (3.5.1) as the value function associated to a infinite horizon control problem of the PDMDP (λ t , Z t ) t . To do so, we first notice that by conditioning on the time jumps we get:

V (t, z, λ) = sup α∈A E α t,z,λ G Z α T = sup α∈A E α t,z,λ ∞ n=0 1 Tn≤T <Tn+1 G Z α Tn = sup α∈A ∞ n=0 E α t,z,λ 1 Tn≤T G Z α Tn P T -T n ≤ T n+1 -T n T n , (3.5.4)
where T n n is the sequence of jump times of N . This process is a jump process with intensity µ s = D m=1 λ m s . Since it holds, conditioned to F Tn :

µ s = D m=1 λ m Tn -λ m 0 e -D j=1 βmj (s-t) , for s ∈ [T n , T n+1 ),
3.5. Model extension to Hawkes Processes then, we have:

P T n+1 -T n ≥ T -T n T n = ∞ T -Tn µ s e -s 0 µudu ds = exp -|λ 0 |(T -T n ) + D m=1 λ m Tn -λ m 0 D j=1 β mj e -D j=1 βmj (T -Tn) -1 . (3.5.5)
Plugging (3.5.5) into (3.5.4), the value function rewrites:

V (t, z, λ) = sup α∈A ∞ n=0 E α t,z,λ 1 Tn≤T G Z α Tn exp -|λ 0 |(T -T n ) + D m=1 λ m Tn -λ m 0 D j=1 β mj e -D j=1 βmj (T -Tn) -1 , (3.5.6)
which completes the step 1. The r.h.s of (3.5.6) can be seen as the value function of an infinite horizon control problem associated to the PDMDP.

Step 2: Let us show that V is the unique solution to (3.5.2).

Notice first that the solutions to the following HJB equation

     G(z) = f (T, z, λ) 0 = ∂f ∂t - D m=1 D j=1 β mj λ m -λ m 0 ∂f ∂λ m + λ m sup a∈Az f t, e a m (z), λ + α m -f t, z, λ , for 0 ≤ t < T.
are the fixed points of the operator T = T 1 • T 2 where T 1 and T 2 are defined as follows:

T 1 : F → f solution of ∂f ∂t - D m=1 D j=1 β mj λ m -λ m 0 ∂f ∂λ m = F (t, z, λ) f (T, z, λ) = G(z),
and:

T 2 : f → - D m=1 λ m sup a∈Az f t, e a m (z), λ + α m -f t, z, λ .
We now use the characteristic method to rewrite the image of T 1 . Let us take function F , and define f = T 1 (F ). Let us fix t ∈ [0, T ] and λ ∈ (R + ) D , and denote by g the function g(s, z) = f (s, z, λ 1 s , ..., λ D s ) where, for m = 1, . . . , D, s → λ m s is a differentiable function defined on [t, T ] as solution to the following ODE:

dλ m s ds = - D j=1 β mj λ m s -λ m 0 , for all t < s ≤ T, λ m t = λ m .
(3.5.7)

For m = 1, . . . , D, basic theory on ODE provides existence and uniqueness of a solution to (3.5.7), which is given by:

λ m s = λ m 0 + λ m -λ m 0 e -D j=1 βmj (s-t) , for s ∈ [t,
T 1 (F ) = f (t, z, λ) = G(z) -
E a t,λ,z f T 1 , Z 1 , λ T1 + α m , (3.5.9)
where T 1 is the first jump time of N larger than t, we denote Z 1 = Z T1 . Equation (3.5.9) shows that the fixed point of T 1 • T 2 is characterized as the fixed point of the operator T defined for any smooth enough function f by:

T (f ) = G(z)e -D m=1 T t λ m s ds + sup a∈Az E a t,λ,z f T 1 , Z 1 , λ T1 + α m , (3.5.10)
where E a t,λ,z [.] stands for the expectation conditioned by the events λ t = λ and Z t = z, when decision a is taken at time t. We recognize here the maximal reward operator of the value function defined in (3.5.6). Basic theory on PDMDP shows that the maximal reward operator T admits V as unique fixed point, which completes step 2.

3.A From uncontrolled to controlled intensity

Remind that the results state in Section 3.3 hold when assuming that the intensities of the orders arrivals are uncontrolled. In particular, we assumed in this section that the market maker has no influence on the next exogenous event that will occur. This can be seen as a weak assumption if the market maker is a small player, but never holds in the case where the latter is a large player. In this section, we show how to alleviate Assumption (Harrivals2) by rewriting the initial control problems (3.3.1) with controlled intensities as a control problems with uncontrolled intensities under a new (controlled) probability measure. The results and proofs in this section are inspired from [START_REF] Brémaud | Point Processes and Queues : Martingale Dynamics[END_REF].

Consider a LOB which can receive at any time limit, cancel, and market orders. Denote by L + (res. L -) the limit sell (resp. buy) order arrival process, received on the ask (resp. buy) side. Denote by C + (resp. C -) the cancel order on the ask (resp. bid) side. Denote by M + (resp. M -) the buy (resp. sell) market order process. The orders arrivals process is then a (4K + 2) dimensional process.

Recall that E is the state space of the order book. The order book is modeled by a jump process Z : [0, T ] → E such that the order arrivals processes have uncontrolled stochastic intensities λ i (a, b), for i = 1, . . . , 4K + 2, that only depend on the bid and ask sides, i.e. (a, b) of the order book under P. We underline that, by assumption, the intensities are uncontrolled under P. Let us fix (α t ) 0≤t≤T ∈ A an admissible control, i.e. a predictive process w.r.t. the natural filtration F t t>0 generated by the uncontrolled orders arrivals processes under P.

(HarrivalsL): We assume in this section that the intensities are Lipschitz and bounded, i.e. there exist [λ] L > 0 and λ ∞ > 0 such that

λ i (a, b) -λ i (a , b ) ≤ [λ] L (|a -a | + |b -b |) , and λ i (a, b) ≤ λ ∞ , for i = 1, . . . , 4K + 2,
for a, a ∈ N K and b, b ∈ (-N) K .

3.A. From uncontrolled to controlled intensity

We want to define the probability P α as the absolutely continuous probability w.r.t. P, which Radon-Nikodym derivative writes:

L α t = dP α dP Ft = 4K+2 i=1 ∞ n=1 µ i α,T i n 1 T i n ≤t exp t 0 1 -µ i s λ i s ds , for 0 ≤ t ≤ T, (3.A.1)
where for i = 1, . . . , 4K + 2, we denote by µ i α,T i n the quotient of the controlled intensity at time T i n of the n th jump of the i th process and the uncontrolled intensity, i.e. denoting by a α and b α the ask and bid where the market order's orders are counted, we define:

µ i α = λ(a α , b α ) λ(a, b) .
Remark 3.A.1. Under (HarrivalsL), it holds:

|λ(a α , b α ) -λ(a, b)| ≤ [λ] L M, for a, a ∈ N K , and b, b ∈ (-N) K , (3.A.2)
where we remind that M stands for the limit number of orders that can be hold by the market maker at the same time in the LOB.

Remark 3.A.2. From Remark 3.A.1, it is straightforward to see that µ i α is bounded under (Har-rivalsL), and moreover:

µ i α ≤ 1 + [λ] L M λ min , for i = 1, . . . , 4K + 2,
where we denote λ min = inf i=1,...,4K+2 inf z∈E λ i (z), and assume the latter to be strictly positive. Note that the bound is uniform w.r.t. the control and the state variables.

Proposition 3.A.1. For every α ∈ A, it holds under (HarrivalsL):

E [L α T ] = 1, (3.A.3)
which implies in particular that P α is well-defined. Moreover, the orders arrivals admit the controlled intensities λ(a α , b α ), for i = 1, . . . , 4K + 2, under P α , where we remind that a α and b α stand for the vector of orders on the ask and the bid sides, where the market maker's orders are counted.

Proof. We divided the proof of Proposition 3.A.1 into two steps.

Step 1: We show (3.A.3).

Let us fix α ∈ A and write the integral representation of (L α t ) 0≤t≤T : for t ∈ [0, T ],

L α t = 1 + 4K+2 i=1 t 0 L α s-µ i α,s -1 d M i s , for i = 1, . . . , 4K + 2, (3.A.4)
where M stands for the local martingale which dynamic writes:

d M i s = dN i s -λ i (a s , b s ) ds. It is then sufficient to show that E T 0 L α s-µ i α,s -1 λ i s ds < +∞, for i = 1, . . . , 4K + 2, (3.A.5)
to get that the

t 0 L α s-µ i α,s -1 d M i s 0≤t≤T
are martingales for i = 1, . . . , 4K + 2 (as proved e.g.

in [START_REF] Brémaud | Point Processes and Queues : Martingale Dynamics[END_REF]), and complete the proof of Step 1, using (3.A.4).

Chapter 3. Algorithmic trading in a LOB model Plugging (3.A.2) into (3.A.1), we get:

L s ≤ µ As ∞ e [λ] L M T , for 0 ≤ s ≤ T, (3.A.6)
where we denote µ ∞ := 1 + [λ] L M λmin , and where (A t ) t∈[0,T ] stands for the sum of all the order arrivals process up to time t, for t ∈ [0, T ]. Moreover, as stated in Remark 3.A.1, we have for all i = 1, . . . , 4K + 2:

| µ i s -1 λ i s (z)| = |λ(a α , b α ) -λ(a, b)| ≤ [λ] L M. (3.A.7)
Plugging (3.A.7) and (3.A.6) into the l.h.s. of (3.A.5), we get:

E T 0 L α s-µ i α,s -1 λ i s ds ≤ T 0 E µ As ∞ e [λ] L M T [λ] L M ds (3.A.8)
Notice that the intensity of A is bounded by λ ∞ , under (HarrivalsL), so that:

E µ As ∞ ≤ e -λ ∞s +∞ n=0 µ n ∞ ( λ ∞ s) n n! ≤ exp { λ ∞ T ( µ ∞ -1)} , for s ∈ [0, T ],
(3.A.9)

Combining (3.A.8) and (3.A.9), we can prove that (3.A.5) holds, which completes the proof of Step 1.

Step 2: We refer to the T3 Theorem in Chapter VI of [START_REF] Brémaud | Point Processes and Queues : Martingale Dynamics[END_REF] for a proof of the second assertion in Proposition 3.A.1.

3.B Dynamics of the controlled order book (simplified version)

In this section, we give the expressions for the dynamics of the controlled order book process (Z t ). The market maker control has been simplified to a couple (la t , lb t ), where la = 1 (resp. 0) if the market maker holds (does not hold) a sell order at the best ask limit, and lb = 1 (resp. 0) if the market maker holds (does not hold) a buying order at the best bid limit. So to speak, the market maker considers to place orders at the best ask limit or at the best bid limit exclusively. In the numerical simulations that we run, we also had to calculate the dynamics of (Z t ) for the set of generalized controls in which the market maker is allowed to post orders on the two first limits at the bid and at the ask side. The expression of the dynamics for the generalized controls are very similar to the ones for the simplified controls.

To understand the dynamics of the rank of the orders of the market maker, we need a model for the cancellation of orders. Suppose for example that the market maker holds an order whose rank is na in the queue, with na < a A -1 (0) . Suppose that the cancel process L C + A -1 (0) jumps. Then two scenarios can occur:

• If the rank of the canceled order is greater than the one of the market maker, then na t stays constant.

• If the rank of the canceled order is smaller than the one of the market maker, then na t = na t -+1.

Model:

We consider a Bernoulli variable X a with parameter:

na -1 a A -1 (0) α δ 1 + a A -1 (0) + 1 -na a A -1 (0) β δ 0 .
We assume that the canceled order is in front of the market maker's order in the queue if X a = 1,

3.B. Dynamics of the controlled order book (simplified version)

and behind it if X a = 0.

We proceed for the bid side as we just did for the ask side. We consider a random variable X b following a Bernoulli law with parameter:

nb -1 |b B -1 (0) | α δ 1 + |b B -1 (0) | + 1 -nb |b B -1 (0) | β δ 0 . 3.B.1 Dynamics of X t et Y t
The dynamic of the amount hold by the market maker on a no-interest-bearing account (X t ) t∈R+ is as follows:

dX t = la t pa t-1 {nat-=1} dM + t -lb t pb t-1 {nbt-=1} dM - t
The market maker's inventory (Y t ) follows the dynamic:

dY t = -la t 1 {nat-=1} dM + t + 1 {nbt-=1} lb t dM - t
where:

• â = sup{a i : i-1 j=1 a j = 0} et b = sup{b i : i-1 j=1 b j = 0}} • M ± t are Cox processes with intensities λ M ± 3.B.2 Dynamics of the a t et b t
We remind that a i is the number of orders located i ticks away from the best buy order. We denote by J the shif t operator that re-index a side of the book when an event occurred on the opposite side.

J L - i , i ∈ {1, . . . , B -1 (0)} is the shift operator that shifts the bid side due to the jump of a L + i for i ∈ {0, K}. We get:

J L - i (a) =   a i+1 , ..., a K , a ∞ , . . . , a ∞ i times  
Dynamics of a i :

Chapter 3. Algorithmic trading in a LOB model

da i = (1 -lb t ) dL + i + lb t dL + i-(A -1 (0)-rbt-) + 1 -lb t + lb t 1 {nbt->1} J M - a i -a i dM -(t) -(1 -lb t )dC + i -lb t dC + i-(A -1 (0)-rbt-) + (1 -la t ) -1 {i=A -1 (0)} dM + t + J C - (a i ) -a i dC - A -1 (0) + (1 -lb t ) A -1 (0)-1 j=1 (J L - j 0,0 (a i ) -a i ) dL - j (t) + lb t A -1 (0)-1 j=1 (J L - j 0,1 (a i ) -a i ) dL - j (t) + la t -1 {nat->1} 1 {i=A -1 (0)} dM + t + J C - (a i ) -a i dC - rat- + lb t rat--1 j=1 (J L - j 1,1 (a i ) -a i ) dL - j (t) + (1 -lb t ) rat--1 j=1 (J L - j 1,0 (a i ) -a i ) dL - j (t)
with J such that:

J C - (a i ) =      a ∞ si i > B -1 (1) -B -1 (0) + K a i-B -1 (1)-B -1 (0) si i > B -1 (1) -B -1 (0) 0 si i ≤ B -1 (1) -B -1 (0) J M - (a i ) = a i-B -1 (1)-B -1 (0) si i > B -1 (1) -B -1 (0) 0 si i ≤ B -1 (1) -B -1 (0) J L - j 0,0 (a i ) = a i+j si i + j ≤ K 0 si i + j < K J L - j 0,1 (a i ) = a i+rbt--j si i + rb t--j ≤ K a ∞ si i + rb t--j > K J L - j 1,0 (a i ) = a i+rat--j si i + ra t--j ≤ K a ∞ si i + ra t--j > K J L - j 1,1 (a i ) =    0 si i + rb t--j < 0 a i+rbt--j si i + rb t--j ≤ K a ∞ si i + rb t--j > K
We remind that b i is the number of buy order located i ticks away from the best sell order.

Dynamics of b i :

db i = -(1 -la t )dL - i -la t dL - i-(A -1 (0)-rat-) + 1 -la t + la t 1 {nat->1} J M + b i -b i dM + (t) + (1 -la t ) dC - i + la t dC - i-(A -1 (0)-rat-) + (1 -lb t ) 1 {i=A -1 (0)} dM - t + J C + (b i ) -b i ) dC + A -1 (0) + (1 -la t ) B -1 (0)-1 j=1 (J L + j 0,0 (b i ) -b i ) dL + j (t) + la t B -1 (0)-1 j=1 (J L + j 1,0 (b i ) -b i ) dL + j (t) + lb t 1 {nbt->1} 1 {i=A -1 (0)} dM - t + J C + (b i ) -b i ) dC + rbt- + la t rbt--1 j=1 (J L + j 1,1 (b i ) -b i ) dL + j (t) + (1 -la t ) rbt--1 j=1 (J L + j 0,1 (b i ) -b i ) dL + j (t)
with J the shift operators:

J C + (b i ) =      b ∞ si i + A -1 (1) -A -1 (0) > K b i-A -1 (1)-A -1 (0) si i > A -1 (1) -A -1 (0) 0 si i ≤ A -1 (1) -A -1 (0) J M + (b i ) =      b ∞ si i + A -1 (1) -A -1 (0) > K b i-A -1 (1)-A -1 (0) si i > A -1 (1) -A -1 (0) 0 si i ≤ A -1 (1) -A -1 (0) J L + j 0,0 (b i ) = b i+j si i + j ≤ K 0 si i + j > K J L + j 1,0 (b i ) = b i-j+A -1 (0) si i -j + A -1 (0) ≤ K b ∞ si i -j + A -1 (0) > K J L + j 0,1 (b i ) = b i+rbt--j si i + rb t--j ≤ K b ∞ si i + rb t--j > K J L + j 1,1 (b i ) =    0 si i + rb t--j < 0 b i+rbt--j si i + rb t--j ≤ K b ∞ si i + rb t--j > K

3.B.3 Dynamics of na t and nb t

Dynamics of na t :

X a has been introduced in part 3.B. It models whether the canceled order is behind or in front of the market maker's order in the queue.

Chapter 3. Algorithmic trading in a LOB model We get:

dna t = la t -X a (1 -lb t ) dC + A -1 (0) (t) + lb t dC + rbt-+ -1 {nat->1} + a A -1 (0) + 1 -na t-1 {nat-=1} dM + t + (2 -na t-) (1 -lb t ) rat-(t)-1 i=1 dL + i + lb t rat-(t)-(A -1 (0)-rbt-)-1 i=1 dL + i + (1 -la t ) a A -1 (0) 1 {a A -1 (0) >1} + (a A -1 (1) + 1)1 {a A -1 (0) =1} -na t-dM + t + lb t (2 -na t-) rbt--1 j=1 dL + j + (a A -1 (0) + 2 -na t-) dL + rbt- + (a A -1 (0) + 1 -na t-) K j=rbt-+1 dL + j + dC + j + a A -1 (0) 1 {a A -1 (0) >1} + (a A -1 (1) + 1)1 {a A -1 (0) =1} -na t-dC + rbt- + (1 -lb t ) (2 -na t-) B -1 (0)-1 j=1 dL + j + (a A -1 (0) + 2 -na t-) dL + A -1 (0) + (a A -1 (0) + 1 -na t-) K j=B -1 (0)+1 dL + j + dC + j + a A -1 (0) 1 {a A -1 (0) >1} + (a A -1 (1) + 1)1 {a A -1 (0) =1} -na t-dC + A -1 (0) + a A -1 (0) + 1 -na t-dM - t + K j=1 dL - j + dC - j dna t = (la t == 0)(-1 -na t-) dM + t + dM - t + (lb t ! = 1) K i=1 dL + i + dL - i + dC + i + dC - i + (lb t == 1) K-(A -1 (0)-rbt-) i=0 dL + i + dC + i + K i=1 dL - i + dC - i + 1 lat=1 1 nat-=-1 + 1 nat-!=-1 1 rat->A -1 (0) a A -1 (0) -na t-dM + t + 1 lbt=1 dC + rbt-+ 1 lbt!=1 dC + A -1 (0) + a A -1 (0) + 1 -na t-1 lbt!=1 K i=1 dL + i + dC + i + 1 lbt=1 K-(A -1 (0)-rbt-) i=1 dL + i + dC + i 3.B.
Dynamics of the controlled order book (simplified version)

Dynamics of nb t :

dnb t = lb t -X b (1 -la t ) dC - B -1 (0) (t) + la t dC - rat-+ -1 {nbt->1} + |b B -1 (0) | + 1 -nb t-1 {nbt-=1} dM - t + (2 -nb t-) (1 -la t ) rbt-(t)-1 i=1 dL - i + la t rbt-(t)-(B -1 (0)-rat-)-1 i=1 dL - i + (1 -lb t ) |b A -1 (0) |1 {|b B -1 (0) |>1} + (|b B -1 (1) | + 1)1 {|b B -1 (0) |=1} -nb t-dM - t + la t (2 -nb t-) rat--1 j=1 dL - j + (|b A -1 (0) | + 2 -nb t-) dL - rat- + (|b A -1 (0) | + 1 -nb t-) K j=rat-+1 dL - j + dC - j + |b A -1 (0) |1 {|b B -1 (0) |>1} + (|b B -1 (1) | + 1)1 {|b B -1 (0) |=1} -nb t-dC - rat- + (1 -la t ) (2 -nb t-) B -1 (0)-1 j=1 dL - j + (|b B -1 (0) | + 2 -nb t-) dL - A -1 (0) + (|b B -1 (0) | + 1 -nb t-) K j=B -1 (0)+1 dL - j + dC - j + |b A -1 (0) |1 {|b B -1 (0) |>1} + (|b B -1 (1) | + 1)1 {|b B -1 (0) |=1} -nb t-dC - A -1 (0) + |b A -1 (0) | + 1 -nb t-dM + t + K j=1 dL + j + dC + j 3.B.

Dynamics of pa and pb

Dynamics of (pa t ) t :

Denoting by δ the tick, we have:

dP A t = δ(1 -la t ) A -1 (1) -ra t-dM + (t) + lb t - rbt--1 i=1 rb t--(A -1 (0) -ra t-) -j dL + i (t) + A -1 (0) -ra t- K j=rbt- dL + i + A -1 (1) -ra t-dC + rbt-+ K j=rbt-+1 A -1 (0) -ra t-dC + j + 1 -lb t - A -1 (0)-1 i=1 ra t--j dL + i (t) + A -1 (0) -ra t- K j=A -1 (0) dL + i + A -1 (1) -ra t-dC + A -1 (0) + K j=A -1 (0)+1 A -1 (0) -ra t-dC + j + 1 {rat-=A -1 (0)} A -1 (0) -ra t-dM - t + K j=1 dL - t + K j=1 dC - t + δla t (A -1 (0) -r A t ) dM + (t) -lb t rbt--(A -1 (0)-rat-)-1 i=1 ra t--j + A -1 (0) -rb t-dL + i (t) -1 -lb t rat--1 i=1 ra t--j dL + i (t)
Dynamics of (pb t ):

dP B t = -δ(1 -lb t ) B -1 (1) -rb t-dM -(t) + la t - rat--1 i=1 ra t--(B -1 (0) -rb t-) -j dL - i (t) + B -1 (0) -rb t- K j=rat- dL - i + B -1 (1) -rb t-dC - rbt-+ K j=rat-+1 B -1 (0) -rb t-dC - j + 1 -la t - B -1 (0)-1 i=1 rb t--j dL - i (t) + B -1 (0) -rb t- K j=B -1 (0) dL - i + B -1 (1) -rb t-dC - A -1 (0) + K j=A -1 (0)+1 B -1 (0) -rb t-dC - j + 1 {rbt-=A -1 (0)} A -1 (0) -rb t-dM + t + K j=1 dL + j + K j=1 dC + j -δlb t (B -1 (0) -r B t ) dM -(t) -la t rat--(B -1 (0)-rbt-)-1 i=1 rb t--j + B -1 (0) -ra t-dL - i (t) -1 -la t rbt--1 i=1 rb t--j dL - i (t)

3.B.5 Dynamics of ra and rb

We remind that ra t denotes the number of ticks between the market maker's order and the best buy order in the order book. We assumed in this simplified control problem that the market maker is allowed to place no more than one order on the best ask and best bid limits. So ra and rb are vectors of size 1 here.

Chapter 3. Algorithmic trading in a LOB model Dynamics of ra:

d ra t = la t 1 {na=1} A -1 (0) -r A t dM + t + (1 -lb t ) rat--1 i=1 i -ra t-dL + i + lb t rbt--(B -1 (0)-rat-)-1 i=1 i + B -1 (0) -rb t--ra t-dL + i + rat--1 i=1 i -ra t-dL - i + B -1 (1) -B -1 (0) dC - rat- 1 -lb t + lb t 1 {nbt->1} B -1 (1) -B -1 (0) dM - t + (l -la t ) lb t rbt--1 j=1 j + B -1 (0) -rb t--ra t-dL + j + A -1 (0) -ra t- K j=rbt- dL + j A -1 (1) -ra t-dC + rbt-+ A -1 (0) -ra t- K j=rbt-+1 dC + j + (1 -lb t ) B -1 (0)-1 j=1 j -ra t-dL + j + A -1 (0) -ra t- K j=B -1 (0) dC + j + dC - j + A -1 (1) -ra t-dC + A -1 (0) + K j=A -1 (0)+1 A -1 (0) -ra t-dC + j + A -1 (0)-1 j=1 (j -ra t-) dL - j + K j=A -1 (0) A -1 (0) -ra t-dL - j + B -1 (1) -ra t dC - B -1 (0) + A -1 (0) -ra t- K j=A -1 (0)+1 dC - j + (1 -lb t ) + lb t 1 {nbt->1} B -1 (1) -ra t-+ lb t 1 {nbt-=1} B -1 (0) -ra t-dM - t + A -1 (1) -ra t-dM + t
We remind that rb t is the number of ticks between the market maker's order and the best sell order in the order book. We divided the proofs of Theorem 3.4.1 and Corollary 3.4.1 into several Lemmas that we state and prove now.

d rb t = lb t 1 {nb=1} B -1 (0) -rb dM - t + (1 -la t ) rbt--1 i=1 i -rb t-dL - i + la t rat--(A -1 (0)-rbt-)-1 i=1 i + A -1 (0) -ra t--rb t-dL - i + rbt--1 i=1 i -rb t-dL + i + A -1 (1) -A -1 (0) dC + rbt- 1 -la t + la t 1 {nat->1} A -1 (1) -A -1 (0) dM + t + (l -lb t ) la t rat--1 j=1 j + A -1 (0) -ra t--rb t-dL - j + K j=rat- B -1 (0) -rb t-dL - j B -1 (1) -rb t-dC - rat-+ K j=rat-+1 B -1 (0) -rb t-dC - j + (1 -la t ) A -1 (0)-1 j=1 j -rb t-dL - j + K j=A -1 (0) B -1 (0) -rb t-dL - j + B -1 (1) -rb t-dC - B -1 (0) + K j=B -1 (0)+1 B -1 (0) -rb t-dC - j + B -1 (0)-1 j=1 (j -rb t-) dL + j + K j=B -1 (0) B -1 (0) -rb t-dL + j + A -1 (1) -rb t dC + A -1 (0) + B -1 (0) -rb t- K j=B -1 (0)+1 dC + j + (1 -la t ) + la t 1 {nat->1} A -1 (1) -rb t-+ la t 1 {nat-=1} A -1 (0) -rb t-dM + t + B -1 (1) -rb t-dM -
Lemma 3.C.1 aims to bound the projection error. It relies on [START_REF] Gyorfi | A Distribution-Free Theory of Nonparametric Regression[END_REF], see p.93, as well as Zador's theorem, stated in section 3.D for sake of completeness.

Lemma 3.C.1. Assume d ≥ 3, and take K = M d+2 points for the optimal quantization of ε n , then it holds under (Hµ) and (HF), as M → +∞,

ε proj n = O 1 M 1/d , (3.C.1)
where we remind that ε proj n := sup a∈A Proj n+1 (F (X n , a, εn )) -F (X n , a, εn ) 2 stands for the average projection error.

Chapter 3. Algorithmic trading in a LOB model

Proof. Let us take η > 0, and observe that

P Proj n+1 F (X n , a, εn+1 ) -F (X n , a, εn+1 ) 2 > η = E M m=1 E 1 X t,(m) n+1 -F (Xn,a,εn+1) > √ η X n , εn+1 = E 1 -µ B F (X n , a, εn+1 ), √ η M ,
where for all x ∈ E and η > 0, B(x, η) denote the ball of center x and radius η. Since x → (1 -x) M is M -Lipschitz, we get by application of Zador's theorem:

P Proj n+1 F (X n , a, εn+1 ) -F (X n , a, εn+1 ) 2 > η ≤ M [F ] L [µ] L εn+1 -ε n+1 2 + E 1 -µ B F (X n , a, ε n+1 ), √ η M = M [F ] L [µ] L K 1/d + E 1 -µ B F (X n , a, ε n+1 ), √ η M + O M K 1/d ,
as the number of points for the quantization of the exogenous noise K goes to +∞, and where M stands for the size of the grids Γ n .

Let us introduce A 1 , ..., A N (η) , a cubic partition of Supp(µ), which is bounded under (Hµ), such that for all j = 1, . . . , N (η), A j has diameter η. Also, Notice that there exists c > 0, which only depends on Supp(µ), such as

N (η) ≤ c η d . (3.C.2)
If x ∈ A j , then A j ⊂ B(x, η), therefore:

E (1 -µ (B(X n , η))) M = N (η) j=1 Aj 1 -µ(B(x, η)) M µ(dx) ≤ N (η) j=1 Aj 1 -µ(A j ) M µ(dx). (3.C.3)
Also notice that: Let L = 2 µ ∞ stands for the diameter of the support of µ. We then get, as M → +∞, Proof. Following the same steps as those used to prove Lemma 3.C.1, we show that:

N (η) j=1 µ(A j ) 1 -µ(A j ) M ≤ N (η) j=1 max z z(1 -z) M ≤ e -1 N (η) M . ( 3
E Proj n+1 F (X n , a, εn+1 ) -F (X n , a, εn+1 ) 2 = ∞ 0 P Proj n+1 F (X n , a, εn+1 ) -F (X n , a, εn+1 ) 2 > η dη ≤ L 2 0 M [F ] L [µ] L K 2/d + P |Proj n+1 F (X n , a, εn+1 ) -F (X n , a, n+1 )| > √ η dη = L 2 0 min 1, e -1 N ( √ η) M dη + O M K 1/d = L 2 0 min 1, cη -d/2 eM dη + O M K 1/d = (c/(eM )) (2/d) 0 1dη + L 2 (c/(eM )) (2/d) cη -d/2 eM dη + O M K 1/d = c2 M 2/d + O M K 1/d , (3.
P Proj n+1 F (x, a, εn+1 ) -F (x, a, εn+1 ) 2 > η = M [F ] L [µ] L K 1/d + E 1 -µ B F (x, a, ε n+1 ), √ η M + O M K 1/d ,
as K → +∞, and moreover,

E 1 -µ B F (x, a, ε n+1 ), √ η M ≤ e -1 N (η) M ,
holds, which is enough to complete the proof of Lemma 3.C.2.

Lemma 3.C.3. Under (HF), for n = 0, . . . , N there exists constant V Q n L > 0 such that for

x, x ∈ E, it holds as M → ∞: , for n = 0, . . . , N :

V Q n (x) -V Q n (x ) ≤ V Q n L |x -x | + O 1 M 1/d . ( 3
   V Q N L ≤ [g] L V Q n L ≤ [f ] L + [F ] L V Q n+1 L
, for n = 0, ..., N -1.

(3.C.8)

Proof. Let us show that by induction that

V Q N is Lipschitz. First, notice that (3.C.7) holds at terminal time n = N , if one define V Q N L as V Q N L = [g] L . Let us take x, x ∈ E. Assume V Q n+1 (x) -V Q n+1 (x ) ≤ V Q n+1 L |x -x | + O 1 M 1/d holds for some n = 0, . . . , N -1. Let us show that V Q n (x) -V Q n (x ) ≤ V Q n L |x -x | + O 1 M 1/d , where V Q n L
is defined in (3.C.8). Notice that, by the dynamic programming principle and the triangular inequality, it holds:

| V Q n (x) -V Q n (x )| ≤ [f ] L |x -x | + sup a E a n V Q n+1 Proj n+1 (F (x, a, εn+1 )) -V Q n+1 Proj n+1 (F (x , a, εn+1 )) ≤ [f ] L |x -x | + V Q n+1 L sup a E Proj n+1 (F (x, a, εn+1 )) -F (x, a, εn+1 ) + O 1 M 1/d ≤ [f ] L + V Q n+1 L [F ] L |x -x | + O 1 M 1/d ≤ V Q n L |x -x | + O 1 M 1/d ,
which completes the proof of (3.C.7).

We now proceed to the proof of Theorem 3.4.1.

Proof. (of Theorem 3.4.1) Combining inequality

|u 1 + u 2 + u 3 | 2 ≤ 3 |u 1 | 2 + |u 2 | 2 + |u 3 | 2 that holds for all u 1 , u 2 , u 3 ∈ R with inequality sup i∈I a i -sup i∈I b i ≤ sup i∈I |a i -b i | that holds for all families (a i ) i∈I
and (a i ) i∈I of reals, and all subset I of R, we have:

V Q n (X n ) -V n (X n ) 2 2 ≤ 3 E sup a∈A E n,Xn V Q n+1 Proj n+1 (F (X n , a, εn+1 )) -V Q n+1 (F (X n , a, εn+1 )) 2 + sup a∈A E n,Xn V Q n+1 (F (X n , a, εn+1 )) -V Q n+1 (F (X n , a, ε n+1 )) 2 + sup a∈A E n,Xn V Q n+1 (F (X n , a, ε n+1 )) -V n+1 (F (X n , a, ε n+1 ))
2 where E n,Xn stands for the expectation conditioned by the state X n at time n. It holds as M → +∞, 3.D. Zador's Theorem using Lemma 3.C.3:

V Q n (X n ) -V n (X n ) 2 2 ≤ 3 V Q n L E sup a E n,Xn |Proj n+1 (F (X n , a, εn+1 )) -F (X n , a, εn+1 )| 2 + sup a E n,Xn |F (X n , a, εn+1 ) -F (X n , a, ε n+1 )| 2 + 3 r ∞ E | V Q n+1 (X n+1 )) -V n+1 (X n+1 ))| 2 + 1 M 1/d (3.C.9)
Under (HF), (3.C.9) can then be rewritten as:

V Q n (X n ) -V n (X n ) 2 2 ≤ 3 V Q n L [F ] 2 L ( Q n ) 2 + ( proj n ) 2 + 3 r ∞ V Q n+1 (X n+1 ) -V n+1 (X n+1 ) 2 2 + 1 M 1/d . ( 3

3.D Zador's Theorem

Theorem 3.D.1 (Zador's theorem). Let us take n = 0, . . . , N , and denote by K the number of points for the quantization of the exogenous noise ε n .

Assume that E |ε n | 2+η < +∞ for some η > 0. Then, there exists a universal constant C > 0 such that: lim

M →+∞ M 1 d εn -ε n 2 = C
Proof. We refer to [START_REF] Graf | Foundations of quantization for probability distributions[END_REF] for a proof of Theorem 3.D.1.

Introduction

The optimal control of McKean-Vlasov (also called mean-field) dynamics is a rather new topic in the area of stochastic control and applied probability, which has been knowing a surge of interest with the emergence of the mean-field game theory. It is motivated on the one hand by the asymptotic formulation of cooperative equilibrium for a large population of particles (players) in mean-field interaction, and on the other hand from control problems with cost functional involving nonlinear functional of the law of the state process (e.g., the mean-variance portfolio selection problem or risk measure in finance).

In this paper, we are interested in McKean-Vlasov (MKV) control problems under partial observation and common noise, whose formulation is described as follows. On a probability space (Ω, F, P) equipped with two independent Brownian motions B and W 0 , let us consider the controlled stochastic MKV dynamics in R n :

dX s = b(X s , P W 0 Xs , α s )ds + σ(X s , P W 0 Xs , α s )dB s + σ 0 (X s , P W 0 Xs , α s )dW 0 s , X 0 = x 0 ∈ R n , (4.1.1)
where P W 0 Xs denotes the conditional distribution of X s given W 0 (or equivalently given F 0 s where F 0 = (F 0 t ) t is the natural filtration generated by W 0 ), and the control α is F 0 -progressive valued in some Polish space A. This measurability condition on the control means that the controller has a partial observation of the state, in the sense that she can only observe the common noise. We make the standard Lipschitz condition on the coefficients b(x, µ, a), σ(x, µ, a), σ 0 (x, µ, a) with respect to

(x, µ) in R n ×P 2 (R n ), uniformly in a ∈ A, where P 2 (R n ) is the set of all probability measures on (R n , B(R n ))
with a finite second-order moment, endowed with the 2-Wasserstein metric W 2 . This ensures the well-posedness of the controlled MKV stochastic differential equation (SDE) (4.1.1). The cost functional over a finite horizon T associated to the stochastic MKV equation (4.1.1) (sometimes called conditional MKV equation) for a control process α, is

J(α) = E T 0 f (X t , P W 0 X t , α t )dt + g(X T , P W 0 X T ) ,
and the objective is to maximize over an admissible set A of control processes the cost functional:

V 0 = sup α∈A J(α). (4.1.2)
The set A of admissible controls usually requires some integrability conditions depending on the growth conditions on f , g, in order to ensure that J(α) is well-defined for α ∈ A (more details will be given in the examples, see Section 4.4). Notice that classical partial observation control problem (without MKV dependence on the coefficients) arises as a particular case of (4.1.1)-(4.1.2). We refer to the introduction in [START_REF] Pham | Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics[END_REF] for the details.

Let us recall from [START_REF] Pham | Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics[END_REF] the dynamic programming equation associated to the conditional MKV control problem (4.1.2). We start by defining a suitable dynamic version of this problem. Let us consider F 0 a sub σ-algebra of F independent of B, W 0 . It is assumed w.l.o.g. that F 0 is rich enough in the sense that

P 2 (R n ) = {L(ξ) : ξ ∈ L 2 (F 0 ; R n )}
, where L(ξ) denotes the law of ξ. Given a control α ∈ A, we consider the dynamic version of (4.1.1) starting from ξ ∈ L 2 (F 0 ; R n ) at time t ∈ [0, T ], and written as:

X t,ξ,α s = ξ + s t b(X t,ξ,α u , P W 0 X t,ξ,α u , α u )du + s t σ(X t,ξ,α u , P W 0 X t,ξ,α u , α u )dB u + s t σ 0 (X t,ξ,α u , P W 0 X t,ξ,α u , α u )dW 0 u , t ≤ s ≤ T.
Let us then define the dynamic cost functional:

J(t, ξ, α) = E T t f (X t,ξ,α s , P W 0 X t,ξ,α s , α s )ds + g(X t,ξ,α T , P W 0 X t,ξ,α T ) , 4.1. Introduction for (t, ξ) ∈ [0, T ] × L 2 (F 0 ; R n ), α ∈ A,
and notice by the law of conditional expectations, and as α is F 0 -progressive that

J(t, ξ, α) = E T t f (P W 0 X t,ξ,α s , α s )ds + ĝ(P W 0 X t,ξ,α s ) ,
where f :

P 2 (R n ) × A → R, ĝ : P 2 (R n ) → R are defined by f (µ, a) = µ(f (•, µ, a)) = R n f (x, µ, a)µ(dx), ĝ(µ) = µ(g(•, µ)) = R n g(x, µ)µ(dx).
Moreover, notice that the conditional law of X t,ξ,α s given W 0 depends on ξ only through its law L(ξ), and we can then define for α ∈ A:

ρ t,µ,α s = P W 0 X t,ξ,α s , for t ≤ s, µ = L(ξ) ∈ P 2 (R n ).
Therefore, the dynamic cost functional J(t, ξ, α) depends on ξ ∈ L 2 (F 0 ; R n ) only through its law L(ξ), and by an abuse of notation, we write J(t, µ, α) = J(t, ξ, α) when µ = L(ξ). We then consider the value function for the conditional MKV control problem (4.1.2), defined on

[0, T ] × P 2 (R n ) by v(t, µ) = sup α∈A J(t, µ, α) = sup α∈A E T t f (ρ t,µ,α s , α s )ds + ĝ(ρ t,µ,α T ) ,
and notice that at time t = 0, when ξ = x 0 is a constant, then

V 0 = v(0, δ x0 ).
It is shown in [START_REF] Pham | Dynamic programming for optimal control of stochastic McKean-Vlasov dynamics[END_REF] that dynamic programming principle (DPP) for the conditional MKV control problem (4.1) holds: for

(t, µ) ∈ [0, T ] × P 2 (R n ), v(t, µ) = sup α∈A E θ t f (ρ t,µ,α s , α s )ds + v(θ, ρ t,µ,α θ ) ,
for any F 0 -stopping time θ valued in [t, T ]. Next, by relying on the notion of differentiability with respect to probability measures introduced by P. L. Lions [START_REF] Lions | Théorie des jeux de champ moyen et applications (mean field games)[END_REF] (see also the lecture notes [START_REF] Cardaliaguet | Notes on mean field games[END_REF]) and the chain rule (Itô's formula) along flow of probability measures (see [START_REF] Buckdahn | Mean-field stochastic differential equations and associated PDEs[END_REF], [START_REF] Chassagneux | A probabilistic approach to classical solutions of the master equation for large population equilibria[END_REF]), we derive the HJB equation for v:

   ∂ t v + sup a∈A f (µ, a) + µ a v(t, µ) + µ ⊗ µ M a v(t, µ) = 0, (t, µ) ∈ [0, T ) × P 2 (R n ), v(T, µ) = ĝ(µ), µ ∈ P 2 (R n ), where for φ ∈ C 2 b (P 2 (R n )), a ∈ A, and µ ∈ P 2 (R n ), a φ(µ) is the function R n → R defined by a φ(µ)(x) = ∂ µ φ(µ)(x).b(x, µ, a) + 1 2 tr ∂ x ∂ µ φ(µ)(x)(σσ + σ 0 σ 0 )(x, µ, a) ,
and

M a φ(µ) is the function R n × R n → R defined by M a φ(µ)(x, x ) = 1 2 tr ∂ 2 µ φ(µ)(x, x )σ 0 (x, µ, a)σ 0 (x , µ, a) .
The HJB equation (4.1) is a fully nonlinear partial differential equation (PDE) in the infinitedimensional Wasserstein space. In general, this PDE does not have an explicit solution except in the notable important class of linear-quadratic MKV control problem. Numerical resolution for MKV control problem or equivalently for the associated HJB equation is a challenging problem due to the nonlinearity of the optimization problem and the infinite-dimensional feature of the Wasserstein space. In this work, our purpose is to investigate a class of MKV control problems which can be reduced to finite-dimensional problems in view of numerical resolution.

Polynomial McKean-Vlasov control problem

Main assumptions

We make two kinds of assumptions on the coefficients of the model: one on the dependence on x and the other on the dependence on µ.

Assumptions: dependence on x: we consider a class of models where the coefficients of the MKV equation are linear w.r.t. the state variable X, i.e., they are in the form

   b(x, µ, a) = b 0 (µ, a) + b 1 (µ, a)x, σ(x, µ, a) = ϑ 0 (µ, a) + ϑ 1 (µ, a)x, σ 0 (x, µ, a) = γ 0 (µ, a) + γ 1 (µ, a)x,
while the running and terminal cost functions are polynomial in the state variable in the sense that they are in the form (for simplicity we present here the one-dimensional case n = 1)

f (x, µ, a) = f 0 (µ, a) + p k=1 f k (µ, a)x k , g(x, µ) = g 0 (µ) + p k=1 g k (µ)x k ,
for some integer p ≥ 1.

Assumptions: dependence on µ: we assume that all the coefficients depend on µ through its first p moments, i.e., they are in the form We assume that the coefficients b0 , b1 , θ0 , θ1 , γ0 , γ1 are Lipschitz w.r.t. the p first arguments uniformly w.r.t. the control argument a ∈ A. This condition will ensure existence and uniqueness of a solution to the finite-dimensional MKV SDE defined later in (4.2.1).

       b 0 (µ, a) = b0 (μ 1 , μ2 , . . . , μp , a), b 1 (µ, a) = b1 (μ 1 , μ2 , . . . , μp ,
Notice that in this case, the functions f and ĝ defined in Remark 4.2.1. In the multidimensional case, we should consider a class of multi-polynomial functions f and g of degree p in the form

f (x, µ, a) = p |k|=0 f k (µ k ) |k |≤p , a x k , g(x, µ) = p |k|=0 g k (µ k ) |k |≤p x k ,
where we use multi-index notations k

= (k 1 , . . . , k n ) ∈ N n , |k| = k 1 + . . . + k n , x k = x k1 1 . . . x kn n for x = (x 1 , . . . , x n ) ∈ R n and µ k = R n x k µ(dx).

Markovian embedding

Given the controlled process X = X α solution to the stochastic MKV dynamics (4.1.1), denote by

Y (k) t = E[X k t |W 0 ], k = 1, . . . , p.
To alleviate the notations, we assume that n = 1 (otherwise multi-indices should be used). From the linear/polynomial assumptions (4.2.1)-(4.2.1), by Itô's formula and taking conditional expectations, we can derive the dynamics of (Y (1) , Y (2) , . . . , Y (p) ) as

dY (k) t = B k (Y (1) t , Y (2)
t , . . . , Y

(p) t , α t )dt + Σ k (Y (1) t , Y (2)
t , . . . , Y while the cost functional is written as

(p) t , α t )dW 0 t , Y (k) 0 = x k 0 , k = 1, . . . , p, (4.
J(α) = E T 0 f (Y (1) t , Y (2)
t , . . . , Y

t , α t )dt + ḡ(Y

T , Y

T , . . . , Y

T ) .

The MKV control problem is then reduced in this polynomial framework into a finite-dimensional control problem with F 0 -adapted controlled variables (Y (1) , Y (2) , . . . , Y (p) ). In the next section, we describe three probabilistic numerical methods for solving finite-dimensional stochastic control problems and will apply in section 4.4 each of these methods to three examples arising from polynomial MKV control problems under partial observation and common noise.

Numerical methods

In this section, we introduce our numerical methods for the resolution of the reduced problem (4.2.1)-(4.2.2).

Let us introduce the process Z α , valued in R d , controlled by an adapted process α taking values in A, solution to

dZ α t = b(Z α t , α t )dt + σ 0 (Z α t , α t )dW 0 t , Z α 0 = z 0 ∈ R d
, and the performance measure Introduce now a time discretization t n = n∆t, n = 0, . . . , N , ∆t = T /N , and denote by A ∆t the space of discrete processes (α tn ) N -1 n=0 such that for all n, n = 0, . . . , N -1, α tn is F 0 tn -measurable. We can write the Euler approximation of the SDE governing the process Z = Z α , with α ∈ A ∆t (to alleviate notations, we sometimes omit the dependence on α when there is no ambiguity, and keep the same notation Z for the discrete and continuous process)

J(t, z, α) = E T t f (Z α t , α t )dt + g(Z α T ) Z α t = z ,
Z tn+1 = Z tn + b(Z tn , α tn )∆t + σ 0 (Z tn , α tn )∆W 0 tn , (4.3.1)
where ∆W 0 tn ∼ N (0, ∆t) is an increment of W 0 . The discrete time approximation of J(t n , z, α) is defined as:

J ∆t (t n , z, α) = E N -1 k=n f (Z t k , α t k )∆t + g(Z t N ) Z tn = z ,
where α ∈ A ∆t .

Value and Performance iteration

For n = 0, . . . , N , consider V ∆t (t n , z) = sup α∈A∆t J ∆t (t n , z, α), the discrete time approximation of the value function at time t n : V t n , z = sup α∈A J(t n , z, α). The dynamic programming principle states that V ∆t (t n , •) 0≤n≤N is solution to the Bellman equation:

   V ∆t (t N , z) = g(z) V ∆t (t n , z) = sup a∈A f (z, a)∆t + E a n,z V ∆t (t n+1 , Z tn+1 ) , n = N -1, . . . , 0, (4.3.2)
where E a n,z [•] denotes the expectation conditioned on the event {Z tn = z} and when using the control α tn = a at time t n . Observe that for n = 0, . . . , N -1, the equation (4.3.2) provides a backward procedure to recursively compute the V ∆t (t n , •) if we know how to analytically compute the conditional expectations E a n,z [V ∆t (t n+1 , Z tn+1 )] for all z ∈ R d and all control a ∈ A. We refer to the procedure in (4.3.2) as value iteration.

An alternative approach to compute V ∆t (t n , •), for n = 0, . . . , N -1, is to notice that once again by the dynamic programming principle, it holds that V ∆t (t n , •) 0≤n≤N is solution to the backward equation

           V ∆t (t N , z) = g(z) V ∆t (t n , z) = sup a∈A f (z, a)∆t + E a n,z N -1 k=n+1 f Z t k , α * t k (Z t k ) ∆t + g(Z t N ) , for n = N -1, . . . , 0, (4.3.3)
where for k = n + 1, . . . , N -1, the control α * t k is the optimal control at time t k defined as follows:

α * t k (z) = argmax a∈A f (z, a)∆t + E a k,z N -1 =k+1 f Z * t , α * t (Z * t ) ∆t + g(Z * t N ) ,
and where Z * t k n≤k≤N is the process Z controlled by the following control α from time t n to t N :

α tn = a, α t k = α * t k for n + 1 ≤ k ≤ N -1.

Numerical methods

For n = 0, . . . , N -1, the scheme (4.3.3) provides once again a backward procedure to compute V ∆t (t n , •), assuming that we know how to analytically compute the conditional expectations

E a n,z N -1 k=n+1 f Z t k , α * t k (Z t k ) ∆t + g(Z t N ) ,
for all z ∈ R d and all control a ∈ A. We refer to the procedure in (4.3.3) as the performance iteration2 .

Except for trivial cases, closed-form formulas for the conditional expectations appearing in the value and policy iteration procedures are not available, and they have to be approximated, which is the main difficulty when implementing both approaches to compute the value functions. In the next section, we discuss different ways to approximate conditional expectations and derive the corresponding estimations of the value functions V ∆t (t n , •) for n = 0, . . . , N -1.

Approximation of conditional expectations

In this subsection, we present three numerical methods that we apply later to conditional MKV problems. Two of these methods belong to the class of Regression Monte Carlo techniques, a family of algorithms whose effectiveness highly relies on the choice of the basis functions used to approximate conditional expectations; the third algorithm, Quantization, approximate the controlled process Z α tn with a particular finite state Markov chain for which expectations can be approximated quickly.

Regression Monte Carlo

In the simpler uncontrolled case, the family of Regression Monte Carlo algorithms is based on the idea of approximating the conditional expectation E V ∆t (t n+1 , Z tn+1 ) Z tn , for n = 0, . . . , N -1, by the orthogonal projection of V ∆t (t n+1 , Z tn+1 ) onto the space generated by a finite family of φ k (Z tn ) k≥1 where (φ k ) k≥1 is a family of basis functions, i.e., a family of measurable real-valued functions defined on R d such that φ k (Z tn ) k≥1 is total in L 2 (σ(Z tn ))3 and such that for all scalars β k and all K ≥ 1, if K k=1 β k φ k (Z tn ) = 0 a.s. then β k = 0, for k = 1, . . . , K.

The expectation E V ∆t (t n+1 , Z tn+1 ) Z tn should then be approximated as follows:

E V ∆t (t n+1 , Z tn+1 ) Z tn ≈ K k=1 β n k φ k (Z tn ),
where K ≥ 1 is fixed and β n = (β n 1 , . . . , β n K ) is defined as:

β n = argmin β∈R K E V ∆t (t n+1 , Z tn+1 ) - K k=1 β k φ k (Z tn ) 2 . (4.3.4)
Notice that β n is defined in (4.3.4) as the minimizer of a quadratic function, and can then be rewritten by straightforward calculations as:

β n = E φ(Z tn )φ(Z tn ) -1 E V ∆t (t n+1 , Z tn+1 )φ(Z tn ) , (4.3.5)
where we use the notation φ = (φ 1 , . . . , φ K ) , and where we assumed that E φ(Z tn )φ(Z tn ) is invertible 4 .

In order to estimate a solution to (4.3.5) we rely on Monte Carlo simulations to approximate expectations with finite sums. Consider the training set { Z m tn , Z m tn+1 } M m=1 at time t n obtained by running M ≥ 1 forward simulations of the process Z from time t 0 = 0 to t n+1 . β n defined in (4.3.5) can then be estimated by

βn = ÂM n -1 1 M M m=1 V ∆t (t n+1 , Z m tn+1 )φ(Z m tn ),
where we denote by ÂM n the estimator 1 M M m=1 φ(Z m tn )φ(Z m tn ) of the covariance matrix

A n = E φ(Z tn )φ(Z tn ) .
The procedure presented above offers a convenient mean to approximate conditional expectations when the dynamics of the process Z are uncontrolled. When controlled, however, one has to account for the effect of the control on the conditional expectations either explicitly, via Control Randomization, or implicitly, via Regress-Later.

Remark 4.3.1. As recently detailed in [START_REF] Bayer | Dynamic programming for optimal stopping via pseudo-regression[END_REF], there is another way of building estimates of β n , for n = 0, . . . , N -1, which is more accurate, less costly to compute, and appears to be more efficient in pratice. The idea is the following: assuming that the basis of functions is orthonormal w.r.t. µ, we have:

β n k := (V ∆t (t n+1 , .), φ k ) L 2 (µ) = E E[V ∆t (t n+1 , Z tn+1 )|Z tn ]φ k (Z tn ) = E V ∆t (t n+1 , Z tn+1 )φ k (Z tn )
Hence, one can build βn , defined below, as a better estimate of β n than βn .

βn k = 1 M M m=1 V ∆t (t n+1 , Z m tn+1 )φ k (Z m tn ), fork = 0, . . . , K.
The general case, where the basis functions are note orthonormal, requires multiplication with the Gram matrix formed by (φ k , φ l ) 1≤k,l≤K , but the idea remains the same.

Control Randomization

In order to explicitly account for the effect of the control, one could directly introduce dependence on the control in the basis function. This basic idea of Control Randomization consists in replacing in the dynamics of Z the endogenous control by an exogenous control (I tn ) 0≤n≤N , as introduced in [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF].

Trajectories of (Z tn , I tn ) 0≤n≤N can then be simulated from time t 0 to time t N . Consider the training set {Z m tn , I m tn } N,M n=0,m=1 , with M ≥ 1, where I m tn are i.i.d. samples from a "training distribution" µ n with support in A. The training set will be used to estimate the optimal β n coefficients for n = 0, . . . , N -1. In the case of value iteration, {V ∆t (t n+1 , Z m tn+1 )} M m=1 is regressed against basis functions (which are, in this context, functions of the state and the control) evaluated at the points {Z m tn , I m tn } M m=1 , as follows:

E a n,z V ∆t (t n+1 , Z tn+1 ) ≈ K k=1 βn k φ k (z, a),
where βn is an estimator of Notice that the basis functions take state and action variables as input in the case of Control Randomization-based method, i.e., their domain is R d ×A. Also, observe that the estimated conditional expectation highly depends on the choice of the randomization for the control 5 . An optimal feedback control at time t n given Z tn = z is approximated by the expression (see Subsection 4.3.4 for more practical details on the computation of the argmax):

β n := argmin β∈R K    E   V ∆t t n+1 , Z tn+1 - K k=1 β k φ k (Z tn , I tn ) 2      , defined as βn = ÂM n -1 1 M M m=1 V ∆t t n+1 , Z m tn+1 φ(Z m tn , I m tn ) ,
αtn (z) = argmax a∈A f (z, a)∆t + K k=1 βn k φ k (z, a) . (4.3.7)
The value function at time t n is then estimated using Control Randomization method and value iteration procedure as

V CR ∆t (t n , z) = f z, αtn (z) ∆t + K k=1 βn k φ k z, αtn (z) , z ∈ R d .
Notice that Control Randomization can be easily employed in a performance iteration procedure by computing controls (4.3.7), keeping in mind that at each time t n we need to re-simulate new trajectories { Zm t k } N,M k=n,m=1 iteratively from the initial condition Zm tn = z, using the estimated optimal strategies (α t k ) N -1 k=n+1 to compute the quantities

N -1 k=n f t k , Zm t k , αt k ( Zm t k ) +g( Zm t N ), for 1 ≤ m ≤ M .

Regress-Later

We present now a regress-later idea in which conditional expectation with respect to Z tn is computed in two stages. First, a conditional expectation with respect to Z tn+1 is approximated in a regression step by a linear combination of basis functions of Z tn+1 . Then, analytical formulas are applied to condition this linear combination of functions of future values on the present value Z tn . For further details, see [START_REF] Glasserman | Simulation for American options: Regression Now or Regression Later?[END_REF], [START_REF] Bender | Least-squares Monte Carlo for backward SDEs[END_REF], [START_REF] Nadarajah | Comparison of least squares Monte Carlo methods with applications to energy real options[END_REF] or [START_REF] Balata | Regress-Later Monte-Carlo for optimal inventory control with applications in energy[END_REF]. With this approach, the effect of the control is factored in implicitly, through its effect on the (conditional) distribution of Z tn+1 conditioned on Z tn .

Unlike the traditional Regress-Now method for approximating conditional expectations (which we discussed so far in the uncontrolled case and in Control Randomization), the Regress-Later approach, as studied in [START_REF] Balata | Regress-Later Monte-Carlo for optimal inventory control with applications in energy[END_REF], imposes conditions on basis functions: Assumption 4.3.1. For each basis function φ k , k = 1, . . . , K, the conditional expectation

φn k (z, a) = E a n,z [φ k (Z tn+1 )]
can be computed analytically.

Using the Regress-Later approximation of the conditional expectation and recalling Assumption 4.3.1 we obtain the optimal control α m tn corresponding to the point Z m tn , sampled independently from a "training distribution" µ n (see Subsection 4.3.3 for further details):

α m tn = argmax a∈A f (Z m tn , a)∆t + K k=1 βn+1 k φn k Z m tn , a .

Chapter 4. A class of numerically solvable MKV control problems

Notice that we are able to exploit the linearity of conditional expectations because βn+1 = argmin

β∈R K M m=1 V ∆t (t n+1 , Z m tn+1 ) - K k=1 β k φ k (Z m tn+1 ) 2 (4.3.8)
is a constant once the training sets at times t k , k = n + 1, . . . , N, are fixed.

The value function at time t n , is then estimated using Regress-Later method and value iteration procedure as

V RL ∆t (t n , Z m tn ) = f (Z m tn , α m tn )∆t + K k=1 βn+1 k φn k Z m tn , α m tn .
Notice that contrary to Control Randomization, Regress-Later does not require the training points to be distributed as Z tn+1 conditioned on Z tn because the projection (4.3.8) is an approximation to an expectation conditional to the measure µ n which can be chosen freely to optimize the precision of the sample estimation. On the other hand Regress-Later, similarly to Control Randomization, can be easily employed in a performance iteration procedure by generating forward trajectories at each time step.

Remark 4.3.1. Recall that the Regress-Later method uses training points that are i.i.d at each time step and independent across time steps. Contrary to other Regression Monte Carlo approaches, Regress-Later does not require to use the information about the conditional distribution during the regression step as that is accounted for in the second step of the method, when conditional expectations are computed analytically.

Quantization

We propose in this section a quantization-based algorithm to numerically solve control problems. We may also refer to the latter as the Q-algorithm or Q in all the numerical examples considered in Section 4.4, where Q stands for Quantization. Let us first introduce some ingredients of Quantization, and then propose different ways of using them to approximate conditional expectations.

Let (E, |.|) be a Euclidean space. We denote by ε a L-quantizer of an E-valued random variable ε, that is a discrete random variable on a grid Γ = {e 1 , . . . , e L } ⊂ E L defined by

ε = Proj Γ (ε) = L =1 e 1 ε∈C (Γ) ,
where C 1 (Γ), . . ., C L (Γ) are the Voronoi cells corresponding to Γ, i.e., they form a Borel partition of E satisfying

C (Γ) ⊂ e ∈ E : |e -e | = min j=1,...,L |e -e j | ,
and where Proj Γ stands for the Euclidean projection on Γ.

The discrete law of ε is then characterized by

p = P[ε = e ] = P[ε ∈ C (Γ)], = 1, . . . , L.
The grid of points (e ) L =1 which minimizes the L 2 -quantization error ε -ε 2 leads to the so-called optimal L-quantizer, and can be obtained by a stochastic gradient descent method, known as Kohonen algorithm or competitive learning vector quantization (CLVQ) algorithm, which also provides as a byproduct an estimation of the discrete law (p ) L =1 . We refer to [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF] for a description of the algorithm, and mention that for the normal distribution, the optimal grids and the weights of the Voronoi tesselations are precomputed for dimension up to 10 and are available on the website http://www.quantize.maths-fi.com.

Numerical methods

In practice, optimal grids of the Gaussian random variable N 1 (0, 1) of dimension 1 with 25 to 50 points, have been used to solve the control problems considered in Section 4.4.

We now propose two ways to approximate conditional expectations. The first approximation belongs to the family of the constant piecewise approximation, and the other one is an improvement on the first one, where the continuity of the approximation w.r.t. the control variable is preserved.

In the sequel, assume that for n = 0, . . . , N -1 we have a set Γ n of points in R d that should be thought of as a training set used for estimating V (t n , •). See Subsection 4.3.3 for more details on how to build Γ n .

Piecewise constant interpolation

We assume here that we already have an estimate of V ∆t (t n+1 , •), the value function at time t n+1 , for n ∈ {0, . . . , N -1}, and we denote by V Q ∆t (t n+1 , •) the estimate. The conditional expectation is then approximated as

E a n,z V Q ∆t t n+1 , Z tn+1 ≈ L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t (z, a, e ) , for z ∈ Γ n ,
where:

• G ∆t is defined, using the notations introduced in (4.3.1), as

G ∆t (z, a, ε) = z + b(z, a)∆t + σ 0 (z, a) √ ∆t ε. (4.3.9)
• Proj Γn (.) stands for the Euclidean projection on Γ n .

• Γ = {e 1 , . . . , e L and p 1≤ ≤L are respectively the optimal L-quantizer and its associated sequence of weights of the exogenous noise ε. See above for more details.

An optimal feedback control at time t n and point z ∈ Γ n is approximated by the expression (see Subsection 4.3.4 for more practical details on the computation of the argmax):

αQ tn (z) = argmax a∈A f (z, a)∆t + L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t (z, a, e ) .
The value function at time t n , is then estimated using the piecewise constant approximation and value iteration procedure as

V Q ∆t (t n , z) = f Z m tn , αQ tn (z) ∆t + L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t z, αQ tn (z), e .
Remark 4.3.2. Clearly, the constant piecewise approximation can be easily extended to control problems of all dimensions d ≥ 1. However the latter is, in most cases, not continuous w.r.t. the control variable since it remains constant on each Voronoi cells (see, e.g., Figure 4.1 p.128). As a result, the optimization process over the control space suffers from high instability and inaccuracy, which implies a poor estimation of the value function V (t n , •).

Semi-linear interpolation Once again, we assume here that we already have V Q ∆t (t n+1 , •), an estimate of the value function at time t n+1 , with n ∈ {0, . . . , N -1}, and wish to provide an estimation of the conditional expectation in the particular case where the controlled process lies in dimension d=1. Consider the following piecewise linear approximation of the conditional expectation, which is continuous w.r.t. the control variable a:

E a n,z V Q ∆t t n+1 , Z tn+1 ≈ L =1 p λ e ,z a V Q ∆t t n+1 , z + + (1 -λ e ,z a ) V Q ∆t t n+1 , z -, for z ∈ Γ n , (4.3.10)
where for all = 1, . . . , L, z -and z + are defined as follows:

• z -and z + are the two closest states in Γ n+1 from G ∆t (z, a, e ), such that z -< G ∆t (z, a, e ) < z + , if such states exist; and, in this case, we define λ e ,z a = G∆t(z,a,e )-z- z+-z-.

• Otherwise, z -and z + are equal and defined as the closest state in Γ Z n+1 from G ∆t (z, a, e ) and we define λ e ,z a = 1. Remark 4.3.3. This second approximation is continuous w.r.t. the control variable, which brings stability and accuracy to the optimal control task (see Subsection 4.3.4), and also ensures an accurate estimate of the value function at time t n . We will mainly use this approximation in the numerical tests (see Section 4.4). Remark 4.3.4. Although the dimension d = 1 plays a central role to define clearly the states z - and z + in (4.3.10), the semi-linear approximation can actually be generalized to a certain class of control problems of dimension greater than 1, using multi-dimensional Quantization (see, e.g., the comments on the Q-algorithm designed to solve the Portfolio Optimization example, in Subsection 4.4.1.2). However, it is not well-suited to solve numerically general control problems in dimension greater than 1. For these cases, other interpolating methods such as the use of Gaussian processes are more appropriated (see, e.g., [START_REF] Ludkovski | Simulation methods for stochastic storage problems: a statistical learning perspective[END_REF] for an introduction on the use of Gaussian processes in Regression Monte Carlo).

The optimal feedback control at time t n and point z ∈ Γ n is approximated as (see Subsection 4.3.4 for more practical details on the computation of the argmax):

αQ tn (z) = argmax a∈A f (z, a)∆t + L =1 p λ e ,z a V Q ∆t t n+1 , z + + (1 -λ e ,z a ) V Q ∆t t n+1 , z - .
The value function at time t n is then estimated using the semi-linear approximation and value iteration procedure as

V Q ∆t (t n , z) = f z, αQ tn (z) ∆t + L =1 p λ e ,z αQ tn (z) V Q ∆t t n+1 , z + + (1 -λ e ,z αQ tn (z) ) V Q ∆t t n+1 , z -,
where z + and z -are defined using the control αQ tn (z). See (4.3.10) for their definitions.

Training points design

We discuss here the choice of the training measure µ and the sets (Γ n ) n=0,...,N -1 used to compute the numerical approximations in Regression Monte Carlo and Quantization. Two cases are considered in this section. The first one is a knowledge-based selection, relevant when the controller knows with a certain degree of confidence where the process has to be driven in order to optimize her reward functional. The second case, on the other hand, is when the controller has no idea where or how to drive the process to optimize the reward functional.

Exploitation only strategy

In the knowledge-based setting there is no need for exhaustive and expensive (in time mainly) exploration of the state space, and the controller can directly choose training sets Γ constructed from distributions µ that assign more points to the parts of the state space where the optimal process is likely to be driven.

In practice, at time t n , assuming we know that the optimal process is likely to stay in the ball centered around the point m n and with radius r n , we chose a training measure µ n centered around m n as, for example N (m n , r 2 n ), and build the training set as sample of the latter. In the Regress-Later setting this can be done straightforwardly, while Control Randomization requires one to select 4.3. Numerical methods a measure for the random control such that the controlled process Z is driven in such area of the state space.

Taking samples according to µ to build grids makes them random. Another choice, which we used in the Quantization-based algorithm, is to use the (deterministic) optimal grid of N (m n , σ 2 n ) with reduced size (typically take 50 points for a problem in dimension 1, 250 for one of dimension 2 when σ 2 n = 1,. . . ), which can be found at www.quantize.maths-fi.com, to reduce the size of the training set and alleviate the complexity of the algorithms. Remark 4.3.5. As the reader will see, we chose the training sets based on the "exploitation only strategy" procedure, i.e. by guessing where to drive optimally the process, when solving the Liquidation Problem introduced in Subsection 4.4.1.

Explore first, exploit later

Explore first: If the agent has no idea of where to drive the process to receive large rewards, she can always proceed to an exploration step to discover favorable subsets of the state space. To do so, the Γ n , for n = 0, . . . , N -1, can be built as uniform grids that cover a large part of the state space, or µ can be chosen uniform on such domain. It is essential to explore far enough to have a well understanding of where to drive and where not to drive the process.

Exploit later:

The estimates for the optimal controls at time t n , n = 0, . . . , N -1, that come up from the Explore first step, are relatively good in the way that they manage to avoid the wrong areas of state space when driving the process. However, the training sets that have been used to compute the estimated optimal control are too sparse to ensure accuracy on the estimation. In order to improve the accuracy, the natural idea is to build new training sets by simulating M times the process using the estimates on the optimal strategy computed from the Explore first step, and then proceed to another estimation of the optimal strategies using the new training sets. This trick can be seen as a two steps algorithm that improves the estimate of the optimal control. Remark 4.3.6. In Control Randomization, multiple runs of the method are often needed to obtain precise estimates, because the initial choice of the dummy control could drive the training points far from where the optimal control would have driven them. In practice, after having computed an approximate policy backward in time, such policy is used to drive M simulations of the process forward in time, which in turn produce control paths that can be fed as a random controls in a new backward procedure, leading to more accurate results. Remark 4.3.7. We applied the "explore first, exploit later" idea to solve the Portfolio Optimization problem introduced in Subsection 4.4.1.

Optimal control searching

Assume in this section that we already have the estimates V ∆t (t k , •) for the value function at time t k , for k = n + 1, . . . , N , and want to estimate V (t n , •) the value function at time t n .

The optimal control searching task consists in optimizing the function6

Q n : (z, •) → f (z, a)∆t + Êa n,z V ∆t (t n+1 , Z tn+1 )
over the control space A, for each z ∈ Γ n , and where we denote by Êa n,z V ∆t (t n+1 , Z tn+1 ) an approximation of E a n,z V ∆t (t n+1 , Z tn+1 ) using Regress-Later, or Control Randomization or Quantizationbased methods (see Subsection 4.3.2). Once again, we remind that importance of this task is motivated Chapter 4. A class of numerically solvable MKV control problems by the dynamic programming principle stating that for all n = 0, . . . , N -1, we can approximate the value function at time n as follows

V ∆t (t n , z) = sup a∈A Q n (z, a),
where V ∆t (t n , •) is our desired estimate of the value function at time n.

Low cardinality control set

In the case where the control space A is discrete (with a relatively small cardinality), one can solve the optimization problem by an exhaustive search over all the available controls without compromising the computational speed.

Remark 4.3.8. Note that in the case where the control space is continuous, one can always discretize the latter in order to rely on the effectiveness of extensive search to solve the optimal control problem. However, the control space discretization brings an error. So the control might have to include a high number of points in the discretization in order to reduce the error thereby causing a considerable slow down of the computations.

High cardinality/continuous control space

If we assume differentiability almost everywhere, as follows from the semi-linear approximation in Quantization, and most choices of basis functions in Regression Monte Carlo, we can carry on the optimization step by using some gradient-based algorithm for optimization of differentiable functions. Actually, many optimizing algorithms (Brent, Golden-section Search, Newton gradient-descent,. . . ) are already implemented in standard libraries of most programming languages like Python (see, e.g., package scipy.optimize), Julia (see, e.g., package Optim.jl), C and C++ (see, e.g., package NLopt).

Remark 4.3.9. When the control is of dimension 1, polynomials of order smaller than 5 are employed as basis functions in Regression Monte Carlo as well as for the running reward f . The optimal control can then be computed analytically as a function of the regression coefficients, since every polynomial equation of order smaller than 4 can be solved by radicals.

Concretely, in all the examples considered in Section 4.4, we used the Golden-section Search or the Brent methods when testing Quantization-based algorithm to find the optimal controls at each point of the grids. These algorithms were very accurate to find the optimal controls, and we made use of Remark 4.3.9 to find the optimal controls using the Regress-Later-based algorithm.

Upper and lower bounds

After completing the backward procedure, we can compute an unbiased estimation of the value of the control policy by using Monte Carlo simulations and sample average. Assume already computed (or simply available) the matrix of regression coefficients, in the case of Regression Monte Carlo, and discrete probability law p for Quantization, we can use this information to implicitly compute the control and simulate forward many trajectories of the controlled process starting from a common initial condition. We can then evaluate the average performance measure by computing the sample average of the rewards collected on each trajectory. Denoting such approximation by V∆t (0, z), and recalling that by definition J ∆t (0, z, α) ≤ J ∆t (0, z, α * ), for all α ∈ A ∆t and where α * represents the optimal control process; it holds V∆t (0, z) ≤ V ∆t (0, z), for z ∈ R d .

The argument above implies that, neglecting the time-discretization error, we obtain a lower bound for V ∆t (0, •) by evaluating the estimated policy. To get an upper bound of the value function via duality, see [START_REF] Hildebrand | Regression based duality approach to optimal control with application to hydro electricity storage[END_REF], [START_REF] Henry-Labordère | Deep primal-dual algorithm for BSDEs: applications of machine learning to CVA and IM[END_REF] based on [START_REF] Rogers | Monte Carlo Valuation of American Options[END_REF], and [START_REF] Bender | Pathwise Dynamic Programming[END_REF].

Numerical methods

Pseudo-codes

In this section, we present the pseudo-code for the three approaches presented in the previous sections. For simplicity, we will only show the algorithms designed using value iteration procedure. However, the performance iteration update rule can be substituted in the codes below provided that forward simulations are run to obtain a pathwise realization of the controlled process and associated rewards.

Pseudo-code for a Regress-Later-based algorithm

We present in Algorithm 4.1 a pseudo-code to estimate V ∆t (t n , •), for n = 0, . . . , N -1, using Value Iteration and based on Regress-Later method. For n = 0, . . . , N -1, we denote by V RL ∆t (t n , •) the derived estimation of V ∆t (t n , •), and will refer to it as the RLMC algorithm in the numerical tests presented in Section 4.4.

Note that we use the same training measure µ at each time step so that there is only one covariance matrix to estimate (since A tn is the same for all n = 0, . . . , N -1). Denote by ÂM the estimator, as defined in (4.3.6).

βn = Â-1 M 1 M M m=1 V RL ∆t (t n+1 , Z m tn+1 )φ(Z m tn+1 ) . 6:
Generate a new layer of i.i.d. training points {Z m tn } M m=1 accordingly to the distribution µ.

7:

For all m = 1, . . . , M do

V RL ∆t (t n , Z m tn ) = sup a∈A f (Z m tn , a)∆t + K k=1 βn k φn k (Z m tn , a) .
8: end for 9: Evaluate the policy to obtain

V RL ∆t . Outputs: { βn k } N,K n,k=1 , V RL ∆t (0, z) for z ∈ R d .

Pseudo-code for a Control Randomization-based algorithm

We present in Algorithm 4.2 a pseudo-code to estimate V ∆t (t n , •, •), for n = 0, . . . , N -1, using Value Iteration and based on Control Randomization method. For n = 0, . . . , N -1, we denote by V CR ∆t (t n , •) the derived estimation of V ∆t (t n , •), and will refer to it as the CR algorithm in the numerical tests presented in Section 4.4. • M : number of training points,

• µ: initial distribution of dummy control,

• K: number of basis functions,

• {φ k } K k=1 : family of basis functions.

(t N , Z m t N ) = g(Z m t N ), m = 1, . . . , M . 4: for n = N -1 to 0 do 5: βn = ( ÂM ) -1 1 M M m=1 V CR ∆t (t n+1 , Z m tn+1 )φ(Z m tn , I m tn )
.

6:

For all m = 1, . . . , M do

V CR ∆t (t n , Z m tn ) = sup a∈A f (Z m tn , a)∆t + K k=1 βn k φ k (Z m tn , a) .
7: end for 8: Evaluate the policy to obtain

V CR ∆t . Outputs: { βn k } N,K n=0,k=1 , V CR ∆t (0, z) for z ∈ R d .

Pseudo-code for a Quantization-based algorithm

We present in Algorithm 4.3 a pseudo-code to estimate V ∆t (t n , •, •), for n = 0, . . . , N -1, using value iteration procedure and based on Quantization method. For n = 0, . . . , N -1, we denote by V Q ∆t (t n , •) the derived estimation of V ∆t (t n , •), and will refer to it as the Q-algorithm in the numerical tests presented in Section 4.4.

Note that we made use of a piecewise constant approximation of conditional expectations to approximate V Q ∆t (t n , •) in order to keep the algorithm simple. Also, note that, as said previously, in most of the numerical tests run in Section 4.4, we will use optimal grids available at www.quantize. maths-fi.com and will take L = 25 to 50 points for the size of the optimal grid of the Gaussian noise ε.

Applications and numerical results

Portfolio Optimization under drift uncertainty

The model

We consider a financial market model with one risk-free asset, assumed to be equal to one, and d risky assets of price process S = (S 1 , . . . , S d ) governed

dS t = diag(S t )(β t dt + σdB 0 t ), S 0 = s 0 ∈ R d ,
where B 0 is a d-dimensional Brownian motion on a filtered probability space (Ω, F, F, P 0 ), σ is the d × d invertible matrix volatility coefficient, assumed to be known and constant. However, the drift (β t ) of the asset (which is typically a diffusion process governed by another independent Brownian • Γ = {e 1 , . . . , e L } , (p ) 1≤ ≤L : the L-optimal grid of the exogenous noise ε, and its associated weights,

1: Initialize the estimated value function at time N :

V Q ∆t (t N , z) = g(z), ∀z ∈ Γ N . 2: for n = N -1 to 0 do 3:
Estimate the value function at time t n as follows:

V Q ∆t (t n , z) = max a∈A f (z, a) ∆t + L =1 p V Q ∆t t n+1 , Proj Γn+1 G ∆t (z, a, e )
, ∀z ∈ Γ n .

4: end for 5: Evaluate the policy to obtain

V Q ∆t . Outputs: α(t n , z) z∈Γn,0≤n≤N -1 , V Q ∆t (0, z) z∈Γ0 .
motion B) is unknown and unobservable like the Brownian motion B 0 . The agent can actually only observe the stock prices S, and we denote by F S the filtration generated by the price process S, which should be view as the available information.

In this context, we shall consider two important classes of optimization problems in finance:

(1) Portfolio Liquidation. We consider the problem of an agent (trader) who has to liquidate a large number y 0 of shares in some asset (we consider one stock, d = 1) within a finite time T , and faces execution costs and market price impact. In contrast with frictionless Merton problem, we do not consider mark-to-market value of the portfolio and instead consider separately the amount on the cash account and the inventory Y , i.e., the position or number of shares held at any time. The strategy of the agent is then described by a real-valued F S -adapted process α, representing the velocity at which she buys (α t > 0) or sells (α t < 0) the asset, and the inventory is thus given by

Y t = y 0 + t 0 α u du, 0 ≤ t ≤ T.
The objective of the trader is to minimize over α the total liquidation cost

J 1 (α) = E 0 T 0 α t S t + f (α t ) dt + (Y T )
where f (.) is an increasing function with f (0) = 0, representing a temporary price impact, and (.) is a loss function, i.e., a convex function with (0) = 0, penalizing the trader when she does not succeed to liquidate all her shares.

(2) Portfolio Selection. The set A of portfolio strategies, representing the amount invested in the assets, consists in all F S -adapted processes α valued in some set A of R d , and satisfying

T 0 |α t | 2 dt < ∞.
The dynamics of wealth process X = X α associated to a portfolio strategy α is then governed by dX t = α t S -1 t dS t = α t .β t dt + α t σdB 0 t , X 0 = x 0 ∈ R, and as in Merton Portfolio Selection problem, the objective of the agent is to maximize over portfolio strategies the utility of terminal wealth

J 2 (α) = E 0 [U (X T )],
where U is a utility function on R, e.g., CARA function U (x) = -exp(-px), p > 0.

Let us show how one can reformulate the above problems into a McKean-Vlasov type problem under partial observation and common noise as described in Section 4.1. We first introduce the socalled probability reference P, which makes the observation price process a martingale. Let us then define the process

Z t = exp - t 0 σ -1 β u dB 0 u - 1 2 t 0 |σ -1 β u | 2 du , 0 ≤ t ≤ T,
which is a (P 0 , F)-martingale (under suitable integrability conditions on β), and defines a probability measure P ∼ P 0 through its density: dP dP 0 Ft = Z t , and under which the process

W 0 t = B 0 t + t 0 σ -1 β u du, 0 ≤ t ≤ T,
is a (P, F)-Brownian motion by Girsanov's theorem, and the dynamics of S is

dS t = diag(S t )σdW 0 t .
Notice that F S = F 0 the filtration generated by W 0 . We also denote by L t = 1/Z t the (P, F)-martingale governed by

dL t = L t σ -1 β t .dW 0 t .
Next, we use Bayes formula and rewrite the gain (resp. cost) functionals of our two portfolio optimization problems as

J 1 (α) = E 0 L t α t (S t + f (α t ))dt + L T (Y T ) = E T 0 L0 t α t (S t + f (α t ))dt + L0 T (Y T ) = E T 0 L0 t α t ( S0 t + f (α t ))dt + L0 T ( Ȳ 0 T ) , J 2 (α) = E L T U (X T ) = E L0 T U (X T ) = E L0 T U ( X0 T )
where

L0 t = E[L t |W 0 ] = P W 0 Lt (d ), X0 t = E[X t |W 0 ] = xP W 0 Xt (dx) = X t , Ȳ 0 t = E[Y t |W 0 ] = yP W 0 Yt (dy) = Y t , S0 t = E[S t |W 0 ] = sP W 0 St (ds) = S t
, and we used the law of conditional expectations and the fact that S, X and Y are F 0 -adapted. This formulation of the functional J 1 (resp. J 2 ) fits into the MKV framework of Section 4.1 with state variables (X, L, β) (resp. (Y, S, L, β))

We now consider the particular case when β is an F 0 -measurable random variable distributed according to some probability distribution ν(db): this corresponds to a Bayesian point of view when the agent's belief about the drift is modeled by a prior distribution. In this case, let us show how our partial observation problem can be embedded into a finite-dimensional full observation Markov control problem. Indeed, by noting that β is independent of the Brownian motion W 0 under P, we have

L0 t = E exp σ -1 β.W 0 t - 1 2 |σ -1 β| 2 t W 0 = F (t, W 0 t ),
where

F (t, w) = exp σ -1 b.w - 1 2 |σ -1 b| 2 t ν(db).

Applications and numerical results

Hence, the functionals J 1 and J 2 can be written as

J 1 (α) = E T 0 F (t, W 0 t )α t (S t + f (α t ))dt + F (T, W 0 T ) (Y T ) , J 2 (α) = E F (T, W 0 T )U (X T ) .
We are then reduced to a (P, F 0 )-control problem with state variables (W 0 , X) for problem (1) and (W 0 , S, Y ) for problem (2), with the following dynamics under P:

dS t = diag(S t )σdW 0 t , S 0 = s 0 ∈ (R + ) d , dX t = α t σdW 0 t , X 0 = 0, dY t = α t dt, Y 0 = y 0 ∈ R + .
Remark 4.4.1. The case where the drift β is modeled by a linear Gaussian process is another example of partial observation. This would lead to the well-known Kalman-Bucy filter, hence to a finite-dimensional control problem. However, for general unobserved drift process β, we fall into an infinite dimensional control problem involving the filter process.

Numerical results

Let us now illustrate numerically the impact of uncertain Bayesian drift on the Portfolio Liquidation problem and the Portfolio Selection problem in dimension d = 1, by considering a Gaussian prior distribution β ∼ ν = N (b 0 , γ 2 0 ), with b 0 ∈ R and γ 0 > 0. In this case, F is explicitly given by:

F (t, w) = σ σ 2 + γ 2 0 t exp 1 2(σ 2 + γ 2 0 t) (-b 2 0 t + 2b 0 σw + γ 2 0 w 2 ) .
1. Portfolio Liquidation. Let us first consider the Portfolio Liquidation problem (1) with a linear price impact function f (a) = γa, γ > 0, and a quadratic loss function (y) = ηy 2 , η > 0. The optimal trading rate is given by (see [START_REF] Pham | Linear quadratic optimal control of conditional McKean-Vlasov equation with random coefficients and applications[END_REF])

α * t = - Y * t T -t + γ/η + 1 2γ 1 T -t + γ/η T t E 0 [S u |F S t ]du -S t
where Y * is the associated inventory with feedback control α * : dY * t = α * t dt, Y * 0 = y 0 . Since we consider a Gaussian prior N (b 0 , γ 2 0 ) for β, the optimal trading rate is explicitly given by

α * t = - 1 T -t + γ/η Y * t + 1 2γ - 1 γ 0 π 2 e - b 2 0 2γ 2 0 erfi b 0 + γ 2 0 (T -t) √ 2γ 0 -erfi b 0 √ 2γ 0 +(T -t+ γ η ) S t ,
where erfi is the imaginary error function, defined as:

erfi(x) = 2 √ π x 0 e t 2 dt.
Remark 4.4.2. In the particular case where the price process is a martingale, i.e., b 0 = 0, and in the limiting case when the penalty parameter η goes to infinity, corresponding to the final constraint Y T = 0, we see that α * t converges to -Y * t /(T -t), hence it becomes independent of the price process, and this leads to an explicit optimal inventory:

Y * t = y 0 T -t
T with constant trading rate α * t = -y 0 /T . We retrieve the well-known VWAP strategy obtained in [START_REF] Almgren | Optimal execution of portfolio transactions[END_REF].

We solve the problem numerically, taking N = 100 for the time discretization, and fixing the other parameters as follows: γ=5, S 0 =6, Y 0 =1, η=100 and σ=0.4. We run two sets of forward Monte Carlo simulations for b 0 = 0.1, T = 1 and b 0 = -0.1, T = 0.5 changing the value of γ 0 . We tested the Regress-Later Monte Carlo (RLMC), the Control Randomization (CR) and the Quantization Chapter 4. A class of numerically solvable MKV control problems (Q) algorithms. In particular, we wanted to compare the performance of these algorithms with (α * tn ) N -1 n=0 , where α * , defined above, is the optimal strategy associated to the continuous-time Portfolio Liquidation problem. We refer to this discrete-time strategy as α * (i.e., re-using the same notation), and we use Opt, or continuous-time optimal strategy when we want to stress the fact that this strategy is optimal for the continuous-time control problem, and not for the discrete time one. We also tested a benchmark strategy (Bench) which consists in liquidating the inventory at a constant rate -y 0 /T . The test consisted in computing the estimates V∆t (t 0 = 0, S 0 = 6, Y 0 = 1) associated to the different algorithms.

We display the results obtained by the different algorithms in Table 4.1 and plot them in Figure 4.2. One can observe in Figure 4.2 that for ∆t = 1 100 the estimations V∆t (t 0 = 0, S 0 = 6, Y 0 = 1) of the value function V ∆t (t 0 = 0, S 0 = 6, Y 0 = 1), provided by RLMC, CR or Q-based methods, are sometimes such that 4.2 the curve Opt). It means that RLMC, CR, or Q-based methods sometimes provide better estimations of the optimal strategy than α * for the discrete time control problem. However, since under suitable conditions (see, e.g., [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF]), the optimal strategy for the discrete time control problem α * ∆t converges toward α * , i.e. we have α * ∆t ----→ ∆t→0 α * , then it holds:

V∆t (t 0 = 0, S 0 = 6, Y 0 = 1) ≤ Ĵ∆t (t 0 = 0, S 0 = 6, Y 0 = 1, α * ), where Ĵ∆t (•, •, •, α * ) is a Monte Carlo estimate of J ∆t (•, •, •, α * ) applying strategy (α * tn ) N -1 n=0 (see in Figure
Ĵ∆t (t 0 = 0, S 0 = 6, Y 0 = 1, α * ) ----→ ∆t→0 V (t 0 = 0, S 0 = 6, Y 0 = 1).
Figure 4.3 shows a sample of the inventory (Y t ) t∈[0,T ] when the agent follows α * and the Quantization algorithm. One can notice that given the chosen penalization parameters, it is optimal to short some stocks at terminal time. Finally, notice that the concavity of the curves comes from the fact that the running cost does not penalize the inventory. If so, we expect the curves of the inventory w.r.t. time to be convex, see, e.g., [START_REF] Gatheral | Dynamical models of market impact and algorithms for order execution[END_REF].

Details on the RL and CR algorithms implementation

The implementation of Regression Monte Carlo algorithms has required intense tuning and the use of the performance iteration technique introduced in Subsection 4.3.3.2 in order to obtain satisfactory results. Paramount is, in addition, the distribution chosen for the training points in Regress-Later and for the initial control in Control Randomization. The problem of finding the best set of data to provide to the backward procedure is similar in the two Regression Monte Carlo algorithms. However little study is available in the literature; for more details on this problem in the Regress-Later setting see [START_REF] Nadarajah | Comparison of least squares Monte Carlo methods with applications to energy real options[END_REF] and [START_REF] Balata | Regress-Later Monte-Carlo for optimal inventory control with applications in energy[END_REF]. In the case of RL algorithm a training measure µ n has been chosen in order to sufficiently explore the state space in the Y dimension, in particular we considered

µ n = U[-0.5, 0.5 + T -tn tn ].
Similarly for CR we seek a distribution of the random control such that the controlled process Y results in having a distribution similar to µ n . In order to achieve such goal we follow the "explore first, exploit later" approach presented in Subsection 4.3.3.2 and use a perturbed version of the empirical distribution of the control (to avoid concentration of the training points) obtained at previous iteration of the method to determine the random control at next iteration of the method.

In order to choose the basis functions, we used the fact that we expect the value function to be convex in the Y dimension with minimum around the optimal inventory level and monotone in the S dimension. For RL algorithm we choose therefore the following set of basis functions: {s, y, y 2 , sy, sy 2 }, where we take the square function y 2 as a general approximator for convex functions around their minima (where we expect the measure µ n to be concentrated). On the other hand, CR requires that we guess what the functional form of the conditional expectation of the value function is with respect to the control process. Considering our argument on square function approximating general convex functions we choose to add the set {α, α 2 , αy, αs} to the set of basis functions used by RL.

Applications and numerical results

Note that there is no need for time-consuming optimal control searching with such a choice of basis functions, as explained in Remark 4.3.9.

Finally note that we observed very high volatility in the quality of the policy estimated by control randomization. For this reason we estimated the policy 50 times, and report in Table 4.1 the results provided by the best performing one; increasing the number of training points further affects the variability only marginally.

Details on the Q algorithm implementation

To numerically solve this example, we used the optimal grid of the Gaussian random variable with L = 50 points, denoted by Γ ε L , to define the grid7 Γ W n = t n Γ ε L that discretizes W tn , the Brownian motion at time t n , and the grid Γ Y n = Y 0 -tn T + t n Γ ε L that discretizes Y tn , the inventory at time t n , for n = 0, ..., N . Note that Γ Y n , for n = 0, ..., N , is centered at point Y 0 -tn T because we guessed that the optimal liquidation rate was close to Y0 T (see Figure 4.3 to check that our guess is correct). We then considered the grid

Γ n = Γ W n × Γ Y n to discretize Z tn = (W tn , Y tn ), n = 0, ..., N .
We first tried to design a quantization algorithm using the following expression for the conditional expectation approximations: ). This approximation belongs to the family of constant piecewise approximations, and is in the spirit of multidimensional component-wise-quantization methods already studied in the literature (see, e.g., [START_REF] Fiorin | Product Markovian quantization of a diffusion process with applications to finance[END_REF]). Unfortunately, as it can be seen in Figure 4.1, approximation (4.4.6) is discontinuous w.r.t. the control variable a in such a way that the optimal control searching task suffered from instability and inaccuracy, which implied bad value function estimations at time n = 0, ..., N -1. We thus had to use a better conditional expectation approximation.

E a n,(w,y) V Q ∆t t n+1 , Proj Γ W n+1 W tn+1 , Proj Γ Y n+1 Y tn+1 (4.4.6) ≈ L =1 p V Q ∆t t n+1 , Proj Γ W n+1 G ∆t ((w, y), a, e ) , Proj Γ Y n+1 G ∆t ((w, y), a, e ) , for (w, y, a) ∈ Γ W n × Γ Y n × A,
We then decided to smooth the previous approximation of the conditional expectations w.r.t. the control variable by considering the following

E a n,(w,y) V Q ∆t t n+1 , Proj Γ W n+1 W tn+1 , Proj Γ Y n+1 Y tn+1 ≈ L =1 p λ e ,(w,y) a V Q ∆t t n+1 , Proj Γ W n+1 G w ∆t ((w, y), a, e ) , y + + (1 -λ e ,(w,y) a ) V Q ∆t t n+1 , Proj Γ W n+1 G w ∆t ((w, y), a, e ) , y -,
where, in the spirit of the semi-linear approximation presented in Subsection 4.3.2, we have for all = 1, ..., L:

• G w ∆t ((w, y), a, e ) and G y ∆t ((w, y), a, e ) respectively stand for the first and the second component of G ∆t ((w, y), a, e ), i.e., G ∆t ((w, y), a, e ) = G w ∆t ((w, y), a, e ), G y ∆t ((w, y), a, e ) . See (4.3.9) for the definition of G ∆t .

→ e∈Γ ε P( ε = e) V Q ∆t t n+1 , Proj Γ W n+1 G ∆t ((w, y), a, e) , Proj Γ Y n+1
G ∆t ((w, y), a, e) . We took n = N -1, w = 0, and y = -0.18 to plot the curve. Observe that the approximation is discontinuous w.r.t. the control variable a in such a way that it makes the search of the minimizer of this function very difficult by usual (gradient descent-based) algorithms. Also, observe that the minimum of the function, which is actually equal to the estimation of the value function at time N -1 at point (w = 0, y = -0.18), suffers from inaccuracy. Table 4.1 -Portfolio Liquidation results. Estimations of the value functions at point (s 0 = 6, y 0 = 1) and time 0 provided by different algorithms. This approximation is a slight generalization (to dimension d=2) of the semi-linear approximation developed in (4.3.10). Its main interest lies in the continuity of the approximation w.r.t. the control variable a, which provides stability and accuracy to the usual (gradient descent-based) algorithms for the optimal controls searching, as can be seen on the numerical results (see, e.g., Table 4 Chapter 4. A class of numerically solvable MKV control problems 1] that the optimal portfolio strategy is explicitly given by

b 0 = 0.1, T = 1 b 0 = -0.1, T = 1/2 γ 0 Opt RLMC CR Q Bench Opt RLMC CR Q Bench 0.1 -1.
α * t = σ 2 + γ 2 0 t σ 2 + γ 2 0 T βt pσ 2 where βt = E 0 [β|F S t ] = σ 2 σ 2 + γ 2 0 t b 0 + γ 2 0 σ 2 + γ 2 0 t ln S t S 0 + 1 2 σ 2 t ,
is the posterior mean of the drift (Bayesian learning on the drift), and the optimal performance by

J 2 (α * ) = -exp -p x 0 + 1 2p ln σ 2 + γ 2 0 T σ 2 - γ 2 0 T σ 2 + γ 2 0 T + b 2 0 2pσ 2 σ 2 T σ 2 + γ 2 0 T .
The Portfolio Selection problem, even though in many aspects similar to the Portfolio Liquidation problem, is interesting in its own right because the control acts only on the variance of the controlled wealth process. We tested the Regress-Later Monte Carlo (RLMC), the Control Randomization (CR) and the Quantization (Q) algorithm on the Portfolio Selection problem. Similarly to what has been done for Portfolio Liquidation problem, we discretized time choosing N = 100 and solved the discrete time problem associated. We considered two set of experiments, b 0 = 0.1, T = 1 and b 0 = -0.1, T = 0.5, for different values of γ 0 ∈ [0, 1], p = 1, σ = 0.4. Given all these different parameters, we compared the performance of these algorithms with the one of the optimal strategy for the continuous-time problem α * (Opt). The general test consists in computing a forward Monte Carlo with 500000 samples, following optimal strategy estimated using different strategies, to provide estimates of V (t 0 = 0, X 0 = 0, W 0 = 0) the value function at time 0.

We present the results of our numerical experiments in Table 4.2. One can see that the Quantization algorithm performs similarly to the theoretical optimal strategy (Opt) for the continuous time problem, which can be interpreted as stability and accuracy of the Q algorithm, and also shows that the time discretization error is almost zero here.

We also present in Figure 4.4 a sample of the wealth of the agent following the optimal strategy and the (Q) estimated one. One can see that the strategies slightly differ when the drift is high, and remain the same when the drift is low. The small difference can be explained by the fact that the optimal strategy (Opt) is not optimal for the discrete time version of the problem.

Details on the Q algorithm implementation

We designed the same Quantization algorithm as the one built to solve the Portfolio Liquidation problem. We nevertheless had to take a larger number of points in the grids to minimize the backpropagation of errors from the borders of the girds; and had to use the "explore first, exploit later" idea (see Subsection 4.3.3.2) to improve the results.

Details on the RL and CR algorithms implementation

When implementing Regression Monte Carlo algorithms, and choosing basis functions, the control on variance implies that low order polynomial can not be used alone, as they can easily cause the control to be bang-bang between the boundaries of its domain. Similarly, piecewise approximations are not very effective, as the dependence on the control is very weak, requiring a high number of local supports and making the computational complexity overwhelming. We tested both value and performance iteration and tried to employ different kinds of basis functions and training points. Unfortunately, both Regress-Later and Control Randomization do not cope well with controlling the dynamics of a process through the variance only. A tailor-made implementation of Regression Monte Carlo to deal with this kind of problems is outside the scope of this paper and further investigation will follow in future work. For now, we chose not to provide results based on RL and CR methods. when the latter follows the continuoustime optimal strategy (Opt) and the (Q) estimated optimal strategy to solve the Portfolio Selection problem. We took σ=0.4, T =1, P =0.1, γ 0 =5, b 0 =0,1. One can see that the two strategies are the same when the drift is low; but Q performs slightly better than Opt when the drift is high, which is a time-discretization effect.

A model of interbank systemic risk with partial observation

The model

We consider the following model of systemic risk inspired by the model in [START_REF] Carmona | Mean field games and systemic risk[END_REF]. The log-monetary reserves of N banks lending to and borrowing from each other are governed by the system

dX i t = κ N N j=1 (X j t -X i t )dt + σX i t ( 1 -ρ 2 dW i t + ρdW 0 t ), i = 1, . . . , N
where W i , i = 1, . . . , N , are independent Brownian motions, representing the idiosyncratic risk of each bank, W 0 is a common noise independent of W i , σ > 0 is given real parameter, ρ ∈ [-1, 1], and where X i 0 , i = 1, ..., N are i.i.d.. The mean-reversion coefficient κ > 0 models the strength of interaction between the banks where bank i can lend to and borrow from bank j with an amount proportional to the difference between their reserves. In the asymptotic regime when N → ∞, the theory of propagation of chaos implies that the reserve state X i of individual banks become independent and identically distributed conditionally on the common noise W 0 , with a state governed by

dX t = κ(E[X t |W 0 ] -X t )dt + σX t ( 1 -ρ 2 dB t + ρdW 0 t )
for some Brownian motion B independent of W 0 .

Let us now consider a central bank, viewed as a social planner, who only observes the common noise and not the reserves of each bank, and can influence the strength of the interaction between the individual banks, through an F 0 -adapted control process α t . The reserve of the representative bank in the asymptotic regime is then driven by

dX t = (κ + α t )(E[X t |W 0 ] -X t )dt + σX t ( 1 -ρ 2 dB t + ρdW 0 t ), X 0 ∼ X 1 0 ,
and we consider that the objective of the central bank is to minimize

J(α) = E T 0 1 2 α 2 t + η 2 (X t -E[X t |W 0 ]) 2 dt + c 2 (X T -E[X T |W 0 ]) 2 ,
where η > 0 and c > 0 penalize the departure of the reserve from the average. This is a MKV control problem under partial observation, but notice that it does not belong to the class of linear quadratic (LQ) MKV problems due to the control α which appears in a multiplicative form with the state. However, it fits into our class of polynomial MKV problem, and can be embedded into standard control problem as follows:

We set Xt = E[X t |W 0 ] and Y t = E[(X t -Xt ) 2 |W 0 ]
. The cost functional is then written as

J(α) = E T 0 1 2 α 2 t + η 2 Y t dt + c 2 Y T
where the dynamics of X and Y are governed by

d Xt = σρ Xt dW 0 t , X0 = x 0 = E[X 0 ], dY t = σ 2 -2(κ + α t ) Y t + σ 2 (1 -ρ 2 ) X2 t dt + 2ρσY t dW 0 t , Y 0 = Var(X 0 ).
We have then reduced the problem to a (P, F 0 )-control problem, with state variables ( X, Y ) of dimension two, which is not LQ but can be solved numerically.

Numerical results

For this problem, in the absence of analytical solution, we decided to compare the estimations of the value function at time 0 provided by our algorithms with a numerical approximation based on

Applications and numerical results

finite difference scheme provided by Mathematica, of the solution to the 2-dimensional HJB equation associated to the systemic risk problem:

         ∂ t V + η 2 y + σ 2 -2κ y + σ 2 (1 -ρ 2 )x 2 ∂ y V + sup a∈A 1 2 a 2 -2ay∂ y V + σ 2 ρ 2 x 2 2 ∂ 2 xx V + 2σ 2 ρ 2 xy∂ xy V + 2σ 2 ρ 2 y 2 ∂ 2 yy V = 0, for (t, x, y) ∈ [0, T ) × R × R + , V (T, x, y) = c 2 y, ∀(x, y) ∈ R × R + .
(4.4.7) We refer to the solution of this partial differential equation (obtained using Mathematica using finite differences as explained below) as the Benchmark (or simply Bench) in the sequel.

We computed V∆t (t 0 = 0, x 0 = 10, y 0 = 0) using RL, CR and Q methods by considering a sample of size 500 000, and using the following parameters T = 1, σ = 0.1, κ = 0.5 and X 0 = 10. We recall that V∆t (t 0 = 0, x 0 , y 0 ) is an estimation of V (0, x 0 = 10, y 0 = 0), the value function at (x 0 , y 0 ) and time 0 (see its definition on the last step of each pseudo-code presented in Subsection 4.3.6).

In Table 4.3 we display the numerical results of experiments run for two situations: we took η = 10, c = 100 and η = 100, ρ = 0.5 and vary the value of ρ in the first case, and vary the value of c in the second one. Plots of the two tables are also available in Figure 4.5. One can observe that the algorithms performs well. Mainly, Bench and Q provide slightly better results than the Regression Monte Carlo-based algorithms (the curves of Bench and Q are below those of the other two). Figure 4.6 shows two examples of paths (X t ) t∈[0,T ] controlled by RLMC (curve "RLMC"), (X t ) t∈[0,T ] naively controlled by α = 0 (curve "uncontrolled"), and the conditional expectation of X ( Xt ) t∈[0,T ] (curve "E(X|W )"). One can see in these two examples that the (RLMC estimated) optimal control is as follows:

• do nothing when the terminal time is far, i.e., take α = 0, not to pay any running cost.

• catch X when the terminal time is getting close, to minimize the terminal cost.

We finally present a sample of paths (Y t ) t∈[0,T ] controlled by the decisions given by Q in Figure 4.7. One can see that the (Q estimated) optimal strategy minimizes the running cost first by letting Y grow; and deals with the terminal cost later by making Y small when the terminal time is approaching.

Details on the RL and CR algorithms implementation

For the implementation of the RL algorithm we decided to use polynomial basis functions up to degree 2. This choice allows us to compute the optimal control analytically as a function of the regression coefficients (see Remark 4.3.9). Compared to other optimization techniques, explicit expression allows for much faster and error-free computations (see Remark 4.3.9). For CR, we used basis functions up to degree 3 in all dimensions to obtain more stable results.

Regarding the choice of the training measure in RL, we employed marginal normal distributions on each dimension. As we know that the inventory dimension Y represents the conditional variance of the original process X, we centered the training distribution µ n at zero but considered only training points Y m n ≥ 0. In CR, on the other hand, we need to carefully choose the distribution of the random control so that the process Y does not become negative. Notice in fact that the Euler approximation, contrary to the original SDE describing Y , does not remain positive and we would therefore need to carefully choose a control to avoid driving Y negative. In order to achieve such goal, without having to worry too much about the control, we modified the Euler approximation of (4.4.7) to feature a reflexive boundary at zero. Such features allow to train the estimated control policy to not overshoot when trying to drive the process Y to zero, without having Y to become negative. Table 4.3 -Results for the systemic risk problem. Estimations of the value function at point (x 0 = 10) at time 0 provided by different strategies. We took T = 1, N = 100, σ = 0.1, κ = 0.5, X 0 = 10.

Details on the Q algorithm implementation

As stated above, it is straightforward that Y > 0 on (0, T ]. However, the Euler scheme used to approximate the dynamics of Y does not prevent the associated process (Y ti ) 0<i≤N to be non-positive. When implementing the Q algorithm for the systemic risk problem, we forced Proj Γ Y i Y ti 0<i≤N to remain positive by simply choosing positive points for the grids Γ Y i that quantize the states of Y ti , at time t i for i = 0, ..., N . Also, given the expression of the instantaneous and terminal reward, one can expect Y to stay close to 0, but we do not have any idea of how small Y should stay for the strategy to be optimal (cf. Figure 4.7 to see a posteriori where Y lies). To deal with this situation, we decided to adopt the "explore first, exploit later" procedure. First, we chose some random grids with a lot of points near 0 and computed the optimal strategy on these grids. Then, we ran forward Monte Carlo simulations and generated an empirical distribution of the quantized Y . Second, we build new grids of Quantization for Y by generating new points according to the empirical distribution that we got from in the previous step. Finally, we computed new (hopefully better) estimations of the optimal strategy by running the Q algorithm using the new grids. The Q strategy performed better after applying this step, but not significantly since our first naive guess for the grids (i.e., before bootstrapping) was already good enough.

Details on the implementation of the deterministic algorithm for the resolution of the HJB

We use the NDSolve function in Mathematica based on finite difference method to solve (4.4.7). Note that usually terminal and boundary conditions are required to get numerical results. The final condition: V (T, x, y) = c

Conclusion

In this work, we have investigated how to use probabilistic numerical methods for some classes of mean field control problem via Markovian embedding. We focused on two types of Regression Monte Carlo methods (namely, Regress-Later and Control Randomization) and Quantization. We have then presented three different examples of applications.

We found that the Regression Monte Carlo algorithms perform well in problems of control of the drift. In such problems, they are much faster than Quantization for similar precision. In particular, we noticed that Regress-Later is usually more reliable than Control Randomization; often the choice of a uniform distribution of the training points on an appropriate interval is sufficient to obtain high- Figure 4.6 -Two realizations of (X t ) t∈[0,T ] controlled by RLMC (curve "RLMC"), (X t ) t∈[0,T ] naively controlled taken α = 0 (curve "uncontrolled"), and X (curve "E(X|W )"). The optimal control for the systemic risk problem (computed by RLMC) is to do nothing at first, and catch X when the terminal time is getting close. Chapter 4. A class of numerically solvable MKV control problems quality estimations. On the other hand Control Randomization is very sensitive to the choice of the distribution of the randomized control, and often a few iterations are necessary before finding a good control distribution. We have also tried to use the performance iteration or path recomputation method, but on the examples we considered, it was very time consuming and did not help much in terms of accuracy. Despite the success of Regression Monte Carlo methods in problems with control on the drift, the example of Portfolio Selection highlighted a possible weakness of these algorithms.

When the control acts on the variance only, we found difficult to make the numerical scheme converge to sensible results within the computational resources available. We realized that the study of these problems and the solution via Regression Monte Carlo methods is outside the scope of this paper. This is probably related to another limitation of this family of methods: the choice of the basis functions for the regression. Indeed, for some problems, a good basis might be very large or might require several steps of trials and errors.

Quantization-based method, on the other hand, provided very stable and accurate results. A first interesting and practical feature of the Q-algorithm is that regressing the value function using quantization-based methods is local. So, first, it can be easily parallelized to provide fast results, and, second, it is easy to check at which points of the grids the estimations suffer from instability and how to change the grid to fix the problem (basically, by adding more points where the estimations need to be improved). Another interesting feature of the quantization methods is that, one can choose the grids on which to learn the value function. It is possible to exploit this feature in the case where one has, a priori, a rough idea of where the controlled process should be driven by the optimal strategy (see, e.g., the liquidation problem). In this case, one should build grids with many points located where the process is supposed to go. In the case where one has no guess of where the optimal process goes, it is always possible to use bootstrapping methods to build better grids iteratively, starting from a random guess for the grid (see, e.g., the systemic risk problem). In both cases, one has to be particularly careful with the borders of the grids that have been built. Indeed, the decisions computed by quantization-based methods at the borders might easily be wrong if the grids do not have a "good shape" at the borders. Unfortunately, the shape of the grid that should be used depends heavily on the problem under consideration. Except in special cases, it seems not possible to avoid the use of deterministic algorithms (such as gradient descent methods or extensive search) to find the optimal action at each point of the grid. A smooth expression of the conditional expectations of the quantized processes is necessary for the deterministic algorithms find optimal strategy efficiently. Once again, the use of parallel computing can alleviate the time-consuming task of searching for the optimal control at each point of the grids.

Chapter 5

Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis1 Abstract This paper develops algorithms for high-dimensional stochastic control problems based on deep learning and dynamic programming. Unlike classical approximate dynamic programming approaches, we first approximate the optimal policy by means of neural networks in the spirit of deep reinforcement learning, and then the value function by Monte Carlo regression. This is achieved in the dynamic programming recursion by performance or hybrid iteration, and regress now or later/quantization methods from numerical probabilities. We provide a theoretical justification of these algorithms. Consistency and rate of convergence for the control and value function estimates are analyzed and expressed in terms of the universal approximation error of the neural networks, and of the statistical error when estimating network function. Numerical results on various applications are presented in a companion paper [START_REF] Bachouch | Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications[END_REF] and illustrate the performance of the proposed algorithms.

keywords Deep learning, dynamic programming, performance iteration, quantization, convergence analysis, statistical risk.

Introduction

A large class of dynamic decision-making problems under uncertainty can be mathematically modeled as discrete-time stochastic optimal control problems in finite horizon. This paper is devoted to the analysis of novel probabilistic numerical algorithms based on neural networks for solving such problems. Let us consider the following discrete-time stochastic control problem over a finite horizon N ∈ N \ {0}. The dynamics of the controlled state process

X α = (X α n ) n valued in X ⊂ R d is given by X α n+1 = F (X α n , α n , ε n+1 ), n = 0, . . . , N -1, X α 0 = x 0 ∈ R d ,
where (ε n ) n is a sequence of i.i.d. random variables valued in some Borel space (E, B(E)), and defined on some probability space (Ω, F, P) equipped with the filtration F = (F n ) n generated by the noise (ε n ) n (F 0 is the trivial σ-algebra), the control α = (α n ) n is an F-adapted process valued in A ⊂ R q , and F is a measurable function from

R d × R q ×E into R d .
Given a running cost function f defined on R d × R q , a terminal cost function g defined on R d , the cost functional associated to a control process α is

J(α) = E N -1 n=0 f (X α n , α n ) + g(X α N ) .
The set C of admissible control is the set of control processes α satisfying some integrability conditions ensuring that the cost functional J(α) is well-defined and finite. The control problem, also called Markov decision process (MDP), is formulated as

V 0 (x 0 ) := inf α∈C J(α),
and the goal is to find an optimal control α * ∈ C, i.e., attaining the optimal value: V 0 (x 0 ) = J(α * ).

Notice that problem (5.1.1)-( 5.1.2) may also be viewed as the time discretization of a continuous time stochastic control problem, in which case, F is typically the Euler scheme for a controlled diffusion process, and V 0 is the discrete-time approximation of a fully nonlinear Hamilton-Jacobi-Bellman equation.

Problem (5.1.2) is tackled by the dynamic programming approach, and we introduce the standard notations for MDP: denote by {P a (x, dx ), a ∈ A, x ∈ X }, the family of transition probabilities associated to the controlled (homogenous) Markov chain (5.1.1), given by P a (x, dx ) = P [F (x, a, ε 1 ) ∈ dx ] 5.1. Introduction and for any measurable function ϕ on X :

P a ϕ(x) = ϕ(x )P a (x, dx ) = E ϕ F (x, a, ε 1 ) .
With these notations, we have for any measurable function ϕ on X , for any α ∈ C,

E[ϕ(X α n+1 )|F n ] = P αn ϕ(X α n ), ∀ n ∈ N.
The optimal value V 0 (x 0 ) is then determined in backward induction starting from the terminal condition

V N (x) = g(x), x ∈ X ,
and by the dynamic programming (DP) formula, for n = N -1, . . . , 0:

Q n (x, a) = f (x, a) + P a V n+1 (x), x ∈ X , a ∈ A, V n (x) = inf a∈A Q n (x, a),
The function Q n is called optimal state-action value function, and V n is the (optimal) value function. Moreover, when the infimum is attained in the DP formula at any time n by a * n (x), we get an optimal control in feedback form given by: α * = (a * n (X * n )) n where X * = X α * is the Markov process defined by

X * n+1 = F (X * n , a * n (X * n ), ε n+1 ), n = 0, . . . , N -1, X * 0 = x 0 .
The DP has a probabilistic formulation: it says that for any control α ∈ A, the value function process augmented with the cumulative costs defined by

S α n := V n (X α n ) + n-1 k=0 f (X α k , α k ), n = 1, . . . , N
is a submartingale, and a martingale for the optimal control α * . This martingale property for the optimal control is a key observation for our algorithms described later.

Remark 5.1.1. We can deal with state/control constraints at any time, which is useful for the applications:

(X α n , α n ) ∈ S a.s., n ∈ N, where S is some given subset of R d × R q . In this case, in order to ensure that the set of admissible controls is not empty, we assume that the sets A(x) := a ∈ R q : (F (x, a, ε 1 ), a) ∈ S a.s.

are non empty for all x ∈ X , and the DP formula reads now as

V n (x) = inf a∈A(x) f (x, a) + P a V n+1 (x) , x ∈ X .
in the finite horizon case, and

V (x) = inf a∈A(x) f (x, a) + P a V (x) , x ∈ X ,
From a computational point of view, it may be more convenient to work with unconstrained state and control variable, hence by relaxing the state and control constraint and introducing into the running cost a penalty function L(x, a): f (x, a) ← f (x, a) + L(x, a), and g(x) ← g(x) + L(x, a). For example, if the constraint set S is in the form:

S = {(x, a) ∈ R d × R q : h k (x, a) = 0, k = 1, . . . , p, h k (x, a) ≥ 0, k = p + 1, .
. . , q}, for some functions h k , then one can take as penalty functions:

L(x, a) = p k=1 µ k |h k (x, a)| 2 + q k=p+1 µ k max(0, -h k (x, a)).
where µ k > 0 are penalization coefficients (large in practice).

Chapter 5. Neural networks algorithms for control problems: convergence analysis

The implementation of the DP formula requires the knowledge and explicit computation of the transition probabilities P a (x, dx ). In situations when they are unknown, this leads to the problematic of reinforcement learning for computing the optimal control and value function by relying on simulations of the environment. The challenging tasks from a numerical point of view are then twofold:

1. Transition probability operator. Calculations for any x ∈ X , and any a ∈ A, of P a V n+1 (x), for n = 0, . . . , N -1. This is a computational challenge in high dimension d for the state space with the "curse of dimensionality" due to the explosion of grid points in deterministic methods.

2. Optimal control. Computation of the infimum in a ∈ A of f (x, a) + P a V n+1 (x) for fixed x and n, and of ân (x) attaining the minimum if it exists. This is also a computational challenge especially in high dimension q for the control space.

The classical probabilistic numerical methods based on DP for solving the MDP are sometimes called approximate dynamic programming methods, see e.g. [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF], [START_REF] Powell | Approximate dynamic programming: solving the curses of dimensionality[END_REF], and consist basically of the two following steps:

(i) Approximate at each time step n the Q n value function defined as a conditional expectation.

This can be performed by regression Monte-Carlo (RMC) techniques or quantization. RMC is typically done by least-square linear regression on a set of basis function following the popular approach by Longstaff and Schwarz [LS01] initiated for Bermudean option problem, where the suitable choice of basis functions might be delicate. Conditional expectation can be also approximated by regression on neural network as in [START_REF] Kohler | Pricing of high-dimensional American options by neural networks[END_REF] for American option problem, and appears as a promising and efficient alternative in high dimension to the linear regression. The main issue in the controlled case concerns the simulation of the endogenous controlled MDP, and this can be overcome by control randomization as in [START_REF] Kharroubi | A numerical algorithm for fully nonlinear HJB equations: an approach by control randomization[END_REF]. Alternatively, quantization method consists in approximating the noise (ε n ) by a discrete random variable on a finite grid, in order to reduce the conditional expectation to a finite sum.

(ii) Control search: Once we get an approximation (x, a) → Qn (x, a) of the Q n value function, the optimal control ân (x) which achieves the minimum over a ∈ A of Q n (x, a) can be obtained either by an exhaustive search when A is discrete (with relatively small cardinality), or by a (deterministic) gradient-based algorithm for continuous control space (with relatively small dimension).

Recently, numerical methods by direct approximation, without DP, have been developed and made implementable thanks to the power of computers: the basic idea is to focus directly on the control approximation by considering feedback control (policy) in a parametric form:

a n (x) = A(x; θ n ), n = 0, . . . , N -1,
for some given function A(., θ n ) with parameters θ = (θ 0 , . . . , θ N -1 ) ∈ R q×N , and minimize over θ the parametric functional

J(θ) = E N -1 n=0 f (X A n , A(x; θ n )) + g(X A N ) ,
where (X A n ) n denotes the controlled process with feedback control (A(., θ n )) n . This approach was first adopted in [START_REF] Kou | EM algorithm and stochastic control[END_REF], who used EM algorithm for optimizing over the parameter θ, and further investigated in [START_REF] Han | Deep learning approximation for stochastic control problems[END_REF], [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF], [START_REF] Henry-Labordère | Deep primal-dual algorithm for BSDEs: applications of machine learning to CVA and IM[END_REF], who considered deep neural networks (DNN) for the parametric feedback control, and stochastic gradient descent methods (SGD) for computing the optimal parameter θ. The theoretical foundation of these DNN algorithms has been recently investigated in [START_REF] Han | Convergence of the deep BSDE method for coupled FBSDEs[END_REF]. Deep learning has emerged recently in machine learning as a successful technique for dealing with high-dimensional problems in speech recognition, computer vision, etc (see e.g. [START_REF] Lecun | Deep learning[END_REF], [START_REF] Goodfellow | Deep learning[END_REF]). Let us mention that DNN approximation in stochastic control has already been explored in 5.2. Preliminaries on DNN and SGD the context of reinforcement learning (RL) (see [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF] and [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]), and called deep reinforcement learning in the artificial intelligence community [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] (see also [START_REF] Li | Deep reinforcement learning: an overview[END_REF] for a recent survey) but usually for infinite horizon (stationary) control problems.

In this paper, we combine different ideas from the mathematics (numerical probability) and the computer science (reinforcement learning) communities to propose and compare several algorithms based on dynamic programming (DP), and deep neural networks (DNN) for the approximation/learning of (i) the optimal policy, and then of (ii) the value function. Notice that this differs from the classical approach in DP recalled above, where we first approximate the Q-optimal state/control value function, and then approximate the optimal control. Our learning of the optimal policy is achieved in the spirit of [START_REF] Han | Deep learning approximation for stochastic control problems[END_REF] by DNN, but sequentially in time though DP instead of a global learning over the whole period 0, . . . , N -1. Once we get an approximation of the optimal policy, and recalling the martingale property (5.1.3), we approximate the value function by Monte-Carlo (MC) regression based on simulations of the forward process with the approximated optimal control. In particular, we avoid the issue of a priori endogenous simulation of the controlled process in the classical Q-approach. The MC regressions for the approximation of the optimal policy and/or value function, are performed according to different features leading to algorithmic variants: Performance iteration (PI) or hybrid iteration (HI), and regress now or regress later/quantization in the spirit of [START_REF] Longstaff | Valuing American options by simulation: a simple least-squares approach[END_REF] or [START_REF] Glasserman | Simulation for American options: Regression Now or Regression Later?[END_REF]. Numerical results on several applications are devoted to a companion paper [START_REF] Bachouch | Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications[END_REF]. The theoretical contribution of the current paper is to provide a detailed convergence analysis of our three proposed algorithms: Theorem 5.4.1 for the NNContPI Algo based on control learning by performance iteration with DNN, Theorem 5.4.2 for the Hybrid-Now Algo based on control learning by DNN and then value function learning by regress-now method, and Theorem 5.4.3 for the Hybrid-LaterQ Algo based on control learning by DNN and then value function learning by regress later method combined with quantization. We rely mainly on arguments from statistical learning and non parametric regression as developed notably in the book [START_REF] Gyorfi | A Distribution-Free Theory of Nonparametric Regression[END_REF], for giving estimates of approximated control and value function in terms of the universal approximation error of the neural networks, and of the statistical error in the estimation of network functions.

The plan of this paper is organized as follows. We recall in Section 5.2 some basic results about deep neural networks (DNN) and stochastic optimization gradient descent methods used in DNN. Section 5.3 is devoted to the description of our three algorithms. We analyze in detail in Section 5.4 the convergence of the three algorithms. Finally the Appendix collect some Lemmas used in the proof of the convergence results.

Preliminaries on DNN and SGD

Neural network approximations

Deep Neural networks (DNN) aim to approximate (complex non linear) functions defined on finitedimensional space, and in contrast with the usual additive approximation theory built via basis functions, like polynomial, they rely on composition of layers of simple functions. The relevance of neural networks comes from the universal approximation theorem and the Kolmogorov-Arnold representation theorem (see [START_REF] Kolmogorov | On the representation of continuous functions of several variables by superpositions of continuous functions of a smaller number of variables[END_REF], [START_REF] Cybenko | Approximations by superpositions of sigmoidal functions[END_REF] or [START_REF] Hornick | Approximation capabilities of multilayer feedforward networks[END_REF]), and this has shown to be successful in numerous practical applications.

We consider here feedforward artificial network (also called multilayer perceptron) for the approximation of the optimal policy (valued in A ⊂ R q ) and the value function (valued in R), both defined on the state space X ⊂ R d . The architecture is depicted in Figure 5.1, and it is mathematically represented by functions

x ∈ X -→ Φ(z; θ) ∈ R o ,
with o = q or 1 in our context, and where θ ∈ Θ ⊂ R p are the weights (or parameters) of the neural networks. The DNN function Φ = Φ L with input layer Φ 0 = (Φ i 0 ) i = x ∈ X composed of d units (or neurons), L -1 hidden layers (with layer composed of d units), and output layer composed of Chapter 5. Neural networks algorithms for control problems: convergence analysis d L = o neurons is obtained by successive composition of linear combination and activation function σ (that is a nonlinear monotone function like e.g. the sigmoid, the rectified linear unit ReLU, the exponential linear unit ELU, or the softmax):

Φ = σ (w Φ -1 + γ ) ∈ R d , = 1, . . . , L,
for some matrix weights (w ) and vector weight (γ ), aggregating into θ = (w , γ ) L =1 . A key feature of neural networks is the computation of the gradient (with respect to the variable x and the weights θ) of the DNN function via a forward-backward propagation algorithm derived from chain rule composition. For example, for the sigmoid activation function σ (y) = 1/(1 + e -y ), and noting that σ = σ (1 -σ ), we have

∂Φ ∂z ij = w ∂Φ -1 ∂z ij Φ i (1 -Φ i ), = 1, . . . , L, i = 1, . . . , d , j = 1, . . . , d while the gradient w.r.t. θ of K(θ) = K(Φ L (.; θ)), for a real-valued differentiable function y ∈ R d L → K(y)
, is given in backward induction by

∆ i := ∂K ∂Φ i Φ i (1 -Φ i ), = L, . . . , 1, i = 1, . . . , d ∂K ∂w ij = Φ j -1 ∆ i , ∂K ∂γ i = ∆ i , ∂K ∂Φ -1 j = d k=1 ∆ k w kj , j = 1, . . . , d -1 .
We refer to the online book [Nie] for a gentle introduction to neural networks and deep learning.

Stochastic optimization in DNN

Approximation by means of DNN requires a stochastic optimization with respect to a set of parameters, which can be written in a generic form as

inf θ E L n (Z n ; θ) ,
where Z n is a random variable from which the training samples Z

n , m = 1, . . . , M are drawn, and L n is a loss function involving DNN with parameters θ ∈ R p , and typically differentiable w.r.t. θ with known gradient D θ L n .

Several basic algorithms are already implemented in TensorFlow for the search of infimum in (5.2.1). Given a training sample of size M , in all the following cases, the sequence (θ k n ) k∈N tends to θ n = argmin θ E L n (Z n ; θ) under suitable assumptions on the learning rate sequence (γ k ) ∞ k=0 .

Description of the algorithms

• Batch gradient descent: (compute the gradient over the full training set). Fix an integer K, and do

θ k+1 n = θ k n -γ k 1 M M m=1 D θ L n (Z (m) n ; θ k n ), for k = 1, . . . , K.
The main problem with the Batch Gradient Descent is that the convergence is very slow and also the computation of the sum can be painful for very large training sets. Hence it makes it very stable, but too slow in most situations.

• Stochastic gradient descent (SGD): (compute the gradient over one random instance in the training set)

θ m+1 n = θ m n -γ m D θ L n (Z (m) n ; θ m n ), m = 1, . . . , M -1.
starting from θ 0 n ∈ R p , with a learning rate γ m . The Stochastic gradient algorithm computes the gradient based on a single random instance in the training set. It is then a fast but unstable algorithm.

• Mini-batch gradient descent: (compute the gradient over random small subsets of the training set, i.e. mini-batches) let M b be an integer than divides M . M b stands for the number of mini-batches and should be taken much smaller than M in the applications. For all k, . . . , M b,

-Randomly draw a subset Z (k,m) n M k+1 m=1 of size M k+1 := M M b in the training set.
iterate:

θ k+1 n = θ k n -γ k 1 M k+1 M k+1 m=1 D θ L n (Z (m) n ; θ k n ).
The mini-batch gradient descent is often considered to be the best trade-off between speed and stability.

The three gradient descents that we just introduced are the first three historical algorithms that has been designed to learn optimal parameters. Other methods such as the Adaptive optimization methods AdaGrad, RMSProp, and finally Adam are also available. Although not well-understood and even questioned (see e.g. [START_REF] Wilson | The marginal value of adaptive gradient methods in machine learning[END_REF]), the latter are often chosen by the practitioners to solve (5.2.1) and appear to provide the best results in most of the situations. For sake of simplicity, we only refer in the sequel to the stochastic gradient descent method, when presenting our algorithms. However, we recommend to test and use different algorithms in order to know which are the ones that provide best and fastest results for a given problem.

Description of the algorithms

We propose algorithms relying on a DNN approximation of the optimal policy that we compute sequentially in time through the dynamic programming formula, and using performance or hybrid iteration. The value function is then computed by Monte-Carlo regression either by a regress now method or a regress later joint with quantization approach. These variants lead to three algorithms for MDP that we detail in this section.

Let us introduce a set A of neural networks for approximating optimal policies, that is a set of parametric functions x ∈ X → A(x; β) ∈ A, with parameters β ∈ R l , and a set V of neural networks functions for approximating value functions, that is a set of parametric functions x ∈ X → Φ(x; θ) ∈ R, with parameters θ ∈ R p .

We are also given at each time n a probability measure µ n on the state space X , which we refer to as a training distribution. Some comments about the choice of the training measure are discussed in Section 5.3.3.

Chapter 5. Neural networks algorithms for control problems: convergence analysis

Hybrid-LaterQ Algo

Given an estimate âM n of the optimal policy at time n, and an estimate V M n+1 of Vn+1 , the regress-later approach for estimating Vn is achieved in two stages: (a) we first regress/interpolate the estimated value V M n+1 X âM n n+1 at time n + 1 by a NN (or alternatively a Gaussian process) Φ(X

âM n n+1
), (b) Analytical formulae are applied to the conditional expectation of this NN of future values X âM n n+1 with respect to the present value X n , and this is obtained by quantization of the noise (ε n ) driving the dynamics (5.1.1) of the state process.

The ingredients of the quantization approximation are described as follows:

• We denote by ε a K-quantizer of the E-valued random variable ε n+1 ε 1 (typically a Gaussian random variable), that is a discrete random variable on a grid Γ = {e 1 , . . . , e K } ⊂ E K defined by

ε = Proj Γ (ε 1 ) := K =1 e 1 ε1∈C (Γ) ,
where C 1 (Γ), . . ., C K (Γ) are Voronoi tesselations of Γ, i.e., Borel partitions of the Euclidian space (E, |.|) satisfying

C (Γ) ⊂ e ∈ E : |e -e | = min j=1,...,K |e -e j | .
The discrete law of ε is then characterized by

p := P[ε = e ] = P[ε 1 ∈ C (Γ)], = 1, . . . , K.
The grid points (e ) which minimize the L 2 -quantization error ε 1 -ε 2 lead to the so-called optimal L-quantizer, and can be obtained by a stochastic gradient descent method, known as Kohonen algorithm or competitive learning vector quantization (CLVQ) algorithm, which also provides as a byproduct an estimation of the associated weights (p ). We refer to [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF] for a description of the algorithm, and mention that for the normal distribution, the optimal grids and the weights of the Voronoi tesselations are precomputed on the website http://www.quantize.mathsfi.com

• Recalling the dynamics (5.1.1), the conditional expectation operator is equal to

P âM n (x) W (x) = E W (X âM n n+1 )|X n = x = E W (F (x, âM n (x), ε 1 )) , x ∈ X ,
that we shall approximate analytically by quantization via:

P âM n (x) W (x) := E W (F (x, âM n (x), ε)) = K =1 p W F (x, âM n (x), e ) .
The two stages of the regress-later are then detailed as follows:

(a) (Later) interpolation of the value function: Given a DNN Φ (.; θ) on R d with parameters θ ∈ R p , we interpolate V M n+1 by

V M n+1 (x) := Φ x; θ M n+1 ,
where θ M n+1 is obtained via SGD (as described in paragraph 5.2.2) from the regression of

V M n+1 (X âM n n+1 ) against Φ X âM n n+1 ; θ , using training samples X (m) n , X âM n ,(m) n+1 , m = 1, . . . , M of X n µ n , and X âM n ,(m) n+1 of X âM n n+1 .

Description of the algorithms

(b) Updating/approximation of the value function: by using the hat operator in (5.3.6) for the approximation of the conditional expectation by quantization, we calculate analytically

V M n (x) := f (x, a) + P âM n V M n+1 (x) = f (x, a) + K =1 p Φ F (x, âM n (x), e ); θ M n+1 .
Remark 5.3.3. Let us discuss and compare the Algos Hybrid-Now and Hybrid-LaterQ. When regressing later, one just has to learn a deterministic function through the interpolation step (a), as the noise is then approximated by quantization for getting analytical formula. Therefore, compared to Hybrid-Now, the Hybrid-LaterQ Algo reduces the variance of the estimate V M n . Moreover, one has a wide choice of loss functions when regressing later, e.g., MSE loss function, L1-loss, relative error loss, etc, while the L2-loss function is required to approximate of condition expectation using regress-now method. However, although quantization is quite easy and fast to implement in small dimension for the noise, it might be not efficient in high-dimension compared to Hybrid-Now. k , m = 1, . . . , M , used at times k = n, . . . , N , for computing the estimated optimal policies âM k , and this is emphasized through the exponent M in the notations.

Training sets design

We discuss here the choice of the training measure µ n used to generate the training sets on which will be computed the estimations. Two cases are considered in this section. The first one is a knowledgebased selection, relevant when the controller knows with a certain degree of confidence where the process has to be driven in order to optimize her cost functional. The second case, on the other hand, is when the controller has no idea where or how to drive the process to optimize the cost functional.

Exploitation only strategy

In the knowledge-based setting, there is no need for exhaustive and expensive (in time mainly) exploration of the state space, and the controller can directly choose training sets Γ n constructed from distributions µ n that assign more points to the parts of the state space where the optimal process is likely to be driven.

In practice, at time n, assuming we know that the optimal process is likely to stay in the ball centered around the point m n and with radius r n , we choose a training measure µ n centered around m n as, for example N (m n , r 2 n ), and build the training set as sample of the latter.

Explore first, exploit later

• Explore first: If the agent has no idea of where to drive the process to receive large rewards, she can always proceed to an exploration step to discover favorable subsets of the state space. To do so, Γ n , the training sets at time n, for n = 0, . . . , N -1, can be built as uniform grids that cover a large part of the state space, or µ can be chosen uniform on such domain. It is essential to explore far enough to have a well understanding of where to drive and where not to drive the process.

• Exploit later: For n = 0, . . . , N -1, the estimates for the optimal controls at time t n that come up from the Explore first step are relatively good in the way that they manage to avoid the wrong areas of state space when driving the process. However, the training sets that have been used to compute the estimated optimal control are too sparse to ensure accuracy on the estimation. In order to improve the accuracy, the natural idea is to build new training sets by simulating M times the process using the estimates on the optimal strategy computed from the Explore first step, and then proceed to another estimation of the optimal strategies using the new training sets. This trick can be seen as a two steps algorithm that improves the estimate of the optimal control.

Convergence analysis

Under this boundedness condition, it is clear that the value function V n is also bounded:

V n ∞ ≤ (N -n) f ∞ + g ∞ , ∀n ∈ {0, ..., N }.
We shall finally assume a Lipschitz condition on the dynamics of the MDP.

(HF) For any e ∈ E, there exists C(e) such that for all couples (x, a) and (x , a ) in X × A:

|F (x, a, e) -F (x , a , e)| ≤ C(e) (|x -x | + |a -a |) .
In the sequel, we define for any M ∈ N * :

ρ M = E sup 1≤m≤M C(ε m ) ,
where the (ε m ) m is a i.i.d. sample of the noise ε. The rate of convergence of ρ M toward infinity will play a crucial role to show the convergence of the algorithms.

Remark 5.4.2. A typical example when (HF) holds is the case where F is defined through the time discretization of an Euler scheme, i.e., as

F (x, a, ε) := b(x, a) + σ(x,
ρ M ≤ [b] L + d[σ] L 2 log(2dM ), (5.4.1)
which implies in particular that

ρ M = M →+∞ O log(M ) .
Let us indeed check the inequality (5.4.1). For this, let us fix some integer M > 0 and let Z := sup 1≤m≤M | m 1 | where m 1 are i.i.d. such that 1 1 ∼ N (0, 1). From Jentzen inequality to the r.v. Z and the convex function z → exp(tz), where t > 0 will be fixed later, we get

exp (tE [Z]) ≤ E exp tZ ≤ E sup 1≤m≤M exp t| m 1 | ≤ M m=1 E exp t| m 1 | ≤ 2M exp t 2 2 ,
where we used the closed-form expression of the moment generating function of the folded normal distribution 2 to write the last inequality. Hence, we have for all t > 0:

E Z ≤ log(2M ) t + t 2 .
We get, after taking t = 2 log(2M ):

E Z ≤ 2 log(2M ). (5.4.2) Since inequality x d ≤ d x ∞ holds for all x ∈ R d , we derive E sup 1≤m≤M C(ε m ) ≤ [b] L + d[σ] L E sup 1≤m≤dM C( m 1 ) ,
and apply (5.4.2) with M = dM , to complete the proof of (5.4.1).

2 The folded normal distribution is defined as the distribution of |Z| where Z ∼ N 1 (µ, σ). Its moment generating function is given by t → exp

σ 2 t 2 2 + µt 1 -Φ -µ σ -σt + exp σ 2 t 2 2 -µt 1 -Φ µ σ -σt
, where Φ is the c.d.f. of N 1 (0, 1).

Convergence analysis

Recall that the approximation of the optimal policy in the NNcontPI algorithm is computed in backward induction as follows: For n = N -1, . . . , 0, generate a training sample for the state X • Compute the approximated policy at time n

âM n ∈ argmin A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 (5.4.4)
where

Ŷ (m),A n+1 = N -1 k=n+1 f X (m),A k , âM k X (m),A k + g X (m),A N , (5.4.5)
with X (m),A k N k=n+1 defined by induction as follows, for m=1, . . . , M :

   X (m),A n+1 = F X m n , A X m n , ε m n+1 X (m),A k = F X (m),A k-1 , A X (m),A k-1 , ε m k , for k = n + 2, . . . , N.
• Compute the estimated value function V M n as in (5.3.2). Remark 5.4.5. In order to simplify the theoretical analysis, we assume that the argmin in (5.4.4) is exactly reached by running batch, mini-batch or stochastic gradient descent, which are the methods that we used to code the algorithm in our companion paper. Remark 5.4.6. The minimization problem in (5.4.4) is actually a problem of minimization over the parameter β (of the neural network A) of the expectation of a function of β and some noises

X (m) n M m=1 , ε m k M,N m=1,k=n+1
, where F is iterated many times. Stochastic-gradient-based methods are chosen for such a task, although the gradient becomes more and more difficult to compute when we are going backward in time, since there are more and more iterations of F involved in the derivatives of the gradients.

The integrand is differentiable if assumption (HF) holds, but it is always possible to apply the stochastic-gradient-based algoritm for certain classes of non-differentiable functions F (see e.g. the gradient-descent implementation in TensorFlow which works with the non-differentiable at 0 ReLu activation functions.).

We now state our main result about the convergence of the NNcontPI algorithm.

Theorem 5.4.1. Assume that there exists an optimal feedback control (a opt k ) N -1 k=n for the control problem with value function V n , n = 0, . . . , N , and let

X n µ. Then, as M → ∞ 4 E V M n (X n ) -V n (X n ) = O ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| , (5.4.6)
where E stands for the expectation over the training set used to evaluate the approximated optimal policies (â M k ) n≤k≤N -1 , as well as the path (X n ) n≤k≤N controlled by the latter. Moreover, as M → ∞5

E M V M n (X n ) -V n (X n ) = O P ρ N -n-1 M γ N -n-1 M η N -n-2 M log(M ) M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| ,
where E M stands for the expectation conditioned by the training set used to estimate the optimal policies (â M k ) n≤k≤N -1 .

Remark 5.4.7. 1. The term

ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M
should be seen as the estimation error. It is due to the approximation of the optimal controls by means of neural networks in A M using empirical cost functional in (5.4.4). We show in section 5.B that this term disappears in the ideal case where the real cost functional (i.e. not the empirical one) is minimized.

2. The rate of convergence depends dramatically on N since it becomes exponentially slower when N goes to infinity. This is a huge drawback for this performance iteration-based algorithm. We will see in the next section that the rate of convergence of value iteration-based algorithms do not suffer from this dramatical dependence on N .

Comment: Since we clearly have V n ≤ V M n , estimation (5.4.6) implies the convergence in L 1 norm of the NNcontPI algorithm, under condition (5.4.3), and in the case where

sup n≤k≤N inf A∈A M E |A(X k ) -a opt k (X k )| -----→ M →+∞ 0
. This is actually the case under some regularity assumptions on the optimal controls, as stated in the following proposition.

Proposition 5.4.1. The two following assertions hold:

1. Assume that a opt k (X k ) ∈ L 1 (µ) for k = n, ..., N -1. Then sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| -----→ M →+∞ 0. (5.4.7)
2. Assume that the function

a opt k is c-Lipschitz for k = n, ..., N -1. Then sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| < c γ M c -2d/(d+1) log γ M c + γ M K -(d+3)/(2d) M . (5.4.8)
Proof. The first statement of Proposition 5.4.1 relies essentially on the universal approximation theorem, and the second assertion is stated and proved in [START_REF] Bach | Breaking the curse of dimensionality with convex neural networks[END_REF]. For sake of completeness, we recall the details in Section 5.E in the Appendix. Remark 5.4.8. Note that the second statement in the above proposition is stronger than the first one since it provides a rate of convergence of the approximation error. Fixing K M and minimizing the r.h.s. of (5.4.8) over γ M , results in

sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| < cK -1 d M 1 + d + 1 2d log K M , when we take γ M = cK d+1 2d
M . Hence, for such a value of γ M , the l.h.s. decreases to 0 with rate proportional to

log(K M ) / d √ K M .
The rest of this section is devoted to the proof of Theorem 5.4.1. Let us introduce some useful notations. Denote by A X the set of Borelian functions from the state space X into the control space A.

For n = 0, . . . , N -1, and given a feedback control (policy) represented by a sequence (A k ) k=n,...,N -1 , with A k in A X , we denote by J

(A k ) N -1 k=n n
the cost functional associated to the policy (A k ) k . Notice that with this notation, we have

V M n = J (â M k ) N -1 k=n n
. We define the estimation error at time n associated to the NNContPI algorithm by

ε esti PI,n := sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -E M J A,(â M k ) N -1 k=n+1 n (X n ) ,

Convergence analysis

with X n µ: It measures how well the chosen estimator (e.g. mean square estimate) can approximate a certain quantity (e.g. the conditional expectation). Of course we expect the latter to cancel when the size of the training set used to build the estimator goes to infinity. Actually, we have Lemma 5.4.1. For n = 0, . . . , N -1, we have

E[ε esti PI,n ] ≤ √ 2 + 16 (N -n) f ∞ + g ∞ √ M
(5.4.9)

+ 16γ M √ M [f ] L 1+ρ M 1-ρ N -n-1 M 1+η M γ M N -n-1 1-ρ M 1+η M γ M + 1+η M γ M N -n-1 ρ N -n M [g] L = O ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M , as M → ∞.
This implies in particular that

ε esti PI,n = O P ρ N -n-1 M γ N -n-1 M η N -n-2 M log(M ) M , as M → ∞, (5.4.10)
where we remind that

ρ M = E sup 1≤m≤M C(ε m ) is defined in (HF).
Proof. The relation (5.4.9) states that the estimation error cancels when M → ∞ with a rate of convergence of order O

ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M
. The proof is in the spirit of the one that can be found in chapter 9 of [START_REF] Gyorfi | A Distribution-Free Theory of Nonparametric Regression[END_REF]. It relies on a technique of symmetrization by a ghost sample, and a wise introduction of additional randomness by random signs. The details are postponed in Section 5.C in the Appendix. The proof of (5.4.10) follows from (5.4.9) by a direct application of Markov inequality.

Let us also define the approximation error at time n associated to the NNContPI algorithm by

ε approx PI,n := inf A∈A M E M J A,(â M k ) N -1 k=n+1 n (X n ) -inf A∈A X E M J A,(â M k ) N -1 k=n+1 n (X n ) ,
where we recall that E M denotes the expectation conditioned by the training set used to compute the estimates (â M k ) N -1 k=n+1 and the one of X n µ. ε approx PI,n measures how well the regression function can be approximated by means of neural networks functions in A M (notice that the class of neural networks is not dense in the set A X of all Borelian functions). Lemma 5.4.2. For n = 0, . . . , N -1, it holds as M → ∞,

E[ε approx PI,n ] = O ρ N -n-1 M γ N -n-1 M η N -n-2 M √ M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| . (5.4.11)
This implies in particular

ε approx PI,n = O P ρ N -n-1 M γ N -n-1 M η N -n-2 M log(M ) M + sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| .
(5.4.12)

Proof. See Section 5.D in Appendix for the proof of (5.4.11). The proof of (5.4.12) then follows by a direct application of Markov inequality. Step 1. Let us denote by

ĴA,(â M k ) N -1 k=n+1 n,M := 1 M M m=1 f X (m) n , A(X (m) n ) + Ŷ (m),A n+1
, the empirical cost function, from time n to N , associated to the sequence of controls (A, (â M k ) N -1 k=n+1 ) and the training set, where we recall that Ŷ (m),A n+1 is defined in (5.4.5). We then have

E M V M n (X n ) = E M J (â M k ) N -1 k=n n (X n ) - Ĵ(â M k ) N -1 k=n n,M + Ĵ(â M k ) N -1 k=n n,M ≤ ε esti PI,n + Ĵ(â M k ) N -1 k=n n,M , (5.4.13)
by definition of V M n and ε esti PI,n . Moreover, for any

A ∈ A M , ĴA,(â M k ) N -1 k=n+1 n,M = ĴA,(â M k ) N -1 k=n+1 n,M -E M J A,(â M k ) N -1 k=n+1 n (X n ) + E M J A,(â M k ) N -1 k=n+1 n (X n ) ≤ ε esti PI,n + E M J A,(â M k ) N -1 k=n+1 n (X n ) . (5.4.14)
Recalling that

âM n = argmin A∈A M ĴA,(â M k ) N -1 k=n+1 n,M
, and taking the infimum over A M in the l.h.s. of (5.4.14) first, and in the r.h.s. secondly, we then get

Ĵ(â M k ) N -1 k=n n,M ≤ ε esti PI,n + inf A∈A M E M J A,(â M k ) N -1 k=n+1 n (X n ) .
Plugging this last inequality into (5.4.13) yields the following estimate

E M V M n (X n ) -inf A∈A M E M J A,(â M k ) N -1 k=n+1 n (X n ) ≤ 2ε esti PI,n . (5.4.15)
Step 2. By definition (5.4.1) of the approximation error, using the law of iterated conditional expectations for J n , and the dynamic programming principle for V n with the optimal control a opt n at time n, we have

inf A∈A M E M J A,(â M k ) N -1 k=n+1 n (X n ) -E M [V n (X n )] = ε approx PI,n + inf A∈A X E M f (X n , A(X n )) + E A n J (â M k ) N -1 k=n+1 n+1 (X n+1 ) -E M f (X n , a opt n (X n )) + E a opt n n V n+1 (X n+1 ) ≤ ε approx PI,n + E M E a opt n n J (â M k ) N -1 k=n+1 n+1 (X n+1 ) -V n+1 (X n+1 ) ,
where E A n [.] stands for the expectation conditioned by X n at time n and the training set, when strategy A is followed at time n. Under the bounded density assumption in (Hd), we then get

inf A∈A M E M J A,(â M k ) N -1 k=n+1 n (X n ) -E M [V n (X n )] ≤ ε approx PI,n + r ∞ J (â M k ) N -1 k=n+1 n+1 (x ) -V n+1 (x ) µ(dx ) ≤ ε approx PI,n + r ∞ E M V M n+1 (X n+1 ) -V n+1 (X n+1 ) , with X n+1 ∼ µ. (5.4.16)
where

X (m),A n+1 = F (X (m) n , A(X (m) n ), ε (m) n+1 ) P A(X (m) n ) (X (m)
n , dx ). (ii) compute the untruncated estimation of the value function at time n

Ṽ M n ∈ argmin Φ∈V M 1 M M m=1 f (X (m) n , âM n (X (m) n )) + V M n+1 (X (m),â M n n+1 ) -Φ(X (m) n ) 2
and set the truncated estimated value function at time n

V M n = max min V M n , V n ∞ , -V n ∞ .
Remark 5.4.9. Notice that we have truncated the estimated value function in (5.4.19) by an a priori bound on the true value function. This truncation step is natural from a practical implementation point of view, and is also used for simplifying the proof of the convergence of the algorithm. The conditions in (5.4.18) for the parameters are weaker than those required in (5.4.3) for the NNcontPI algo by performance iteration, which implies a much faster convergence w.r.t. the size of the training set. However, one should keep in mind that unlike the performance iteration procedure, the value iteration one is biased since the computation of V M n+1 X A n+1 are biased future rewards when decision A is taken at time n and estimated optimal strategies are taken at time k ≥ n + 1.

We now state our main result about the convergence of the Hybrid-Now algorithm.

Theorem 5.4.2. Assume that there exists an optimal feedback control (a opt k ) N -1 k=n for the control problem with value function V n , n = 0, . . . , N , and let X n µ. Then, as M → +∞

E M | V M n (X n ) -V n (X n )| = O P γ 4 M K M log(M ) M 1 2(N -n) + γ 4 M ρ 2 M η 2 M log(M ) M 1 4(N -n) + sup n≤k≤N inf Φ∈V M E M |Φ(X k ) -V k (X k )| 2 1 2(N -n) + sup n≤k≤N inf A∈A M E |A(X k ) -a opt k (X k )| 1 2(N -n)
, where E M stands for the expectation conditioned by the training set used to estimate the optimal policies (â M k ) n≤k≤N -1 .

Comment: Theorem 5.4.2 states that the estimator for the value function provided by hybridnow algorithm converges in L 1 (µ) when the size of the training set goes to infinity. Note that the term γ 4

M K M log(M ) M 1 2(N -n)
stands for the estimation error made by estimating empirically the value functions using neural networks, and

γ 4 M ρ 2 M η 2 M log(M ) M 1 4(N -n)
stands for the estimation error made by estimating empirically the optimal control using neural networks. The following quantity

sup n≤k≤N inf Φ∈V M E |Φ(X k ) -V k (X k )| 2
stands for the approximation error made by estimating the value function as a neural network function in V M , and the term sup

n≤k≤N inf A∈A M E |A(X k ) -a opt k (X k )|
is the one made by estimating the optimal control as a neural network function in A M .

In order to prove Theorem 5.4.2, let us first introduce the estimation error at time n associated

Convergence analysis

to the Hybrid-Now algorithm by

ε esti HN,n := sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -E A M,n,Xn f (X n , A(X n )) + V M n+1 X n+1 ,
where

Ŷ (m),A n+1 = V M n+1 X (m),A n+1
, and

X (m),A n+1 = F X (m) n , A(X (m) n ), ε m n+1 .
We have the following bound on this estimation error:

Lemma 5.4.3. For n = 0, ..., N -1, it holds:

E ε esti HN,n ≤ √ 2 + 16 (N -n) f ∞ + g ∞ + 16[f ] L √ M + 16 ρ M η M γ 2 M √ M = M →∞ O ρ M η M γ 2 M √ M .
(5.4.20)

Proof. See Section 5.F in Appendix.

Remark 5.4.10. The result stated by lemma 5.4.3 is sharper than the one stated in Lemma 5.4.1 for the performance iteration procedure. The main reason is that we can make use of the γ M η M -Lipschitz-continuity of the estimation of the value function at time n + 1.

We secondly introduce the approximation error at time n associated to the Hybrid Now algorithm by

ε approx HN,n := inf A∈A M E M f X n , A(X n ) + Ŷ A n+1 -inf A∈A X E M f X n , A(X n ) + Ŷ A n+1 ,
where

Ŷ A n+1 := V M n+1 (F (X n , A(X n ), ε n+1 )).
We have the following bound on this approximation error: Lemma 5.4.4. For n = 0, ..., N -1, it holds:

ε approx HN,n ≤ ([f ] L + V n+1 ∞ [r] L ) inf A∈A X E M A(X n ) -a opt n (X n ) + 2 r ∞ E M V n+1 (X n+1 ) -V M n+1 (X n+1 ) .
Proof. See Section 5.G in Appendix.

Proof of Theorem 5.4.2

Observe that not only the optimal strategy but also the value function is estimated at each time step n = 0, ..., N -1 using neural networks in the hybrid algorithm. It spurs us to introduce the following auxiliary process ( V M n ) N n=0 defined by backward induction as:

V M N (x) = g(x), for x ∈ X , V M n (x) = f (x, âM n (x)) + E V M n+1 (F (x, âM n (x), ε n+1 )) , for x ∈ X ,
and we notice that for n = 0, ..., N -1, V M n is the quantity estimated by V M n .

Step 1. We state the following estimates: for n = 0, ..., N -1,

0 ≤ E M V M n (X n ) -inf a∈A f (X n , a) + E a M,n,Xn V M n+1 (X n+1 ) ≤ 2ε esti HN,n + ε approx HN,n , (5.4.21)
Chapter 5. Neural networks algorithms for control problems: convergence analysis and,

E M V M n (X n ) -inf a∈A f (X n , a) + E a M,n,Xn V M n+1 (X n+1 ) 2 (5.4.22) ≤ 2 ((N -n) f ∞ + g ∞ ) 2ε esti HN,n + ε approx HN,n
, where E M,n,Xn stands for the expectation conditioned by the training set and X n .

Let us first show the estimate (5.4.21). Note that inequality

V M n (X n ) -inf a∈A f (X n , a) + E a M,n,Xn V M n+1 (X n+1 ) ≥ 0
holds because âM n cannot do better than the optimal strategy. Take its expectation to get the first inequality in (5.4.21). Moreover, we write

E M V M n (X n ) ≤ E M f X n , âM n (X n ) + V M n+1 X âM n n+1 ≤ inf A∈A M E M f (X n , A(X n )) + V M n+1 X A n+1 + 2ε esti HN,n ,
which holds by the same arguments as those used to prove (5.4.15). We deduce that

E M V M n (X n ) ≤ inf A∈A X E M f (X n , A(X n )) + V M n+1 X A n+1 + ε approx HN,n + 2ε esti HN,n ≤ E M inf a∈A f (X n , a) + E a M V M n+1 (X n+1 ) X n + ε approx HN,n + 2ε esti HN,n .
This completes the proof of the second inequality stated in (5.4.21).

On the other hand, noting:

V M n (X n ) -inf a∈A f (X n , a) + E a M V M n+1 (X n+1 ) X n ≤ 2 ((N -n) f ∞ + g ∞ ) ,
and using (5.4.21), we obtain the inequality (5.4.22).

Step 2. We state the following estimation: for all n ∈ {0, ..., N } Let us first show the following relation:

V M n (X n ) -V M n (X n ) M,1 = O P γ 2 M K M log(M ) M + inf Φ∈V M Φ(X n ) -V M n (X n ) M,1 + inf A∈A X A(X n ) -a opt n (X n ) M,1 + V n+1 (X n+1 ) -V M n+1 (X n+1 ) M,1 , where . M,p = E M [|.| p ] 1 p , i.
E M V M n (X n ) -V M n (X n ) 2 = O P γ 4 M K M log(M ) M + inf Φ∈V M E |Φ(X n ) -V M n (X n )| 2 .
Chapter 5. Neural networks algorithms for control problems: convergence analysis from assumption (HF-LQ). Now, recall by Zador theorem about optimal quantization (see [START_REF] Graf | Foundations of quantization for probability distributions[END_REF]) that there exists some positive constant C such that

lim M →+∞ L 2 d M εn+1 -ε n+1 2 2 = C.
By using Zador theorem in ((HF-LQ)) and with inequality ε Q 1 ≤ ε Q 2 , we obtain the bound (5.4.30) for the quantization error.

Step 2: We show: as M → ∞,

Ṽ trunc n+1 (X n+1 ) -V M n+1 (X n+1 ) M,1 = O P log(M ) M + inf Φ∈V M Φ(X n+1 ) -V n+1 (X n+1 ) M,1 + V n+1 (X n+1 ) -V M n+1 (X n+1 ) 2 2 .
Denote by

Rn+1 Ṽ trunc n+1 = 1 M M m=1 Ṽ trunc n+1 (X (m) n+1 ) -V M n+1 (X (m) n+1 )
2 the empirical quadratic risk, and by

R n+1 Ṽ trunc n+1 = E M Ṽ trunc n+1 (X n+1 ) -V M n+1 (X n+1 )
2 its associated quadratic risk. From the central limit theorem, we have

Rn+1 Ṽ trunc n+1 -R n+1 Ṽ trunc n+1 σ M,n+1 √ M L ----→ M →∞ N (0, 1)
where σ M,n+1 is the standard variation conditioned by the training set, defined as

σ 2 M,n+1 = Var M Ṽ trunc n+1 (X n+1 ) -V M n+1 (X n+1 ) 2 . Use inequality Ṽ trunc n+1 (X n+1 ) -V M n+1 (X n+1 ) ≤ (N -n) f ∞ + g ∞ to bound σ M,n+1
by a constant that does not depend on M , and get

R n+1 Ṽ trunc n+1 = O P log(M ) M + 1 M M m=1 Ṽ trunc n+1 (X (m) n+1 ) -V M n+1 (X (m) n+1 ) 2 ,
which, after noticing that

1 M M m=1 Ṽ trunc n+1 (X (m) n+1 ) -V M n+1 (X (m) n+1 ) 2 ≤ 1 M M m=1 Ṽ M n+1 (X (m) n+1 ) -V M n+1 (X (m) n+1 ) 2 , implies: R n+1 Ṽ trunc n+1 = O P log(M ) M + inf Φ∈V M 1 M M m=1 Φ(X (m) n+1 ) -V M n+1 (X (m) n+1 ) 2 .
(5.4.31)

Once again from the central limit theorem, we derive:

inf Φ∈V M 1 M M m=1 Φ(X (m) n+1 ) -V M n+1 (X (m) n+1 ) 2 = O P log(M ) M + inf Φ∈V M Φ(X n+1 ) -V M n+1 (X n+1 ) 2 2 .
(5.4.32)

5.A. Localization

5.A Localization

In this section, we show how to relax the boundedness condition on the state space by a localization argument.

Let R > 0. Consider the localized state space BX d (0, R) := X ∩ { x d ≤ R}, where . d is the Euclidean norm of R d . Let Xn 0≤n≤N be the Markov chain defined by its transition probabilities as

P Xn+1 ∈ O Xn = x, a = O r(x, a; y)dπ R • µ(y), for n = 0, . . . , N -1, for all Borelian O in BX d (0, R)
, where π R is the Euclidean projection of R d on BX d (0, R), and π R • µ is the pushforward measure of µ. Notice that the transition probability of X admits the same density r, for which (Hd) holds, w.r.t. π R • µ.

Define ( V R n ) n as the value function associated to the following stochastic control problem for Xn

N n=0 :      V R N (x) = g(x), V R n (x) = inf α∈C E N -1 k=n f Xk , α k + g XN , for n = 0, . . . , N -1, (5.A.1) for x ∈ BX d (0, R).
By the dynamic programming principle, ( V R n ) n is solution of the following Bellman backward equation:

   V R N (x) = g(x) V R n (x) = inf a∈A f (x, a) + E a n V R n+1 π R F (x, a, ε n+1 ) , ∀x ∈ B X d (0, R),
where, again, π R is the Euclidean projection on B X d (0, R).

Lemma 5.A.1. V R n is Lipschitz for n ∈ {0, ..., N }, and moreover the following bound on its Lipschitz coefficient holds:

[ V R n ] L ≤ 1 [F ] n L 1 -[F ] N -n L 1 -[F ] L [f ] L + [F ] N -n L [g] L , (5.A.2)
with the standard convention 1-x n 1-x := n if x = 1.

Proof: Using the dynamic programming equation, assumption (HR), and also the fact that π R is 1-Lipschitz as a projection on the convex B d (0, R) straightforwardly yields:

[ V R N ] L ≤ g L , [ V R n ] L ≤ [f ] L + [F ] L V R n+1 L , for n = {0, ..., N -1}.
Proceed by induction to prove (5.A.2).

We shall assume two conditions on the measure µ.

(Hloc) µ is such that:

E |π R (X) -X| ----→ R→∞ 0 and P (|X| > R) ----→ R→∞ 0, where X ∼ µ.
Using the dominated convergence theorem, it is straightforward to see that (Hloc) holds if µ admits a moment of order 1.

Chapter 5. Neural networks algorithms for control problems: convergence analysis Proposition 5.A.1. Let X n ∼ µ. It holds:

E V R n π R (X n ) -V n X n ≤ V ∞ [r] L E [|π R (X n ) -X n |] + 2P (|X n | > R) 1 -r N -n ∞ 1 -r ∞ + [g] L r N -n ∞ E [|π R (X n ) -X n |] ,
where we denote V ∞ = sup 0≤k≤N V k ∞ , and use the convention 1-x p 1-x = p for x = 0 and p > 1. Consequently, for all n = 0, ..., N , we get under (Hloc):

E V R n π R (X n ) -V n X n ----→ R→∞ 0,
where X n ∼ µ.

Comment: Proposition 5.A.1 states that if X is not bounded, the control problem (5.A.1) associated to a bounded controlled process X can be as close as desired, in L 1 (µ) sense, to the original control problem by taking R large enough. Moreover, as stated before, the transition probability of X admits the same density r as X w.r.t. the pushforward measure π R • µ.

Proof of Proposition 5.A.1. Take X n ∼ µ and write:

E V R n (π R (X n )) -V n (X n ) ≤ E V R n (X n ) -V n (X n ) 1 |Xn|≤R + E V R n (π R (X n )) -V n (X n ) 1 |Xn|≥R . (5.A.3)
Let us first bound the first term in the r.h.s. of (5.A.3), by showing that, for n = 0, . . . , N -1:

E V R n (X n ) -V n (X n ) 1 |Xn|≤R ≤ r ∞ E V R n+1 (π R (X n+1 )) -V n+1 (X n+1 ) + [r] L V n+1 ∞ E [|π R (X n+1 ) -X n+1 |] , with X n+1 ∼ µ.
Take x ∈ Bd (0, R) and notice that

V R n (x) -V n (x) ≤ inf a∈A A V R n+1 (π R (y)) -V n+1 (
≤ r ∞ E V R n+1 (π(X n+1 )) -V n+1 (X n+1 ) + [r] L V n+1 ∞ E [|π R (X n+1 ) -X n+1 |] , where X n+1 ∼ µ.
It remains to inject this bound in the expectation to obtain (Hloc).

To bound the second term in the r.h.s. of (5.A.3), notice that

V R n (π R (X n )) -V n (X n ) ≤ 2 V n ∞ holds a.s., which implies: E V R n (π R (X n )) -V n (X n ) 1 |Xn|≥R ≤ 2 V n ∞ P (|X n | > R) .
Use the dynamic programming equation and the standard inequality

| inf i a i -inf i b i | ≤ sup i |a i -b i | to write: E V R n (X n ) -V n (X n ) ≤ E sup a∈A V R n+1 π F (X n , a, ε n+1 ) -V n+1 F (X n , a, ε n+1 ) + E V n π R (X n ) -V n (X n ) ≤ 3 V R n 2 L + [V n ] 2 L E π R (X n ) -X n 2 + E V R n π R (X n ) -V n π R (X n ) 2 .
(5.A.4)

5.B. Forward evaluation of the optimal controls in A M

Use the programming equation and assumption (Hd) to bound the last term in the r.h.s. of (5.A.4):

E V R n π R (X n ) -V n π R (X n ) ≤ E sup a∈A E a n,Xn V R n+1 π R F (π R (X n ), a, ε n+1 ) -V n+1 π R F (π R (X n ), a, ε n+1 ) ≤ r ∞ E V R n+1 π R (X n+1 ) -V n+1 (π R (X n+1 )) ≤ r ∞ E V R n+1 π R (X n+1 ) -V n+1 (X n+1 ) + E V n+1 X n+1 -V n+1 (π R (X n+1 )) ≤ r ∞ [V n+1 ] L E π R (X n ) -X n + E V R n+1 π R (X n+1 ) -V n+1 (X n+1 ) .(5.A.5)
Inject (5.A.5) in (5.A.4) to obtain:

E V R n (π R (X n )) -V n (X n ) ≤ [V n ] L + r ∞ [V n+1 ] L E π R (X n ) -X n + r ∞ E V R n+1 π R (X n+1 ) -V n+1 (X n+1 ) .
Hence, by induction it holds

E V R n (π R (X n )) -V n (X n ) ≤ E π R (X 1 ) -X 1 N -1 k=n [V k ] L + r ∞ V k+1 L r k-n ∞ + r N -n ∞ [g] L .
Plugging (Hloc) and (Hloc) into (5.A.3) yields:

E V R n (π R (X n )) -V n (X n ) ≤ r ∞ E V R n+1 (π R (X n+1 )) -V n+1 (X n+1 ) + [r] L V n+1 ∞ E [|π R (X n+1 ) -X n+1 |] + 2 V n ∞ P (|X n | > R) ,
with X n and X n+1 i.i.d. following the law µ. The result stated in proposition 5.A.1 then follows by induction.

5.B Forward evaluation of the optimal controls in A M

We evaluate in this section the real performance of the best controls in A M . Let (a A M n ) N -1 n=0 be the sequence of optimal controls in the class of neural networks A M , and denote by (J A M n ) 0≤n≤N the cost functional sequence associated to (a A M n ) N -1 n=0 and characterized as solution of the Bellman equation:

   J A M N (x) = g(x) J A M n (x) = inf A∈A M f (x, A(x)) + E A n,Xn [J A M n+1 (X n+1 )] ,
where E A n,Xn [•] stands for the expectation conditioned by X n and when the control A is applied at time n.

In this section, we are interested in comparing J A M n to V n . Note that V n (x) ≤ J A M n (x) holds for all x ∈ X , since A M is included in the set of the Borelian functions of X . We can actually show the following:

Chapter 5. Neural networks algorithms for control problems: convergence analysis Proposition 5.B.1. Assume that there exists a sequence of optimal feedback controls (a opt n ) 0≤n≤N -1 for the control problem with value function V n , n = 0, . . . , N . Then it holds, as M → ∞:

E J A M n (X n ) -V n (X n ) = O sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| .
(5.B.1) Remark 5.B.1. Notice that there is no estimation error term in (5.B.1), since the optimal strategies in A M are defined as those minimizing the real cost functionals in A M , and not the empirical ones.

Proof of Proposition 5.B.1. Let n ∈ {0, ..., N -1}, and X n ∼ µ. Take A ∈ A M , and denote

J A n (X n ) = f (x, A(x)) + E A n,Xn [J A M n+1 (X n+1 )]. Clearly, we have J A M n = min A∈A M J A n . Moreover: E J A n (X n ) -V n (X n ) ≤ E |f (X n , A(X n )) -f (X n , a opt n (X n ))| + E |J A M n+1 (F (X n , A(X n ), ε n+1 )) -V n+1 (F (X n , a opt n (X n ), ε n+1 ))| ≤ [f ] L E |a opt n (X n ) -A(X n )| + E |V n+1 (F (X n , A(X n ), ε n+1 )) -V n+1 (F (X n , a opt n (X n ), ε n+1 ))| + E |J A M n+1 (F (X n , A(X n ), ε n+1 )) -V n+1 (F (X n , A(X n ), ε n+1 ))| . (5.B.2)
Applying assumption (Hd) to bound the last term in the r.h.s. of (5.B.2) yields

E J A n (X n ) -V n (X n ) ≤ [f ] L + V n+1 ∞ [r] L E |a opt n (X n ) -A(X n )| + r ∞ E |J A M n+1 (X n+1 ) -V n+1 (X n+1 )| ,
which holds for all A ∈ A M , so that:

E J A M n (X n ) -V n (X n ) ≤ [f ] L + V n+1 ∞ [r] L inf A∈A M E |a opt n (X n ) -A(X n )| + r ∞ E |J A M n+1 (X n+1 ) -V n+1 (X n+1 )| .
(5.B.1) then follows directly by induction.

5.C Proof of Lemma 5.4.1

The proof is divided into four steps.

Step 1: Symmetrization by a ghost sample. We take ε > 0 and show that for

M > 2 (N -n) f ∞+ g ∞ 2 ε 2
, the following holds:

P sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -E J A,(â M k ) N -1 k=n+1 n (X n ) > ε ≤ 2P sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -f (X (m) n , A(X (m) n )) -Ŷ (m),A n+1 > ε 2 ,(5.C.1)
where:

• (X (m) k
) 1≤m≤M,n≤k≤N is a copy of (X

Y (m),A n+1 := N -1 k=n+1 f (X (m),A k , âM k (X (m),A k )) + g(X (m),A N ).
Let A * ∈ A M be such that:

1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 -E J A * ,(â M k ) N -1 k=n+1 n (X n ) > ε
if such a function exists, and an arbitrary function in A M if such a function does not exist. Note that

1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 -E J A * ,(â M k ) N -1 k=n+1 n (X n ) is a r.v.,
P M E M J A * ,(â M k ) N -1 k=n+1 n (X n ) - 1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 > ε 2 ≤ Var M J A * ,(â M k ) N -1 k=n+1 n (X n ) M (ε/2) 2 ≤ (N -n) f ∞ + g ∞ 2 M ε 2 ,
where we have used 0 ≤ J

A * ,(â M k ) N -1 k=n+1 n (X n ) ≤ (N -n) f ∞ + g ∞ which implies Var M J A * ,(â M k ) N -1 k=n+1 n (X n ) = Var M J A * ,(â M k ) N -1 k=n+1 n (X n ) - (N -n) f ∞ + g ∞ 2 ≤ E J A * ,(â M k ) N -1 k=n+1 n (X n ) - (N -n) f ∞ + g ∞ 2 2 ≤ (N -n) f ∞ + g ∞ 2 4 .
Thus, for M > 2

(N -n) f ∞+ g ∞ 2 ε 2
, we have

P M E M J A * ,(â M k ) N -1 k=n+1 n (X n ) - 1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 ≤ ε 2 ≥ 1 2 . ( 5

.C.2)

Hence:

P sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -f (X (m) n , A(X m n )) -Ŷ (m),A n+1 > ε 2 ≥ P 1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 -f (X (m) n , A * (X (m) n )) -Ŷ (m),A * n+1 > ε 2 ≥ P 1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 -E M J A * ,(â M k ) N -1 k=n+1 n (X n ) > ε, 1 M M m=1 f (X (m) n , A * (X (m) n )) + Ŷ (m),A * n+1 -E M J A * ,(â M k ) N -1 k=n+1 n (X n ) ≤ ε 2 .
5.C. Proof of Lemma 5.4.1

Step 3: Introduction of additional randomness by random signs. Let (r m ) 1≤m≤M be i.i.d. Rademacher r.v. 6 . We show that: ) n≤k≤N and (ε

E sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -f (X (m) n , A(X (m) n )) -Ŷ (m),A n+1 ≤ 4E sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 . ( 5
(m)
k ) n≤k≤N are i.i.d., their joint distribution remain the same if one randomly interchanges the corresponding components. Thus, it holds for ε ≥ 0:

P sup A∈A M 1 M M m=1 f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -f (X m n , A(X m n )) -Ŷ (m),A n+1 > ε = P sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 -f (X m n , A(X m n )) -Ŷ (m),A n+1 > ε ≤ P sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 > ε 2 + P sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 > ε 2 ≤ 2P sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 > ε 2 .
It remains to integrate on R + w.r.t. ε to get (5.C.5).

Step 4: We show that

E sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 ≤ (N -n) f ∞ + g ∞ √ M + [f ] L + [f ] L N -1 k=n+1 1 + η M γ M k-n [F ] k-n L + η N -n M γ N -n M [F ] N -n L [g] L = O γ N -n M η N -n M √ M , as M → ∞. (5.C.6)
Adding and removing the cost obtained by control 0 at time n yields: We now bound the first term of the r.h.s. of (5.C.7). By Cauchy-Schwartz inequality, and recalling that (r m ) 1≤m≤M are i.i.d. with zero mean such that r 2 m = 1, we get

E sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) + Ŷ (m),A n+1 ≤ E 1 M M m=1 r m f (X (m) n , 0) + Ŷ (m),0 n+1 + E sup A∈A M 1 M M m=1 r m f (X (m) n , A(X (m) n )) -f (X (m) n , 0) + Ŷ (m),A n+1 - Ŷ (
E 1 M M m=1 r m f (X (m) n , 0) + Ŷ (m),0 n+1 ≤ 1 M E M m=1 r m f (X (m) n , 0) + Ŷ (m),0 n+1 2 ≤ 1 √ M (N -n) f ∞ + g ∞
Turn now to the second term of (5.C.7). By the Lipschitz continuity of f , it stands:

E sup A∈A M M m=1 r m f (X (m) n , A(X (m) n )) -f (X (m) n , 0) + Ŷ (m),A n+1 - Ŷ (m),0 n+1 ≤ [f ] L E sup A∈A M M m=1 r m A(X (m) n ) + E sup A∈A M M m=1 r m Ŷ (m),A n+1 - Ŷ (m),0 n+1 ≤ [f ] L + [f ] L N -1 k=n+1 1 + η M γ M k-n E   sup 1≤m≤M k j=n+1 C ε m j   + [g] L 1 + η N -n M γ N -n M E   sup 1≤m≤M N j=n+1 C ε m j   E sup A∈A M M m=1 r m A(X (m) n )
where we condition by the exogenous noise, use assumption (HF-PI) and the η M γ M -Lipschitz continuity of the estimated optimal controls at time k, for k = n + 1, . . . , N -1. Now, notice first that

E sup 1≤m≤M N k=n+1 C (ε m k ) ≤ N k=n+1 E sup 1≤m≤M C (ε m k ) ≤ ρ N -n M ,
and moreover:

E sup A∈A M M m=1 r m A(X (m) n ) ≤ γ M E sup |v|2≤1/R M m=1 r m (v T X (m) n ) + ≤ γ M E sup |v|2≤1/R M m=1 r m v T X (m) n ,
where R > 0 is a bound for the state space (see e.g. the discussion on the Frank-Wolfe step p.10 of [START_REF] Bach | Breaking the curse of dimensionality with convex neural networks[END_REF] for a proof of this inequality), which implies by Cauchy-Schwarz inequality:

E sup A∈A M M m=1 r m A(X (m) n ) ≤ γ M R E M m=1 r m X (m) n 2 ≤ γ M √ M
since the (r m ) m are i.i.d. Rademacher r.v. Plug first (5.C.9) and (5.C.10) into (5.C) to obtain

E sup A∈A M M m=1 r m f (X (m) n , A(X (m) n )) -f (X (m) n , 0) + Ŷ (m),A n+1 - Ŷ (m),0 n+1 ≤ [f ] L + [f ] L N -1 k=n+1 1 + η M γ M k-n ρ k-n M + [g] L 1 + η N -n M γ N -n M ρ N -n M γ M √ M .
     J A,(â) N -1 k=n+1 N (x) = g(x) J A,(â) N -1 k=n+1 n (x) = f (x, A(x)) + E A n,x J A,(â) N -1 k=n+1 n+1 (X n+1 ) ,
where E A n,x [•] stands for the expectation conditioned by {X n = x} when feedback control A is followed at time n.

Take n ∈ {1, ..., N }. It holds:

ε approx PI,n := inf A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -inf A∈A X E M J A,(â) N -1 k=n+1 n (X n ) = inf A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -E [V n (X n )] + E [V n (X n )] -inf A∈A X E M J A,(â) N -1 k=n+1 n (X n ) ≤ inf A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -E [V n (X n )] ,
(5.D.1)

where the last inequality stands because the value function is smaller than the cost functional associated to any other strategy. We then apply the dynamic programming principle to obtain:

min A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -E [V n (X n )] ≤ inf A∈A M E M f X n , A(X n ) + E A n J (â) N -1 k=n+1 n+1 (X n+1 ) -E f X n , a opt n (X n ) + E a opt n [V n+1 (X n+1 )] . (5.

D.2)

To bound the r.h.s. of (5.D.2), first observe that for A ∈ A M :

E M f X n , A(X n ) + E A n J (â) N -1 k=n+1 n+1 (X n+1 ) -E f X n , a opt n (X n ) + E a opt n [V n+1 (X n+1 )] ≤ E |f X n , A(X n ) -f X n , a opt n (X n ) | + E M E A n J (â) N -1 k=n+1 n+1 (X n+1 ) -E a opt n V n+1 (X n+1 ) ≤ [f ] L + V n+1 ∞ [r] L E |A(X n ) -a opt n (X n )| + r ∞ E M J (â) N -1 k=n+1 n+1 (X n+1 ) -V n+1 (X n+1 ) , (5.

D.3)

where we used twice assumption (Hd) at the second-last line of (5.D.3). Inject inequality

E M J (â) N -1 k=n+1 n+1 (X n+1 ) ≤ inf A∈A M E M J A,(â) N -1 k=n+2 n+1 (X n+1 ) + 2ε esti n+1
Chapter 5. Neural networks algorithms for control problems: convergence analysis into (5.D.3) to obtain:

E M f X n , A(X n ) + E A n J (â) N -1 k=n+1 n+1 (X n+1 ) -E f X n , a opt n (X n ) + E a opt n [V n+1 (X n+1 )] ≤ [f ] L + V n+1 ∞ [r] L E |A(X n ) -a opt n (X n )| + r ∞ inf A∈A M E M J A,(â) N -1 k=n+2 n+1 (X n+1 ) -V n+1 (X n+1 ) + 2 r ∞ ε esti n+1 .
(5.D.4) Plugging (5.D.4) into (5.D.2) yields inf

A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -E [V n (X n )] ≤ r ∞ inf A∈A M E M J A,(â) N -1 k=n+2 n+1 (X n+1 ) -V n+1 (X n+1 ) + 2 r ∞ ε esti n+1 + [f ] L + V n+1 ∞ [r] L inf A∈A M E |A(X n ) -a opt n (X n )| ,
which implies by induction, as M → +∞:

E inf A∈A M E M J A,(â) N -1 k=n+1 n (X n ) -E [V n (X n )] = O sup n+1≤k≤N -1 E ε esti k + sup n≤k≤N -1 inf A∈A M E |A(X n ) -a opt n (X n )| .
We now use Lemma 5.4.1 to bound the expectations of the ε esti PI,k for k = n + 1, . . . , N -1, and plug the result into (5.D.1) to complete the proof of Lemma 5.4.2.

5.E Function approximation by neural networks

We assume a opt n (X n ) ∈ L 2 (µ), and show the relation (5.4.7) in Proposition 5.4.1. The universal approximator theorem applies for

A ∞ := ∞ M =1 A M ,
and states that for all ε > 0, there exists a neural network a * in A ∞ such that:

sup n≤k≤N -1 a opt k -a * ∞ < ε V d (X )
,

where V d (X ) stands for the volume of compact set X seen as a compact of the euclidean space R d . By integrating, we then get:

sup n≤k≤N -1 X a opt k (x) -a * (x) dµ(x) < ε,
Also, notice that A M M ≥1 is increasing, which implies that A ∞ = lim M →+∞ A M , and gives the existence of M > 0, that depends on ε, such that a * ∈ A M .

Therefore, we have shown that for n = 0, ..., N - We use the Lipschitz-continuity of f as follows, to bound its second term:

1 sup n≤k≤N -1 inf A∈A M E |A(X k ) -a opt k (X k )| ----→ M →∞ 0, with X k ∼ µ,
E sup A∈A M M m=1 r m f (X (m) n , A(X (m) n )) -f (X (m) n , 0) + Ŷ (m),A n+1 - Ŷ (m),0 n+1 ≤ [f ] L E sup A∈A M M m=1 r m A(X (m) n ) + E sup A∈A M M m=1 r m Ŷ (m),A n+1 - Ŷ (m),0 n+1 , ≤ ([f ] L + ρ M η M γ M ) E sup A∈A M M m=1 r m A(X (m) n )
where we condition by the exogenous noise, use assumption (HF), and the η M γ M -Lipschitz continuity of the estimated value fonction at time n + 1.

By using the same arguments as those presented in the proof of Lemma 5.4.1, we can first bound

E sup A∈A M M m=1 r m A(X (m)
n ) as follows:

E sup A∈A M M m=1 r m A(X (m) n ) ≤ γ M √ M ,
and then conclude that (5.F.3) holds.

Step 5: Conclusion Combining(5.F.1),(5.F.2) and (5.F.3) results in the bound on the estimation error as stated in (5.4.20).

5.G Proof of Lemma 5.4.4

We divide the proof of Lemma 5.4.4 into two steps.

First write

ε approx HN,n ≤ inf A∈A M E M f (X n , A(X n )) + V M n+1 X A n+1 -E [V n (X n )] + E [V n (X n )] -inf A∈A X E M f (X n , A(X n )) + V M n+1 X A n+1 .
(5.G.1)

Step 1: We show

inf A∈A M E M f (X n , A(X n )) + V M n+1 X A n+1 -E [V n (X n )] ≤ ([f ] L + V n+1 ∞ [r] L ) inf A∈A X E M A(X n ) -a opt n (X n ) + r ∞ E M V n+1 (X n+1 ) -V M n+1 (X n+1 ) .
(5.G.2) Take A ∈ A M , and apply the dynamic programming principle to write

E M f (X n , A(X n )) + V M n+1 X A n+1 -E [V n (X n )] ≤ [f ] L E M A(X n ) -a opt n (X n ) + E M E M V M n+1 (X A n+1 ) -E M V n+1 X a opt n n+1 ≤ ([f ] L + V n+1 ∞ [r] L ) E M A(X n ) -a opt n (X n ) + E M V M n+1 (X A n+1 ) -V n+1 (X A n+1 ) ,
where we used (Hd) at the second-to-last line. By using one more time assumption (Hd), we then get:

E M f (X n , A(X n )) + V M n+1 X A n+1 -E [V n (X n )] ≤ ([f ] L + V n+1 ∞ [r] L ) E M A(X n ) -a opt n (X n ) + r ∞ E M V M n+1 (X n+1 ) -V n+1 (X n+1 ) , with X n+1 ∼ µ,
which is the result stated in (5.G.2).

Step 2: We show

E [V n (X n )] -inf A∈A X E M f (X n , A(X n )) + V M n+1 X A n+1 ≤ r ∞ E M V n+1 (X n+1 ) -V M n+1 (X n+1 ) . (5.G.3) Write E [V n (X n )] -inf A∈A X E M f (X n , A(X n )) + V M n+1 X A n+1 ≤ inf A∈A X E M f (X n , A(X n )) + V n+1 X A n+1 -inf A∈A X E M f (X n , A(X n )) + V M n+1 X A n+1 ≤ inf A∈A X E M V n+1 X A n+1 -V M n+1 X A n+1 ≤ r ∞ E M V n+1 (X n+1 ) -V M n+1 (X n+1 )
which completes the proof of (5.G.3).

Step 3 Conclusion:

We complete the proof of Lemma 5.4.4 by plugging (5.G.2) and (5.G.3) into (5.G.1).

5.H Some useful Lemmas for the proof of Theorem 5.4.2

Fix M ∈ N * , let x 1 , . . . , x M ∈ R d , and set x M = (x 1 , . . . , x M ). Define the distance d 2 (f, g) between f : R d → R and g : R d → R by

d 2 (f, g) = 1 M M m=1 |f (x m ) -g(x m )| 2 1/2 . An ε-cover of V (w.r.t. the distance d 2 ) is a set of functions f 1 , . . . , f P : R d → R such that min p=1,...,P d 2 (f, f p ) < ε, for f ∈ V.
Let N 2 (ε, V, x M ) denote the size of the smallest ε-cover of V w.r.t. the distance d 2 , and set by convention N 2 (ε, V, x M ) = +∞ if there does not exist any ε-cover of V of finite size. c 2 E e (Y -m(X)) 2 /c 2 -1|X ≤ σ 2 a.s.

N 2 (ε, V, x M ) is called 2 -ε-covering number of V on x M .

Introduction

This paper is devoted to the numerical resolution of discrete-time stochastic control problem over a finite horizon. The dynamics of the controlled state process X = (X n ) n valued in R d is given by

X n+1 = F (X n , α n , ε n+1 ), n = 0, . . . , N -1, X 0 = x 0 ∈ R d ,
where (ε n ) n is a sequence of i.i.d. random variables valued in some Borel space (E, B(E)), and defined on some probability space (Ω, F, P) equipped with the filtration F = (F n ) n generated by the noise (ε n ) n (F 0 is the trivial σ-algebra), the control α = (α n ) n is an F-adapted process valued in A ⊂ R q , and F is a measurable function from R d × R q ×E into R d . Given a running cost function f defined on R d × R q and a terminal cost function g defined on R d , the cost functional associated with a control process α is

J(α) = E N -1 n=0 f (X n , α n ) + g(X N ) .
The set A of admissible controls is the set of control processes α satisfying some integrability conditions ensuring that the cost functional J(α) is well-defined and finite. The control problem, also called Markov decision process (MDP), is formulated as

V 0 (x 0 ) := inf α∈A J(α),
and the goal is to find an optimal control α * ∈ A, i.e., attaining the optimal value: V 0 (x 0 ) = J(α * ).

Notice that problem (6.1.1)-(6.1.3) may also be viewed as the time discretization of a continuous time stochastic control problem, in which case, F is typically the Euler scheme for a controlled diffusion process.

It is well-known that the global dynamic optimization problem (6.1.3) can be reduced to local optimization problems via the dynamic programming (DP) approach, which allows to determine the value function in a backward recursion by

V N (x) = g(x), x ∈ R d , V n (x) = inf a∈A Q n (x, a), with Q n (x, a) = f (x, a) + E V n+1 (X n+1 ) X n = x, α n = a , (x, a) ∈ R d ×A.
Moreover, when the infimum is attained in the DP formula (6.1.4) at any time n by a * n (x) ∈ arg min a∈A Q n (x, a), we get an optimal control in feedback form (policy) given by: α * = (a * n (X * n )) n where X * is the Markov process defined by

X * n+1 = F (X * n , a * n (X * n ), ε n+1 ), n = 0, . . . , N -1, X * 0 = x 0 .
The practical implementation of the DP formula may suffer from the curse of dimensionality and large complexity when the state space dimension d and the control space dimension are high. In [START_REF] Huré | Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis[END_REF], we proposed algorithms relying on deep neural networks for approximating/learning the optimal policy and then eventually the value function by performance/policy iteration or hybrid iteration with Monte Carlo regressions now or later. This research led to three algorithms, namely algorithms NNcontPI, Hybrid-Now and Hybrid-LaterQ that are recalled in Section 6.2. In Section 6.3, we perform some numerical and comparative tests for illustrating the efficiency of our different algorithms, on 100-dimensional nonlinear PDEs examples as in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] and quadratic Backward Stochastic Differential equations as in [START_REF] Chassagneux | Numerical simulation of quadratic BSDEs[END_REF]. We present numerical results for an option hedging problem in finance, and energy storage problems arising in the valuation of gas storage and in microgrid management. Numerical results and comparisons to quantization-type algorithms Qknn, introduced in this paper as an efficient algorithm to numerically solve low-dimensional control problems, are also provided. Finally, we conclude in Section 6.4 with some comments about possible extensions and improvements of our algorithms.

Algorithms

Algorithms

We introduce in this section four neural network-based algorithms for solving the discrete-time stochastic control problem (6.1.1)-(6.1.3). The convergence of these algorithms have been analyzed in detail in our companion paper [START_REF] Huré | Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis[END_REF]. We also introduce at the end of this section a quantization and k-nearest-neighbor-based algorithm (Qknn) that will be used as benchmark when testing our algorithms on low-dimensional control problems.

We are given a class of deep neural networks (DNN) for the control policy represented by the parametric functions x ∈ R d → A(x; β) ∈ A, with parameters β ∈ R q , and a class of DNN for the value function represented by the parametric functions: x ∈ R d → Φ(x; θ) ∈ R, with parameters θ ∈ R p . Recall that these DNN functions A and Φ are compositions of linear combinations and nonlinear activation functions, see [START_REF] Goodfellow | Deep learning[END_REF].

Additionally, we shall be given a sequence of probability measures on the state space R d , that we call training measure and denoted (µ n ) N -1 n=0 , and which should be seen as dataset providers to learn the optimal strategies and the value functions at time n = 0, . . . , N -1. Remark 6.2.1 (Training sets design). Two cases are considered for the choice of the training measure µ n used to generate the training sets on which will be computed the estimates at time n. The first one is a knowledge-based selection, relevant when the controller knows with a certain degree of confidence where the process has to be driven in order to optimize her cost functional. The second case is when the controller has no idea where or how to drive the process to optimize the cost functional.

(1) Exploitation only strategy

In the knowledge-based setting, there is no need for exhaustive and expensive (in time mainly) exploration of the state space, and the controller can take a training measure µ n that assigns more points in the region of the state space that is likely to be visited by the optimally-driven process.

In practice, at time n, assuming we know that the optimal process is likely to lie in a region D, we choose a training measure in which the density assigns lot of weight to the points of D, for example U(D), the uniform distribution in D.

(2) Explore first, exploit later When the controller has no idea where or how to drive the process to optimize the cost functional, we suggest the latter to adopt the following two-step approach:

(i) Explore first: If the agent has no idea of where to drive the process to receive large rewards, she can always proceed to an exploration step to discover favorable subsets of the state space. To do so, the training sets Γ n at time n, for n = 0, . . . , N -1, can be built as uniform grids that cover a large part of the state space, or µ can be chosen uniform on such domain. It is essential to explore far enough to acquire a good understanding of where to drive and where not to drive the process.

(ii) Exploit later: The estimates for the optimal controls at time n, n = 0, . . . , N -1, that come up from the Explore first step, are relatively good in the way that they manage to avoid the poor regions when driving the process. However, the training sets that have been used to compute the estimated optimal control are too sparse to ensure accuracy on the estimation. In order to improve the accuracy, the natural idea is to build new training sets by simulating M times the process using the estimates on the optimal strategy computed from the Explore first step, and then proceed to another estimation of the optimal strategies using the new training sets. This trick can be seen as a two-step algorithm that improves the final estimate of the optimal control.

Remark 6.2.2 (Choice of Neural Networks). Unless otherwise specified, we use Feedback Neural Networks with two or three hidden layers and d+10 neurons per hidden layer. We tried sigmoid, tanh, ReLU and ELU activation functions and noticed that ELU is most often the one providing the best results in our applications. We normalize the input data of each neural network in order to speed up the training of the latter.

Remark 6.2.3 (Neural Networks Training). We use the Adam optimizer, as implemented in Ten-sorFlow, with learning rate set to 0.001 or 0.005, which are the default values in TensorFlow, to train the optimal strategy and the value function computed from the algorithms described later.

In order to force the weights of biases of the neurons to stay small, we use an L2 regularization with parameter mainly set to 0.01, but the value can change in order to make sure that the regularization term is neither too strong or too weak when added to the loss when training neural networks.

We consider a large enough number of mini-batches of size 64 or 128 for the training, depending essentially empirically on the dimension of the problem. We use at least 10 epochs 2 and stop the training when the loss computed on a validation set of size 100 stops decreasing. We noticed that taking more than one epoch really improves the quality of the estimates. Remark 6.2.4 (Constraints). The proposed algorithms can deal with state and control constraints at any time, which is useful in several applications:

(X α n , α n ) ∈ S a.s., n ∈ N,
where S is some given subset of R d × R q . In this case, in order to ensure that the set of admissible controls is not empty, we assume that the sets A(x) := a ∈ R q : (F (x, a, ε 1 ), a) ∈ S a.s.

are non empty for all x ∈ S, and the DP formula now reads

V n (x) = inf a∈A(x) f (x, a) + P a V n+1 (x) , x ∈ S.
From a computational point of view, it may be more convenient to work with unconstrained state/control variables, hence by relaxing the state/control constraint and introducing into the running cost a penalty function L(x, a): f (x, a) ← f (x, a) + L(x, a), and g(x) ← g(x) + L(x, a). For example, if the constraint set S is in the form: S = {(x, a) ∈ R d × R q : h k (x, a) = 0, k = 1, . . . , p, h k (x, a) ≥ 0, k = p + 1, . . . , q}, for some functions h k , then one can take as penalty functions:

L(x, a) = p k=1 µ k |h k (x, a)| 2 + q k=p+1 µ k max(0, -h k (x, a)).
where µ k > 0 are penalization coefficients (large in practice).

Control Learning by Performance Iteration

We present in this section Algorithm 6.1, which combines an optimal policy estimation by neural networks and the dynamic programming principle. We rely on the performance iteration procedure, i.e. paths are always recomputed up to the terminal time N .

Algorithm NNContPI

Our first algorithm, refereed to as NNContPI, and described in Algorithm 6.1, is well-designed for control problems with continuous control space such as R q or a ball in R q . The main idea is to first parametrize the optimal control using a neural network in which the activation function for the output layer takes values in the control space. For example, one can take the identity function as activation function for the output layer if the control space is R q ; or the sigmoid function if the control space is [0, 1], or more generally a bounded set. Then, it remains to learn the optimal parameters of the neural network using a training set, which is given as initial positions of law µ n and random noises.

Algorithms

Algorithm 6.1 NNContPI

Input: the training distributions (µ n ) N -1 n=0 ; 1: for n = N -1, . . . , 0 do 2: Compute βn ∈ argmin β∈R q E f X n , A(X n ; β) + N -1 k=n+1 f X β k , âk X β k + g X β N
where X n ∼ µ n and where X β k N k=n+1 is defined by induction as:

X β n+1 = F X n , A X n ; β , ε n+1 X β k+1 = F X β k , âk X β k ; β , ε k+1 , for k = n + 1, . . . , N -1.
for m = 1, . . . , M ;

3:

Set ân = A(.; βn ); ân is the estimate of the optimal policy at time n 4: end for Output: estimates of the optimal strategy (â n ) N -1 n=0 ;

Algorithm ClassifPI

In the special case where the control space A is finite, i.e., Card(A) = L < ∞ with A = {a 1 , . . . , a L }, a classification method can be used: consider a DNN that takes state x as input and returns a probability vector p(x; β) = (p (x; β)) L =1 with parameters β. Algorithm 6.2, presented below, is based on this idea, and is called ClassifPI.

L =1 p (X n ; β) f (X n , a ) + N -1 k=n+1 f X k , âk (X k ) + g(X N ) ,
where

X n ∼ µ n on R d , X n+1 = F (X n , a , ε n+1 ), X k+1 = F (X k , âk (X k ), ε k+1 ), for k = n + 1, .
. . , N -1 and = 1, . . . , L;

3:

Set ân = a ˆ n (x) with ˆ n (x) ∈ arg max =1,...,L p (x; βn ); ân is the estimate of the optimal policy at time n 4: end for Output: estimates of the optimal strategy (â n ) N -1 n=0 ;

Note that, when using Algorithms 6.1 and 6.2, the estimate of the optimal strategy at time n highly relies on the estimates of the optimal strategy at time n + 1, . . . , N -1, that have been computed previously. In particular, the practitioner who wants to use Algorithms 6.1 and 6.2 needs to keep track of the estimates of the optimal strategy at time n+1, . . . , N -1 in order to compute the estimate of the optimal strategy at time n. Remark 6.2.5. In practice, for n = N -1, ..., 0, one should approximate the expectations (6.2.1) and (6.2.3) by its empirical mean, i.e. consider a sample from µ n for the initial position at time n, and other samples from the law ε k , for k = n + 1, . . . , N , in order to generate a finite number of paths Chapter 6. Neural networks algorithms for control problems: numerical applications (X β k ) N k=n+1 on which to estimate the expectations (6.2.1) and (6.2.3) using Monte Carlo method.

Control and value function learning by double DNN

We present in this section two algorithms, which in contrast with Algorithms 6.1 or 6.2, only keep track on the estimates of the value function and optimal control at time n + 1 in order to build an estimate of the value function and optimal control at time n.

Regress Now (Hybrid-Now)

The Algorithm 6.3, refereed to as Hybrid-Now, combines optimal policy estimation by neural networks and dynamic programming principle, and relies on an hybrid procedure between value and performance iteration.

Algorithm 6.3 Hybrid-Now

Input: the training distributions (µ n ) N -1 n=0 ; 1: Set VN = g; 2: for n = N -1, . . . , 0 do 3: Compute: βn ∈ argmin β∈R q E f X n , A(X n ; β) + Vn+1 (X β n+1 )
where X n ∼ µ, and X β n+1 = F X n , A(X n ; β) , ε n+1 );

4:

Set ân = A(.; βn ); ân is the estimate of the optimal policy at time n; -estimate of the optimal strategy (â n ) N -1 n=0 ; -estimate of the value function ( Vn ) N -1 n=0 ;

Remark 6.2.6. One can combine different features from Algorithms 6.1, 6.2 and 6.3 to solve specific problems, as it has been done for example in Section 6.3.4, where we designed Algorithm 6.6 to solve a Smart Grid Management problem.

Regress Later and Quantization (Hybrid-LaterQ)

The Algorithm 6.4, called Hybrid-LaterQ, combines regress-later and quantization methods to build estimates of the value function. The main idea behind Algorithm 6.4 is to first interpolate the value function at time n + 1 by a set of basis functions, which is in the spirit of the regress-later-based algorithms, and secondly regress the interpolation at time n using quantization. The usual regresslater approach requires the ability to compute closed-form conditional expectations, which limits the stochastic dynamics and regression bases that can be considered. The use of quantization avoids this limitation and makes the regress-later algorithm more generally applicable.

Let us first recall the basic ingredients of quantization. We denote by ε a K-quantizer of the R dvalued random variable ε n+1 ε 1 (typically a Gaussian random variable), that is a discrete random The grid points (e ) which minimize the L 2 -quantization error ε 1 -ε 2 lead to the so-called optimal Lquantizer, and can be obtained by a stochastic gradient descent method, known as Kohonen algorithm or competitive learning vector quantization (CLVQ) algorithm, which also provides as a byproduct an estimation of the associated weights (p ). We refer to [START_REF] Pagès | Optimal quantization methods and applications to numerical problems in finance[END_REF] for a description of the algorithm, and mention that for the normal distribution, the optimal grids and the weights of the Voronoi tesselations are precomputed on the website http://www.quantize.maths-fi.com.

Quantization is mainly used in Algorithm 6.4 to efficiently approximate the expectations: recalling the dynamics (6.1.1), the conditional expectation operator for any functional W is equal to

P âM n (x) W (x) = E W (X âM n n+1 )|X n = x = E W (F (x, âM n (x), ε 1 )) , x ∈ R d ,
that we shall approximate analytically by quantization via:

P âM n (x) W (x) := E W (F (x, âM n (x), ε)) = K =1 p W F (x, âM n (x), e ) .
Observe that the solution to (6.2.7) actually provides a neural network Φ(.; θn+1 ) that interpolates Ṽn+1 . Hence the Algorithm 6.4 contains an interpolation step, and moreover, any kind of distance in R d can be chosen as a loss to compute θn+1 . In (6.2.7), we decide to take the L 2 -loss, mainly because it is the one that worked the best in our applications. Remark 6.2.7 (Quantization). In dimension 1, we used the optimal grids and weights with L = 21 points, to quantize the reduced and centered normal law N (0, 1); and took 100 points to quantize the reduced and centered normal law in dimension 2, i.e. N 2 (0, 1). All the grids and weights for the optimal quantization of the normal law in dimension d are available in http://www.quantize.maths-fi. com for d = 1, . . . , 100. 6.2.2.3 Some remarks on Algorithms 6.3 and 6.4

As in Remark 6.2.5, all the expectations written in our pseudo-codes in Algorithm 6.3 and 6.4 should be approximated by empirical mean using a finite training set. Algorithms 6.3 or 6.4 are quite efficient to use in the usual case where the value function and the optimal control at time n are very close to the value function and the optimal control at time n + 1, which happens e.g. when the value function and the optimal control are approximations of the time discretization of a continuous in time value function and an optimal control. In this case, it is recommended to follow the two steps procedure:

(i) initialize the parameters (i.e. weights and bias) of the neural network approximations of the value function and the optimal control at time n to the ones of the neural network approximations of the value function and the optimal control at time n + 1.

Chapter 6. Neural networks algorithms for control problems: numerical applications Algorithm 6.4 Hybrid-LaterQ Input:

-the training distributions (µ n ) N -1 n=0 ; -The grid {e 1 , . . . , e L } of L points in E, with weights p 1 , . . . , p L for the quantization of the noise ε n ;

1: Set VN = g; 2: for n = N -1, . . . , 0 do 3: Compute: βn ∈ argmin β∈R q E f X n , A(X n ; β) + Vn+1 (X β n+1 )
where X n ∼ µ n , and X β n+1 = F X n , A(X n ; β) , ε n+1 );

4:

Set ân = A(.; βn ); ân is the estimate of the optimal policy at time n -estimate of the optimal strategy (â n ) N -1 n=0 ; -estimate of the value function ( Vn ) N -1 n=0 ;

Chapter 6. Neural networks algorithms for control problems: numerical applications Algorithm 6.5 Qknn Input:

-Grids Γ k , k = 0, . . . , N in R d ; -Grid {e 1 , . . . , e L } of L points in E, with weights p 1 , . . . , p L for the quantization of ε n ;

1: Set VN = g; 2: for n = N -1, . . . , 0 do 3:

Compute for (z, a) ∈ Γ n × A,

Qn (z, a) = f (z, a) + L =1 p V n+1 Proj Γn+1 F (z, a, e ) ,
where Proj Γn+1 is the Euclidean projection over Γ n+1 ;

4:
Qn is the approximated Q-value3 at time n Set V n (z) = Qn z, Ân (z) , ∀z ∈ Γ n ;

7:

V n is the estimate by quantization of the value function 8: end for Output:

-estimate of the optimal strategy (â n ) N -1 n=0 ; -estimate of the value function ( Vn ) N -1 n=0 ;

6.3. Numerical applications

Numerical applications

In this section, we test the Neural-Networks-based algorithms presented in Section 6.2 on different examples. In high-dimension, we took the same example as already considered in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] so that we can directly compare our results to theirs. In low-dimension, we compared the results of our algorithms to the ones provided by Qknn, which has been introduced in Section 6.2 as an excellent benchmark for low-dimensional control problems.

A semilinear PDE

We consider the following semilinear PDE with quadratic growth in the gradient:

   ∂v ∂t + ∆ x v -|D x v| 2 = 0, (t, x) ∈ [0, T ) × R d , v(T, x) = g(x), x ∈ R d . (6.3.1)
By observing that for any p ∈ R d , -|p| 2 = inf a∈R d [|a| 2 + 2a.p], the PDE (6.3.1) can be written as a Hamilton-Jacobi-Bellman equation

   ∂v ∂t + ∆ x v + inf a∈R d |a| 2 + 2a.D x v] = 0, (t, x) ∈ [0, T ) × R d , v(T, x) = g(x), x ∈ R d ,
hence associated with the stochastic control problem

v(t, x) = inf α∈A E T t |α s | 2 ds + g(X t,x,α T ) ,
where X = X t,x,α is the controlled process governed by

dX s = 2α s ds + √ 2dW s , t ≤ s ≤ T, X t = x,
W is a d-dimensional Brownian motion, and the control process α is valued in A = R d . The time discretization (with time step h = T /N ) of the control problem (6.3.2) leads to the discrete-time control problem (6.1.1)-(6.1.2)-(6.1.3) with

X α n+1 = X α n + 2α n h + √ 2hε n+1 =: F (X α n , α n , ε n+1 ), n = 0, . . . , N -1,
where (ε n ) n is a sequence of i.i.d. random variables of law N (0, I d ), and the cost functional

J(α) = E N -1 n=0 h|α n | 2 + g(X α N ) .
On the other hand, it is known that an explicit solution to (6.3.1) (or equivalently (6.3.1)) can be obtained via a Hopf-Cole transformation (see e.g. [START_REF] Chassagneux | Numerical simulation of quadratic BSDEs[END_REF]), and is given by

v(t, x) = -ln E exp -g(x + √ 2W T -t ) , (t, x) ∈ [0, T ] × R d .
We choose to run tests on two different examples that have already been considered in the literature:

Test 1 Some recent numerical results have been obtained in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] (see Section 4.3 in [EHJ17]) when T = 1 and g(x) = ln( 1 2 (1 + |x| 2 )) in dimension d = 100 (see Table 2 and Figure 3 in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF]). Their method is based on neural network regression to solve the BSDE representation associated to the PDE (6.3.1), and provide estimates of the value function at time 0 and state 0 for different values of a coefficient γ. We plotted the results of the Hybrid-Now algorithm in Figure 6.1. Hybrid-Now Chapter 6. Neural networks algorithms for control problems: numerical applications Table 6.1 -Value function at time 0 and state 0 w.r.t. γ, computed with the Y&R, Hybrid-Now, Hybrid-Later and Qknn algorithms. Bench reports the MC estimates of the closed-form formula (6.3.3).

γ

Y&R Hybrid-LaterQ Hybrid-Now Qknn Bench 1.0 -0.402 -0.456 -0.460 -0.461 -0.464 0.5 -0.466 -0.495 -0.507 -0.508 -0.509 0.1 -0.573 -0.572 -0.579 -0.581 -0.586 0.0 -0.620 -1.000 -1.000 -1.000 -1.000

took one hour to achieves a relative error of 0.11%, using a 4-cores 3GHz intel Core i7 CPU. We want to highlight the fact that the algorithm presented in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] only needed 330 seconds to provide a relative error of 0.17%. However, to our experience, it is difficult to reduce the relative error from 0.17% to 0.11% using their algorithm. Also, we believe that the computation time of our algorithm can easily be alleviated; some ideas in that direction are discussed in Section 6.4.

We also considered the same problem in dimension d = 2, for which we plotted the first component of X w.r.t. time in Figure 6.3, for five different paths of the Brownian motion, where for each ω, the agent follows either the naive (α = 0) or the Hybrid-Now strategy. One can see that both strategies are very similar when the terminal time is far; but the Hybrid-Now strategy clearly forces X to get closer to 0 when the terminal time gets closer, in order to reduce the terminal cost.

Implementation details on the algorithms presented in Section 6.2:

As one can guess from the representation of v in (6.3.2), it is probably optimal to drive the process X around 0. Hence we decided to take µ n := ( nT N ) 1/2 N d (0, I d ) as a training measure at time n to learn the optimal strategy and value function at time n, for n = 0, . . . , N -1.

Test 2 Tests of the algorithms have also been run in dimension 1 with the terminal cost g(x) = -x γ 1 0≤x≤1 -1 1≤x and γ ∈ (0, 1). This problem has already been considered in [START_REF] Richou | Etude théorique et numérique des équations différentielles stochastiques rétrogrades[END_REF], where the author used the algorithm presented in [START_REF] Richou | Numerical simulation of BSDEs with drivers of quadratic growth[END_REF], which is based on the BSDE representation of the PDE (6.3.1). Their estimates of the value function at time 0 and state 0, when γ = 1, 0.5, 0.1, 0, are available in [START_REF] Richou | Etude théorique et numérique des équations différentielles stochastiques rétrogrades[END_REF], and have been reported in column Y &R of Table 6.1. Also, the exact values for the value function have been computed for these values of γ by Monte Carlo using the closed-form formula (6.3.3), and are reported in the column Bench of Table 6.1. Tests of the Hybrid-Now and Hybrid-LaterQ algorithms have been run, and the estimates of the value function at time 0 and state x = 0 are reported in the Hybrid-Now and Hybrid-LaterQ columns. We also tested Qknn and reported its results in column Qknn. Note that Qknn is particularly well-suited to 1-dimensional control problems. In particular, it is not time-consuming since the dimension of the state space is d=1. Actually, it provides the fastest results, which is not surprising since the other algorithms need time to learn the optimal strategy and value function through gradient-descent method at each time step n = 0, . . . , N -1. Moreover, Table 6.1 reveals that Qknn is the most accurate algorithm on this example, probably because it uses local methods in space to estimate the conditional expectation that appears in the expression of the Q-value.

We end this paragraph by giving some implementation details for the different algorithms as part of Test 2:

• Y&R: The algorithm Y&R converged only when using a Lipschitz version of g. The following approximation was used to obtained the results in Table 6.1:

g N (x) = g(x) if x ∈ [0, N -1 1-γ ] -N x otherwise.
• Hybrid-Now: We used N = 40 time steps for the time-discretization of [0, T ]. The value functions and optimal controls at time n = 0, . . . , N -1 are estimated using neural networks with 3 hidden layers and 10+5+5 neurons. strategy estimated by the Hybrid-Now algorithm (opt) and the naive strategy α = 0 (bench). The dimension of the semilinear control problem has been set to d=2. Observe that, as expected, the strategy designed by the Hybrid-Now algorithm is to not influence the diffusion of X when the terminal time is far in order to avoid any running cost; and to try to make X small when the terminal time gets close, in order to minimize the terminal cost.

• Hybrid-LaterQ: We used N = 40 time steps for the time-discretization of [0, T ]. The value functions and optimal controls at time n = 0, . . . , N -1 are estimated using neural networks with 3 hidden layers containing 10+5+5 neurons; and 51 points for the quantization of the exogenous noise.

• Qknn: We used N = 40 time steps for the time-discretization of [0, T ]. We take 51 points to quantize the exogenous noise, ε n ∼ N (0, 1), for n = 0, . . . , N ; and decided to use the 200 points of the optimal grid of N 2 (0, 1) for the state space discretization.

The main conclusion regarding the results in this semilinear PDE problem is that Hybrid-Now provides better estimates of the solution to the PDE in dimension d=100 than the previous results available in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] but requires more time to do so.

Hybrid-Now and Hybrid-Later provide better results than those available in [START_REF] Richou | Numerical simulation of BSDEs with drivers of quadratic growth[END_REF] to solve the PDE in dimension 2; but are outperformed by Qknn, which is arguably very accurate.

Option hedging

Our second example comes from a classical hedging problem in finance. We consider an investor who trades in q stocks with (positive) price process (P n ) n , and we denote by (α n ) valued in A ⊂ R q the amount held in these assets on the period (n, n + 1]. We assume for simplicity that the price of the riskless asset is constant equal to 1 (zero interest rate). It is convenient to introduce the return process as: R n+1 = diag(P n ) -1 (P n+1 -P n ), n = 0, . . . , N -1, so that the self-financed wealth process of the investor with a portfolio strategy α, and starting from some capital w 0 , is governed by

W α n+1 = W α n + α n .R n+1 , n = 0, . . . , N -1, W α 0 = w 0 .
Given an option payoff h(P N ), the objective of the agent is to minimize over her portfolio strategies α her expected square replication error

V 0 = inf α∈A E h(P N ) -W α N ,
where is a convex function on R. Assuming that the returns R n , n = 1, . . . , N are i.i.d, we are in a (q + 1)-dimensional framework of Section 5.1 with X α = (W α , P ) with ε n = R n valued in E ⊂ R q , with the dynamics function

F (w, p, a, r) = w + a.r p + diag(p)r, x = (w, p) ∈ R × R q , a ∈ R q , r ∈ E,
the running cost function f = 0 and the terminal cost g(w, p) = (h(p) -w). We test our algorithm in the case of a square loss function, i.e. (w) = w 2 , and when there is no portfolio constraints A = R q , and compare our numerical results with the explicit solution derived in [START_REF] Bertsimas | Hedging derivative securities and incomplete markets: an ε-arbitrage approach[END_REF]: denote by ν(dr) the distribution of R n , by ν = E[R n ] = rν(dr) its mean, and by M2 = E[R n R n ] assumed to be invertible; we then have

V n (w, p) = K n w 2 -2Z n (p)w + C n (p)
where the functions K n > 0, Z n (p) and C n (p) are given in backward induction, starting from the terminal condition

K N = 1, Z N (p) = h(p), C N (p) = h 2 (p),
and for n = N -1, . . . , 0, by

K n = K n+1 1 -ν M -1 2 ν , Z n (p) = Z n+1 (p + diag(p)r)ν(dr) -ν M -1 2 Z n+1 (p + diag(p)r)rν(dr), C n (p) = C n+1 (p + diag(p)r)ν(dr) - 1 K n+1 Z n+1 (p + diag(p)r)rν(dr) M -1 2 Z n+1 (p + diag(p)r)rν(dr) , so that V 0 = K 0 w 2 0 -2Z 0 (p 0 )w 0 + C 0 (p 0 )
, where p 0 is the initial stock price. Moreover, the optimal portfolio strategy is given in feedback form by α * n = a * n (W * n , P n ), where a * n (w, s) is the function

a * n (w, p) = M -1 2 Z n+1 (p + diag(p)r)rν(dr) K n+1 -νw ,
and W * is the optimal wealth associated with α * , i.e., W * n = W α * n . Moreover, the initial capital w * 0 that minimizes V 0 = V 0 (w 0 , p 0 ), and called (quadratic) hedging price is given by

w * 0 = Z 0 (p 0 ) K 0 .
Test Take N = 6, and consider one asset q = 1 with returns modeled by a trinomial tree:

ν(dr) = π + δ r+ + π 0 δ 0 + π -δ r-, π 0 + π + + π -= 1,
with r + = 5%, r -= -5%, π + = 60%, π -= 30%. Take p 0 = 100, and consider the call option h(p) = (p -κ) + with κ = 100. The price of this option is defined as the initial value of the portfolio that minimizes the terminal quadratic loss of the agent when the latter follows the optimal strategy associated with the initial value of the portfolio. In this test, we want to determine the price of the call and the associated optimal strategy using different algorithms.

Remark 6.3.1. The option hedging problem is linear-quadratic, hence belongs to the class of problems where the agent has ansatzes on the optimal control and the value function. Indeed, we expect here the optimal control to be affine w.r.t. w and the value function to be quadratic w.r.t. w. For these kind of problems, the algorithms presented in Section 6.2 can easily be adapted so that the expressions of the estimators satisfy the ansatzes. See (6.3.4) and (6.3.5) for the option hedging problem.

Chapter 6. Neural networks algorithms for control problems: numerical applications Hybrid-LaterQ (green dashes). We draw the value function in red for comparison. One can observe that all the algorithms estimate the price to be 4.5, but Hybrid-LaterQ is better than Hybrid-Now at reducing the quadratic risk.

Numerical results

In Figure 6.4, we plot the value function at time 0 w.r.t w 0 , the initial value of the portfolio, when the agent follows the theoretical optimal strategy (benchmark), and the optimal strategy estimated by the Hybrid-Now or Hybrid-LaterQ algorithms. We perform forward Monte Carlo using 10,000 samples to approximate the lower bound of the value function at time 0 (see [START_REF] Henry-Labordère | Deep primal-dual algorithm for BSDEs: applications of machine learning to CVA and IM[END_REF] for details on how to get an approximation of the upper-bound of the value function via duality). One can observe that while all the algorithms give a call option price approximately equal to 4.5, Hybrid-LaterQ clearly provides a better strategy than Hybrid-Now to reduce the quadratic risk of the terminal loss.

We plot in Figure 6.5 three different paths of the value of the portfolio w.r.t the time n, when the agent follows either the theoretical optimal strategy (red), or the estimated one using Hybrid-Now (blue) or Hybrid-LaterQ (green). We set w 0 = 100 for these simulations.

Comments on Hybrid-Now and Hybrid-LaterQ The Option Hedging problem belongs to the class of the linear-quadratic control problems for which we expect the optimal control to be affine w.r.t. w and the value function to be quadratic w.r.t. w. It is then natural to consider the following classes of controls A M and functions F M to properly approximate the optimal controls and the values functions at time n=0, . . . , N -1:

A M := (w, p) → A(x; β) • 1, w ; β ∈ R p , (6.3.4) F M := (w, p) → Φ(x; θ) • 1, w, w 2 ; θ ∈ R p , (6.3.5)
where β describes the parameters (weights+bias) associated with the neural network A and θ describes those associated with the neural network Φ. The notation stands for the transposition, and • for the inner product. Note that there are 2 (resp. 3) neurons in the output layer of A (resp. Φ), so that the inner product is well-defined in (6.3.5) and (6.3.4).

Valuation of energy storage

We present a discrete-time version of the energy storage valuation problem studied in [START_REF] Carmona | Valuation of energy storage: an optimal switching approach[END_REF]. We consider a commodity (gas) that has to be stored in a cave, e.g. salt domes or aquifers. The manager of

Numerical applications

Hybrid-Now Hybrid-LaterQ Figure 6.5 -Three simulations of the agent's wealth w.r.t. the time n when, for each ω, the latter follows the theoretical optimal strategy (red), the estimated one using Hybrid-Now (blue) and the one using Hybrid-LaterQ (green). We took w0 = 100. Observe that the process is driven similarly to the optimally controlled process, when the agent follows the estimated optimal strategy using Hybrid-LaterQ or Hybrid-Now.

such a cave aims to maximize the real options value by optimizing over a finite horizon N the dynamic decisions to inject or withdraw gas as time and market conditions evolve. We denote by (P n ) the gas price, which is an exogenous real-valued Markov process modeled by the following mean-reverting process:

P n+1 = p(1 -β) + βP n + ξ n+1 ,
where β < 1, and p > 0 is the stationary value of the gas price. The current inventory in the gas storage is denoted by (C α n ) n and depends on the manager's decisions represented by a control process α = (α n ) valued in {-1, 0, 1}: α n = 1 (resp. -1) means that she injects (resp. withdraws) gas with an injection (resp. withdrawal) rate a in (C

α n ) (resp. a out (C α n )) requiring (causing) a purchase (resp. sale) of b in (C α n ) ≥ a in (C α n ) (resp. b out (C α n ) ≤ a out (C α n ))
, and α n = 0 means that she is doing nothing. The difference between b in and a in (resp. b out and a out ) indicates gas loss during injection/withdrawal. The evolution of the inventory is then governed by

C α n+1 = C α n + h(C α n , α n ), n = 0, . . . , N -1, C α 0 = c 0 ,
where we set

h(c, a) =    a in (c) for a = 1 0 for a = 0 -a out (c) for a = -1,
and we have the physical inventory constraint:

C α n ∈ [C min , C max ], n = 0, . . . , N.
The running gain of the manager at time n is f (P n , C α n , α t ) given by

f (p, c, a) =    -b in (c)p -K 1 (c) for a = 1 -K 0 (c) for a = 0 b out (c)p -K -1 (c) for a = -1,
Chapter 6. Neural networks algorithms for control problems: numerical applications and K i (c) represents the storage cost in each regime i = -1, 0, 1. The problem of the manager is then to maximize over α the expected total profit

J(α) = E N -1 n=0 f (P n , C α n , α n ) + g(P N , C α N ) ,
where a common choice for the terminal condition is

g(p, c) = -µp(c 0 -c) + ,
which penalizes for having less gas than originally, and makes this penalty proportional to the current price of gas (µ > 0). We are then in the 2-dimensional framework of Section 5.1 with X α = (P, C α ), and the set of admissible controls in the dynamic programming loop is given by:

A n (c) = a ∈ {-1, 0, 1} : c + h(c, a) ∈ [C min , C max ], c ∈ [C min , C max ] , n = 0, . . . , N -1.
Test We fixed the parameters as follows, to run our numerical tests:

a in (c) = b in (c) = 0.06, a out (c) = b out (c) = 0.25 K i (c) = 0.01c C max = 8, C min = 0, c 0 = 4, p = 5, β = 0.5, ξ n+1 N (0, σ 2
) with σ 2 = 0.05, and µ = 2 in the terminal penalty function, N =30.

Numerical results

We plotted in Figure 6.6 the estimates of the value function at time 0 w.r.t. a in using Qknn, as well as the reward function (6.3.8) associated to the naive do-nothing strategy α = 0 (see Bench in figure 6.6). As expected, the naive strategy performs well when a in is small compared to a out , since, in this case, it takes time to fill the cave, so that the agent is likely to do nothing in order to avoid any penalization at terminal time. When a in is of the same order as a out , it is easy to fill up and empty the cave, so the agent has more freedom to buy and sell gas in the market without worrying about the terminal cost. Observe that the value function is not monotone, due to the fact that the C component in the state space takes its value in a bounded and discrete set (see (6.3.7)). Table 6.2 provides the estimates of the value function using the ClassifPI, Hybrid-Now and Qknn algorithms. Observe first that the estimates provided by Qknn are larger than those provided by the other algorithms, meaning that Qknn outperforms the other algorithms. The second best algorithm is ClassifPI, while Hybrid-Now performs poorly and clearly suffers from instability, due to the discontinuity of the running rewards w.r.t. the control variable.

Finally, Figures 6.7, 6.8, 6.9 provide the optimal decisions w.r.t. (P, C) at times 5, 10, 15, 20, 25, 29 estimated respectively by the Qknn, ClassifPI and Hybrid-Now algorithms. As expected, one can observe on each plot that the optimal strategy is to inject gas when the price is low, to sell gas when the price is high, and to make sure to have a volume of gas greater than c 0 in the cave when the terminal time is getting closer to minimize the terminal cost.

Let us now comment on the implementation of the algorithms:

• Qknn: Table 6.2 shows that once again, due to the low-dimensionality of the problem, Qknn provides the best value function estimates. The estimated optimal strategies, shown on Figure 6.7, are very good estimates of the theoretical ones. The three decision regions on Figure 6.7 are natural and easy to interpret: basically it is optimal to sell when the price is high, and to buy when it is low. However, a closer look reveals that the waiting region (where it is optimal to do nothing) has an unusual triangular-based shape, due essentially to the discreteness of the space on which the C component of the state space takes its values. We expect this shape to be very hard to reproduce with the DNN-based algorithms proposed in Section 6.2. estimated by Qknn, by running a forward Monte Carlo with a sample of size 100,000 (blue). We also plotted the cost functional associated with the naive passive strategy α = 0 (Bench). See that for small values of ain such as 0.06, doing nothing is a reasonable strategy. Observe also that the value function is not monotone w.r.t. ain which is due to the dynamics of C (6.3.7).

Numerical applications

• ClassifPI: As shown on Figure 6.8, the ClassifPI algorithm manages to provide accurate estimates for the optimal controls at time n = 0, . . . , N -1. However, the latter is not able to catch the particular triangular-based shape of the waiting region, which explains why Qknn performs better.

• Hybrid-Now: As shown on Figure 6.9, Hybrid-Now only manages to provide relatively poor estimates, compared to ClassifPI and Qknn, of the three different regions at time n = 0, . . . , N -1. In particular, the regions suffer from instability.

We end this paragraph by providing some implementation details for the different algorithms we tested.

• Qknn: We used the extension of Algorithm 6.5 introduced in the paragraph "semi-linear interpolation" of the Section 3.2.2. in [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF] and used projection of each state on its k=2-nearest neighbors to get an estimate of the value function which is continuous w.r.t. the control variable at each time n = 0, . . . , N -1. The optimal control is computed at each point of the grids using the Brent algorithm, which is a deterministic function optimizer already implemented in Python4 .

• Implementation details for the neural network-based algorithms: We use neural networks with two hidden layers, ELU activation functions5 and 20 + 20 neurons . The output layer contains 3 neurons with softmax activation function for the ClassifPI algorithm and no activation function for the Hybrid-Now one. We use a training set of size M=60,000 at each time step. Note that given the expression of the terminal cost, the ReLU activation functions (Rectified Linear Units) could have been deemed a better choice to capture the shape of the value functions, but our tests revealed that ELU activation functions provide better results. At time n = 0, . . . , N -1, we took µ n = U(C min , C max ) as training measure.

We did not use the pre-train trick discussed in Section 6.2.2.3, which explains the instability in the decisions that can be observed in Figure 6.9.

The main conclusion of our numerical comparisons on this energy storage example is that ClassifPI, the DNN-based classification algorithm designed for stochastic control problems with discrete control space, appears to be more accurate than the more general Hybrid-Now. Nevertheless, ClassifPI was not able to capture the unusual triangle-based shape of the optimal control as well as Qknn did.

Microgrid management

Finally, we consider a discrete-time model for power microgrid inspired by the continuous-time models developed in [START_REF] Heymann | Continuous optimal control approaches to microgrid energy management[END_REF] and [START_REF] Jiang | An approximate dynamic programming algorithm for monotone value functions[END_REF]; see also [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF]. The microgrid consists of a photovoltaic (PV) power plant, a diesel generator and a battery energy storage system (BES), hence using a mix of fuel and renewable energy sources. These generation units are decentralized, i.e., installed at a rather small scale (a few kW power), and physically close to electricity consumers. The PV produces electricity from solar panels with a generation pattern (P n ) n depending on the weather conditions. The diesel generator has two modes: on and off. Turning it on consumes fuel, and produces an amount of power α n . The BES can store energy for later use but has limited capacity and power. The aim of the microgrid management is to find the optimal planning that meets the power demand, denoted by (D n ) n , while minimizing the operational costs due to the diesel generator. We denote by R n = D n -P n , the residual demand of power: when R n > 0, one should provide power through diesel or battery, and when R n < 0, one can store the surplus power in the battery.

The optimal control problem over a fixed horizon N is formulated as follows. At any time n = 0, . . . , N -1, the microgrid manager decides the power production of the diesel generator, either by turning it off: α n = 0, or by turning it on, hence generating a power α n valued in [A min , A max ] with 0 < A min < A max < ∞. There is a fixed cost κ > 0 associated with switching from the on/off mode to the other one off/on, and we denote by M α n the mode valued in {0 = off, 1 = on} of the generator right before time n, i.e., M α n+1 = 1 αn =0 . When the diesel generator and renewable provide a surplus of power, the excess can be stored into the battery (up to its limited capacity) for later use, and in case of power insufficiency, the battery is discharged for satisfying the power demand. The input power process I α for charging the battery is then given by

I α n = (α n -R n ) + ∧ (C max -C α n ),
where C max is the maximum capacity of the battery with current charge C α , while the output power process O α for discharging the battery is given by

O α n = (R n -α n ) + ∧ C α n .

Numerical applications

Figure 6.9 -Estimated optimal decisions at times 5, 10, 15, 20, 25, 29 w.r.t. (P,C) for the energy storage valuation problem using Hybrid-Now. Injection (a=-1) in purple, store (a=0) in blue and withdraw (a=1) in yellow. Observe the instability in the decisions which come from the fact that we did not pre-train the neural networks (see Section 6.2.2.3)

Numerical applications

The control space {0} ∪ [A min , A max ] is a mix between a discrete space and a continuous space, which is challenging for algorithms with neural networks. We actually use a mixture of classification and standard DNN for the control: (p 0 (x; θ), π(x; β)) valued in [0, 1] × [A min , A max ], where p 0 (x; θ) is the probability of turning off in state x, and π(x; β) is the amount of power when turning on with probability 1 -p 0 (x; θ). In other words,

X n+1 = F (X n , 0, ε n+1 ) with probability p 0 (X n ; θ n ) F (X n , π(X n ; β n ), ε n+1 ) with probability 1 -p 0 (X n ; θ n )
The pseudo-code of algorithm, designed for this problem, is written in Algorithm 6.6, and we refer to the latter as ClassifHybrid, in the sequel. Note in particular that it is an Hybrid version of ClassifPI. Algorithm 6.6 ClassifHybrid

Input: the training distributions (µ n ) N -1 n=0 ; 1: Set VN = g; 2: for n = N -1, . . . , 0 do 3: Compute ( β0 n , β1 n ) ∈ argmax β 0 ,β 1 E p 0 (X n ; β 0 ) f (X n , 0) + Vn+1 f ( X0 n+1 + (1 -p 0 (X n ; β 0 )) f (X n , π(X n ; β 1 )) + Vn+1 X1,β 1 n+1 , where X n µ n , X0 n+1 = F (X n , 0, ε n+1 ), and X1,β 1 n+1 = F (X n , π(X n ; β 1 ), ε n+1 ) 4: Compute θn ∈ argmin θ E p 0 X n ; β0 n f (X n , 0) + Vn+1 f ( X0 n+1 -Φ(.; θ) 2 + 1 -p 0 X n ; β0 n f (X n , π(X n ; β 1 n )) + Vn+1 X1, β1 n n+1 -Φ(.; θ) 2 ; 5:
Set Vn = Φ(.; θn ); Vn is the estimate of the value function at time n 6: end for Output:

-estimate of the optimal strategy (â n ) N -1 n=0 ; -estimate of the value function ( Vn ) N -1 n=0 ;

Test We took the following parameters to compare Qknn and ClassifHybrid:

N = 30 or 200, R = 0.1, = 0.9, σ = 0.2, C min = 0, C max = 1 or 4, C 0 = 0, K = 2, γ = 2, κ = 0.2, Q -= 10, R 0 = 0.1, A min = 0.05, A max = 10 Q + = 1000.
Results Figure 6.10 shows the Qknn-estimated optimal decisions to take at times n = 1, 10, 28 in the cases where m = M n = 0 and m = M n = 1. If the generator is off at time n, i.e. m = 0, the blue curve separates the region where it is optimal to keep it off and the one where it is optimal to generate power. If the generator is on at time n, i.e. m = 1, the blue curve separates the region where it is optimal to turn it off and the one where it is optimal to generate power. A colorscale is Chapter 6. Neural networks algorithms for control problems: numerical applications

Discussion and conclusion

Our proposed algorithms are well-designed and provide accurate estimates of optimal control and value function associated with various high-dimensional control problems. Also, when tested on low-dimensional problems, they performed as well as the Monte Carlo-based or quantization-based methods, which have shown their efficiency in low dimension, see e.g. [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF] and [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF].

The presented algorithms suffer from a rather high time-consuming cost due to the expensive training of 2(N -1) neural networks to learn the value functions and optimal controls at times n = 0, . . . , N -1. However, the agent can easily alleviate the computation time. A first trick consists in reducing the number of neural networks by partially or totally ignoring the dynamic programming principle (DPP), as it has been done e.g. in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF]. The use of one unique Recurrent Neural Networks (RNN) (in the case where the DPP is totally ignored) or a few of them (in the partialignored case) can also be considered to learn the optimal controls, either all at the same time (first case), or group by group in a backward way (second case). We refer to [START_REF] Chan-Wai-Nam | Machine Learning for semi linear PDEs[END_REF] for algorithms in this spirit. Another trick consists in learning faster the value functions and optimal controls at times n = 0, . . . , N -1 by pre-training the neural networks. The way to proceed in that direction is to initialize at time n the weights and bias of the value function estimator Vn to the ones of Vn+1 . We then rely on the continuity of the value function w.r.t. the time n to expect that the weights will not change much from time n to n + 1, hence do not need to be train for a long time anymore. Another benefit from the pre-training task is to get the stability of the estimates w.r.t. the time, which is also a very pleasant feature.

Part III

Deep learning for BSDEs and PDEs

Chapter 7 Some machine learning schemes for high-dimensional nonlinear PDEs1 Abstract We propose new machine learning schemes for solving high dimensional nonlinear partial differential equations (PDEs). Relying on the classical backward stochastic differential equation (BSDE) representation of PDEs, our algorithms estimate simultaneously the solution and its gradient by deep neural networks. These approximations are performed at each time step from the minimization of loss functions defined recursively by backward induction. The methodology is extended to variational inequalities arising in optimal stopping problems. We analyze the convergence of the deep learning schemes and provide error estimates in terms of the universal approximation of neural networks. Numerical results show that our algorithms give very good results till dimension 50 (and certainly above), for both PDEs and variational inequalities problems. For the PDEs resolution, our results are very similar to those obtained by the recent method in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] when the latter converges to the right solution or does not diverge. Numerical tests indicate that the proposed methods are not stuck in poor local minima as it can be the case with the algorithm designed in [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF], and no divergence is experienced. The only limitation seems to be due to the inability of the considered deep neural networks to represent a solution with a too complex structure in high dimension.

Key words: Deep neural networks, nonlinear PDEs in high dimension, optimal stopping problem, backward stochastic differential equations.

MSC Classification: 60H35, 65C20, 65M12.

Neural networks as function approximators

Like the second methodology, our approach relies on BSDE representation of the PDE and deep learning approximations: we first discretize the BSDE associated to the PDE by an Euler scheme, but in contrast with [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF], we adopt a classical backward resolution technique. On each time step, we propose to use some machine learning techniques to estimate simultaneously the solution and its gradient by minimizing a loss function defined recursively by backward induction, and solving this local problem by a stochastic gradient algorithm. Two different schemes are designed to deal with the local problems:

(1) The first one tries the estimate the solution and its gradient by a neural network.

(2) The second one tries only to approximate the solution by a neural network while its gradient is estimated directly with some numerical differentiation techniques.

The proposed methodology is then extended to solve some variational inequalities, i.e., free boundary problems related to optimal stopping problems.

Convergence analysis of the two schemes for PDEs and variational inequalities is provided and shows that the approximation error goes to zero as we increase the number of time steps and the number of neurons/layers whenever the gradient descent method used to solve the local problems is not trapped in a local minimum.

In the last part of the paper, we test our algorithms on different examples. When the solution is easy to represent by a neural network, we can solve the problem in quite high dimension (at least 50 in our numerical tests). We show that the proposed methodology is superior to the algorithm proposed in [HJE17] that often does not converge or is trapped in a local minimum far away from the true solution. We then show that when the solution has a very complex structure, we can still solve the problem but only in moderate dimension: the neural network used is not anymore able to represent the solution accurately in very high dimension. Finally, we illustrate numerically that the method is effective to solve some system of variational inequalities: we consider the problem of American options and show that it can be solved very accurately in high dimension (we tested until 40).

The outline of the paper is organized as follows. In Section 7.2, we give a brief and useful reminder for neural networks. We describe in Section 7.3 our two numerical schemes and compare with the algorithm in [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF]. Section 7.4 is devoted to the convergence analysis of our machine learning algorithms, and we present in Section 7.5 several numerical tests.

Neural networks as function approximators

Multilayer (also called deep) neural networks are designed to approximate unknown or large class of functions. In contrast to additive approximation theory with weighted sum over basis functions, e.g. polynomials, neural networks rely on the composition of simple functions, and appear to provide an efficient way to handle high-dimensional approximation problems, in particular thanks to the increase in computer power for finding the "optimal" parameters by (stochastic) gradient descent methods.

We shall consider feedforward (or artificial) neural networks, which represent the basic type of deep neural networks. Let us recall some notation and basic definitions that will be useful in our context. We fix the input dimension d 0 = d (here the dimension of the state variable x), the output dimension d 1 (here d 1 = 1 for approximating the real-valued solution to the PDE, or d 1 = d for approximating the vector-valued gradient function), the global number L + 1 ∈ N \ {1, 2} of layers with m , = 0, . . . , L, the number of neurons (units or nodes) on each layer: the first layer is the input layer with m 0 = d, the last layer is the output layer with m L = d 1 , and the L -1 layers between are called hidden layers, where we choose for simplicity the same dimension m = m, = 1, . . . , L -1.

A feedforward neural network is a function from R d to R d1 defined as the composition

x ∈ R d -→ A L • • A L-1 • . . . • • A 1 (x) ∈ R d1 .
(7.2.1)

Chapter 7. Deep learning schemes for high-dimensional PDEs

Here A , = 1, . . . , L are affine transformations: A 1 maps from R d to R m , A 2 , . . . , A L-1 map from R m to R m , and A L maps from R m to R d1 , represented by

A (x) = W x + β ,
for a matrix W called weight, and a vector β called bias term, : R → R is a nonlinear function, called activation function, and applied component-wise on the outputs of A , i.e., (x 1 , . . . , x m ) = ( (x 1 ), . . . , (x m )). Standard examples of activation functions are the sigmoid, the ReLu, the Elu, tanh.

All these matrices W and vectors β , = 1, . . . , L, are the parameters of the neural network, and can be identified with an element θ ∈ R Nm , where

N m = L-1 =0 m (1 + m +1 ) = d(1 + m) + m(1 + m)(L -2) + m(1 + d 1 )
is the number of parameters, where we fix d 0 , d 1 , L, but allow growing number m of hidden neurons. We denote by Θ m the set of possible parameters: in the sequel, we shall consider either the case when there are no constraints on parameters, i.e., Θ m = R Nm , or when the total variation norm of the neural networks is smaller than γ m , i.e.,

Θ m = Θ γ m := θ = (W , β ) : |W l | ≤ γ m , = 1, . . . , L , with γ m ∞, as m → ∞.
We denote by Φ m (.; θ) the neural network function defined in (7.2.1), and by N N d,d1,L,m (Θ m ) the set of all such neural networks Φ m (.; θ) for θ ∈ Θ m , and set

N N d,d1,L = m∈N N N d,d1,L,m (Θ m ) = m∈N N N d,d1,L,m (R Nm ),
as the class of all neural networks within a fixed structure given by d, d 1 , L and .

The fundamental result of Hornick et al. [START_REF] Hornik | Multilayer feedforward networks are universal approximators[END_REF] justifies the use of neural networks as function approximators:

Universal approximation theorem (I): N N d,d1,L is dense in L 2 (ν) for any finite measure ν on R d , whenever is continuous and non-constant.

Moreover, we have a universal approximation result for the derivatives in the case of a single hidden layer, i.e. L = 2, and when the activation function is a smooth function, see [START_REF] Hornik | Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks[END_REF].

Universal approximation theorem (II): Assume that is a (non constant) C k function. Then, N N d,d1,2 approximates any function and its derivatives up to order k, arbitrary well on any compact set of R d .

Deep learning-based schemes for semi-linear PDEs

The starting point for our probabilistic numerical schemes to the PDE (7.1.1) is the well-known (see [START_REF] Pardoux | Adapted solution of a backward stochastic differential equation[END_REF]) nonlinear Feynman-Kac formula via the pair

(Y, Z) of F-adapted processes valued in R × R d , solution to the BSDE Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s , 0 ≤ t ≤ T, (7.3.1)
related to the solution u of (7.1.1) via

Y t = u(t, X t ), 0 ≤ t ≤ T,
and when u is smooth:

Z t = σ (t, X t )D x u(t, X t ), 0 ≤ t ≤ T.
Chapter 7. Deep learning schemes for high-dimensional PDEs -Initialize from an estimation U

(1)

N of u(t N , .) with U

(1)

N = g -For i = N -1, . . . , 0, given U (1) i+1 , use a pair of deep neural networks (U i (.; θ), Z i (.; θ)) ∈ N N d,1,L,m (R Nm ) × N N d,d,L,m (R Nm )
for the approximation of (u(t i , .), σ (t i , .)D x u(t i , .)), and compute (by SGD) the minimizer of the expected quadratic loss function

     L(1) i (θ) := E U (1) i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 θ * i ∈ arg min θ∈R Nm L1 i (θ). (7.3.6)
Then, update:

U (1) i = U i (.; θ * i ), and set Z (1) i = Z i (.; θ * i ). (2) Second version: -Initialize with U (2) N = g -For i = N -1, . . . , 0, given U (2)
i+1 , use a deep neural network U i (.; θ) ∈ N N d,1,L,m (Θ m ), and compute (by SGD) the minimizer of the expected quadratic loss function

           L(2) i (θ) := E U (2) i+1 (X ti+1 )- F (t i , X ti , U i (X ti ; θ), σ (t i , X ti ) Dx U i (X ti ; θ), ∆t i , ∆W ti ) 2 θ * i ∈ arg min θ∈Θm L2 i (θ), (7.3.7)
where Dx U i (.; θ) is the numerical differentiation of U i (.; θ). Then, update:

U (2) i = U i (.; θ * i ), and set Z (2) i = σ (t i , .) Dx U i (.; θ * i ).
Remark 7.3.1. For the first version of the scheme, one can use independent neural networks, respectively for the approximation of u(t i , .) and for the approximation of σ (t i , .)D x u(t i , .). In other words, the parameters are divided into a pair θ = (ξ, η) and we consider neural networks U i (.; ξ) and Z i (.; η).

In the sequel, we refer to the first and second version of the new scheme above as DBDP1 and DBDP2, where the acronym DBDP stands for deep learning backward dynamic programming.

The intuition behind DBDP1 and DBDP2 is the following. For simplicity, take f = 0, so that F (t, x, y, z, h, ∆) = y + z ∆. The solution u to the PDE (7.1.1) should then approximately satisfy (see (7.3.4))

u(t i+1 , X ti+1 ) ≈ u(t i , X ti ) + D x u(t i , X ti ) σ(t i , X ti )∆W ti .
Consider the first scheme DBDP1, and suppose that at time i + 1, U

i+1 is an estimation of u(t i+1,. ). The quadratic loss function at time i is then approximately equal to

L(1) i (θ) ≈ E u(t i+1 , X ti+1 ) -U i (X ti ; θ) -Z i (X ti ; θ) ∆W ti 2 ≈ E u(t i , X ti ) -U i (X ti ; θ) 2 + ∆t i σ (t i , X ti )D x u(t i , X ti ) -Z i (X ti ; θ) 2 .

Convergence analysis

• we solve smaller problems that are less prone to be trapped in local mimimizer. The memory needed in [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] can be a problem when taking too many time steps.

• at each time step, we initialize the weights and bias of the neural network to the weights and bias of the previous time step treated : this methodology always used in iterative solvers in PDE methods permits to have a starting point close to the solution, and then to avoid local minima too far away from the true solution. Besides the number of gradient iterations to achieve is rather small after the first resolution step.

The small disadvantage is due to the Tensorflow structure. As it is done in python, the global graph creation takes much time as it is repeated for each time step and the global resolution is a little bit time consuming : as the dimension of the problem increases, the time difference decreases and it becomes hard to compare the computational time for a given accuracy when the dimension is above 5.

Extension to variational inequalities: scheme RDBDP

Let us consider a variational inequality in the form min -∂ t u -Lu -f (t, x, u, σ D x u), u -g = 0, t ∈ [0, T ), x ∈ R d , u(T, x) = g(x), x ∈ R d .

(7.3.8) which arises, e.g., in optimal stopping problem and American option pricing in finance. It is known, see e.g. [START_REF] Karoui | Reflected Solutions of Backward SDEs, and related obstacle problems for PDEs[END_REF], that such variational inequality is related to reflected BSDE of the form

Y t = g(X T ) + T t f (s, X s , Y s , Z s ) ds - T t Z s dW s + K T -K t , (7.3.9)
Y t ≥ g(X t ), 0 ≤ t ≤ T, where K is an adapted non-decreasing process satisfying T 0 Y t -g(X t ) dK t = 0.

The extension of our DBDP1 scheme for such variational inequality, and refereed to as RDBDP scheme, becomes

• Initialize U N = g
• For i = N -1, . . . , 0, given U i+1 , use a pair of (multilayer) neural network (U i (.; θ), Z i (.; (7.3.10) Then, update: U i = max U i (.; θ * i ), g], and set Ẑi = Z(.; θ * i ). Remark 7.3.2. As for the scheme DBDP1, two neural networks can be used respectively for the approximation of u(t i , .) and for the approximation of σ (t i , .)D x u(t i , .).

Convergence analysis

The main goal of this section is to prove convergence of the DBDP schemes towards the solution (Y, Z) to the BSDE (7.3.1) (or reflected BSDE (7.3.9) for variational inequalities), and to provide a rate of convergence that depends on the approximation errors by neural networks.

Convergence of DBDP1

We assume the standard Lipschitz conditions on µ and σ, which ensures the existence and uniqueness of an adapted solution X to the forward SDE (7.1.2) satisfying for any p > 1, E sup We shall make the standing usual assumptions on the driver f and the terminal data g.

(H1) (i) There exists a constant [f ] L > 0 such that the driver f satisfies:

|f (t 2 , x 2 , y 2 , z 2 ) -f (t 1 , x 1 , y 1 , z 1 )| ≤ [f ] L |t 2 -t 1 | 1/2 + |x 2 -x 1 | + |y 2 -y 1 | + |z 2 -z 1 | ,
for all (t 1 , x 1 , y 1 , z 1 ) and (t 2 , x 2 , y 2 , z 2 )

∈ [0, T ] × R d × R × R d . Moreover, sup 0≤t≤T
|f (t, 0, 0, 0)| < ∞.

(ii) The function g satisfies a linear growth condition.

Recall that Assumption (H1) ensures the existence and uniqueness of an adapted solution (Y, Z) to (7.3.1) satisfying

E sup 0≤t≤T |Y t | 2 + T 0 |Z t | 2 dt < ∞.
From the linear growth condition on f in (H1), and (7.4.1), we also see that where E i denotes the conditional expectation given F ti . Since Z is a L 2 -projection of Z, we know that ε Z (π) converges to zero when |π| goes to zero. Moreover, as shown in [START_REF] Zhang | A numerical scheme for BSDE's[END_REF], when the terminal condition g is also Lipschitz, we have ε Z (π) = O(|π|).

Convergence analysis

Let us first investigate the convergence of the scheme DBDP1 in (7.3.6), and define (implicitly)

   V ti := E i U (1)
i+1 (X ti+1 ) + f (t i , X ti , V ti , Z ti )∆t i Z ti :=

1 ∆ti E i U (1)
i+1 (X ti+1 )∆W ti , (7.4.5) for i = 0, . . . , N -1. Notice that V ti is well-defined for |π| small enough (recall that f is Lipschitz) by a fixed point argument. By the Markov property of the discretized forward process (X ti ) i=0,...,N , we note that there exists some deterministic functions vi and ẑi s.t.

V ti = vi (X ti ), and Z ti = ẑi (X ti ), i = 0, . . . , N -1. (7.4.6)

Moreover, by the martingale representation theorem, there exists an R d -valued square integrable process ( Z t ) t such that

U (1)
i+1 (X ti+1 ) = V ti -f (t i , X ti , V ti , Z ti )∆t i + ti+1 ti Z s dW s , (7.4.7)

and by Itô isometry, we have

Z ti = 1 ∆t i E i ti+1 ti
Z s ds , i = 0, . . . , N -1.

Let us now define a measure of the (squared) error for the DBDP1 scheme by E ( U (1) , Z (1) ), (Y, Z) := max i=0,...,N -1

E Y ti -U (1) i (X ti ) 2 + E N -1 i=0 ti+1 ti Z t -Z (1)
i (X ti ) 2 dt .

Our first main result gives an error estimate of the DBDP1 scheme in terms of the L 2 -approximation errors of vi and ẑi by neural networks U i and Z i , i = 0, . . . , N -1, assumed to be independent (see Remark 7.3.1), and defined as

ε N ,v i := inf ξ E vi (X ti ) -U i (X ti ; ξ) 2 , ε N ,z i := inf η E ẑi (X ti ) -Z i (X ti ; η) 2 .
Here, we fix the structure of the neural networks with input dimension d, output dimension d 1 = 1 for U i , and d 1 = d for Z i , number of layers L, and m neurons for the hidden layers, and the parameters vary in the whole set R Nm where N m is the number of parameters. From the universal approximation theorem (I) ([HSW89]), we know that ε N N,v i and ε N N,z i converge to zero as m goes to infinity, hence can be made arbitrary small for sufficiently large number of neurons.

Theorem 7.4.1. (Consistency of DBDP1) Under (H1), there exists a constant C > 0, independent of π, such that

E ( U (1) , Z (1) ), (Y, Z) ≤ C E g(X T ) -g(X T ) 2 + |π| + ε Z (π) + N -1 i=0 N ε N ,v i + ε N ,z i .
(7.4.8)

Remark 7.4.1. The error contributions for the DBDP1 scheme in the r.h.s. of estimation (7.4.8) consists of four terms. The first three terms correspond to the time discretization of BSDE, similarly as in [START_REF] Bouchard | Discrete-time approximation and Monte-Carlo simulation of backward stochastic differential equations[END_REF], [START_REF] Gobet | A regression-based Monte Carlo method to solve backward stochastic differential equations[END_REF], namely (i) the strong approximation of the terminal condition (depending on the forward scheme and the terminal data g), and converging to zero, as |π| goes to zero, with a rate |π| when g is Lipschitz by (7.4.2) (see [START_REF] Avikainen | On irregular functionals of SDEs and the Euler scheme[END_REF] for irregular g), (ii) the strong approximation of the forward Euler scheme, and the L 2 -regularity of Y , which gives a convergence of order |π|, (iii) the L 2 -regularity of Z, which converges to zero, as |π| goes to zero, with a rate |π| when g is Lipschitz.

Finally, the better the neural networks are able to approximate/learn the functions vi and ẑi at each time i = 0, . . . , N -1, the smaller is the last term in the error estimation.

Chapter 7. Deep learning schemes for high-dimensional PDEs Proof of Theorem 7.4.1.

In the following, C will denote a positive generic constant independent of π, and that may take different values from line to line.

Step 1. Fix i ∈ {0, . . . , N -1}, and observe by (7.3.1), (7.4.5) that

Y ti -V ti = E i Y ti+1 -U (1)
i+1 (X ti+1 ) + E i ti+1 ti f (t, X t , Y t , Z t ) -f (t i , X ti , V ti , Z ti ) dt .

By using Young inequality: (a + b) 2 ≤ (1 + γ∆t i )a 2 + (1 + 1 γ∆ti )b 2 for some γ > 0 to be chosen later, Cauchy-Schwarz inequality, the Lipschitz condition on f in (H1), and the estimation (7.4.2) on the forward process, we then have

E Y ti -V ti 2 ≤ (1 + γ∆t i )E E i Y ti+1 -U (1) i+1 (X ti+1 ) 2 + 4[f ] 2 L ∆t i 1 + 1 γ∆t i |∆t i | 2 + E ti+1 ti Y t -V ti 2 dt + E ti+1 ti Z t -Z ti 2 dt ≤ (1 + γ∆t i )E E i Y ti+1 - Û(1)
i+1 (X ti+1 ) 2 (7.4.9)

+ 4 [f ] 2 L γ (1 + γ∆t i ) C|π| 2 + 2∆t i E Y ti -V ti 2 + E ti+1 ti Z t -Z ti 2 dt ,
where we use in the last inequality the L 2 -regularity (7.4.4) of Y .

Recalling the definition of Z as a L 2 -projection of Z, we observe that By multiplying equation (7.3.1) between t i and t i+1 by ∆W ti , and using Itô isometry, we have together with (7.4.5)

∆t i Zti -Z ti = E i ∆W ti Y ti+1 -U (1) i+1 (X ti+1 ) + E i ∆W ti ti+1 ti f (t, X t , Y t , Z t ) dt = E i ∆W ti Y ti+1 -U (1) i+1 (X ti+1 ) -E i Y ti+1 -U (1)
i+1 (X ti+1 )

+ E i ∆W ti ti+1 ti f (t, X t , Y t , Z t ) dt .
By Cauchy-Schwarz inequality, and law of iterated conditional expectations, this implies Step 2. By using Young inequality in the form:

∆t i E Zti -Z ti 2 ≤ 2d E Y ti+1 -U (1) i+1 (X ti+1 ) 2 -E E i Y ti+1 -U (
(a + b) 2 ≥ (1 -|π|)a 2 + (1 -1 |π| )b 2 ≥ (1 -|π|)a 2 - 1 |π| b 2 , we have E Y ti -V ti 2 = E Y ti -U (1)
i (X ti ) + U

(1)

i (X ti ) -V ti 2 ≥ (1 -|π|)E Y ti -U (1) i (X ti ) 2 - 1 |π| E U (1)
i (X ti ) -V ti 2 .

(7.4.13)

By plugging this last inequality into (7.4.12), we then get for |π| small enough

E Y ti -U (1) i (X ti ) 2 ≤ (1 + C|π|)E Y ti+1 -U (
+ CN E V ti -U (1)
i (X ti ) 2 .

From discrete Gronwall's lemma (or by induction), and recalling the terminal condition Y t N = g(X T ), U

i (X t N ) = g(X T ), the definition ε Z (π) of the L 2 -regularity of Z, and (7.4.3), this yields max i=0,...,N -1

E Y ti -U (1)
i (X ti ) 2 ≤ CE g(X T ) -g(X T ) 2 + C|π| + Cε Z (π)

+ CN N -1 i=0 E V ti -U (1)
i (X ti ) 2 . (7.4.14)

Step 3.

= E V ti -U i (X ti ; ξ) + f (t i , X ti , U i (X ti ; ξ), Z i (X ti ; η)) -f (t i , X ti , V ti , Z ti ) ∆t i 2 + ∆t i E Z ti -Z i (X ti ; η) 2 .
By using Young inequality: (a + b) 2 ≤ (1 + γ∆t i )a 2 + (1 + 1 γ∆ti )b 2 , together with the Lipschitz condition on f in (H1), we clearly see that

Li (θ) ≤ (1 + C∆t i )E V ti -U i (X ti ; ξ) 2 + C∆t i E Z ti -Z i (X ti ; η) 2 .
(7.4.16)

On the other hand, using Young inequality in the form: (a + b) 2 ≥ (1 -γ∆t i )a 2 + (1 -1 γ∆ti )b 2 ≥ (1 -γ∆t i )a 2 -1 γ∆ti b 2 , together with the Lipschitz condition on f , we have

Li (θ) ≥ (1 -γ∆t i )E V ti -U i (X ti ; ξ) 2 - 2∆t i [f ] 2 L γ E V ti -U i (X ti ; ξ) 2 + E Z ti -Z i (X ti ; η) 2 + ∆t i E Z ti -Z i (X ti ; η) 2 .
Chapter 7. Deep learning schemes for high-dimensional PDEs

By choosing γ = 4[f ] 2 L , this yields Li (θ) ≥ (1 -C∆t i )E V ti -U i (X ti ; ξ) 2 + ∆t i 2 E Z ti -Z i (X ti ; η) 2 .
(7.4.17)

Step 4. Fix i ∈ {0, . . . , N -1}, and take θ * i = (ξ * i , η * i ) ∈ arg min θ L(1) i (θ) so that U

(1) i = U i (.; ξ * i ), and Z

(1) i = Z i (.; η * i ). By (7.4.15), notice that θ * i ∈ arg min θ Li (θ). From (7.4.17) and (7.4.16), we then have for all θ = (ξ, η)

(1 -C∆t i )E V ti -U (1) i (X ti ) 2 + ∆t i 2 E Z ti -Z (1) i (X ti ) 2 ≤ Li (θ * i ) ≤ Li (θ) ≤ (1 + C∆t i )E V ti -U i (X ti ; ξ) 2 + C∆t i E Z ti -Z i (X ti ; η) 2 .
For |π| small enough, and recalling (7.4.6), this implies

E V ti -U (1) i (X ti ) 2 + ∆t i E Z ti -Z (1) i (X ti ) 2 ≤ Cε N ,v i + C∆t i ε N ,z i . ( 7
E Y ti -U (1)
i (X ti ) 2 ≤ CE g(X T ) -g(X T ) 2 + C|π| + Cε Z (π)

+ C N -1 i=0 N ε N ,v i + ε N ,z i , (7.4.19)
which proves the consistency of the Y -component in (7.4.8).

Step 5. Let us finally prove the consistency of the Z-component. From (7.4.10) and (7.4.11), we have for any i = 0, . . . , N -1:

E ti+1 ti Z t -Z ti 2 dt ≤ E ti+1 ti Z t -Zti 2 dt + 2d|π|E ti+1 ti |f (t, X t , Y t , Z t )| 2 dt + 2d E Y ti+1 -U (1) i+1 (X ti+1 ) 2 -E E i Y ti+1 -U (
+ 2d N -1 i=0 E Y ti -U (1) i (X ti ) 2 -E E i Y ti+1 -U (1)
i+1 (X ti+1 )

2
where we change the indices in the last summation. Now, from (7.4.9), (7.4.13), we have

2d E Y ti -U (1) i (X ti ) 2 -E E i Y ti+1 -U (1) i+1 (X ti+1 ) 2 ≤ 1 + γ|π| 1 -|π| -1 E E i Y ti+1 - Û(1) i+1 (X ti+1 ) 2 + 8d[f ] 2 L γ 1 + γ|π| 1 -|π| C|π| 2 + |π|E Y ti -V ti 2 + E ti+1 ti Z t -Z ti 2 dt + 2d |π|(1 -|π|) E U (1)
i (X ti ) -V ti 2 .

Convergence analysis

We now choose γ = 24d[f ] 2 L so that

8d[f ] 2 L γ
(1 + γ|π|)/(1 -|π|) ≤ 1/2 for |π| small enough, and by plugging into (7.4.20), we obtain (note also that (1 + γ|π|)/(1 -|π|) -1 = O(|π|)):

1 2 E N -1 i=0 ti+1 ti Z t -Z ti 2 dt ≤ ε Z (π) + C|π| + C max i=0,...,N E Y ti -U (1) i (X ti ) 2 + 1 2 |π| N -1 i=0 E Y ti -V ti 2 + CN N -1 i=0 E U (1) i (X ti ) -V ti 2 ≤ Cε Z (π) + C|π| + C max i=0,...,N E Y ti -U (1) i (X ti ) 2 + CN N -1 i=0 E U (1) i (X ti ) -V ti 2 ≤ CE g(X T ) -g(X T ) 2 + C|π| + Cε Z (π) + C N -1 i=0 N ε N ,v i + ε N ,z i , (7.4.21)
where we used (7.4.12) and (7.4.3) in the second inequality, and (7.4.18) and (7.4.19) in the last inequality.

By writing that

E ti+1 ti Z t -Z (1) i (X ti ) 2 dt ≤ 2E ti+1 ti Z t -Z ti 2 dt + 2∆t i E Z ti -Z (1)
i (X ti ) 2 , and using (7.4.18), (7.4.21), we obtain after summation over i = 0, . . . , N -1, the required error estimate for the Z-component as in (7. 4.19), and this ends the proof.

Convergence of DBDP2

We shall consider neural networks with one hidden layer, m neurons with total variation smaller than γ m (see Section 7. Let us investigate the convergence of the scheme DBDP2 in (7.3.7) with neural networks in N N d,1,2,m (Θ γ m ), and define for i = 0, . . . , N -1:

   V ti := E i U (2)
i+1 (X ti+1 ) + f (t i , X ti , V ti , Z ti )∆t i = vi (X ti ),

Z ti := 1 ∆ti E i U (2)
i+1 (X ti+1 )∆W ti = ẑi (X ti ).

(7.4.24)

A measure of the (squared) error for the DBDP2 scheme is defined similarly as in DBDP1 scheme:

E ( U (2) , Z (2) ), (Y, Z) := max i=0,...,N -1 E Y ti -U (2) i (X ti ) 2 + E N -1 i=0 ti+1 ti Z t -Z (2)
i (X ti ) 2 dt .

Our second main result gives an error estimate of the DBDP2 scheme in terms of the L 2approximation errors of vi and its derivative (which exists under assumption detailed below) by neural networks U i ∈ N N d,1,2,m (Θ γ m ), i = 0, . . . , N -1, and defined as

ε N ,m i := inf θ∈Θ γ m E vi (X ti ) -U i (X ti ; θ) 2 + ∆t i E σ (t i , X ti ) D x vi (X ti ) -D x U i (X ti ; θ) 2 ,
which are expected to be small in view of the universal approximation theorem (II), see discussion in Remark 7.4.2.

We also require the additional conditions on the coefficients:

(H2) (i) The functions x → µ(t, .), σ(t, .) are C 1 with bounded derivatives uniformly w. Proof. For simplicity of notations, we assume d = 1, and only detail the arguments that differ from the proof of Theorem 7.4.8. From (7.4.24), and the Euler scheme (7.3.3), we have vi (x) = ṽi (x) + ∆t i f (t i , x, vi (x), ẑi (x)), ṽi (x) := E ûi+1 (X x ti+1 ) , x ∈ R d , ẑi (x) = 1 ∆t i E ûi+1 (X x ti+1 )∆W ti , X x ti+1 = x + µ(t i , x)∆t i + σ(t i , x)∆W ti .

Under assumption (H2)(i), and recalling that ûi+1 = U i+1 (.; θ * i+1 ) is C 2 with bounded derivatives, we see that ṽi is C 1 with D x ṽi (x) = E 1 + D x µ(t i , x)∆t i + D x σ(t i , x)∆W ti D x ûi+1 (X x ti+1 ) = E D x ûi+1 (X x ti+1 ) + ∆t i R i (x) (7.4.26) R i (x) := D x µ(t i , x)E D x ûi+1 (X x ti+1 ) + σ(t i , x)D x σ(t i , x)E D 2 x ûi+1 (X x ti+1 ) ,

where we use integration by parts in the second equality. Similarly, we have

      
ẑi (x) = σ(t i , x)E D x ûi+1 (X x ti+1 ) , D x ẑi (x) = D x σ(t i , x)E D x ûi+1 (X x ti+1 ) + σ(t i , x)E D 2 x ûi+1 (X x ti+1 ) + ∆t i σ(t i , x)G i (x) G i (x) := D x µ(t i , x)E D 2

x ûi+1 (X x ti+1 ) + σ(t i , x)D x σ(t i , x)E D 3 x ûi+1 (X x ti+1 ) .

E V ti -U (2) i (X ti ) 2 + ∆t i E Z ti -Z (2) i (X ti ) 2 ≤ CE vi (X ti ) -U i (X ti ; θ) 2 + C∆t i E Z ti -σ(t i , X ti ) Dx U i (X ti ; θ) 2 ,
for all θ ∈ Θ N , and then with (7.4.28), and by definition of ε N N,v,2 i :

E V ti -U (2) i (X ti ) 2 + ∆t i E Z ti -Z (2)
E Y ti -U (2)
i (X ti ) 2 ≤ CE g(X T ) -g(X T ) 2 + C|π| + Cε Z (π)

+ CN N -1 i=0 E V ti -U (2)
i (X ti ) 2 .

Plugging (7.4.29) into this last inequality, together with (7.4.22), gives the required estimation (7.4.25) for the Y -component. Finally, by following the same arguments as in Step 5 in the proof of (7.4.1), we obtain the estimation (7.4.25) for the Z-component.

Remark 7.4.2. The universal approximation theorem (II) [START_REF] Hornik | Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks[END_REF] is valid on compact sets, and one cannot conclude a priori that the error of network approximation ε N ,m i converge to zero as m goes to infinity. Instead, we have to proceed into two steps: (i) Localize the error by considering

ε N ,m,K i := inf θ∈Θ γ m E ∆ i (X ti ; θ)1 |Xt i |≤K ,
where we set ∆ i (x; θ) := |v i (x) -U i (x; θ)| 2 + ∆t i σ (t i , x) D x vi (x) -D x U i (x; θ) 2 .

(ii) Consider an increasing family of neural networks Θ γ N -1 m ⊂ . . . ⊂ Θ γ i m ⊂ . . . ⊂ Θ γ 0 m on which to minimize the approximation errors by backward induction at times t i , i = N -1, . . . , 0, and where, γ i m is defined by γ i m := γ ϕ N -1-i (m) , with ϕ : N → N an increasing function, and where we use the notation ϕ k := ϕ • ... • ϕ (composition k times). where . 2 is the L 2 -norm on (Ω, F, P).

Remark 7.4.3. The estimation (7.4.32) implies that max i=0,...,N -1

E Y ti -Ûi (X ti ) 2 ≤ C |π| + N N -1 i=0 εN i ,
which is of the same order than the error estimate in Theorem 7.4.1 when g is Lipschitz.

Proof. Let us introduce the discrete-time approximation of the reflected BSDE

     Y π t N = g(X t N ) Ỹ π ti = E i [Y π ti+1 ] + f (t i , X ti , Ỹ π ti )∆t i Y π
ti = max Ỹ π ti ; g(X ti ) , i = 0, . . . , N -1. Fix i = 0, . . . , N -1. From (7.4.31), (7.4.33), we have Li (θ) := Ṽti -U i (X ti ; θ) + f (t i , X ti , U i (X ti ; θ)) -f (t i , X ti , Ṽti ) ∆t i 2 .

| Ỹ π ti -Ṽti | ≤ E i Y π ti+1 -U i+1 (X ti+1 ) + ∆t i f (t i , X ti , Ỹ π ti ) -f (t i , X ti , Ṽti ) ≤ E i Y π ti+1 -U i+1 (X ti+1 ) + [f ] L ∆t i | Ỹ π ti -Ṽti |,
From the Lipschitz condition on f , and by Minkowski inequality, we have for all θ

(1 -[f ] L ∆t i ) Ṽti -U i (X ti ; θ) 2 ≤ Li (θ) ≤ (1 + [f ] L ∆t i ) Ṽti -U i (X ti ; θ) 2 .
Chapter 7. Deep learning schemes for high-dimensional PDEs Take now θ * i ∈ arg min θ Li (θ) = arg min θ Li (θ). Then, from the above relations, we have

(1 -[f ] L ∆t i ) Ṽti -U i (X ti ; θ * i ) 2 ≤ (1 + [f ] L ∆t i ) Ṽti -U i (X ti ; θ)
2 , for all θ, and so Ṽti -U i (X ti ; ξ * i ) 2 ≤ (1 + C|π|) εN i .

(7.4.36)

Recalling that V ti = max[ Ṽti ; g(X ti )], and Ûi (X ti ) = max[U i (X ti ; ξ * i ); g(X ti )], and since | max(a, c)max(b, c)| ≤ |a -b|, we deduce that

E V ti -Ûi (X ti ) 2 ≤ C ε N N,ṽ i + ∆t i ε N N,z i .
By taking θ = θ * i in (7.4.35), recalling that U i (X ti ) = max[U i (X ti ; θ * i ); g(X ti )], Y π ti = max[ Ỹ π ti ; g(X ti )], and since | max(a, c) -max(b, c)| ≤ |a -b|, we obtain by using (7.4.36)

Y π ti -U i (X ti ) 2 ≤ (1 + C|π|) Y π ti+1 -U i+1 (X ti+1 ) 2 + εN i .
By induction, this yields max i=0,...,N -1

Y π ti -U i (X ti ) 2 ≤ C N -1 i=0 εN i ,
and we conclude with (7.4.34).

We finally turn to the general case when f may depend on z, and study the convergence of the RDBDP scheme (7.3.10) towards the variational inequality (7.

E Y ti -U i (X ti ) 2 + E N -1 i=0 ti+1 ti Z t -Z i (X ti ) 2 dt .
Let us define from the scheme (7.3.10)

        
Ṽti := E i U i+1 (X ti+1 ) + f (t i , X ti , Ṽti , Zti )∆t i = ṽi (X ti ), Zti := 1 ∆t i E i U i+1 (X ti+1 )∆W ti = zi (X ti ),

V ti := max[ Ṽti ; g(X ti )], i = 0, . . . , N -1.

(7.4.37)

Our final main result gives an error estimate of the RDBDP scheme in terms of the L 2 -approximation errors of ṽi and zi by neural networks U i and Z i , i = 0, . . . , N -1, assumed to be independent (see Remark 7.3.1), and defined as

ε N ,ṽ i := inf ξ E ṽi (X ti ) -U i (X ti ; ξ) 2 , ε N ,z i := inf η E zi (X ti ) -Z i (X ti ; η) 2 .
The result is obtained under one of the following additional assumptions (H3) g is C 1 , and g, D x g are Lipschitz.

or (H4) σ is C 1 , with σ, D x σ both Lipschitz, and g is C 2 , with g, D x g, D 2

x g all Lipschitz. Proof. Let us introduce the discrete-time approximation of the reflected BSDE

               Y π t N = g(X t N ) Z π ti = 1 ∆t i E i Y π ti+1 ∆W ti , Ỹ π ti = E i [Y π ti+1 ] + f (t i , X ti , Ỹ π ti , Z π ti )∆t i Y π ti = max Ỹ π ti ; g(X ti
) , i = 0, . . . , N -1. Fix i = 0, . . . , N -1. By writing that

Ỹ π ti -Ṽti = E i Y ti+1 -U i+1 (X ti+1 ) + ∆t i f (t i , X ti , Ỹ π ti , Z π ti ) -f (t i , X ti , Ṽti , Zti ) ,
and proceeding similarly as in Step 1 in the proof of Theorem 7.4.1, we have by Young inequality and Lipschitz condition on f

E Ỹ π ti -Ṽti 2 ≤ (1 + γ∆t i )E E i Y π ti+1 -U i+1 (X ti+1 ) 2 + 2 [f ] 2 L γ 1 + γ∆t i ∆t i E Ỹ π ti -Ṽti 2 + ∆t i E Z π ti -Zti 2 .
(7.4.41) From (7.4.37), (7.4.39), Cauchy-Schwarz inequality, and law of iterated conditional expectations, we have similarly as in Step 1 in the proof of Theorem 7.4.1:

∆t i E Z π ti -Zti 2 ≤ 2d E Y π ti+1 -U i+1 (X ti+1 ) 2 -E E i Y π ti+1 -U i+1 (X ti+1 ) 2 .
Then, by plugging into (7.4.41) and choosing γ = 4d[f ] 2 L , we have for |π| small enough:

E Ỹ π ti -Ṽti 2 ≤ (1 + C|π|)E Y π ti+1 -U i+1 (X ti+1 ) 2 .
Next, by using Young inequality as in Step 2 in the proof of Theorem 7.4.1, we obtain for all θ = (ξ, ζ):

E Ỹ π ti -U i (X ti ; ξ) 2 ≤ (1 + C|π|)E Y π ti+1 -U i+1 (X ti+1 ) 2 + CN E Ṽti -U i (X ti ; ξ) 2 . (7.4.42)
On the other hand, by the martingale representation theorem, there exists an R d -valued square integrable process ( Zt ) t such that U i+1 (X ti+1 ) = Ṽti -f (t i , X ti , Ṽti , Zti )∆t i + Zt dt , and Li (θ) := E Ṽti -U i (X ti ; ξ) + f (t i , X ti , U i (X ti ; ξ), Z i (X ti ; η)) -f (t i , X ti , Ṽti , Zti ) ∆t i 2

+ ∆t i E Zti -Z i (X ti ; η) 2 .
By the same arguments as in Step 3 in the proof of Theorem 7.4.1, using Lipschitz condition on f and Young inequality, we show that for all θ = (ξ, η)

(1 -C∆t i )E Ṽti -U i (X ti ; ξ) 2 + ∆t i 2 E Zti -Z i (X ti ; η) 2 ≤ Li (θ) ≤ (1 + C∆t i )E Ṽti -U i (X ti ; ξ) 2 + C∆t i E Zti -Z i (X ti ; η) 2 .
By taking θ * i = (ξ * i , η * i ) ∈ arg min θ Li (θ) = arg min θ Li (θ), it follows that for |π| small enough

E Ṽti -U i (X ti ; ξ * i ) 2 + ∆t i E Zti -Z i (X ti ; η * i ) 2 ≤ Cε N ,ṽ i + C∆t i ε N ,z i .
By plugging into (7.4.42), recalling that U i (X ti ) = max[U i (X ti ; ξ * i ); g(X ti )], Y π ti = max[ Ỹ π ti ; g(X ti )], and since | max(a, c) -max(b, c)| ≤ |a -b|, we obtain

E Y π ti -U i (X ti ) 2 ≤ (1 + C|π|)E Y π ti+1 -U i+1 (X ti+1 ) 2 + CN ε N ,ṽ i + ∆t i ε N ,z

Numerical results

In the first two subsections, we compare our schemes DBDP1 (7.3.6), DBDP2 (7.3.7) and the scheme proposed by [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] on some examples of PDEs and BSDEs.

We first test our algorithms on some PDEs with bounded solutions and quite a simple structure (see section 7.5.1), and then try to solve some PDEs with unbounded solutions and more complex structures (see section 7.5.2). Our goal is to emphasize that solutions with simple structure easily represented by a neural network can be evaluated by our method even in very high-dimension, whereas the solution with complex structure can only be evaluated in moderate dimension.

Finally, we apply the scheme described in section 7.3.3 to an American option problem and show its accuracy in high dimension (see section 7.5.3).

If not specified, we use in the sequel a fully connected feedforward network with two hidden layers, and d + 10 neurons on each hidden layer, to implement our schemes (7.3.6) and (7.3.7). We choose tanh as activation function for the hidden layers in order to avoid some explosion while calculating the numerical gradient Z in scheme (7.3.7) and choose identity function as activation function for the output layer. We renormalize the data before entering the network. We use Adam Optimizer, implemented in TensorFlow and mini-batch with 1000 trajectories for the stochastic gradient descent.

Numerical results

Table 7.1 -Estimate of u(0, x 0 ) where d = 1 and x 0 = 1. Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 1.4686938.

Averaged value Standard deviation DBDP1

1.46332 0.01434 DBDP2 1.4387982 0.01354 7.5.1 PDEs with bounded solution and simple structure

We begin with a simple example in dimension one. It is not hard to find test cases where the scheme proposed in [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] fails even in dimension one. In fact the latter scheme works well for small maturities and with a starting point close to the solution.

It is always interesting to start by testing schemes in dimension one as one can easily compare graphically the numerical results to the theoretical solution. Then we take some examples in higher dimensions and show that our method seems to work well when the dimension increases higher.

An example in 1D

We take the following parameters for the BSDE problem defined by (7.1.2) and (7. -1 2 sin(x) cos(x)e T -t 2 + 1 2 (yz) 2 g(x) = cos(x).

for which, the explicit analytic solution is equal to u(t, x) = e T -t 2 cos(x).

We want to estimate the solution u and its gradient D x u from our schemes. This example is interesting, because with T = 1, the method proposed in [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF], initializing u(0, .) as the solution of the associated linear problem associated (f = 0) and randomly initializing D x u(0, .) works very well. However, for T = 2, the method in [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] always fails on our test whatever the choice of the initialization: the algorithm is either trapped in a local minimum when the initial learning rate associated to the gradient method is too small or explodes when the learning rate is taken higher. This numerical failure is not dependent on the considered network: using some LSTM networks as in [START_REF] Chan-Wai-Nam | Machine Learning for semi linear PDEs[END_REF] gives the same result.

Because of the high non-linearity, we discretize the BSDE using N = 240 time steps, and implemented hidden layers with d + 10 = 11 neurons. Figure 7.1 (resp. Figure 7.2) depicts the estimated functions u(t, .) and D x u(t, .) estimated from DBDP1 (resp. DBDP2) scheme. u(t, .) and its estimate at time t = 1.

Z and its estimate at time t = 1. u(t, .) and its estimate at time t = 0.0091. Z and its estimate at time t = 0.0091. We take N = 120 in the Euler scheme, and d + 10 neurons for each hidden layer. We take 1000 trajectories in mini batch, use data renormalization, and check the loss convergence every 50 iterations. For this small maturity, the scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] generally converges, and we give the results obtained with the same network and initializing the scheme with the linear solution of the problem. Results in dimension 5 to 50 are given in Tables 7.2, 7.3, 7.4 and 7.5. Both schemes (7.3.6) and (7.3.7) work well with results very close to the solution and close to the results calculated by the scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF]. As the dimension increases, scheme (7.3.6) seems to be the most accurate.

Remark 7.5.1. In dimension 50, the initial learning rate in scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] is taken small in order to avoid a divergence of the method. In fact, running the test 3 times (with 10 runs each time), we observed convergence of the algorithm two times, and in the last test: one of the ten run exploded, and another one clearly converged to a wrong solution. Notice that the structure of the solution is more complex than in the first example. We aim at evaluating the solution at x = 0.51I d . We take 120 time steps for the Euler time discretization and d + 10 neurons in each hidden layers. As shown in Figures 7.3 and 7.4 as well as in Table 7.6, the three schemes provide accurate and stable results in dimension d = 1.

In dimension 2, the three schemes provide very accurate and stable results, as shown in Figures 7.5 and 7.6, as well as in Table 7.7.

Above dimension 3, the scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] always explodes no matter the chosen initial learning rate and the activation function for the hidden layers (among the tanh, ELU, ReLu and sigmoid ones). Besides, taking 3 or 4 hidden layers does not improve the results.

We reported the results obtained in dimension d = 5 and 8 in Table 7.8 and 7.9. Scheme (7.3.6) seems to work better than scheme (7.3.7) as the dimension increases. Note that the standard deviation increases with the dimension of the problem.

When d ≥ 10, schemes (7.3.6) and (7.3.7) both fail at providing correct estimates of the solution, as shown in Table 7.10. Increasing the number of layers or neurons does not improve the result.

Averaged value Standard deviation DBDP1 0.5715359 0.0038 DBDP2 0.5707974 0.00235 Scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] 0.57145 0.0006 Table 7.7 -Estimate of u(0, x 0 ), where d = 2 and x 0 = 0.51I 2 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 0.570737.

Chapter 7. Deep learning schemes for high-dimensional PDEs u(t, .) and its estimate at time t = 0.5. Z and its estimate at time t = 0.5 u(t, .) and its estimate at time t = 0.0085. Z and its estimate at time t = 0.0085.

Application to American options

Consider the stock price X t = (X 1 t , . . . , X d t ) of d assets with the following dynamics under the risk neutral probability measure: dX i t = rX i t dt + σ i X i t dW i t , where W . = (W 1 . , . . . , W d . ) is a d-dimensional Brownian Motion, σ = (σ 1 , . . . , σ d ) ∈ R d , and r is the risk-free rate.

The value at time t of an American option with payoff g and maturity T is given by: u(t, x) = sup Error on solution at date t = 0.5.

Error on solution at date t = 0.0085. NC NC Table 7.8 -Estimate of u(0, x 0 ), where d = 5 and x 0 = 0.51I 5 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 0.87715.

as proved e.g. in [START_REF] Jaillet | Variational Inequalities and the Pricing of American Options[END_REF].

Let us define the change of function v by: u(t, x) = e rt v(t, log(x)), which is solution of the following variational inequality min (-∂ t v -Lv, v -ĝ) = 0, on [0, T ) × R d v(T, .) = ĝ, on R d , (7.5.3) where ĝ(t, x) = e -rt g(e x ),

Lv = 1 2 d i=1 σ 2 i D 2 xi v ii + d i=1 (r - 1 2 σ 2 i )D xi v i .
In this section, we test the scheme described in section 7.3.3 on (7.5.3) in the special case of a geometrical put with strike K = 1 , T = 1, r = 0.05, X i 0 = 1, σ i = 0.2 for i = 1 to d, and payoff (K -d i=1 X i t ) + , as considered previously in [START_REF] Bouchard | Monte-Carlo valuation of American options: facts and new algorithms to improve existing methods[END_REF]. In dimension d, the case boils down to the resolution of an American option in dimension d = 1 so that it can be very accurately estimated e.g. with a tree-based method. Results given in Table 7.11 show that scheme (7.3.10) is very accurate for the pricing of American options.

Averaged value Standard deviation DBDP1

1.169441 0.02537 DBDP2 1.0758344 0.00780 Scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] NC NC Table 7.9 -Estimate of u(0, x 0 ), where d = 8 and x 0 = 0.51I 8 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 1.1603167.

Numerical results

Averaged value Standard deviation DBDP1 -0.3105 0.02296 DBDP2 -0.3961 0.0139 Scheme [START_REF] Han | Overcoming the curse of dimensionality: Solving highdimensional partial differential equations using deep learning[END_REF] NC NC

  , µ, a) = b 0 (µ, a) + b 1 (µ, a)x, σ(x, µ, a) = ϑ 0 (µ, a) + ϑ 1 (µ, a)x, σ 0 (x, µ, a) = γ 0 (µ, a) + γ 1 (µ, a)x, We also assume that the terminal and instantaneous rewards are polynomial w.r.t. x, i.e. f (x, µ, a) = f 0 (µ, a) + p k=1 f k (µ, a)x k , g(x, µ) = g 0 (µ) + p k=1 g k (µ)x k , with p ≥ 1.

  12 σ is often called activation function in the deep learning literature. Classical examples of activation functions are: the ReLu function: x → max(x, 0), the ELU function: x → exp(x) -1 if x ≤ 0 x otherwise , or the sigmoid function: x → 1 1+exp(-x) .

3 :

 3 while max(∆θ n , ∆β n ) > ε do 4:

  Estimate of the optimal quantity of energy to get from the generator at time n = 1 w.r.t. the couple of state variables (R, C), in the case where m = 0 or m = 1, using the Classif-Hybrid algorithm.

Figure 1 . 2 -

 12 Figure 1.2 -Estimates of the optimal quantity of energy to get from the generator using Classif-Hybrid or Qknn.

  .4.1 page 225), and we present here the simplified version of the result: Theorem 1.3.1. (Consistence of DBDP1) Under classical regularity conditions on the coefficients of 1.3. Neural networks for PDEs and BSDEs the BSDE, there exists a constant C > 0, independent of the time discretization π, such that

b 0

 0 , µ, a) = b 0 (µ, a) + b 1 (µ, a)x, σ(x, µ, a) = ϑ 0 (µ, a) + ϑ 1 (µ, a)x, σ 0 (x, µ, a) = γ 0 (µ, a) + γ 1 (µ, a)x, Nous considérons aussi que la récompense terminale et la récompense instantanée sont polynomiaux enx f (x, µ, a) = f 0 (µ, a) + p k=1 f k (µ, a)x k , g(x, µ) = g 0 (µ) + p k=1 g k (µ)x k , avec p ≥ 1.Hypothèses sur la dépendance en µ : Nous supposons que les coefficients du modèle dépendent de µ à travers les p premiers moments de cette dernière, i.e., (µ, a) = b0 (μ 1 , μ2 , . . . , μp , a), b 1 (µ, a) = b1 (μ 1 , μ2 , . . . , μp , a) ϑ 0 (µ, a) = θ0 (μ 1 , μ2 , . . . , μp , a), ϑ 1 (µ, a) = θ1 (μ 1 , μ2 , . . . , μp , a) γ 0 (µ, a) = γ0 (μ 1 , μ2 , . . . , μp , a), γ 1 (µ, a) = γ1 (μ 1 , μ2 , . . . , μp , a) f k (µ, a) = fk (μ 1 , μ2 , . . . , μp , a), g k (µ) = ḡk (μ 1 , μ2 , . . . , μp ), k = 0, . . . , p, Chapter 2. Introduction (in French) où, pour µ ∈ P p (R), on note μk = x k µ(dx), k = 1, . . . , p.

  (a) Estimation de la quantité optimale d'énergie à se faire fournir au temps n = 1 en fonction du couple de variables d'état (R, C), dans le cas où m = 0 ou m = 1, en utilisant l'algorithme Qknn basé sur de la régression locale. La zone au-dessous de la courbe bleue est celle où il est optimal d'éteindre (ou de laisser éteint) le générateur. La zone au-dessus de la courbe bleue est celle où il est optimal de se faire délivrer de l'énergie.

  Estimation de la quantité optimale d'énergie à se faire fournir au temps n = 1 en fonction du couple de variables d'état (R, C), dans le cas où m = 0 ou m = 1, en utilisant l'algorithme Classif-Hybrid.

Figure 2 . 2 -

 22 Figure 2.2 -Quantité optimale d'énergie à se faire délivrer, estimée en utilisant Classif-Hybrid et Qknn.

  Estimations des Z ti pour i = 0, . . . , N ; La convergence de RDBDP est étudiée dans le Chapitre 7 (voir Théorème 7.4.4 page 235), et nous présentons ici une version simplifiée du résultat : Theorem 2.3.3. (Consistance of RDBDP) Sous des hypothèses classiques de régularité des coefficients, il existe une constante C > 0, indépendante de π, telle que

  3.2 présentés ci-dessus, le terme d'erreurs d'approximation suivant apparaît : Part I Quantization and regression-type methods for MDPs. Application to market making and Mckean-Vlasov control problems.

  Order Book; pure-jump controlled process; High-Frequency Trading; Queuing model; High-dimensional Stochastic Control; Markov Decision Process; Quantization; Local Regression Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 3.2 Model setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.2.1 Order book representation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.2.2 Market maker strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.3 Existence and characterization of the optimal strategy . . . . . . . . . . 67 3.3.1 Definition and well-posedness of the value function . . . . . . . . . . . . . . 67 3.3.2 Markov Decision Process formulation of the market-making problem . . . . 69 3.3.3 Proof of Theorem 3.3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.4 Numerical Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.4.1 Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 3.4.2 Presentation and rate of convergence of the Qknn algorithm . . . . . . . . . 76 Chapter 3. Algorithmic trading in a LOB model 3.4.3 Qknn agorithm applied to the order book control problem (3.3.1) . . . . . . 79 3.4.4 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 3.5 Model extension to Hawkes Processes . . . . . . . . . . . . . . . . . . . . 86 3.A From uncontrolled to controlled intensity . . . . . . . . . . . . . . . . . . 90 3.B Dynamics of the controlled order book (simplified version) . . . . . . . 92 3.B.1 Dynamics of Xt et Yt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.B.2 Dynamics of the at et bt . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 3.B.3 Dynamics of nat and nbt . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 3.B.4 Dynamics of pa and pb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.B.5 Dynamics of ra and rb . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 3.C Proof of Theorem 3.4.1 and Corollary 3.4.1 . . . . . . . . . . . . . . . . . 101 3.D Zador's Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

  But |na| ≤ √ M |a| + M and |nb| ≤ √ M |b| + M are bounded by a linear function of (a, b), also |ra| and |rb| are bounded by the universal constant √ M K, so assertion 4 in Definition 3.3.1 holds.

  Proposition 3.3.3. If b is a bounding function for (Z t ), then b(t, z) := b(z)e γ(z)(T -t) , with γ(z) = γ 0 (4K + 2)Λ 1 + |a| + |b| and γ 0 > 0 is a bounding function for the MDP, i.e. for all t ∈ [0, T ], z ∈ E, a ∈ A z , we have:

  .i.d. sequence of r.v. following law µ. Γ n should be seen as a training sampling to estimate the value function V n at time n.

  and where we denoted by [r] L the Lipschitz constant of r w.r.t. x.

  .4.10) It remains to plug (3.4.9) and (3.4.10) into (3.4.8) to complete the proof of Proposition 3.4.1.

Figure 3 . 4 -

 34 Figure 3.4 -Symmetrical intensities, Size of the grids: 100000. Short terminal time: T=1. Notice that the Qknn strategy reduces the variance of the P&L, but the expected wealth when following Qknn strategy (StratOpt2lim) is the same as the one following the naive strategy (Strat11).

Figure 3 . 5 -Figure 3

 353 Figure 3.5 -Symmetrical and constant intensities. Size of the grids: 6000. The computed optimal strategy is less efficient than the naive strategy.

Figure 3

 3 Figure 3.7 -P&L of the market maker following the optimal strategy and following the naive strategy 11. Symmetrical state dependent intensities. Long Terminal Time: T=10. Notice that the Qknn strategy does better than the naive strategy when the intensities are state dependent.

3. 5 .

 5 Figure 3.10 -P&L of the market maker who follows the optimal strategy and following the naive strategy 11. Long Terminal Time. Constant and symmetrical intensities for the arrivals of orders.Notice that the strategies computed by Qknn algorithm when taking A2lim performs as well as the one computed on the two best limits of the order book exclusively. Then, in this setting, placing orders only at the best-ask and best-bid seems to be the the optimal strategy.

  T ], and m = 1, . . . , D. then g(t, z) = G(z) -T t F (s, z, λ s ) ds, which finally leads to the following expression of T 1 (F ):

  z, λ s ds. (3.5.8) Replacing F by T 2 (f ) in (3.5.8), we get that f is fixed point of T 1 • T 2 if and only if: f (t, λ, z) + D m=1 T t λ m s f s, z, λ s ds = G(z) -e a m (z), λ s + α m ds. Chapter 3. Algorithmic trading in a LOB model Notice ∂f (s, λ s , z)e -e a m (z), λ s + α m , so that: f (t, λ, z) = G(z)e -e a m (z), λ s + α m ds = G(z)e -

  3.C. Proof of Theorem 3.4.1 and Corollary 3.4.1 Dynamics of rb:

t 3 .

 3 C Proof of Theorem 3.4.1 and Corollary 3.4.1

.C. 4 )

 4 Combining (3.C.3) and (3.C.4) leads toE (1 -µ (B(X n , η))) M ≤ e -1 N (η) M . (3.C.5) 3.C. Proof of Theorem 3.4.1 and Corollary 3.4.1

a∈A

  C.6) where c is defined as c := d d-2 c e 1/d , and where we used (3.C.5) and (3.C.2) to go from the second to the third line. It remains to take K = M d+1 points for the optimal quantization of the exogenous noise, and then take square root of equality (3.C.6), in order to derive (3.C.1).Lemma 3.C.2. Assume d ≥ 3, take K = M d+2 points for the optimal quantization of ε n , and let x ∈ E. Then it holds under (Hµ) and (HF), as M → +∞: Proj n+1 (F (x, a, εn )) -F (x, a, εn ) 2 , stands for the laterprojection error at state x.

  .C.7) Chapter 3. Algorithmic trading in a LOB model Moreover, following bounds holds on V Q n L

  .4.5) then follows by induction, which completes the proof of Theorem 3.4.1. Proof. (of Corollary 3.4.1) Corollary 3.4.1 is straightforward by plugging the bound for the projection error provided by Lemma 3.C.1 and the one of the quantization error provided by the Zador's Theorem into (3.4.5).

Chapter 4 .

 4 A class of numerically solvable MKV control problems

  a) ϑ 0 (µ, a) = θ0 (μ 1 , μ2 , . . . , μp , a), ϑ 1 (µ, a) = θ1 (μ 1 , μ2 , . . . , μp , a) γ 0 (µ, a) = γ0 (μ 1 , μ2 , . . . , μp , a), γ 1 (µ, a) = γ1 (μ 1 , μ2 , . . . , μp , a) f k (µ, a) = fk (μ 1 , μ2 , . . . , μp , a), g k (µ) = ḡk (μ 1 , μ2 , . . . , μp ), k = 0, . . . , p, where, given µ ∈ P p (R), we denote by μk = x k µ(dx), k = 1, . . . , p.

  (4.1)-(4.1) are given by f (µ, a) = f0 (μ 1 , μ2 , . . . , μp , a) + p k=1 fk (μ 1 , μ2 , . . . , μp , a)μ k =: f (μ 1 , μ2 , . . . , μp , a), ĝ(µ) = ḡ0 (μ 1 , μ2 , . . . , μp ) + p k=1 ḡk (μ 1 , μ2 , . . . , μp )μ k =: ḡ(μ 1 , μ2 , . . . , μp ).

  2.1)where, by convention y 0 = 1, y -1 = 0,B k (y 1 , y 2 , . . . , y p , a) = k b0 (y 1 , . . . , y p , a)y k-1 + k b1 (y 1 , . . . , y p , a)y k + k(k -1) 2 ( θ0 (y 1 , . . . , y p , a)) 2 y k-2 + k(k -1) 2 ( θ1 (y 1 , . . . , y p , a)) 2 y k +k(k -1) θ0 (y 1 , . . . , y p , a) θ1 (y 1 , . . . , y p , a)y k-1 + k(k -1) 2 (γ 0 (y 1 , . . . , y p , a)) 2 y k-2 + k(k -1) 2 (γ 1 (y 1 , . . . ,y p , a)) 2 y k +k(k -1)γ 0 (y 1 , . . . , y p , a)γ 1 (y 1 , . . . , y p , a)y k-1 , k = 1, . . . , p, Σ k (y 1 , y 2 , . . . , y p , a) = k (γ 0 (y 1 , . . . , y p , a)y k-1 + γ1 (y 1 , . . . , y p , a)y k ) , k = 1, . . . , p,

Chapter 4 .

 4 A class of numerically solvable MKV control problems which assesses the average performance of the control.

4. 3 .

 3 Numerical methodswhere φ = (φ 1 , . . . , φ K ) and matrix A n = E φ(Z tn , I tn )φ(Z tn , I tn ) .

Algorithm 4 . 1

 41 Regress-Later Monte Carlo algorithm (RLMC) -Value iteration Inputs: • M : number of training points, • µ: distribution of training points, • K: number of basis functions, • {φ k } K k=1 : family of basis functions. 1: Estimate the covariance matrix ÂM . 2: Generate i.i.d. training points {Z m t N } M m=1 accordingly to the distribution µ. 3: Initialize the value function V RL ∆t (t N , Z m t N ) = g(Z m t N ), ∀m = 1, . . . , M . 4: for n = N -1 to 0 do 5:

Chapter 4 .

 4 A class of numerically solvable MKV control problems Algorithm 4.2 Control Randomization algorithm (CR) -Value iteration Inputs:

4. 4 .

 4 Applications and numerical results Algorithm 4.3 Quantization algorithm (Q) -Value iteration Inputs: • Γ k , k = 0, . . . , N : grids of training points in R d ,

  where the first and second components of the process Z = (W, Y ) are projected respectively on the grids Γ W n and Γ Y n ; and Proj Γ W n (resp. Proj Γ Y n ) stands for the Euclidean projection of the first (resp. second) component of Z = (W, Y ) on Γ W n (resp. Γ Y n

Chapter 4 .

 4 Figure 4.1 -Plot of the quantized-based piecewise-constant approximation of the conditional expectation CondExp:a → e∈Γ ε P( ε =

  w,y),a,e )-y-y+-y-in the first case of the definition of y -and y + above; λ

  b0 = -0.1 and T = 0.5.

Figure 4 . 2 -

 42 Figure 4.2 -Results for the Portfolio Liquidation problem. Estimation of the value function at point (s 0 = 6, y 0 = 1) at time 0 provided by different strategies w.r.t. γ 0 . We took γ=5, S 0 =6, Y 0 =1, η=100 and σ=0.4.

Figure 4 . 3 -

 43 Figure 4.3 -Simulation of (Y t ) t∈[0,T ] using the (continuous-time) optimal strategy (Opt), the (Q) estimated one, and the Benchmark strategy (Bench) to solve the Portfolio Liquidation problem. We took T = 1, σ = 0.4, γ 0 = 1, b 0 = 0.1, S 0 = 6, Y 0 = 1, N = 100, γ = 5, η = 100.

Table 4 .Figure 4 . 4 -

 444 Figure 4.4 -3 simulations of the agent's wealth (X t ) t∈[0,T ] when the latter follows the continuoustime optimal strategy (Opt) and the (Q) estimated optimal strategy to solve the Portfolio Selection problem. We took σ=0.4, T =1, P =0.1, γ 0 =5, b 0 =0,1. One can see that the two strategies are the same when the drift is low; but Q performs slightly better than Opt when the drift is high, which is a time-discretization effect.

Chapter

5

 5 and η = 100.

Figure 4 . 5 -

 45 Figure 4.5 -Results for the systemic risk problem. Estimations of the value function at time 0 using different algorithms w.r.t. ρ and c. We took T =1, N =100, σ=0.1, κ=0.5, x 0 =10.

Figure 4 . 7 -

 47 Figure 4.7 -Sample of (Y t ) t∈[0,T ] controlled by Q. The (Q) estimated optimal control for the systemic risk problem is to initially let Y become large, and then reduce its value when the approaching the terminal time.

AMS

  65C05, 90C39, 93E35 Contents 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 5.2 Preliminaries on DNN and SGD . . . . . . . . . . . . . . . . . . . . . . . 143 5.2.1 Neural network approximations . . . . . . . . . . . . . . . . . . . . . . . . . 143 5.2.2 Stochastic optimization in DNN . . . . . . . . . . . . . . . . . . . . . . . . 144 5.3 Description of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . 145 5.3.1 Control learning by performance iteration . . . . . . . . . . . . . . . . . . . 146 5.3.2 Control learning by hybrid iteration . . . . . . . . . . . . . . . . . . . . . . 147 5.3.3 Training sets design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 5.3.4 Comparison of the algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . 150 5.4 Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

Figure 5 . 1 -

 51 Figure 5.1 -Representation of a neural network with d = 3, 2 hidden layers, d 1 = d 2 = 4, d 3 = 1.

Remark 5 . 3 . 4 .

 534 We point out that the estimated value function V M n in Hybrid-Now or Hybrid-LaterQ depend on training samples X (m)

  a)ε, with b and σ Lipschitz-continuous w.r.t. the couple (x, a), and ε ∼ N (0, I d ), where I d is the identity matrix of size d × d. Indeed, in this case, it is straightforward to see that C(ε) = [b] L + [σ] L ε d , where [b] L and [σ] L stand for the Lipschitz coefficients of b and σ, and . d stands for the Euclidean norm in R d . Moreover, one can show that:

  . . . , M from the training distribution µ, and samples of the exogenous noise ε

Chapter 5 .

 5 Neural networks algorithms for control problems: convergence analysis Proof of Theorem 5.4.1.

  e. . M,p stands for the L p norm conditioned by the training set, for p ∈ {1, 2}. The proof relies on Lemma 5.H.1 and Lemma 5.H.2 (see Section 5.H in Appendix) which are proved respectively in[START_REF] Kohler | Nonparametric regression with additional measurement errors in the dependent variable[END_REF] (see their Theorem 3) and[START_REF] Kohler | Pricing of high-dimensional American options by neural networks[END_REF].

  y) r (x, a; π R (y)) dµ(y) + |V n+1 (y)| |r(x, a; π R (y)) -r(x, a; y)| dµ(y)

k•

  ) 1≤m≤M,n≤k≤N generated from an independent copy of the exogenous noises (ε (m) k ) 1≤m≤M,n≤k≤N , and independent copy of initial positions at time n, (X (m) n ) M m=1 , following the same control âM k at time k= n + 1, . . . , N -1, and control A at time n, 5.C. Proof of Lemma 5.4.1 We remind that Y (m),A n+1 has already been defined in (5.4.5), and we similarly define

  which implies that A * also depends on ω ∈ Ω. Denote by P M the probability conditioned by the training set of exogenous noises (ε (m) k ) 1≤m≤M,n≤k≤N and initial positions (X (m) k ) 1≤m≤M,n≤k≤N , and recall that E M stands for the expectation conditioned by the latter. Application of Chebyshev's inequality yields

.C. 5 )

 5 Since for each m = 1, ..., M the set of exogenous noises (ε (m) k

  m),0 n+1 . (5.C.7) Chapter 5. Neural networks algorithms for control problems: convergence analysis

( 5 .

 5 C.11) Plug then (5.C.8) and (5.C.11) into (5.C.7) to get (5.C.6). 5.D. Proof of Lemma 5.4.2 Step 5: Conclusion Plug (5.C.6) into (5.C.5) and combine it with (5.C.3) to obtain the bound on the estimation error, as stated in (5.4.9) of Lemma 5.4.1. 5.D Proof of Lemma 5.4.2 Let (â M k ) N -1 k=n+1 be the sequence of estimated controls at time k = n + 1, ..., N -1. Take A ∈ A M and remind that we denote by J A,(â) N -1 k=n+1 n the cost functional associated to the control A at time n, and âM k at time k = n + 1, . . . , N -1. The latter is characterized as solution of the Bellman equation

5 .

 5 G. Proof ofLemma 5.4.4

Lemma 5 .

 5 H.1. Let (X, Y ) be a random variable. Assume |Y | ≤ L a.s. and let m(x) = E[Y |X = x].Assume Y -m(X) is sub-Gaussian in the sense that max m=1,...,M

Algorithm 6. 2

 2 ClassifPI Input: the training distributions (µ n ) N -1 n=0 ; 1: for n = N -1, . . . , 0 do 2: Compute ân (x) = a ˆ n (x) with ˆ n (x) ∈ arg max =1,...,L p (x; βn ) βn ∈ arg min β∈R q E

E

  (f (X n , ân (X n )) + Vn+1 (X βn n+1 ) -Φ(X n ; θ) Set Vn = Φ(.; θn );Vn is the estimate of the value function at time n 7: end for Output:

6. 2

 2 . Algorithms variable on a grid Γ = {e 1 , . . . , e L } ⊂ (R d ) L defined by ε = Proj Γ (ε 1 ) := L =1 e 1 ε1∈C (Γ) , where C 1 (Γ), . . ., C K (Γ) are Voronoi tesselations of Γ, i.e., Borel partitions of the Euclidian space (E, |.|) satisfying C (Γ) ⊂ e ∈ E : |e -e | = min j=1,...,L |e -e j | . The discrete law of ε is then characterized by p := P[ε = e ] = P[ε 1 ∈ C (Γ)], = 1, . . . , L.

Ep

  Vn+1 (X βn n+1 ) -Φ(X n+1 ; θ) 2 and set Ṽn+1 = Φ(.; θn+1 ); interpolation at time n + 1 6:SetVn (x) = f (x, ân (x)) + L =1 Ṽn+1 F (x, ân (x), e ) ;Vn is the estimate by quantization of the value function at time n

5 :

 5 Compute the optimal control at time nÂn (z) ∈ argmin a∈A Qn (z, a) , ∀z ∈ Γ n ;(6.2.9) use classical optimization algorithms of deterministic functions for this step 6:

6. 3 .Figure 6

 36 Figure 6.1 -Relative error of the Hybrid-Now estimate of the value function at time 0 w.r.t the number of mini-batches used to build the Hybrid-Now estimators of the optimal strategy. The value functions have been computed running three times a forward Monte Carlo with a sample of size 10,000, following the optimal strategy estimated by the Hybrid-Now algorithm.

Figure 6 . 3 -

 63 Figure 6.3 -Pathwise comparison of the first component of X w.r.t. time when the agent follows the optimal

Figure 6 . 4 -

 64 Figure6.4 -Estimates of the value function at time 0 w.r.t. w 0 using Hybrid-Now (blue line) or Hybrid-LaterQ (green dashes). We draw the value function in red for comparison. One can observe that all the algorithms estimate the price to be 4.5, but Hybrid-LaterQ is better than Hybrid-Now at reducing the quadratic risk.

Figure 6 . 7 -

 67 Figure 6.7 -Estimated optimal decisions at times 5, 10, 15, 20, 25, 29 w.r.t. (P,C) for the energy storage valuation problem using Qknn. Injection (a=-1) in red, store (a=0) in black and withdraw (a=1) in blue.

Figure 6 . 8 -

 68 Figure 6.8 -Estimated optimal decisions at times 5, 10, 15, 20, 25, 29 w.r.t. (P,C) for the energy storage valuation problem using ClassifPI. Injection (a=-1) in purple, store (a=0) in blue and withdraw (a=1) in yellow.

4 -Figure 6

 46 Figure6.10 -Estimated optimal decisions at time 1, 10 and 28, using Qknn, with N = 30 time steps. The region under the blue line is the one where it is optimal to turn the generator off if m=1 (i.e. the generator was on at time n-1), or keep it off if m = 0 (i.e. the generator was off at time n-1).

Figure 6 Figure 6

 66 Figure 6.11 -Estimated optimal decisions at time 1, 10 and 28, using ClassifHybrid, with N = 30 time steps.

  θ)) ∈ N N d,1,L,m (R Nm ) × N N d,d,L,m (R Nm ), and compute (by SGD) the minimizer of the expected quadratic loss function   Li (θ) := E U i+1 (X ti+1 ) -F (t i , X ti , U i (X ti ; θ), Z i (X ti ; θ), ∆t i , ∆W ti ) 2 θ * i ∈ arg min θ∈R Nm Li (θ).

0≤t≤TE

  |X t | p < C p (1 + |x 0 | p ),(7.4.1)for some constant C p depending only on p, b, σ and T . Moreover, we have the well-known error estimate with the Euler scheme X = X π defined in (7.3.3) with a time grid π = {t 0 = 0 < t 1 < . . . < t N = T }, with modulus |π| s.t. N |π| is bounded by a constant depending only on T (hence independent of N ):max i=0,...,N -1 |X ti+1 -X ti+1 | 2 + sup t∈[ti,ti+1] |X t -X ti | 2 = O(|π|). (7.4.2)Here, the standard notation O(|π|) means that lim sup |π|→0 |π| -1 O(|π|) < ∞.

E T 0 |f

 0 (t, X t , Y t , Z t )| 2 dt < ∞. (7.4.3) Moreover, we have the standard L 2 -regularity result on Y : max i=0,...,N -1 E sup t∈[ti,ti+1] |Y t -Y ti | 2 = O(|π|).(7.4.4)Let us also introduce the L 2 -regularity of Z: ε Z (π) := E

  Zti 2 dt + ∆t i E Zti -Z ti 2 .(7.4.10)

  Fix i ∈ {0, . . . , N -1}. By using relation (7.4.7) in the expression of the expected quadratic loss function in (7.3.6), and recalling the definition of Z ti as a L 2 -projection of Z t , we have for all parameters θ = (ξ, η) of the neural networks U i (.; ξ) and Z i (.; η)L(1) i (θ) = Li (θ) + E ti+1 ti Z t -Z ti 2 dt (7.4.15) with Li (θ) :

2Z

  By summing over i = 0, . . . , N -1, we get (recall (7.4.3)) t -Z ti 2 dt ≤ ε Z (π) + C|π| + 2dE g(X T ) -g(X T ) 2 (7.4.20)

  2), a C 3 activation function with linear growth condition, and bounded derivatives, e.g., a sigmoid activation function, or a tanh function: this class of neural networks is then represented by the parametric set of functionsN N d,1,2,m (Θ γ m ) := x ∈ R d → U(x; θ) = m i=1 c i (a i .x + b i ) + b 0 , θ = (a i , b i , c i , b 0 ) m i=1 ∈ Θ γ m , with Θ γ m := θ = (a i , b i , c i , b 0 ) m i=1 : max i=1,...,m |a i | ≤ γ m , m i=1 |c i | ≤ γ m ,for some sequence (γ m ) m converging to ∞, as m goes to infinity, and such that neural networks in N N d,1,2,m (Θ γ m ) have their first, second and third derivatives uniformly bounded w.r.t. the state variable x. More precisely, there exists some constant C depending only on d and the derivatives of s.t. for any U ∈ N N d,1,2,m (Θ γ m ),x∈R d ,θ∈Θ γ m D x U(x; θ) ≤ Cγ 2 m , sup x∈R d ,θ∈Θ γ m D 2 x U(x; θ) ≤ Cγ 3 m ,andsup x∈R d ,θ∈Θ γ m D 3 x U(x; θ) ≤ Cγ 4 m . (7.4.23) Chapter 7. Deep learning schemes for high-dimensional PDEs

  r.t. (t, x) ∈ [0, T ] × R d . (ii) The function (x, y, z) → f (t, .) is C 1 with bounded derivatives uniformly w.r.t. (t, x, y, z) in [0, T ] × R d × R × R d .Theorem 7.4.2. (Consistency of DBDP2) Under (H1)-(H2), there exists a constant C > 0, independent of π, such that E ( U (2) , Z (2) ), (Y, Z) ≤ C E g(X T ) -g(X T )

  Denoting by fi (x) = f (t i , x, vi (x), ẑi (x)), it follows by the implicit function theorem, and for |π| small enough, that vi is C 1 with derivative given byD x vi (x) = D x ṽi (x) + ∆t i D x fi (x) + D y fi (x)D x vi (x) + D z fi (x)D x ẑi (x)and thus by (7.4.26)-(7.4.27)1 -∆t i D y fi (x) σ(t i , x)D x vi (x) = ẑi (x) + ∆t i σ(t i , x) R i (x) + D x fi (x) + D z fi (x)D x ẑi (x) .Under (H2), by the linear growth condition on σ, and using the bounds on the derivatives of the neural networks in N N d,1,2,m (Θ γ m ) in (7.4.23), we then haveE σ(t i , X ti )D x vi (X ti ) -Z ti 2 ≤ C(γ 6 m + |π| 2 γ 8 m )|π| 2 . (7.4.28)Next, by the same arguments as in Steps 3 and 4 in the proof of Theorem 7.4.1 (see in particular (7.4.18)), we have for |π| small enough,

i (X ti ) 2 ≤

 2 Cε N N,v,2 i + C(γ 6 m + |π| 2 γ 8 m )|π| 3 . (7.4.29)On the other hand, by the same arguments as in Steps 1 and 2 in the proof of Theorem 7.4.1 (see in particular (7.4.14)), we have max i=0,...,N -1

7. 4 .

 4 Convergence analysisTheorem 7.4.3. (Case f independent of z: Consistency of RDBDPbis) Let Assumption (H1) hold, with g Lipschitz. Then, there exists a constant C > 0, independent of π, such that max i=0,...,N -1 Y ti -Ûi (X ti )

≤ ( 1 +.

 1 from the Lipschitz condition on f in (H1), and then for |π| small enoughỸ π ti -Ṽti 2 C|π|) Y π ti+1 -U i+1 (X ti+1 ) 2By Minkowski inequality, this yields for all θỸ π ti -U i (X ti ; θ) 2 ≤ (1 + C|π|) Y π ti+1 -U i+1 (X ti+1 ) 2 + Ṽti -U i (X ti ; θ) 2 . (7.4.35)On the other hand, by the martingale representation theorem, there exists an R d -valued square integrable process ( Zt ) t such thatU i+1 (X ti+1 ) = Ṽti -f (t i , X ti , Ṽti )∆t i + ti+1 ti Z s dW s ,and the expected squared loss function of the DBDP3 scheme can be written as Li (θ) = Li (θ) + E

  3.8) related to the solution (Y, Z) of the reflected BSDE (7.3.9) by showing an error estimate for E ( U , Z ), (Y, Z) := max i=0,...,N -1

7. 4 .

 4 Convergence analysis Theorem 7.4.4. (Consistency of RDBDP) Let Assumption (H1) hold. There exists a constant C > 0, independent of π, such that E ( U , Z ), (Y, Z) ≤ C ε(π) + under (H3), and ε(π) = O(|π|) under (H4).

  under (H3), and ε(π) = O(|π|) under (H4).

  ti+1 ti Z s dW s , Chapter 7. Deep learning schemes for high-dimensional PDEs and the expected squared loss function of the RDBDP scheme can be written as Li (θ) = Li (θ) + E ti+1 ti Zt -Zti 2 dt , where we notice by Itô isometry that Zti = 1 ∆ti E i ti+1 ti

.

  Combining with (7.4.40), this proves the error estimate (7.4.38) for the Y -component. The error estimate (7.4.38) for the Z-component is proved along the same arguments as in Step 5 in the proof of Theorem 7.4.1, and is omitted here.

  3.1):σ = 1, µ = 0.2, T = 2, d = 1, (7.5.1) f (t, x, y, z) = (cos(x)(e T -t2+ σ 2 2 ) + µ sin(x))e T -t2

Figure 7 . 1 -

 71 Figure 7.1 -Estimates of u and Z using DBDP1. We took the parameters defined in (7.5.1) and set x 0 = 1.

Figure 7 . 2 -

 72 Figure 7.2 -Estimates of u and Z using DBDP2. We took the parameters defined in (7.5.1) and set x 0 = 1.

7. 5

 5 .1.2 Increasing the dimensionWe extend the example from the previous section to the following d-dimensional problem:d ≥ 1, σ = 1 √ d I d , µ = 0.2 d 1I d , T = 1, f (t, x, y, z) = (cos(x)(e x) cos(x)e T -t 2 + 1 2d (u(1I d .z)) 2 , g(x) = cos(x), with x = d i=1 x i .

7. 5 . 2

 52 PDEs with unbounded solution and more complex structureIn this section with take the following parametersσ = 1 √ d I d , µ = 0, T = 1, f (x, y, z) = k(x)k is chosen such that the solution to the PDE is equal tou(t, x) = T -t d d i=1 (sin(x i )1 xi<0 + x i 1 x1≥0 ) + cos d i=1 ix i .

Figure 7 . 3 -

 73 Figure 7.3 -Estimates of u and Z using DBDP1. We took the parameters defined in (7.5.2), with d = 1, and set x 0 = 0, 5.

τ

  ∈T t,T E[e -rτ g(X τ )],where T t,T is the set of stopping time with values in [t, T ], and is solution of the variational inequalitymin -∂ t u -Lu, u -g = 0, on [0, T ) × (0, ∞) d u(T, .) = g, on (0, ∞) d , with Lu(t, x) = 1 2 d i=1 σ 2 i x 2 i D 2 xi u(t, x) + r d i=1 x i D xi u(t, x) -ru(t, x),Chapter 7. Deep learning schemes for high-dimensional PDEs

Figure 7

 7 Figure 7.6 -Algebric error of the estimate of u using scheme (7.3.7). We took the parameters in (7.5.2) and set d = 2 and x 0 = 0.51I d . Averaged value Standard deviation DBDP1 0.8666 0.013 DBDP2 0.83646 0.00453 Scheme [HJE17] NC NC

  General framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.3.2 Numerical methods available in the literature . . . . . . . . . . . . . . . . . 21 1.3.3 Contributions (Chapter 7) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 1.3.4 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

	Chapter 1. Introduction
	1.1 Quantization and regression methods for MDPs. Applica-
	tions to market-making and McKean-Vlasov-type control
	problem with partial information
	1.1.1 Quantization and regression methods for Markov decision processes
	(MDPs).

1.1 Quantization and regression methods for MDPs. Applications to marketmaking and McKean-Vlasov-type control problem with partial information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1.1 Quantization and regression methods for Markov decision processes (MDPs). 2 1.1.2 Application of the MDPs to market-making (Chapter 3) . . . . . . . . . . . 6 1.1.3 Application of the MDPs to McKean-Vlasov type control problems (Chapter 4) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2 Neural networks for MDPs . . . . . . . . . . . . . . . . . . . . . . . . . . 10 1.2.1 Representation of functions by neural networks & training . . . . . . . . . . 11 1.2.2 Neural networks for MDPs . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2.3 Contributions (Chapters 5 and 6) . . . . . . . . . . . . . . . . . . . . . . . . 15 1.2.4 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.3 Neural networks for PDEs and BSDEs . . . . . . . . . . . . . . . . . . . 19 1.3.1

  The algorithms NNcontPI and ClassifPI are tested numerically in several examples in Chapter 6: ClassifPI provided good results in the Valuation of Energy Storage problem, presented in Section 6.3.3 page 200, but is beaten by Qknn which is well-designed for this problem (see Table6.2 page 203).

  5.2 for examples on which NNZ provides poor estimates or diverges).

	Chapter 1. Introduction
	Question 3a:	What stable and accurate scheme can we propose to solve BSDEs in high di-
		mension?
	Question 3a(bis):	What stable and accurate scheme can we propose to solve PDEs in high dimen-
		sion?
	Question 3b:	What stable and accurate scheme can we propose to solve RBSDEs in high
		dimension?
	Question 3b(bis): What stable and accurate scheme can we propose to solve QVIs in high dimen-
		sion?

Table 1 .

 1 2 -Estimates of the price of an American option using RDBDP algorithm. We reported very accurate estimates of the price (computed by a tree-based algorithm after applying a trick to reduce the dimension of the problem to one) to the benchmark column. The mean and standard deviation are computed on 40 independent simulations. RDBDP is precise up to dimension 40! can then be solved in a very accurate way using conventional methods such as the binomial tree model. The benchmark is then a binomial tree-based algorithm, which provides very accurate results. As we can see on Table1.2, RDBDP appears to provide very accurate results up to dimension 40.

	Chapter 1. Introduction
	Dimension nb steps	RDBDP	std	benchmark
	1	80	0.0613818 0.00019	0.060903
	5	80	0.107650 0.00016	0.10738
	10	80	0.129923 0.00016	0.12996
	20	80	0.15050	0.00010	0.1510
	40	160	0.16758	0.00016	0.1680

  Méthodes de Quantification et de régression pour les MDPs. Application au market-making et aux problèmes de contrôle de type McKean-Vlasov avec information partielle

	2.1.1 Méthodes de Quantification et de régression pour les problèmes de
	décisions Markoviennes (MDP)

Contents 2.1 Méthodes de Quantification et de régression pour les MDPs. Application au market-making et aux problèmes de contrôle de type McKean-Vlasov avec information partielle . . . . . . . . . . . . . . . . . . . . . . . 30 2.1.1 Méthodes de Quantification et de régression pour les problèmes de décisions Markoviennes (MDP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.1.2 Application des MDPs au market-making (Chapitre 3) . . . . . . . . . . . . 34 2.1.3 Application des MDPs aux problèmes de contrôle de type McKean-Vlasov avec information partielle (Chapitre 4) . . . . . . . . . . . . . . . . . . . . . 36 2.2 Réseaux de neurones pour les MDPs . . . . . . . . . . . . . . . . . . . . 38 2.2.1 Représentation de fonctions par réseaux de neurones & apprentissage . . . 39 2.2.2 Revue de littérature sur les réseaux de neurones pour les MDPs . . . . . . . 42 2.2.3 Contributions (Chapitres 5 et 6) . . . . . . . . . . . . . . . . . . . . . . . . 44 2.2.4 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 2.3 Réseaux de neurones pour les EDPs et les EDSRs . . . . . . . . . . . . 48 2.3.1 Cadre général . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 2.3.2 Méthodes numériques existantes dans la littérature . . . . . . . . . . . . . . 50 2.3.3 Contributions (Chapitre 7) . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.3.4 Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 2.1

  2.1.2. Dans le cas où la fonction valeur est point fixe d'un opérateur, il existe d'autres algorithmes issu de l'apprentissage par renforcement, comme Temporal Difference Learning ou le Deep Q-learning, et qui marchent beaucoup mieux que la simple utilisation d'un argument de Picard que nous venons de présenter, pour approximer cette dernière par le schéma (2.1.5). Nous référons à[START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] pour une introduction exhaustive de ces derniers, et référons à[START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] pour l'apprentissage du contrôle optimal pour battre les joueurs professionnels à des jeux vidéos comme ATARI, en utilisant un algorithme basé sur le Q-learning. Enfin, nous citons[START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF] et[START_REF] Silver | Mastering the game of Go without human knowledge[END_REF] où est expliqué comment jouer mieux que les joueurs professionnels au jeu de go en utilisant notamment des méthodes de Monte Carlo Tree Search pour explorer efficacement l'espace de contrôle.

  4 . Nous référons à[START_REF] Balata | Regress-Later Monte Carlo for optimal control of Markov processes[END_REF] pour une présentation rigoureuse théorique de la méthode regress-later appliquée aux problèmes de contrôle stochastique à horizon fini et temps discret. Cette méthode est particulièrement efficace dans le cas

	2.1. Quantification et régression pour les MDPs. Application au market-making et aux problèmes
	de contrôle de type McKean-Vlasov avec information partielle
	où la régression des fonctions de base peut se faire de manière analytique, i.e. on possède des formules
	fermées pour le calcul des espérances conditionnelles, car alors la régression est quasi-instantanée,
	mais aussi ca évite d'avoir à surveiller une erreur supplémentaire due à la régression (et veiller à ne
	pas qu'elle explose). Nous référons à [Ala+19] pour une comparaison des algorithmes regress-later
	avec regress-now sur un exemple concret de management de micro-grille, dans lequel regress-later
	propose de meilleures estimées. Dans le cas où il n'existe pas de formule fermée pour la régression
	des fonctions de base choisies, on peut toujours approximer ces régressions par différentes méthodes
	comme la quantification.

  On modélise un carnet d'ordres par K limites du côté ask (ask-side), et K limites du côté bid (bid-side), comme proposé dans[START_REF] Abergel | Limit Order Books[END_REF]. On note pa t le best-ask à l'instant t, qui est le prix le moins cher auquel le marché est prêt à vendre l'actif à l'instant t, et par pb t le best-bid à l'instant t, qui est le prix le plus cher auquel un participant du marché est prêt à acheter l'actif à l'instant t. On appelle A l'ensemble des stratégies admissibles de ce dernier, i.e. les processus F-prévisibles, où F est la filtration engendré par les processus d'arrivée d'ordres, qui rendent compte des positions des ordres limites du market maker sur le carnet d'ordres.

		2.1. Quantification et régression pour les MDPs. Application au market-making et aux problèmes
			de contrôle de type McKean-Vlasov avec information partielle
		Modèle et représentation mathématique du problème
		Soit K ≥ 1. On représente le carnet d'ordres par le couple a t , b t = a 1 t , ..., a K t , b 1 t , . . . , b K t	où
			• a i t est le nombre d'ordres limites de vente i ticks plus cher que pb t .
			• b i t est le nombre d'ordres limites d'achat i ticks moins cher que pa t .
		Le carnet d'ordres peut recevoir à tout moment trois types d'ordre émis par les participants et qui
		a, e ) modifient le couple (a t , b t ) : des ordres de vente ou d'achat au marché, des ordres limites de vente ou , ∀z ∈ Γ n . d'achat ; ou bien des ordres d'annulations sur l'ask-side ou le bid-side. On modélise les arrivées de ces
		ordres par des processus ponctuels.
	4: end for	On considère maintenant un market-maker qui observe le carnet d'ordres et peut choisir à tout
	Sortie :	moment envoyer des ordres limites au marché.
	-α(t n , z) z∈Γn,0≤n≤N -1 : estimation de la stratégie optimale ;
	-V Q ∆t (0, z) z∈Γ0 : estimation de la fonction valeur ;
	2.1.2 Application des MDPs au market-making (Chapitre 3)
	Nous nous intéressons dans cette section à un problème de market-making, que nous résolvons théo-
	riquement en utilisant la théorie des MDPs, et numériquement en utilisant Qknn. Nous présentons
	d'abord le problème, puis nos contributions théoriques et numériques, disponibles dans le Chapitre 3.
	2.1.2.1 Présentation du problème de market-making
	On considère un titre financier qui s'échange sur un marché financier, ainsi qu'un carnet d'ordres, qui
	est un recueil de toutes les propositions de ventes et d'achats de ce titre. Les participants du marché
	observent le carnet d'ordres et peuvent à tout moment décider de :
	• acheter ou vendre le titre au meilleur prix (on parle d'ordre au marché)
	• émettre de nouvelles propositions de vente ou d'achat de l'actif à un prix donné (on parle
	d'ordres limites)
	• annuler leurs ordres limites avant que ces derniers n'aient été exécutés (on parle d'ordre d'an-
	nulation)
	Les prix auxquels les participants ont la possibilité d'émettre des ordres sont divisés en ticks. Un
	market-maker est une personne qui agit sur un carnet d'ordres en émettant des propositions de vente
	ou d'achat un titre à certains prix (i.e. il n'émet que des ordres limites et des ordres d'annulation). La
	fonction principale de ce dernier est de liquéfier le marché, par ses propositions de vente et d'achat à
	différent prix. Une stratégie suivie par le market-maker, appelée stratégie de market-making, consiste
	à émettre des ordres limites de vente et d'achat sur le marché pour le liquéfier, tout en maximisant
	une quantité d'intérêt propre au market-maker, comme sa richesse terminale par exemple.
	Remark 2.1.3. Il faut différencier un ordre limite qui est un ordre qu'un participant envoie pour
	proposer d'acheter ou de vendre le titre à un certain prix, d'un ordre au marché qui est un ordre envoyé
	par un participant pour acheter ou vendre le titre (au meilleur prix). Il n'y a pas de coût de transaction
	lorsque le participant envoie un ordre limite au marché (parce qu'essentiellement, le marché considère
	que ce dernier liquéfie le marché), alors que l'emission d'un ordre au marché s'accompagne de frais
	de transaction.

  Figure 2.1 -Histogramme de la richesse terminale du market-maker lorsque celui-ci suit une bonne stratégie naïve (bleu clair), et la stratégie optimale estimée par Qknn (bleu foncé). On remarque que la richesse terminale est meilleure lorsque le market-maker suit les décisions estimées par Qknn, et ceci en limitant les grosses pertes.

	Chapter 2. Introduction (in French)
	Contributions numériques :
	Remark 2.1.4. L'espace d'états de la représentation MDP du problème de market-making est dis-
	cret, sparse et grand ; et l'espace de contrôle est de cardinal fini et très petit. Les méthodes de type
	regress-now et regress-later, présentées ci-dessus ne semblent pas adaptées puisqu'il est difficile de
	choisir convenablement les fonctions de base à prendre lorsque l'espace d'états est discret, sparse et
	de dimension moyenne (≈10), pour éviter la sur-estimation et la sous-estimation.
	1 t≤T ,
	avec λ l'intensité de saut du processus ponctuel des ordres limites. Plus précisément, nous montrons
	dans la Section 3.3 du Chapitre 3 le théorème suivant :
	Theorem 2.1.2. T admet un unique point fixe v qui coïncide avec la fonction valeur du problème de
	market-making.

  MDPs aux problèmes de contrôle de type McKean-Vlasov avec information partielle (Chapitre 4)

	2.1. Quantification et régression pour les MDPs. Application au market-making et aux problèmes
	de contrôle de type McKean-Vlasov avec information partielle
	un espace de probabilité (Ω, F, P), et d'un processus stochastique X dont la dynamique est contrôlée
	ainsi :
	Nous proposons dans le Chapitre 4 une méthode basée sur le Markovian embedding pour réduire les
	problèmes de type McKean-Vlasov avec information partielle à des MDPs standards avec notamment
	un espace d'états de dimension finie. Des résultats numériques sont présentés pour divers problèmes de
	type McKean-Vlasov, où les méthodes regress-later, regress-now, Qknn et des méthodes des différence
	finie sont testées et comparées. Nous présentons d'abord les problèmes de contrôle de type McKean-
	Vlasov avec information partielle, qui est le cadre de cette section, puis nos contributions théoriques
	et numériques pour résoudre ces derniers.
	2.1.3.1 Présentation des problèmes de contrôle de type McKean-Vlasov avec informa-
	tion partielle
	On considère un problème de contrôle de type McKean-Vlasov (MKV) sous information partielle et
	bruit commun, qui est la donnée de deux mouvements Browniens indépendants B et W 0 définis sur

Table 2

 2

	.1 -Estimations de la fonction valeur du problème de risque systémique pour différent jeux
	de paramètres ; en utilisant regress-later (RLMC), un algorithme de randomisation du contrôle (CR),
	Qknn ainsi qu'une méthode de discrétisation d'EDP. Plus la valeur est petite, plus l'algorithme est
	performant : ici Qknn n'est jamais battu !
	rappelons ici que les grandes lignes : nous nous intéressons à un problème de contrôle stochastique à
	temps discret et horizon fini (MDP), i.e. nous cherchons

  .2.2) où µ est une mesure de probabilité, et où G est une fonction différentiable et assez régulière pour que la dérivation sous le signe E de J soit licite. Ce type de problème est celui auquel nous faisons face lorsque l'on veut apprendre les paramètres optimaux pour représenter la fonction valeur ou le contrôle optimal dans les problèmes de contrôle.

	Example 2.2.1. Si l'on représente le contrôle au temps n par le réseau de neurones A(.; β n ) et la
	fonction valeur au temps n par le réseau de neurones Φ(.; θ n ) ; alors : en utilisant le schéma (2.2.1),
	le paramètre optimal β * n pour représenter le contrôle optimal au temps n vérifie :

  N -1.

	2.2. Réseaux de neurones pour les MDPs
	Algorithm 2.5 NNContPI
	Entrée : une suite de distributions d'entraînement (µ n
	Commentaires sur NNcontPI
	Au temps n, seuls les paramètres de la représentation du contrôle au temps n sont appris, et
	ceci en utilisant les contrôles déjà appris aux temps n + 1, . . . , N -1. Une excellente conséquence
	de cette propriété est que l'on peut améliorer énormément la qualité et le temps de training en
	usant de la méthode de pré-training.
	L'apprentissage du paramètre optimal β n au temps n repose sur les mesures de training µ k , k =
	n, . . . , N -1, choisies par l'agent. Le choix des mesures de training peut s'avérer être une tâche
	difficile. Il est important 14 que la densité de la mesure de training µ k choisie donne du poids
	dans les régions qui ont de fortes chances d'être visitées fréquemment par le processus contrôlé
	optimalement à l'instant k. Ceci implique que l'agent ait déjà une idée vague des régions où le
	processus optimal est susceptible d'aller 15 .
	L'apprentissage des paramètres optimaux à la date n, en utilisant NNcontPI et ClassifPI, passe
	par la re-simulation de trajectoires de la date n à la date N , comme on peut le voir dans
	Algorithme 2.5 ; ce qui implique que ces méthodes sont très volatiles, surtout dans le cas où N
	est grand, ou que les récompenses f et g sont complexes.

  Les algorithmes NNcontPI et ClassifPI sont testés numériquement sur des exemples dans le Chapitre 6 : ClassifPI obtient de très bons résultats dans l'exemple Valuation of Energy Storage présenté dans la section 6.3.3 page 200, même s'il se fait battre par Qknn 19 qui est certainement mieux adapté au problème (voir le tableau 6.2 page 203).

	Chapter 2. Introduction (in French)
	espérance conditionnelle conditionnée au training set utilisé pour estimer les contrôles
	optimaux (â M k ) n≤k≤N -1 .
	Le théorème 2.2.1 stipule 18 que la fonction valeur au temps n, calculée à partir des estimations
	de la stratégie optimale du temps n au temps N -1, est proche de la vraie fonction valeur au temps
	n ; et la différence tend en moyenne vers 0 lorsque la taille du training set grandit et que l'espace des
	réseaux de neurones grossit. Ce résultat est intéressant notamment parce qu'il prend en compte la
	taille du training set utilisé dans la procédure de SGD, dans le résultat de convergence.

  ; estimées soit par l'algorithme Classif-Hybrid, soit par un algorithme Qknn 21 qui fait figure de benchmark. Dans la figure 2.2, le cas m = 1 réfère à la situation où la batterie est déjà activée avant le temps 1 et où une désactivation au temps 1 entraînerait un coût de switching ; le cas où m = 0 réfère à la situation où la batterie est désactivée juste avant le temps 1 et où une activation au temps 1 entraînerait un coût de switching. Nous constatons que les décisions optimales estimées par les deux algorithmes sont très proches et très régulières. De plus elle font sens : quand la demande résiduelle est grande, i.e. R grand, et que la batterie est vide, i.e. C petit, il est optimal d'activer le générateur de courant et de générer de l'énergie pour répondre à la demande et éviter de se faire pénaliser ; en revanche si la demande est faible et que les batteries sont pleines : le gestionnaire n'a pas besoin d'activer le générateur, et peut se contenter d'utiliser sa batterie pour répondre à la demande.En regardant les fonctions valeurs disponible dans le tableau 6.4 page 211, on s'aperçoit que Classif-Hybrid fait en fait mieux que le benchmark !

	• Valuation of Energy Storage, où l'espace de contrôle est fini, et l'espace d'états a une composante
	continue et une composante discrète.
	• Smart Grid Management, où l'espace d'états est continu, et le contrôle admet une composante
	continue et une composante discrète.
	En particulier Classif-Hybrid a donné d'excellents résultats dans l'exemple de Smart Grid Manage-
	ment, introduit dans la section 6.3.4 page 204, et que nous présentons succinctement ici. Le problème
	Smart Grid Management est celui que se pose un gestionnaire d'énergie (type EDF) qui a une bat-

terie, de charge C, qu'il peut recharger avec de l'énergie solaire P qu'il reçoit, et qui doit subvenir à la demande D de ses consommateurs. Pour ce faire, ce dernier peut à tout temps choisir d'activer un générateur de courant et se délivrer la quantité d'énergie qu'il désire, ou bien de décharger sa batterie. L'agent peut recharger sa batterie avec l'énergie solaire ou bien avec le générateur de courant. Il peut aussi choisir de désactiver la batterie. Tout changement de mode de la batterie entraîne un coût de switching (ceci afin d'empêcher le gestionnaire d'activer et désactiver la batterie trop souvent pour ne pas détériorer la batterie). Le gestionnaire est pénalisé à chaque fois qu'il change le régime de la batterie (coût de switching), et lorsqu'il ne répond pas à la demande D du consommateur. Il paie de plus un coût lorsqu'il se fournit avec le générateur de courant, et celui-ci est proportionnel au courant délivré. Dans la figure 2.2 ci-dessous, extraite des figures 6.11 page 212 et 6.10 page 211, on peut voir la quantité optimale de courant à se faire délivrer par le générateur en fonction de la demande Chapter 2. Introduction (in French) résiduelle R := D -P

  .3. Réseaux de neurones pour les EPDs et les EDSRs où l'on note ∆W ti := W ti+1 -W ti . Pour simplifier la notation, nous notons X le processus discrétisé, et omettons ainsi la dépendance de ce dernier par rapport à la grille de discrétisation en temps π.. L'approximation de (2.3.2) est alors donnée par le schéma d'Euler associé dans la représentation forward (2.3.1) :

  Theorem 2.3.2. (Consistance de DBDP2) Sous des hypothèses de régularité classiques, il existe une constante C > 0, indépendante de π telle que22

, N ; Remark 2.3.2. Calculer ∇U i (.; θ) est du même ordre de complexité que calculer U i (.; θ), lorsque l'on utilise la différentiation automatique rétrograde, implémentée nativement dans Tensorflow ou PyTorch. Du coup, l'algorithme DBDP2 ne perd pas de temps à calculer ∇U i (.; θ). La convergence de DBDP2 est étudiée dans le Chapitre 7 (voir Théorème 7.4.2 page 230), et nous présentons ici une version simplifiée du résultat :

  Table2.2 -Estimation du prix d'une option américaine par l'algorithme RDBDP. Les prix de référence sont des excellentes approximations du prix de l'américaine, calculées en utilisant une astuce pour réduire la dimension à 1. La moyenne et l'écart-type sont calculés sur 40 lancements indépendants de RDBDP. RDBDP est très précis jusqu'à la dimension 40 !

	Dimension nb pas	valeur	std	référence
	1	80	0.0613818 0.00019 0.060903
	5	80	0.107650 0.00016 0.10738
	10	80	0.129923 0.00016 0.12996
	20	80	0.15050	0.00010	0.1510
	40	160	0.16758	0.00016	0.1680

  a bound on the intensity rate of N t . On the other hand, there exists a constant c 2 > 0 such that d(|a| + |b|) t ≤ c 2 dL t so that: E α t,|a|0+|b|0 [|a| t + |b| t ] ≤ |a| 0 + |b| 0 + c 3 t 0 E [|a| s + |b| s ] ds. Applying Gronwall's inequality, we then get: E α t,|a|0+|b|0 [|a| t + |b| t ] ≤ (|a| 0 + |b| 0 ) e c3t .

	(3.3.4)
	Plugging (3.3.4) into (3.3.3) finally leads to:

  -and y + are the two closest states in Γ Y n+1 from G y ∆t ((w, y), a, e ), such that y -< G y ∆t ((w, y), a, e ) < y + if such point exists; y -and y + are equal to the closest state in Γ Y n+1 from G y ∆t ((w, y), a, e ) otherwise.

		347 -1.356	-1.278 -1.368 -1.318 3.689 3.687	3.995 3.686 4.144
	0.2 -1.385 -1.390	-1.283 -1.401 -1.348 3.682 3.682	3.847 3.679 4.138
	0.3 -1.445 -1.446	-1.314 -1.460 -1.402 3.670 3.674	4.034 3.667 4.126
	0.4 -1.523 -1.524	-1.323 -1.556 -1.485 3.655 3.674	4.128 3.650 4.108
	0.5 -1.642 -1.637	-1.348 -1.673 -1.585 3.636 3.664	4.243 3.630 4.088
	0.6 -1.783 -1.777	-1.425 -1.826 -1.711 3.611 3.640	4.386 3.607 4.064
	0.7 -1.973 -1.927	-1.513 -2.018 -1.870 3.581 3.613	4.783 3.572 4.029
	0.8 -2.213 -2.003	-1.637 -2.243 -2.057 3.545 3.575	5.142 3.537 3.992
	0.9 -2.526 -2.457	-1.819 -2.516 -2.288 3.500 3.530	5.345 3.498 3.952
	1	-2.918 -2.801	-1.806 -2.829 -2.560 3.453 3.513	6.765 3.452 3.903
	e ,(w,y) • y • λ a

  .1). 2. Portfolio Selection. Consider the Portfolio Selection problem with one risky asset. We choose a CARA utility function U (x) = -exp(-px), with p > 0. It has been shown in [BGP16, Corollary 4.4. Applications and numerical results

		-1.2
		-1.4										4.2
														Opt
														RLMC
		-1.6										4.1		Q Bench
												4
		-1.8
												3.9
	V(t=0,Y 0 =1)	-2 -2.2									0 =1) V(t=0,Y	3.8
												3.7
		-2.4										3.6
		-2.6										3.5
		-2.8			RLMC Opt							0.1 3.4	0.2	0.3	0.4	0.5	γ 0	0.6	0.7	0.8	0.9	1
					Q
					Bench
		0.1 -3	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1
						γ 0
				b0 = 0.1 and T = 1.

  4. A class of numerically solvable MKV control problems

	ρ	RLMC CR	Q	Bench
	0.1 8.88 0.2 8.73 0.3 8.42 0.4 8.02 0.5 7.61 0.6 6.93 0.7 5.94 0.8 4.86	9.12 8.76 8.94 8.98 8.69 8.77 8.69 8.32 8.48 8.25 7.91 8.06 7.73 7.37 7.51 6.97 6.68 6.79 6.07 5.78 5.87 4.82 4.62 4.67	c RLMC 0 7.79 1 7.88 5 8.22 10 8.63 25 9.69 50 11.08	CR 7.78 7.87 8.23 8.64 9.76 11.27 10.94 10.97 Q Bench 7.77 7.79 7.86 7.88 8.21 8.23 8.61 8.62 9.61 9.62
	0.9 3.32	3.10 3.02 2.97	ρ = 0.5 and η = 100.
		c = 100 and η = 10.

Table 6 .

 6 2 -V (0, P0, C0) estimates for different values of ain, using the optimal strategy provided by the ClassifPI , Hybrid-Now and Qknn algorithms, with aout = 0.25, P0 = 4 and C0 = 4. Estimate of the value function at time 0 w.r.t. ain, when the agent follows the optimal strategy

	a in	Hybrid-Now ClassifPI Qknn α = 0
	0.06 -0.99	-0.71	-0.66	-1.20
	0.10 -0.70	-0.38	-0.34	-1.20
	0.20 -0.21	0.01	0.12	-1.20
	0.30 -0.10	0.37	0.37	-1.20
	0.40	0.10	0.51	0.69	-1.20

  Contents 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 7.2 Neural networks as function approximators . . . . . . . . . . . . . . . . . 219 7.3 Deep learning-based schemes for semi-linear PDEs . . . . . . . . . . . . 220 7.3.1 The deep BSDE scheme of [HJE17] . . . . . . . . . . . . . . . . . . . . . . . 221 7.3.2 New schemes: DBDP1 and DBDP2 . . . . . . . . . . . . . . . . . . . . . . 221 7.3.3 Extension to variational inequalities: scheme RDBDP . . . . . . . . . . . . 223 7.4 Convergence analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.4.1 Convergence of DBDP1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

7.4.2 Convergence of DBDP2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 7.4.3 Convergence of RDBDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232

  X t , Y t , Z t )| 2 dt ,7.4. Convergence analysis and thus for |π| small enough:E Y ti -V ti 2 ≤ (1 + C|π|)E Y ti+1 -U X t ,Y t , Z t )| 2 dt . (7.4.12)

				(1) i+1 (X ti+1 )	2 + C|π| 2
	+ CE	ti+1	Z t -Zti	2 dt + C|π|E	ti+1	|f (t,
		ti				ti
						i+1 (X ti+1 ) 1)	2
		ti+1
	+ 2d∆t i E		|f (t, X t , Y t , Z t )| 2 dt .			(7.4.11)
		ti
	Then, by plugging (7.4.10) and (7.4.11) into (7.4.9), and choosing γ = 8d[f ] 2

L , we have

E Y ti -V ti 2 ≤ C∆t i E Y ti -V ti 2 + (1 + γ∆t i )E Y ti+1 -U (1) i+1 (X ti+1 ) 2 + C|π| 2 + CE ti+1 ti Z t -Zti 2 dt + C∆t i E ti+1 ti |f (t,

Table 7 .

 7 Chapter 7. Deep learning schemes for high-dimensional PDEs 2 -Estimate of u(0, x 0 ) where d = 5 and x 0 = 1I 5 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 0.46768.

		Averaged value Standard deviation
	DBDP1	0.4637038	0.004253
	DBDP2	0.46335	0.00137
	Scheme [HJE17]	0.46562	0.0035
		Averaged value Standard deviation
	DBDP1	-1.3895	0.00148
	DBDP2	-1.3913	0.000583
	Scheme [HJE17]	-1.3880	0.00155

Table 7

 7 .3 -Estimate of u(0, x 0 ) where d = 10 and x 0 = 1I 10 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is -1.383395.

Table 7 .

 7 4 -Estimate of u(0, x 0 ) where d = 20 and x 0 = 1I 20 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 0.6728135. Table7.5 -Estimate of u(0, x 0 ) where d = 50 and x 0 = 1I 50 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 1.5909.

		Averaged value Standard deviation
	DBDP1	0.6760	0.00274
	DBDP2	0.67102	0.00559
	Scheme [HJE17]	0.68686	0.002402

Table 7

 7 .6 -Estimate of u(0, x 0 ), where d = 1 and x 0 = 0.5. Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is 1.37758.

Table 7 .

 7 10 -Estimate of u(0, x 0 ), where d = 10 and x 0 = 0.51I 10 . Average and standard deviation observed over 10 independent runs are reported. The theoretical solution is -0.2148861.

	Dimension nb step	value	std	reference
	1	10	0.06047	0.00023 0.060903
	1	20	0.060789 0.00021 0.060903
	1	40	0.061122 0.00015 0.060903
	1	80	0.0613818 0.00019 0.060903
	5	10	0.10537	0.00014 0.10738
	5	20	0.10657	0.00011 0.10738
	5	40	0.10725	0.00012 0.10738
	5	80	0.107650 0.00016 0.10738
	10	10	0.12637	0.00014 0.12996
	10	20	0.128292 0.00011 0.12996
	10	40	0.12937	0.00014 0.12996
	10	80	0.129923 0.00016 0.12996
	20	10	0.1443	0.00014	0.1510
	20	20	0.147781 0.00012	0.1510
	20	40	0.149560 0.00012	0.1510
	20	80	0.15050	0.00010	0.1510
	40	10	0.15512	0.00018	0.1680
	40	20	0.16167	0.00015	0.1680
	40	40	0.16487	0.00011	0.1680
	40	80	0.16665	0.00013	0.1680
	40	160	0.16758	0.00016	0.1680

Table 7 .

 7 11 -Estimates of the American option using RDBDP. Average and standard deviation over 40 independent runs for different numbers of time steps are reported.

i.e. stochastic control problems in which the coefficients of the dynamic of the controlled process depend on the law of the controlled process itself

This scheme is often referred to as Bellman equation in the literature of stochastic control.

in particular, assuming that T is contracting.

using optimization algorithms such as the gradient descent-type methods.

and not the value function at time n+1!

The algorithm has been successfully tested on the market-making problem, introduced in Section 1.1.2, which is of dimension 12.

We refer e.g. to[START_REF] Graf | Foundations of quantization for probability distributions[END_REF] for a proof of existence of such a r.v.

see Proposition 3.3.1 page 70

see Theorem 3.3.1 page 72

We rely here on the fact that it takes more or less the same time to compute the output of a neural network or its derivatives at one point, when using reverse backpropagation to compute the derivative. All the deep learning languages such as Tensorflow or PyTorch use this method to compute the derivatives of neural networks.

As we will see later, an important feature of the new algorithm that we design in the second part of the thesis is that the training of the neural networks for the optimal control and the Q-value are uncoupled.

The pre-training trick is also referred to as transfer learning trick in the literature.

The notation x M = O(y M ) when M → ∞, means that the ratio |x M |/|y M | is bounded when M goes to infinity.

The notation x M = O P (y M ) when M → ∞ means that there exists c > 0 such as P |x M | > c|y M | → 0 when M goes to infinity.

and not Q-value, as proposed in the Actor-critic algorithms presented above.

and not from time n to N , as it was the case in Algorithm NNcontPI.

Qknn was introduced in Section 1.1.1.3 page 5, and can be considered as an excellent benchmark.

The acronym stands for Deep Galerkin Method.

We refer to[START_REF] Zhang | A numerical scheme for BSDE's[END_REF] for more details on the regularity of Z

we refer to 1.3.4 for more details on this term

see Theorem 1.3.1 for the notations.

we refer to 1.3.4 for more details on this term.

i.e. des problèmes de contrôle où la dynamique du processus contrôlé dépend de la loi de ce dernier.

Le schéma (2.1.3) est communément appelé équation de Bellman dans la littérature du contrôle stochastique.

sous lesquelles, en particulier, T est contractant.

et non la fonction valeur au temps n+1 !

Nous référons par exemple à [GL00] pour l'existence d'une telle v.a.

voir Proposition 3.3.1 page

7 voir Théorème 3.3.1 page 72

et donc des méthodes de régression locale et de quantification

ces fonctions sont souvent appelées les "couches" (ou layer, en anglais) du réseau de neurones dans la littérature de l'apprentissage profond.

Des exemples classiques de fonctions d'activation sont : la fonction ReLu : x → max(x, 0),

Une caractérisation importante des nouveaux algorithmes que nous proposerons plus tard dans l'introduction, est que l'on a découplé l'apprentissage des deux paramètres optimaux.

La méthode de pré-training porte aussi le nom de transfer learning dans la littérature du deep learning.

en baissant le paramètre d'apprentissage η dans la SGD notamment.

surtout en grande dimension !

Dans le cas où l'agent n'a aucune idée de quelle mesure de training choisir, il peut toujours utiliser des méthodes de bootstrapping. Nous référons à la section 4.3.3 page 118 pour plus de détails à ce sujet.

La notation x M = O(y M ) quand M → ∞, signifie que le ratio |x M |/|y M | est borné quand M tend vers l'infini.

La notation x M = O P (y M ) quand M → ∞ signifie qu'il existe c > 0 telle que P |x M | > c|y M | → 0 quand M tend vers l'infini.

remarquons que l'on a toujours V M n ≥ Vn, par définition.

L'algorithme Qknn a été présenté dans la Section 2.1.1.3

et non de n à N comme dans l'algorithme NNcontPI.

Qknn introduit dans la section 2.1.1.3 page 33.

Voir théorème 2.3.1 pour les notations utilisées.

N -1 i=0 N ε N ,ṽ i , ce que nous avons interprété comme la somme des erreurs d'approximation par réseaux de neurones de la fonction u qui sont faites à chaque pas de temps. Il aurait été plus naturel d'avoir la somme d'erreurs suivante N -1 i=0 ε N ,ṽ i dans les théorèmes, i.e. une vraie somme d'erreurs d'approximation par réseaux de neurones, où le N n'apparaît pas dans la somme. Le N est en fait un artefact de calcul, dont nous n'avons pas pu nous défaire.

This Chapter is based on a paper written in collaboration with Frédéric Abergel and Huyên Pham.

Note that we restrict ourselves to the feedback controls here

The value function for the market-making problem is by definition a non-decreasing function w.r.t. the time component

This Chapter is based on a paper written in collaboration with Alessandro Balata, Mathieu Laurière, Huyên Pham, and Isaque Pimentel. This paper is published in ESAIM Proceedings and surveys, 65, (2019)

This procedure is also referred to as the "policy iteration" in the literature.

L 2 (σ(Zt n )) is the space of the square-integrable σ(Zt n )-measurable r.v.

If the assumption does not hold, the least squares problem can still be solved via SVD approach, which is consistent with regression techniques. See e.g. chapter 8 in[START_REF] Gobet | Monte-Carlo methods and stochastic processes: from linear to non-linear[END_REF].

Basically, different randomized controls may drive the process Z to very different locations, and the estimations will suffer from inaccuracy on the states that have been rarely visited.

often referred to as the Q-function, or action-value function, in the reinforcement learning literature. Be aware that Q stands here for the "Quality" of an action taken in a given state, and in particular does not refer to Quantization.

We use the notation tB = {tb, b ∈ B}, where t ∈ R and B is a set.

y is already given by (4.4.7). However, the boundary conditions on V (t, 0, y) and V (t, x, 0) are missing, except the trivial condition consisting of V (t, 0, 0) = 0. We then provided the HJB without boundary conditions to the Mathematica function NDSolve, and let the latter add artificial boundary conditions by itself to output results.

This Chapter is based on a paper written in collaboration with Huyên Pham, Achref Bachouch and Nicolas Langrené.

The notation x M = O(y M ) as M → ∞, means that the ratio |x M |/|y M | is bounded as M goes to infinity.

The notation x M = O P (y M ) as M → ∞, means that there exists c > 0 such that P |x M | > c|y M | → 0 as M goes to infinity.

The probability mass function of a Rademacher r.v. is by definition 1 2 δ -1 + 1 2 δ 1 .

We denote by epoch one pass of the full training set.

The Q-value at time n, denoted by Qn, is defined as the function that takes the couple state-action (x, a) as argument, and returns the expected optimal reward earned from time n to time N when the process X is at state x and action a is chosen at time n; i.e. Qn :R d × R q ∈ (x, a) → f (x, a) + E a n,x [V n+1 (X n+1 )].

We could have chosen other algorithms to optimize the Q-value, but, in our tests, Brent was faster than the other choices that we tried, such as GoldenSearch, and always provided accurate estimates of the optimal controls.

The Exponential Linear Unit (ELU) activation function is defined asx → exp(x) -1 if x ≤ 0 x if x > 0 .

This Chapter is based on a paper written in collaboration with Huyên Pham and Xavier Warin.

Remerciements

Acknowledgments.

Most of this work was realized while the third author was a postdoctoral fellow at NYU Shanghai; the support of a research discretionary fund from the NYU-ECNU Institute of Mathematical Sciences and the support provided by the CEMRACS for his stay at CIRM are gratefully acknowledged. The authors's research is part of the ANR project CAESARS (ANR-15-CE05-0024). Therefore, by minimizing over θ this quadratic loss function, via SGD based on simulations of (X ti , X ti+1 , ∆W ti ) (called training data in the machine learning language), one expects the neural networks U i and Z i to learn/approximate better and better the functions u(t i , .) and σ (t i , )D x u(t i , ) in view of the universal approximation theorem [HSW90]. Similarly, the second scheme DPDP2, which uses only neural network on the value functions, learns u(t i , .) by means of the neural network U i , and σ (t i , )D x u(t i , ) via σ (t i , ) Dx U i . The rigorous arguments for the convergence of these schemes will be derived in the next section.

The advantages of our two schemes, compared to the Deep BSDE algorithm, are the following:

Deep learning for Markov decision processes (MDPs)

Chapter 5. Neural networks algorithms for control problems: convergence analysis

Control learning by performance iteration

This algorithm, refereed in short as NNcontPI Algo, is designed as follows:

• For n = N -1, . . . , 0, we keep track of the approximated optimal policies âk , k = n + 1, . . . , N -1, and approximate the optimal policy at time n by ân = A(.; βn ) with βn ∈ arg min

where X n µ n , Xβ n+1 = F (X n , A(X n ; β), ε n+1 ) P A(Xn;β) (X n , dx ), and for k = n + 1, . . . , N -1,

Given estimate âM k of âk , k = n+1, . . . , N -1, the approximated policy ân is estimated by using a training sample X , and optimizing over the parameters β ∈ R l of the NN A(.; β) ∈ A, the expectation in (5.3.1) by stochastic gradient descent method (or its variants) as described in Section (5.2.2).

We then get an estimate of the optimal policy at any time n = 0, . . . , N -1 by:

where βM n is the "optimal" parameter resulting from the SGD in (5.3.1) with a training sample of size M . This leads to an estimated value function given at any time n by

,

where E M is the expectation conditioned on the training set (used for computing âM k k ), and Xn,x k , for m = 1, . . . , M , used at time k = n, . . . , N , is emphasized through the exponent M in the notations.

Remark 5.3.1. The NNcontPI Algo can be viewed as a combination of the DNN algorithm designed in [START_REF] Han | Deep learning approximation for stochastic control problems[END_REF] and dynamic programming. In the algorithm presented in [START_REF] Han | Deep learning approximation for stochastic control problems[END_REF], which totally ignores the dynamic programming principle, one learns all the optimal controls A(.; β n ), n = 0, . . . , N -1 at the same time, by performing one unique stochastic gradient descent. This is efficient as all the parameters of all the NN are getting trained at the same time, using the same mini-batches. However, when the number of layers of the global neural network gathering all the NN A(.; β n ), n = 0, . . . , N -1 is large (say N -1 n=0 n ≥ 100, where n is the number of layers in A(., β n )), then one is likely to observe vanishing or exploding gradient problems that will affect the training of the weights and biais of the first layers of the global NN (see [G 17] for more details). Therefore, it may be more reasonable to make use of the dynamic programming structure when N is large, and learn the optimal policy sequentially as proposed in our NNcontPI Algo. Notice that a similar idea was already used in [START_REF] Guyon | Uncertain volatility model: a Monte-Carlo approach[END_REF] in the context of uncertain volatility model where the authors use a specific parametrization for the feedback control instead of a DNN adopted more generally here.

Remark 5.3.2. The NNcontPI Algo does not require value function iteration, but instead is based on performance iteration by keeping track of the estimated optimal policies computed in backward recursion. The value function is then computed in (5.3.2) as the gain functional associated to the estimated optimal policies (â M k ) k . Consequently, it provides usually a low bias estimate but induces possibly high variance estimate and large complexity, especially when N is large.

Control learning by hybrid iteration

Instead of keeping track of all the approximated optimal policies as in the NNcontPI Algo, we use an approximation of the value function at time n + 1 in order to compute the optimal policy at time n. The approximated value function is then updated at time n by relying on the martingale property (5.1.3) under the optimal control. This leads to the following generic algorithm: Generic Hybrid Algo 1. Initialization: VN = g 2. For n = N -1, . . . , 0, (i) Approximate the optimal policy at time n by ân = A(.; βn ) with βn ∈ arg min

where X n µ n , XA(.,β) n+1 = F (X n , A(X n ; β), ε n+1 ) P A(Xn;β) (X n , dx ).

(ii) Updating: approximate the value function by

The approximated policy ân is estimated by using a training sample (X

n+1 ), m = 1, . . . , M of (X n , ε n+1 ) to simulate X n , X A(.;β) n+1

, and optimizing over the parameters β ∈ R l of the NN A(.; β) ∈ A, the expectation in (5.3.3) by stochastic gradient descent method (or its variants) as described in Section (5.2.2). We then get an estimate âM n = A .; βM n . The approximated value function written as a conditional expectation in (5.3.4) is estimated according to a Monte Carlo regression, either by a regress now method (in the spirit of [START_REF] Kohler | Pricing of high-dimensional American options by neural networks[END_REF]) or a regress later (in the spirit of [START_REF] Glasserman | Simulation for American options: Regression Now or Regression Later?[END_REF] and [START_REF] Balata | Regress-Later Monte-Carlo for optimal inventory control with applications in energy[END_REF]) joint with quantization approach, and this leads to the following algorithmic variants detailed in the two next paragraphs.

Hybrid-Now Algo

Given an estimate âM n of the optimal policy at time n, and an estimate V M n+1 of Vn+1 , we estimate Vn by neural networks regression, i.e.,

. In other words, we have

where θM n is the "optimal" parameter resulting from the SGD in (5.3.5) with a training sample of size M .

Chapter 5. Neural networks algorithms for control problems: convergence analysis

Comparison of the algorithms

We end this section with a synthetic comparison of the proposed algorithms. We emphasize the pros (+) and cons (-) of the three proposed algorithms in terms of bias estimate for the value function, variance, complexity and dimension for the state space.

Algo

Bias

Convergence analysis

This section is devoted to the convergence of the estimator V M n of the value function V n obtained from a training sample of size M and using DNN algorithms listed in Section 5.3.

Training samples rely on a given family of probability distributions µ n on X , for n = 0, . . . , N , refereed to as training distribution (see Section 5.3.3 for a discussion on the choice of µ). For sake of simplicity, we consider that µ n does not depend on n, and denote then by µ the training distribution. We shall assume that the family of controlled transition probabilities has a density w.r.t. µ, i.e., P a (x, dx ) = r(x, a; x )µ(dx ).

We shall assume that r is uniformly bounded in (x, x , a) ∈ X 2 × A, and uniformly Lipschitz w.r.t. (x, a), i.e., (Hd) There exists some positive constants r ∞ and [r] L s.t.

Remark 5.4.1. Assumption (Hd) is usually satisfied when the state and control space are compacts. While the compactness on the control space A is not explicitly assumed, the compactness condition on the state space X turns out to be more crucial for deriving estimates on the estimation error (see Lemma 5.4.1), and will be assumed to hold true for simplicity. Actually, this compactness condition on X can be relaxed by truncation and localization arguments (see proposition 5.A.1 in the appendix) by considering a training distribution µ such that (Hd) is true and which admits a moment of order 1, i.e. |y|dµ(y) < +∞.

We shall also assume some boundedness and Lipschitz condition on the reward functions:

Chapter 5. Neural networks algorithms for control problems: convergence analysis Remark 5.4.3. Under (Hd), (HR) and (HF), it is straightforward to see from the dynamic programming formula 5.1 that V n is Lipschitz for all n = 0, . . . , N , with a Lipschitz coefficient [V n ] L , which -can be bounded by the minimum of the two following bounds:

which holds since we have by standard arguments:

Note that we use the usual convention 1-x p 1-x = p for p ∈ N * and x = 0. The Lipschitz continuity of V n plays a significant role to prove the convergence of the Hybrid and the LaterQ algorithms described and studied in sections 5.4.2 and 5.4.3.

Control learning by performance iteration (NNcontPI)

In this paragraph, we analyze the convergence of the NN control learning by performance iteration as described in Section 5.3.1. Actually, we shall consider neural networks for the optimal policy with one hidden layer, K neurons with total variation 3 smaller than γ, kernel bounded by η, Relu activation function for the hidden layer, and activation function σ A for the output layer (in order to ensure that the NN is valued in A): this is represented by the parametric set of functions

where . is the Euclidean norm in R d .

Let K M , η M and γ M be sequences of integers such that

(5.4.3)

We denote by A M := η M A γ M k M the class of neural network for policy with norm η M on the kernel a = (a ij ), K M neurons and norm γ M that satisfy conditions (5.4.3).

.

3 The total variation for the class of NN A γ K is equal to K i=0 |c ij | with the notations above. See e.g. [START_REF] Bach | Breaking the curse of dimensionality with convex neural networks[END_REF] for a general definition.

Convergence analysis

Step 3. From (5.4.15) and (5.4.16), we have

(5.4.17)

By induction, this implies

.

Use the estimations (5.4.10) for ε esti PI,n in Lemma 5.4.1, and (5.4.12) for ε approx PI,n in Lemma 5.4.2, and observe that Vn (X n ) ≥ V n (X n ) holds a.s., to complete the proof of (5.4.1). Finally, the proof of (5.4.6) is obtained by taking expectation in (5.4.17), and using estimations (5.4.9) and (5.4.11).

Hybrid-Now algorithm

In this paragraph, we analyze the convergence of the hybrid-now algorithm as described in Section 5.3.2.1. We shall consider neural networks for the value function estimation with one hidden layer, K neurons with total variation γ, kernel bounded by η, a sigmoid activation function σ for the hidden layer, and no activation function for the output layer (i.e. the last layer): this is represented by the parametric set of functions

Let η M , K M and γ M be integers such that:

where we remind that ρ M is defined in (HF).

In the sequel we denote by V M := η M V γ M K M the space of neural networks for the estimated value functions at time n = 0, . . . , N -1, parametrized by the values η M , γ M and K M that satisfy (5.4.18). We also consider the class A M of neural networks for estimated feedback optimal control at time n = 0, . . . , N -1, as described in Section 5.4.1, with the same parameters η M , γ M and K M .

Recall that the approximation of the value function and optimal policy in the hybrid-now algorithm is computed in backward induction as follows:

Convergence analysis

For this, take

M , let δ > δ M , and denote

Apply Lemma 5.H.2 to obtain:

where N 2 (ε, V, x M 1 ) stands for the ε-covering number of V on x M 1 , which is introduced in section 5.H, and where the last line holds since we assumed

, which implies that (5.H.1) holds by (5.4.23). Therefore, by application of Lemma 5.H.1, it holds:

and this completes the proof of (5.4.2).

Next, let us show inf

.

For this, take some arbitrary Φ ∈ V M and split

Chapter 5. Neural networks algorithms for control problems: convergence analysis

To bound the last term in the r.h.s. of (5.4.25), we write

Use the dynamic programming principle, assumption (Hd) and (5.4.22) to get:

.

We then notice that

, and use Lemma 5.4.4 to bound ε approx HN,n . By plugging into (5.4.25), and using the estimations in Lemmas 5.4.3 and 5.4.4, we obtain the estimate (5.4.24). Together with (5.4.2), this proves the required estimate(5.4.2). By induction, we get as M → ∞,

which completes the proof of Theorem 5.4.2.

Hybrid-LaterQ algorithm

In this paragraph, we analyze the convergence of the Hybrid-LaterQ algorithm described in Section 5.3.2.2.

We shall make the following assumption on F to ensure the convergence of the Hybrid-LaterQ algorithm.

(HF-LQ) Assume F to be such that:

1. (Estimation error Assumption) (HF) holds, i.e. for all e ∈ E, there exists C(e) > 0 such that for all couples (x, a) and (x , a ) in X × A:

Convergence analysis

Recall that for all integer M > 0, ρ M is defined as

where the (ε m ) m is a i.i.d. sample of the noise ε.

(Quantization Assumption

) There exists a constant [F ] L > 0 such that for all (x, a) ∈ X × A and all pair of r.v. (ε, ε ), it holds:

As for the hybrid-now algorithm, we shall consider neural networks for the value function estimation with one hidden layer, K neurons with total variation γ, kernel bounded by η, a sigmoid activation function σ for the hidden layer, and no activation function for the output layer (i.e. the last layer), which is represented by the parametric set of function η V γ K . Let η M , K M and γ M be integers such that:

(5.4.26)

In the sequel we denote by V M := η M V γ M K M the space of neural networks parametrized by the values η M , γ M and K M that satisfy (5.4.26). We also consider the class A M of neural networks for estimated feedback optimal control at time n = 0, . . . , N -1, as described in Section 5.4.1, with the same parameters η M , γ M and K M .

Recall that the approximation of the value function and optimal policy in the Hybrid-LaterQ algorithm is computed in backward induction as follows: (i) compute the approximated policy at time n

where

(5.4.27)

and set the truncated interpolation of the value function at time n + 1

(iii) update/compute the estimated value function

where εn is a L-optimal quantizer of ε n on the grid {e 1 , . . . , e L } with weights (p 1 , . . . , p L ).

Chapter 5. Neural networks algorithms for control problems: convergence analysis Remark 5.4.11. 1. It is straightforward to see that the neuronal network functions in V M are Lipschitz with Lipschitz coefficient bounded by η M γ M . We highly rely on this property to show the convergence of the Hybrid-LaterQ algorithm.

2. Note that (5.4.27) is an interpolation step. In the pseudo-code above, we decided to interpolate the value function Ṽ Q n using neural networks in V M by reducing an empirical quadratic norm. However, we could have chosen other families of functions and other loss criterion to minimize. Gaussian processes have been recently reconsidered to interpolate functions, see [START_REF] Ludkovski | Simulation methods for stochastic storage problems: a statistical learning perspective[END_REF].

We now state our main result about the convergence of the Hybrid-LaterQ algorithm.

Theorem 5.4.3. Assume that there exists an optimal feedback control (a opt k ) N -1 k=n for the control problem with value function V n , n = 0, . . . , N . Take X n µ, and let L M be a sequence of integers such that

Take L M points for the optimal quantization of the exogenous noise. Then, it holds as M → ∞:

(5.4.28)

Comment: Theorem 5.4.3, combined to Proposition 5.4.1, show that estimator V M n provided by Hybrid-LaterQ algorithm is consistent, i.e. converges in L 1 toward the value function V n at time n when the number of points for the regression and quantization goes to infinity. Remark 5.4.12. Note that the dimension d of the state space appears (explicitly) in the quantization error written in (5.4.28), as well as (implicitly) in the approximation errors associated to the value functions and optimal control learning. See for example (5.4.8) for an explicit dependence of the approximation error on d.

In order to prove Theorem 5.4.3, let us introduce the estimation error at time n associated to the Hybrid-LaterQ algorithm by

where

, and E A M,n,Xn [.] stands for the expectation conditioned by the training set and X n when decision A has been taken at time n.

We have the following bound on this estimation error: Lemma 5.4.5. For n = 0, . . . , N -1, it holds:

Convergence analysis

Moreover,

Remark 5.4.13. The result stated in Lemma 5.4.5 is the same as the one stated in Lemma 5.4.3 for the hybrid-now algorithm. This result can actually be proved using the same arguments, so we omit the proof here.

Next, we introduce the approximation error at time n associated to the Hybrid-LaterQ algorithm by

where

We have the following bound on this approximation error, which is similar to the one stated in Lemma 5.4.4 for the Hybrid-Now algorithm. The proof is similar and is thus omitted here. Lemma 5.4.6. For n = 0, ..., N -1, it holds:

Proof of Theorem 5.4.3.

We split the L 1 norm as follows:

(5.4.29)

where ( V M n ) n is defined as:

and ( V opt n ) n is defined as:

1 p stands for the L p -norm conditioned by the training set, for p ∈ {1, 2}.

Step 1: The first term in the r.h.s. of (5.4.29) is the quantization error. We show that

is Lipschitz, for n ∈ {0, ..., N }, with its Lipschitz coefficient bounded by η M γ M , we thus get:

Convergence analysis

Indeed, first write

, where Φ = argmin

. Then apply the Central limit theorem to get (5.4.32).

Plugging (5.4.32) into (5.4.31) leads to

Apply the triangular inequality to finally obtain:

It remains to notice that

, to obtain inequality ((HF-LQ)).

Step 3: let us show

(5.4.33)

Note that once again it holds

Chapter 5. Neural networks algorithms for control problems: convergence analysis which can be shown by similar arguments as those used to prove of inequality (5.4.21). It remains to bound the estimation and approximation errors by using estimations (5.4.5) and (5.4.6) to get (5.4.33).

Step 4: We show

(5.4.34)

, where X n+1 ∼ µ. For this, denote by ( V n ) 0≤n≤N the following auxiliary process:

and notice that we have under assumption (Hd):

, as stated in (5.4.34).

Step 5 Conclusion: By plugging (5.4.30), (5.4.33) and (5.4.34) into (5.4.29), we derive the following bound for the l.h.s. of (5.4.29):

, as M → +∞.

By induction, we get for n = 0, . . . , N -1:

which is the result stated in Theorem 5.4.3.

Conclusion

This paper develops new machine learning algorithms for high-dimensional Markov decision process. We propose and compare three algorithms based on dynamic programming, performance/hybrid iteration and neural networks for approximating the control and value function. The main theoretical contribution is to provide a detailed convergence analysis for each of these algorithms: by using least squares neural network regression, we state error estimates in terms of the universal approximation error of neural networks, and of the statistical risk when estimating the network function. Numerical tests on various applications are presented in a companion paper [START_REF] Bachouch | Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications[END_REF].

Chapter 5. Neural networks algorithms for control problems: convergence analysis

) is measurable w.r.t. the σ-algebra generated by the training set, so that conditioning by the training set and injecting (5.C.2) yields

, where we use the definition of A * to go from the second-to-last to the last line. The proof of (5.C.1) is then completed.

Step 2: We show that

, and notice

The second term in the r.h.s. of (5.C.4) comes from (5.C.1). It remains to write the latter as an expectation to obtain (5.C.3).

5.F. Proof of Lemma 5.4.3

which is the required result stated in (5.4.7).

We now show (5.4.8) of proposition 5.4.1: As stated in section 4.7 of [Bac17]: proposition 6 in [START_REF] Bach | Breaking the curse of dimensionality with convex neural networks[END_REF] shows that we can approximate a c-Lipschitz function by a γ 1 -norm less than γ M and uniform error less than c γ M c -2d/(d+1) log γ M c , and proposition 1 in [START_REF] Bach | Breaking the curse of dimensionality with convex neural networks[END_REF] shows that a function with γ 1 less than γ M may be approximated with K M neurons with uniform error γ M K -(d+3)/(2d) M . Thus, given K M and γ M , there exists a neural network a * in V M such that

5.F Proof of Lemma 5.4.3

We prove Lemma 5.4.3 in four steps. Since the proof is very similar to the one of Lemma 5.4.1, we only detail the arguments that are modified.

Step 1: Symmetrization by a ghost sample. We take ε > 0 and show that for M > 2

where:

• we define

.

Proof: Since V M n the estimated value function at time n, for n=0, ..., N -1, is bounded by construction (we truncated the estimation at the last step of the pseudo-code of the Hybrid algorithm), the proof is the same as the one in step 1 of Lemma 5.4.1.

Chapter 5. Neural networks algorithms for control problems: convergence analysis

Step 2: The following result holds

(5.F.1)

Proof: same as step 2 in the proof of Lemma 5.4.1.

Step 3: Introduction of additional randomness by random signs.

The following result holds:

.F.2)

Proof: same as step 3 in the proof of Lemma 5.4.1.

Step 4: We show that

Adding and removing the cost obtained by control 0 at time n yields:

The first term in the r.h.s. in (5.F.4) is bounded as in the proof of Lemma 5.4.1 by

5.H. Some useful Lemmas for the proof of Theorem 5.4.2 for some c, σ > 0. Let γ M , L ≥ 1 and assume that the regression function is bounded by L and that γ M -----→

for some V M of functions Φ : R d → [-γ M , γ M ] and some random variables Ȳ1 , ..., ȲM which are bounded by L, and denote

Then there exists constants c 1 , c 2 > 0 which depend only on σ and c such that for any δ M > 0 with

for all δ ≥ δ M and all g ∈ V M ∪ {m} we have as M → +∞:

Proof. See [START_REF] Kohler | Nonparametric regression with additional measurement errors in the dependent variable[END_REF].

Lemma 5.H.2. Let V M be defined as in Section 5.4.2. For any ε > 0, we have

.

Proof. See Lemma 5.1 in [START_REF] Kohler | Pricing of high-dimensional American options by neural networks[END_REF].

Chapter 6

Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications 1

Abstract This paper presents several numerical applications of deep learning-based algorithms that have been introduced in [START_REF] Huré | Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis[END_REF]. Numerical and comparative tests using TensorFlow illustrate the performance of our different algorithms, namely control learning by performance iteration (algorithms NNcontPI and ClassifPI), control learning by hybrid iteration (algorithms Hybrid-Now and Hybrid-LaterQ), on the 100-dimensional nonlinear PDEs examples from [START_REF] Han | Deep learning-based numerical methods for highdimensional parabolic partial differential equations and backward stochastic differential equations[END_REF] and on quadratic backward stochastic differential equations as in [START_REF] Chassagneux | Numerical simulation of quadratic BSDEs[END_REF]. We also performed tests on low-dimension control problems such as an option hedging problem in finance, as well as energy storage problems arising in the valuation of gas storage and in microgrid management. Numerical results and comparisons to quantization-type algorithms Qknn, as an efficient algorithm to numerically solve low-dimensional control problems, are also provided; and the corresponding codes are available on https://github.com/comeh/.

Key words:

Deep learning, algorithms, performance iteration, value iterations, Monte-Carlo, quantization.

Algorithms

(ii) take a very small learning rate parameter, for the Adam optimizer, that guarantees the stability of the parameters' updates from the gradient-descent based learning procedure.

Doing so, one obtains stable estimates of the value function and optimal control, which is desirable. We highlight the fact that this stability procedure cannot be implemented in most of the algorithms proposed in the literature (for example the ones presented in [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF] which are based on regress-now, regress-later or quantization).

Quantization with k-nearest-neighbors: Qknn-algorithm

Algorithm 6.5 presents the pseudo-code of an algorithm based on the quantization and k-nearest neighbors methods, called Qknn, which will be the benchmark in all the low-dimensional control problems that will be considered in Section 6.3 to test NNContPI, ClassifPI, Hybrid-Now and Hybrid-Later. Also, comparisons of Algorithm 6.5 to other well-known algorithms on various control problems in low-dimension are performed in [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF], which show in particular that Algorithm 6.5 works very well to solve low-dimensional control problems. Actually, in our experiments, Algorithm 6.5 always outperforms the other algorithms based either on regress-now or regress-later methods whenever the dimension of the problem is low enough for Algorithm 6.5 to be feasible.

Just as it has been done in Section 6.2.2.2, we consider an L-optimal quantizer of the noise ε n , i.e. a discrete random variable εn valued in a grid {e 1 , . . . , e L } of L points in E, and with weights p 1 , . . . , p L . We also consider grids Γ k , k = 0, . . . , N of points in R d , which are assumed to properly cover the region of R d that is likely to be visited by the optimally driven process X at time k = 0, . . . , N -1. These grids can be viewed as samples of well-chosen training distributions where more points are taken in the region that is likely to be visited by the optimally driven controlled process (see Remark 6.2.1 for details on the choice of the training measure). Remark 6.2.8. The estimate of the Q-value at time n given by (6.2.8) is not continuous w.r.t. the control variable a, which might cause some stability issues when running Qknn, especially during the optimization procedure (6.2.9). We refer to Section 3.2.2. in [START_REF] Balata | A class of finitedimensional numerically solvable McKean-Vlasov control problems[END_REF] for a detailed presentation of an extension of Algorithm 6.5 where the estimates of the Q value function Q n is continuous w.r.t. the control variable.

Chapter 6. Neural networks algorithms for control problems: numerical applications Here, we denote p + = max(p, 0). Assuming for simplicity that the battery is fully efficient, the capacity charge (C α n ) n of the BES, valued in [0, C max ], evolves according to the dynamics

The imbalance process defined by

represents how well we are doing for satisfying electricity supply: the ideal situation occurs when S α n = 0, i.e., perfect balance between demand and generation. When S α n > 0, this means that demand is not satisfied, i.e., there is missing power in the microgrid, and when S α n < 0, there is an excess of electricity. In order to ensure that there is no missing power, we impose the following constraint on the admissible control:

but penalize the excess of electricity when S α n < 0 with a proportional cost Q -> 0. We model the residual demand as a mean-reverting process:

where (ε n ) n are i.i.d., R ∈ R, and < 1. The goal of the microgrid manager is to find the optimal (admissible) decision α that minimizes the functional cost

where (.) is the cost function for fuel consumption: (0) = 0, and e.g. (a) = Ka γ , with K > 0, γ > 0. This stochastic control problem fits into the 3-dimensional framework of Section 5.1 (see also Remark 6.2.4) with control α valued in A = {0} × [A min , A max ], X α = (C α , M α , R), noise ε n+1 , starting from an initial value (C α 0 , M α 0 , R 0 ) = (c 0 , 0, r 0 ) on the state space [0, C max ] × {0, 1} × R, with dynamics function

zero terminal cost g = 0, and control constraint

Remark 6.3.2. The state/space constraint is managed in our NN-based algorithm by introducing a penalty function into the running cost (see Remark 6.2.4):

with large Q + taken much larger than Q -. Doing so, the NN-based estimate of the optimal control learns not to take any forbidden decision.

Chapter 6. Neural networks algorithms for control problems: numerical applications Table 6.3 -Estimates of the value function at time 0 and state (C0 = 0, M0 = 0, R0 = 0.1), for N = 30 and Cmax = 1, using Qknn and ClassifHybrid algorithms. Note that ClassifHybrid got a better result than Qknn on this problem.

Mean std ClassifHybrid 33.34 0.31 Qknn 35.37 0.34 available on the right to inform how much power it is optimal to generate in both cases. Observe that the optimal decisions are quite intuitive: for example, if the demand is high and the battery is empty, then it is optimal to generate a lot of energy. Moreover, it is optimal to turn the generator off if the demand is negative or if the battery is charged enough to meet the demand. We plot in Figure 6.11 the estimated optimal decisions at times n = 1, 10, 28, using the Hybrid-Now algorithm, with N = 30 time steps. See that the decisions are similar to the ones given using Qknn.

Note that the plots in Figure 6.10 and 6.11 look much better than the ones obtained in [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF] in which algorithms based on regress-now or regress-later are used (see in particular Figure 4 in [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF]); hence Qknn and ClassifHybrid are more stable than the algorithms proposed in [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF].

We report in Table 6.3 the result for the estimates of the value function with N =30 time steps, obtained by running 10 times a forward Monte Carlo with 10,000 simulations using the optimal strategy estimated using Qknn and ClassifHybrid algorithms. Observe that Hybrid-Now performs better than Qknn. However, Qknn run in less than a minute whereas Hybrid-Now needed seven minutes to run. We also report in Table 6.4 the result for the estimatest of the value function with N =200 time steps, obtained by running 20 times a forward Monte Carlo with 10,000 simulations using the Qknnestimated optimal strategy. Figure 6.12 shows two simulations of (C, M, R) controlled using the Qknn-estimated optimal strategy, where N = 200 has been chosen. Observe in particular the natural behavior of the Qknn-decisions which consists in turning the generator on when the demand cannot be met by the battery, and turn it off when the demand is negative or when the battery is charged enough to meet the demand. Note that the plots are similar to the ones plotted in Figure 9 of [START_REF] Alasseur | Regression Monte Carlo for microgrid management[END_REF].

Comments on Qknn: Note that there is no need to use a penalization method with the Qknnalgorithm to constrain the control to stay in A n (x), where x is the state at time n, since, for all state x, we can simply search for the optimal control associated in A n (x), using e.g. the Brent algorithm. For n = 0, . . . , N -1, we took the training set as follows:

and where Γ 1 is the optimal grid for the quantization of N (0, 1), available in http://www.quantize.maths-fi.com, with 51 points. This choice of training points for the C component corresponds to the exploration procedure discussed in Remark 6.2.1, whereas we chose the best grid with 51 points for the (uncontrolled) R component. Comments on ClassifHybrid: We took 100 mini-batches of size 300 and took 100 epochs to run the algorithm. We chose the following training distribution at time n: µ n = U(C min , C max ) × U({0, 1}) × P Rn , where P Rn is the law of the (uncontrolled) residual demand at time t n . Note that such a choice of training distribution means that we want to explore all the available states for the controlled components of the controlled process (C, M, R) in order to learn the optimal strategy globally.

The microgrid management problem is very challenging for our algorithms because the control space {0} ∪ [a min , a max ] is a mix of discrete and continuous space, moreover the choice of the optimal control is subject to constraints. We designed ClassifHybrid, an Hybrid version of ClassifPI, to solve this problem. ClassifHybrid provided very good estimates and actually managed to perform better than Qknn.

Introduction

This paper is devoted to the resolution in high dimension of nonlinear parabolic partial differential equations (PDEs) of the form

and a second-order generator L defined by

Here µ is a function defined on

with values in M d the set of d × d matrices, and L is the generator associated to the forward diffusion process:

with W a d-dimensional Brownian motion on some probability space (Ω, F, P) equipped with a filtration F = (F t ) 0≤t≤T satisfying the usual conditions.

Due to the so called "curse of dimensionality", the resolution of nonlinear PDEs in high dimension has always been a challenge for scientists. Until recently, only the BSDE (Backward Stochastic Differential Equation) approach first developed in [START_REF] Pardoux | Adapted solution of a backward stochastic differential equation[END_REF] was available to tackle this problem: using the time discretization scheme proposed in [START_REF] Bouchard | Discrete-time approximation and Monte-Carlo simulation of backward stochastic differential equations[END_REF], some effective algorithms based on regressions manage to solve non linear PDEs in dimension above 4 (see [START_REF] Gobet | A regression-based Monte Carlo method to solve backward stochastic differential equations[END_REF][START_REF] Lemor | Rate of convergence of an empirical regression method for solving generalized backward stochastic differential equations[END_REF]). However this approach is still not implementable in dimension above 6 or 7 : the number of basis functions used for the regression still explodes with the dimension.

Quite recently some new methods have been developed for this problem, and three kind of methodologies have emerged:

• Some are based on the Feyman-Kac representation of the PDE. Branching techniques [START_REF] Henry-Labordere | Branching diffusion representation of semilinear PDEs and Monte Carlo approximation[END_REF] have been studied and shown to be convergent but only for small maturities and some small nonlinearities. Some effective techniques based on nesting Monte Carlo have been studied in [War18b; War18a]: the convergence is proved for semi-linear equations. Still based on this Feyman-Kac representation some machine learning techniques permitting to solve a fixed point problem have been used recently in [START_REF] Chan-Wai-Nam | Machine Learning for semi linear PDEs[END_REF]: numerical results show that it is efficient and some partial demonstrations justify why it is effective.

• Another class of methods is based on the BSDE approach and the curse of dimensionality issue is partially avoided by using some machine learning techniques. [HJE17; EHJ17] propose a deep learning based technique called Deep BSDE. Based on an Euler discretization of the forward underlying SDE X t , the idea is to view the BSDE as a forward SDE, and the algorithm tries to learn the values u and z = σ Du at each time step of the Euler scheme by minimizing a global loss function between the forward simulation of u till maturity T and the target g(X T ).

• At last, using some machine learning representation of the solution, [START_REF] Sirignano | DGM: A deep learning algorithm for solving partial differential equations[END_REF] proposes to use the automatic numerical differentiation of the solution to solve the PDE on a finite domain.

7.3. Deep learning-based schemes for semi-linear PDEs

The deep BSDE scheme of [HJE17]

The DBSDE algorithm proposed in [HJE17; EHJ17] starts from the BSDE representation (7.3.1) of the solution to (7.1.1), but rewritten in forward form as:

The forward process X in equation (7.1.2), when it is not simulatable, is numerically approximated by an Euler scheme X = X π on a time grid: π = {t 0 = 0 < t 1 < . . . < t N = T }, with modulus |π| = max i=0,...,N -1 ∆t i , ∆t i := t i+1 -t i , and defined as

where we set ∆W ti := W ti+1 -W ti . To alleviate notations, we omit the dependence of X = X π on the time grid π as there is no ambiguity (recall that we use the notation X for the forward diffusion process). The approximation of equation (7.1.1) is then given formally from the Euler scheme associated to the forward representation (7.3.2) by

In [HJE17; EHJ17], the numerical approximation of u(t i , X ti ) is designed as follows: starting from an estimation U 0 of u(0, X 0 ), and then using at each time step t i , i = 0, . . . , N -1, a multilayer neural network x ∈ R d → Z i (x; θ i ) with parameter θ i for the approximation of x → σ (t i , x)D x u(t i , x):

one computes estimations U i of u(t i ; X ti ) by forward induction via:

for i = 0, . . . , N -1. This algorithm forms a global deep neural network composed of the neural networks (7.3.5) of each period, by taking as input data (in machine learning language) the paths of (X ti ) i=0,...,N and (W ti ) i=0,...,N , and giving as output U N = U N (θ), which is a function of the input and of the total set of parameters θ = (U 0 , θ 0 , . . . , θ N -1 ). The output aims to match the terminal condition g(X t N ) of the BSDE, and one then optimizes over the parameter θ the expected square loss function:

This is obtained by stochastic gradient descent-type (SGD) algorithms relying on training input data.

New schemes: DBDP1 and DBDP2

The proposed scheme is defined from a backward dynamic programming type relation, and has two versions:

(1) First version:

Chapter 7. Deep learning schemes for high-dimensional PDEs

The localized approximation error at time t i , for 0 ≤ i ≤ N -1, should then be rewritten as

and the non-localized one as

converges to zero, as m goes to infinity, for any K > 0, as claimed by the universal approximation theorem (II) [START_REF] Hornik | Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks[END_REF]. On the other hand, from the expressions of vi , D x vi in the above proof of Theorem 7.4.2, we see under (H1)-(H2), and from (7.4.23) that for all

, for some positive constant C independent of m, π. We deduce by Cauchy-Schwarz and Chebyshev's inequalities that for all K > 0, and θ ∈ Θ γ i m , i = 0, . . . , N -1,

where we used (7.4.1) in the last inequality. This shows that

and thus, in theory, the error ε N ,m i,N can be made arbitrary small by suitable choices of large m and K.

Convergence of RDBDP

In this paragraph, we study the convergence of machine learning schemes for the variational inequality (7.3.8).

We first consider the case when f does not depend on z, so that the component Y t = u(t, X t ) solution to the reflected BSDE (7.3.9) admits a Snell envelope representation, and we shall focus on the error on Y by proposing an alternative to scheme (7.3.10), refereed to as RDBDPbis scheme, which only uses neural network for learning the function u:

(7.4.30) Then, update:

Let us also define from the scheme (7.4.30)

V ti := max[ Ṽti ; g(X ti )], i = 0, . . . , N -1.

(7.4.31)

Our next result gives an error estimate of the scheme (7.4.30) in terms of the L 2 -approximation errors of ṽi by neural networks U i , i = 0, . . . , N -1, and defined as

E ṽi (X ti ) -U i (X ti ; θ) 2 .

7.5. Numerical results u(t, .) and its estimate at time t = 0.5. Z and its estimate at time t = 0.5 u(t, .) and its estimate at time t = 0.0085. Z and its estimate at time t = 0.0085. Error on solution at date t = 0.5.

Error on solution at date t = 0.0085. The present thesis deals with numerical schemes to solve Markov Decision Problems (MDPs), partial differential equations (PDEs), quasi-variational inequalities (QVIs), backward stochastic differential equations (BSDEs) and reflected backward stochastic differential equations (RBSDEs). The thesis is divided into three parts.

The first part focuses on methods based on quantization, local regression and global regression to solve MDPs. Firstly, we present a new algorithm, named Qknn, and study its consistency. A time-continuous control problem of market-making is then presented, which is theoretically solved by reducing the problem to a MDP, and whose optimal control is accurately approximated by Qknn. Then, a method based on Markovian embedding is presented to reduce McKean-Vlasov control problem with partial information to standard MDP. This method is applied to three different McKean-Vlasov control problems with partial information. The method and high accuracy of Qknn is validated by comparing the performance of the latter with some finite difference-based algorithms and some global regression-based algorithm such as regress-now and regress-later. In the second part of the thesis, we propose new algorithms to solve MDPs in high-dimension. Neural networks, combined with gradient-descent methods, have been empirically proved to be the best at learning complex functions in high-dimension, thus, leading us to base our new algorithms on them. We derived the theoretical rates of convergence of the proposed new algorithms, and tested them on several relevant applications. In the third part of the thesis, we propose a numerical scheme for PDEs, QVIs, BSDEs, and RBSDEs. We analyze the performance of our new algorithms, and compare them to other ones available in the literature on several tests, which illustrates the efficiency of our methods to estimate complex solutions in high-dimension.

Keywords:

Deep Learning, MDPs, PDEs, optimal stopping time, BSDEs, RBSDEs, McKean-Vlasov, global regression, local regression, quantization, limit order book, algorithmic-trading, high-dimension.

Numerical Methods and Deep Learning for Stochastic Control Problems and Partial Differential Equations

La thèse porte sur les schémas numériques pour les problèmes de décisions Markoviennes (MDPs), les équations aux dérivées partielles (EDPs), les équation différentielles stochastiques rétrogrades (EDSRs), ainsi que les équations différentielles stochastiques rétrogrades réfléchies (EDSRs réfléchies). La thèse est divisée en trois parties.

La première partie porte sur des méthodes numériques pour résoudre les MDPs, à base de quantification et de régression locale ou globale. Un problème de market-making est proposé: il est résolu théoriquement en le réécrivant comme un MDP; et numériquement en utilisant le nouvel algorithme. Dans un second temps, une méthode de Markovian embedding est proposée pour réduire des problèmes de type McKean-Vlasov avec information partielle à des MDPs. Cette méthode est mise en oeuvre sur trois différents problèmes de type McKean-Vlasov avec information partielle, qui sont par la suite numériquement résolus en utilisant des méthodes numériques à base de régression et de quantification. Dans la seconde partie de la thèse, on propose de nouveaux algorithmes pour résoudre les MDPs en grande dimension. Ces derniers reposent sur les réseaux de neurones, qui ont prouvé en pratique être les meilleurs pour apprendre des fonctions en grande dimension. La consistance des algorithmes proposés est prouvée, et ces derniers sont testés sur de nombreux problèmes de contrôle stochastique, ce qui permet d'illustrer leurs performances. Dans la troisième partie de la thèse, on s'intéresse à des méthodes basées sur les réseaux de neurones pour résoudre les EDPs, EDSRs et EDSRs réfléchies. La convergence des algorithmes proposés est prouvée; et ces derniers sont comparés à d'autres algorithmes récents de la littérature sur quelques exemples, ce qui permet d'illustrer leurs très bonnes performances.

Mots-clés:

Réseaux de neurones, MDPs, EDPs, Temps d'arrêt optimal, EDPRs, EDPRs réfléchies, McKean-Vlasov, régression globale, regression locale, quantification, carnet d'ordres, algorithmic-trading, grande-dimension.