This thesis is focused on the convergence analysis of some popular stochastic approximation methods in use in the machine learning community with applications to optimization and reinforcement learning. The first part of the thesis is devoted to a popular algorithm in deep learning called Adam used for training neural networks. This variant of stochastic gradient descent is more generally useful for finding a local minimizer of a function. Assuming that the objective function is differentiable and non-convex, we establish the convergence of the iterates in the long run to the set of critical points under a stability condition in the constant stepsize regime. Then, we introduce a novel decreasing stepsize version of Adam. Under mild assumptions, it is shown that the iterates are almost surely bounded and converge almost surely to critical points of the objective function. Finally, we analyze the fluctuations of the algorithm by means of a conditional central limit theorem.

In the second part of the thesis, in the vanishing stepsizes regime, we generalize our convergence and fluctuations results to a stochastic optimization procedure unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm (S-NAG), and the widely used Adam algorithm. We conclude this second part by an avoidance of traps result establishing the non-convergence of the general algorithm to undesired critical points, such as local maxima or saddle points. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest. A chapter of this thesis is also devoted to some non-asymptotic guarantees under a clipping of the effective stepsizes. We control here the expected norm of the gradient along the iterations in the stochastic setting. We also establish convergence rates in terms of the function value gap using the Kurdyka-Łojasiewicz property.

Finally, the last part of this thesis which is independent from the two previous parts, is concerned with the analysis of a stochastic approximation algorithm for reinforcement learning. In this last part, we propose an analysis of an online target-based actor-critic algorithm with linear function approximation in the discounted reward setting. Our algorithm uses three different timescales: one for the actor and two for the critic. Instead of using the standard single timescale temporal difference (TD) learning algorithm as a critic, we use a two timescales target-based version of TD learning closely inspired from practical actor-critic algorithms implementing target networks. First, we establish asymptotic convergence results for both the critic and the actor under Markovian sampling. Then, we provide a finite-time analysis showing the impact of incorporating a target network into actor-critic methods.

Tout d'abord, je souhaiterai remercier très chaleureusement mes directeurs de thèse Pascal et Walid. Dès ma deuxième année d'école à Télécom, en suivant vos cours, j'ai tout de suite été saisi par votre sérieux, votre rigueur et votre talent pédagogique. Merci pour votre présence, votre soutien et votre encadrement lors de mes années de thèse, même en temps de confinement. Merci pour toutes ces journées de riches échanges mathématiques au tableau (et sur zoom). Merci pour votre investissement dans cette thèse, votre rigueur, votre bienveillance et votre sympathie. J'ai eu la chance de découvrir le monde de la recherche scientifique grâce à vous, j'ai beaucoup appris à vos côtés et je vous en suis infiniment reconnaissant.

I would like to thank Prof. Vivek Borkar and Prof. Sébastien Gadat for having reviewed my thesis. It is really a great honor for me. Prof. Vivek Borkar, I took my first steps in stochastic approximation using your remarkable monograph. Prof. Sébastien Gadat, je voudrai vous remercier pour tous vos commentaires constructifs lors des différents rounds de relecture des articles de journaux constituant cette thèse et qui ont grandement contribué à les améliorer. I also would like to thank Prof. Robert Gower, Prof. Niao He and Prof. Edouard Pauwels for accepting to be part of my jury.

Je tiens aussi à remercier Olivier Fercoq pour ses cours d'optimisation et nos discussions mathématiques dont l'une a notamment inspiré une partie des calculs constituant le quatrième chapitre de cette thèse. Merci aussi à Robert Gower et Alexandre Gramfort qui m'ont inspiré et donné envie de continuer dans le domaine de l'optimisation grâce à leur cours de M2. Merci à Télécom Paris et à l'IMT d'avoir financé ma thèse via le programme "Futur et Ruptures".

Durant ma thèse, j'ai eu la chance de croiser le chemin de merveilleuses personnes. Merci à Adil de m'avoir passé le flambeau de l'approximation stochastique et de la meilleure des manières, j'ai pu suivre de près la fin de ta thèse pendant mon stage et tu as toujours aimablement répondu à mes questions, merci pour tes conseils avisés. A très bientôt j'espère ! Merci à Alex, j'ai eu la chance de découvrir le monde de la recherche avec Romain et toi à travers le monde fascinant des kernels dès mon stage ! Merci à tous les anciens de Télécom avec qui j'ai partagé de bons moments au détour d'une conversation scientifique, autour d'un verre à la Butte-aux-Cailles, d'un Five ou d'un resto à Paris 13, autour d'un repas au self, d'un Baby au foyer ou en conférence à travers le monde. Merci à Mathurin, Pierre A. (qui ont brillamment encadré mes TPs de M2 en Optim !), Alexandre G., Moussab, Pierre L., Kévin, Charles (et ses haches), Eugène, Hamid, Mastane, Anna, Robin, Jérôme-Alexis et Simon.

Je souhaiterai aussi remercier ici les doctorants du laboratoire qui ont adouci mon quotidien. Merci à Sholom pour nos échanges mathématiques stimulants et notre fructueuse collaboration. J'espère qu'il y en aura d'autres ! Merci à Guillaume (et Marie !) pour sa motivation communicative et nos pauses interminables à refaire le monde en semaine ou en weekend ! Nous avons aussi traversé une bonne partie de la pandémie ensemble à Saclay ! Merci aussi à Nidham, Pierre C., Amaury, Emile, Kamélia et Kimia. Merci également à la nouvelle génération saclaisienne qui porte le flambeau à son tour: Anass, Arturo, Dimitri, Emilia, Junjie, Luc, Rémi, Tamim et tous ceux que j'oublie pour les moments de partage ensemble.

Je remercie également mes amis en dehors de la thèse qui m'ont permis de m'évader vers d'autres cieux. Merci à Anas pour nos ballades parisiennes nocturnes improvisées et nos plans vacances. Merci aussi à Ali O. (et la mémorable pastèque niçoise) et Ali M., Simo et Réda pour nos dernières vacances partagées ensemble. A très bientôt pour de nouvelles aventures ! Enfin, je souhaiterai ici remercier toute ma famille pour son soutien et avec qui j'ai du temps à rattraper. J'ai une pensée pour mon grand-père qui m'a toujours encouragé à pousser mes études le plus loin possible et à continuer dans cette voie de la recherche scientifique. Je tiens particulièrement à remercier mes parents et mon frère pour leur soutien indéfectible tout au long de ma vie, y compris dans les moments les plus difficiles de cette thèse. Vous avez toujours su être là pour moi. Vous m'avez donné tous les moyens pour réussir et je vous en suis infiniment reconnaissant.

Résumé

Cette thèse est centrée autour de l'analyse de convergence de certains algorithmes d'approximation stochastiques utilisés en machine learning appliqués à l'optimisation et à l'apprentissage par renforcement. La première partie de la thèse est dédiée à un célèbre algorithme en apprentissage profond appelé adam, utilisé pour entraîner des réseaux de neurones. Cette célèbre variante de la descente de gradient stochastique est plus généralement utilisée pour la recherche d'un minimiseur local d'une fonction. En supposant que la fonction objective est différentiable et non convexe, nous établissons la convergence des itérées au temps long vers l'ensemble des points critiques sous une hypothèse de stabilité dans le régime des pas constants. Ensuite, nous introduisons une nouvelle variante de l'algorithme adam à pas décroissants. Nous montrons alors sous certaines hypothèses réalistes que les itérées sont presque sûrement bornées et convergent presque sûrement vers des points critiques de la fonction objective. Enfin, nous analysons les fluctuations de l'algorithme par le truchement d'un théorème central limite conditionnel.

Dans la deuxième partie de cette thèse, dans le régime des pas décroissants, nous généralisons nos résultats de convergence et de fluctuations à une procédure d'optimisation stochastique unifiant plusieurs variantes de descente de gradient stochastique comme la méthode de la boule pesante, l'algorithme stochastique de Nesterov accéléré ou encore le célèbre algorithme adam, parmi d'autres. Nous concluons cette partie par un résultat d'évitement de pièges qui établit la non convergence de l'algorithme général vers des points critiques indésirables comme les maxima locaux ou les points-selles. Ici, le principal ingrédient est un nouveau résultat indépendant d'évitement de pièges pour un contexte non-autonome. Un chapitre de cette thèse est également consacré à des garanties non-asymptotiques pour une large classe d'algorithmes adaptatifs sous une hypothèse de clipping des pas effectifs. Nous établissons en particulier une vitesse de convergence de la norme du gradient le long des itérations (ou de son espérance) dans les cas déterministe et stochastique. Nous montrons également des vitesses de convergence des valeurs de la fonction objective en utilisant l'hypothèse faible de Kurdyka-Łojasiewicz. Enfin, la dernière partie de cette thèse qui est indépendante des deux premières parties est dédiée à l'analyse d'un algorithme d'approximation stochastique pour l'apprentissage par renforcement. Dans cette dernière partie, dans le cadre des processus décisionnels de Markov avec critère de récompense γ-pondéré, nous proposons une analyse d'un algorithme acteur-critique en ligne intégrant un réseau cible et avec approximation de fonction linéraire. Notre algorithme utilise trois échelles de temps distinctes: une échelle pour l'acteur et deux autres pour la critique. Au lieu d'utiliser l'algorithme de différence temporelle (TD) standard à une échelle de temps, nous utilisons une version de l'algorithme TD à deux échelles de temps intégrant un réseau cible inspiré des algorithmes acteur-critique utilisés en pratique. Tout d'abord, nous établissons Coordinatewise quotient vector with coordinates u i /v i when v i = 0 for every i ∈ R d |u| Vector whose i-th coordinate is given by |u i | |u| Vector whose i-th coordinate is given by

|u i | • Standard Euclidean norm x 2 v := i v i x 2 i for every x ∈ R d , v ∈ (0, +∞) d . d(z, A) := inf{ z -z : z ∈ A} if z ∈ R d and A is a non-empty subset of R d 1 Introduction 1.1 Motivations 1.1.1 Non-convex stochastic optimization
Nowadays, machine learning is becoming more and more pervasive in society through several applications ranging from machine translation, speech and image recognition to online advertising and even robotics, to name a few. More advanced applications such as personalized medicine and autonomous vehicles in critical societal domains such as healthcare and transportation are also on the horizon. One of the main pillars supporting machine learning is the mathematical field of optimization. Indeed, typically, the numerical implementation of machine learning methods requires the minimization of complex loss functions measuring the inadequacy of a model to available data. Optimal parameters of the model resulting from this optimization step are then used to make decisions based on yet unseen data.

During the last decades, the access to a massive amount of data and the increase in the computing power have revolutionized the field of machine learning. This evolution led to a renewed interest in deep learning and opened the way for a fast development of deep neural networks fueling tremendous advances in machine learning. As a consequence, several new challenges emerged from optimization problems arising in machine learning. First, the large available volume of data induce challenging large-scale optimization problems (see for e.g., [START_REF] Bottou | Optimization methods for large-scale machine learning[END_REF] for a review). This challenge stimulated the design of stochastic algorithms capable of learning online. Second, the success of deep learning brought to the forefront optimization problems which are typically non-convex due to the non-convexity inherited from neural networks as functions of their weights.

More formally, in this thesis, we consider the problem of finding a local minimizer of the expectation F (x) := E(f (x, ξ)) w.r.t. x ∈ R d , where f ( . , ξ) is a possibly nonconvex function depending on some random variable (r.v.) ξ. The distribution of ξ is assumed unknown, but revealed online by the observation of independent and identically distributed (iid) copies (ξ n : n ≥ 1) of the r.v. ξ. For instance, this general formulation encompasses our motivating learning example: the parameter x then refers to the weights of the neural network at stake whereas the random variable ξ models the data and the function f can be seen as a loss function quantifying the goodness of the prediction model parameterized by x using the data point ξ whereas the function F represents the training loss of the model. The formulation of this stochastic non-convex optimization problem goes beyond this motivating learning problem and finds other applications in diverse sectors such as energy or finance.

From stochastic gradient descent to adaptive gradient methods

In this section, we will gradually introduce the stochastic optimization algorithms at the heart of this thesis, namely adaptive gradient methods.

SGD. Historically introduced by [START_REF] Robbins | A stochastic approximation method[END_REF], Stochastic Gradient Descent (SGD) is the most classical algorithm to search for a local minimizer of the function F . Given the formulation of our optimization problem and assuming that the function f (•, ξ) is differentiable for every fixed value of ξ, the iterates of SGD initialized at some point x 0 ∈ R d can be written as follows:

x n+1 = x n -γ n ∇f (x n , ξ n+1 ) , (1.1)
where (γ n ) is a sequence of positive stepsizes and ∇f (x, ξ) denotes the gradient of the mapping x → f (x, ξ) w.r.t. x for every fixed value of ξ. We briefly highlight that neural networks may involve some activation functions with points where the aforementioned mapping is nondifferentiable (such as the ReLU function, i.e., the positive part function). This setting is beyond the scope of this thesis and we refer to the recent works of [START_REF] Davis | Stochastic subgradient method converges on tame functions[END_REF]; [START_REF] Majewski | Analysis of nonsmooth stochastic approximation: the differential inclusion approach[END_REF]; [START_REF] Bolte | Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning[END_REF]; [START_REF] Bianchi | Convergence of constant step stochastic gradient descent for non-smooth non-convex functions[END_REF] for this specific case. Nevertheless, smooth activation functions such as the widely used sigmoid function or the smooth ReLU still lead to the differentiable setting under study in this thesis.

Instead of computing the full gradient of the objective function F (which may not even be possible), SGD uses a single stochastic gradient at each iteration. As such, this canonical algorithm is particularly suitable for large-scale machine learning problems involving a large number of data samples and has become the workhorse for many machine learning problems and especially for deep learning. In this method, the update rule (1.1) depends on the parameter γ n called the learning rate, which is generally assumed constant or vanishing.

Although widely used, this algorithm has at least two limitations. First, the choice of the learning rate is generally difficult; large learning rates result in large fluctuations of the estimate, whereas small learning rates induce slow convergence. Second, a common learning rate is used for every coordinate despite the possible discrepancies in the values of the gradient vector's coordinates.

Adaptive stepsizes. To alleviate these limitations, an idea which has made its way into the machine learning community is that of adjusting the learning rate coordinate-wise, as a function of the past values of the squared gradient vectors' coordinates. This modification can be seen as a diagonal preconditioning of the stochastic gradient in SGD based on past observed gradients. The independent works of [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] and [START_REF] Mcmahan | Adaptive bound optimization for online convex optimization[END_REF] in the context of online convex optimization led the way to a new class of algorithms that are sometimes referred to as "adaptive1 gradient methods". As proposed by [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF], AdaGrad consists of dividing the learning rate by the square root of the sum of previous gradients squared componentwise. The idea was to give larger learning rates to highly informative but infrequent features instead of using a fixed predetermined schedule. This is particularly relevant in applications such as click through rate prediction for online advertising and text classification where many features only occur rarely with only a few number of non-zero features while few occur very often. We refer to (Duchi et al., 2011, Section 1.3) and (McMahan and Streeter, 2010, Section 1.2) for examples showing how adaptive methods can outperform standard methods when gradients are sparse.

However, in practice, the division by the cumulative sum of squared gradients may generate small learning rates, thus freezing the iterates too early. Several works proposed heuristical ways to set the learning rates using a less aggressive policy. [START_REF] Tieleman | Lecture 6.e-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] introduced an unpublished, yet popular, algorithm referred to as RmsProp where the cumulative sum used in AdaGrad is replaced by a moving average of squared gradients.

Momentum. Besides adaptive stepsizes, another popular modification of vanilla SGD is the use of momentum. Introduced by the seminal work of [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF] (see Section 2 therein) in the deterministic setting, the heavy ball method augments the standard gradient descent method with an additive inertial term corresponding to the difference between two successive iterates of the algorithm. This inertial term which was later called momentum in the modern machine learning community (see for e.g., [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF]) led to accelerated convergence rates of the heavy ball method in comparison to deterministic gradient descent for the optimization of twice differentiable strongly convex functions. Later, [START_REF] Nesterov | A method of solving a convex programming problem with convergence rate O(1/k 2 )[END_REF] proposed the Nesterov accelerated gradient descent method for convex optimization with an "optimal" convergence rate of O(1/k2 ) of the value function gap, outperforming gradient descent. Since the seminal works of Polyak and Nesterov, momentum methods hold the promise of acceleration and several recent works shed the light on this acceleration phenomenon [START_REF] Wibisono | A variational perspective on accelerated methods in optimization[END_REF][START_REF] Wilson | A lyapunov analysis of accelerated methods in optimization[END_REF]. Although the acceleration benefits of momentum methods in the deterministic optimization setting do not necessarily carry over to the stochastic nonconvex optimization setting 2 as also recently advocated by [START_REF] Kidambi | On the insufficiency of existing momentum schemes for stochastic optimization[END_REF], the seminal work of Nesterov together with the renewed interest in first order optimization algorithms for large-scale machine learning fostered the design of similar methods in the context of non-convex stochastic optimization (see for e.g. [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF]; [START_REF] Gadat | Stochastic heavy ball[END_REF]). Apart from possible acceleration, we point out that momentum has also a smoothing effect on the stochastic gradients used in the algorithm.

Adam. We are now ready to introduce the most popular algorithm among the family of adaptive gradient methods we have introduced so far: Adam (for Adaptive Momentum estimation). Proposed by [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], Adam combines the advantages of both AdaGrad, RmsProp and momentum3 methods. The notorious algorithm thus combines the assets of inertial methods with an adaptive per-coordinate learning rate selection for automatic tuning. Since 2015, the popular Adam algorithm has become widely used in deep learning applications and implemented in massively used deep learning libraries such as TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF] and PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]. As originally proposed in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], at each timestep n ∈ N, Adam produces a triplet of iterates (x n , m n , v n ) ∈ R d × R d × R d using the following recursions:

(Adam)                  m n+1 = αm n + (1 -α)∇f (x n , ξ n+1 ) (1.2a) v n+1 = βv n + (1 -β)∇f (x n , ξ n+1 ) 2 (1.2b) mn+1 = m n+1 /(1 -α n+1 ) (1.2c) vn+1 = v n+1 /(1 -β n+1 ) (1.2d) x n+1 = x n -γ mn+1 /(ε + vn+1 ) , (1.2e)
where α, β ∈ [0, 1) are exponential decay rates hyperparameters for the moving averages (typically α = 0.9 and β = 0.999), γ ∈ R * + is the stepsize, ε ∈ R * + is a constant introduced for numerical stability (typically ε = 10 -8 ) and the iterates are initialized with m 0 = v 0 = 0 and some x 0 ∈ R d . We bring to the attention of the reader that all the operations on vectors are coordinatewise throughout this thesis. If x, y are two vectors on R d , we denote by x y, x 2 , x/y, |x|, |x| the vectors on R d whose i-th coordinates are respectively given by x i y i , x 2 i , x i /y i , |x i |, |x i |. As previously described, the algorithm uses momentum as can be seen in steps (1.2a),(1.2e), and computes an adaptive learning rate (γ/(ε + vn+1 ) as can be appreciated in the recursions (1.2b) and (1.2e). Finally, as can be observed in steps (1.2c) and (1.2d) above, the algorithm includes a so-called bias correction step. Acting on the current estimate of the gradient vector, this step is especially useful during the early iterations. We shall provide some comments on this step later on in this introduction when stating our contributions and a more in-depth explanation of this step can be found in the subsequent chapters (see Remark 1 in Chapter 2). We highlight that adaptive gradient methods are still first order optimization algorithms in the sense that they only have access to stochastic gradients without resorting to more computationally demanding information such as Hessians. Compared to SGD, adaptive gradient methods have similar computational complexity while requiring less hyperparameter tuning. One additional interesting feature of adaptive gradient methods which is also true for Newton-like algorithms, is that a multiplication of the objective function by a constant automatically impacts the (adaptive) learning rate which is divided by the same constant (see Eq. (1.2e)).

A major part of this thesis is dedicated to the analysis of the algorithms belonging to the class of adaptive gradient methods we have introduced. In particular, AdaGrad, RMSProp and Adam all belong to this class. As we will expand on it later on, our analysis will also encompass stochastic momentum methods including stochastic Nesterov accelerated gradient.

Challenges. As opposed to the vanilla Stochastic Gradient Descent (SGD), the study of such algorithms is more elaborate, for three reasons. First, the update of the iterates involves a so-called momentum term, or inertia, which has the effect of "smoothing" the increment between two consecutive iterates. Second, as far as adaptive algorithms are concerned, the update also depends on some additional variable (a.k.a. the (adaptive) learning rate) computed online as a function of the history of the computed gradients. As a consequence of this special learning rate, the update rule of the algorithms is not linear as a function of the stochastic gradients. The momentum term and the learning rate together endow the algorithms with a "memory" induced by the use of past stochastic gradient information. In contrast, vanilla SGD is memoryless. Third, the update equation at the time index n is likely to depend on n, making these systems inherently non-autonomous. Indeed, apart from the stepsizes depending explicitely on n, the update rules of the algorithms can also feature an additional explicit dependence on n (see (1.2c) and (1.2d) above).

Actor-critic methods in Reinforcement Learning

In this section, we introduce the Reinforcement Learning (RL, [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]) problem we deal with in the last part of this manuscript. This part is independent from the rest of the thesis.

RL has witnessed a huge success in a wide range of applications such as game playing [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF], robotic manipulation [START_REF] Gu | Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates[END_REF][START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF], dialogue generation in natural language processing [START_REF] Li | Deep reinforcement learning for dialogue generation[END_REF], data centre cooling regulation [START_REF] Lazic | Data center cooling using model-predictive control[END_REF], traffic signal control (Prashanth andBhatnagar, 2010, 2011) even in large-scale urban networks [START_REF] Chen | Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control[END_REF], self-driving cars [START_REF] Kiran | Deep reinforcement learning for autonomous driving: A survey[END_REF] or clinical decision support in critical care [START_REF] Liu | Reinforcement learning for clinical decision support in critical care: comprehensive review[END_REF], to name a few.

In this thesis, we are concerned with sequential decision making problems under uncertainty formulated as Markov Decision Processes (MDP) [START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF]. In this model, an agent learns how to act optimally by interacting with a possibly unknown environment via trial and error. 4We denote by S = {s 1 , • • • , s n } the finite set of states of the environment and A the finite set of possible actions the agent can execute. We use the notation P(X) for the set of probability measures on the set X where X can either be S, A or S × A . Let p : S×A → P(S) be the state transition probability kernel which gives the probability of moving to the next state given the current state and action. The immediate reward5 function R : S × A → R provides for every s ∈ S, a ∈ A the single-stage expected reward R(s, a) when action a ∈ A is executed in state s ∈ S. A randomized stationary policy, which we will simply call a policy, is a mapping π : S → P(A) specifying for each s ∈ S, a ∈ A the probability π(a|s) of selecting action a in state s. This policy describes the agent's behavior strategy in the MDP model. At each time step t ∈ N, the RL agent in a state S t ∈ S executes an action A t ∈ A with probability π(A t |S t ), transitions into a state S t+1 ∈ S with probability p(S t+1 |S t , A t ) and observes a real random reward R t+1 (which we will suppose to be bounded). We denote by P ρ,π the probability distribution of the Markov chain (S t , A t ) issued from the MDP controlled by the policy π with initial state distribution ρ. The notation E ρ,π refers to the associated expectation. We will use E π whenever there is no dependence on ρ. The sequence (R t ) is such that (s.t.) E π [R t+1 |S t , A t ] = R(S t , A t ) . The objective is then to find an optimal policy maximizing the expected cumulative future rewards 6 :

max π J(π) := E ρ,π   ∞ t=0 γ t R t+1   , (1.3)
where the discount factor γ ∈ (0, 1) favors immediate rewards over delayed ones. We consider the setting where the dynamic of the environment is not explicitely known which renders the computation of the objective function J(π) intractable. This dynamic will be revealed online over time thanks to the observation of the environment's successive states following the executed actions. This is the so-called model free approach.

The first fundamental class of RL methods consists of value-based methods using the so-called value function (respectively action-state value function) which quantifies how good is each state (or state-action pair). Given a policy π, the value function V π : S → R and the action-value function (also called Q-function) Q π : S × A → R are defined for every state s ∈ S, action a ∈ A by:

V π (s) := E π   ∞ t=0 γ t R t+1 |S 0 = s   and Q π (s, a) := E π   ∞ t=0 γ t R t+1 |S 0 = s, A 0 = a   .
The optimal value function V * : S → R and Q-function Q * : S×A → R can then be defined for every s ∈ S, a ∈ A by: V * (s) := max π V π (s) and Q * (s, a) := max π Q π (s, a) .Valuebased methods rely on the fundamental property of the optimal policy π * stipulating that π * (s) = arg max a∈A Q * (s, a) . Therefore, in order to compute the optimal policy, one looks for the optimal Q-value function without requiring the knowledge of the reward or transition probabilities. Since this function Q * satisfies a dynamic programming (Bellman) equation and can then be seen as the fixed point of the so-called Bellman equation, the function Q * can be estimated using a stochastic algorithm for fixed point search. This leads to one of the most famous RL algorithms belonging to the class of value-based methods as proposed by [START_REF] Watkins | Learning from delayed rewards[END_REF]: Q-learning. We refer for instance to (Borkar and Chandak, 2021, Section 2.1) or (Avrachenkov et al., 2021, Section 2.1) for a nice self-contained derivation of the algorithm.

A second class of methods consists of the so-called policy-based methods. In these methods, one directly searchs the optimal policy π * . The main representant of this class is the policy gradient method. The idea is to parameterize the policy π by a vector parameter θ ∈ R d (i.e. consider a family of policies {π θ : θ ∈ R d } ) and directly maximize the objective function J(π θ ) by performing a stochastic gradient ascent. For this, the gradient of the objective function is then given by the so-called policy gradient theorem7 [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] as follows:

∇J(θ) = 1 1 -γ • E ( S, Ã) ∼ µ ρ,θ [∆ π θ ( S, Ã) ∇ ln π θ ( Ã| S)] ,
(1.4)

for every θ ∈ R d , where the advantage function8 ∆ π : S × A → R is defined for every policy π and every (s, a) ∈ S × A by ∆ π (s, a) := Q π (s, a) -V π (s) , the initial state S 0 follows an initial probability distribution ρ over states 9 and the gradient is w.r.t. the parameter θ (i.e., the gradient of the function θ → ln π θ ). Here, the couple of r.v.s ( S, Ã) follows the discounted state-action occupancy measure µ ρ,θ ∈ P(S × A) defined for all (s, a) ∈ S × A by: µ ρ,θ (s, a) := d ρ,θ (s) π θ (a|s) where d ρ,θ (s

) := (1 -γ) ∞ t=0 γ t P ρ,π θ (S t = s) (1.5)
is a probability measure over the state space S known as the discounted state-occupancy measure. We refer the reader to Section 5.3 of the self-contained Chapter 5 for more precisions and rigorous regularity conditions under which this theorem holds. In policy gradient methods, the unknown advantage function ∆ π (or Q-function Q π ) is estimated via Monte Carlo simulation using trajectories generated by the MDP (Monte Carlo rollouts). Denoting by ∆π θ t such an estimate of ∆ π θ t at time t ∈ N and given Eq. (1.4) and a sequence of positive stepsizes (α t ), we obtain the following recursion:

θ t+1 = θ t + α t 1 1 -γ ∆π θ t ( St , Ãt )∇ ln π θt ( Ãt | St ) , (1.6)
where the couple ( St , Ãt ) is generated following an online procedure (that we do not explicit here for conciseness, see Section 5.4 for details) guaranteeing that the couple of r.v.s will "asymptotically" follow the unknown distribution µ ρ,θ . One of the shortcomings of such methods is the large variance induced by the gradient estimators. Moreover, a new gradient is estimated regardless of previous estimates as the policy evolves.

Finally, a third class of methods combines value-based and policy-based methods. This is the class of actor-critic methods [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF][START_REF] Konda | Actor-critic-type learning algorithms for markov decision processes[END_REF][START_REF] Konda | On actor-critic algorithms[END_REF] consisting of two coupled iterative algorithms. The first one called the actor updates the current policy using a stochastic gradient ascent strategy with an increment depending on the advantage function (and actually only on the value function V π θ , see the definition of ∆ π θ and details in Section 5.4.1). This last function being unknown, a second incremental (value-based) algorithm called the critic estimates its value all along the iterations. The critic evolves on a faster timescale using larger stepsizes than for the actor, thereby simulating the effect of nested loops where the estimation of the critic takes place in an inner loop useful for the actor evolving in an outer loop.

In the case of complex MDPs with a large number of states and/or actions, estimating the value at each state-action pair to fill the entire table of state-action values becomes intractable because of the "curse of dimensionality" as computing solutions to dynamic programming equations (that value functions satisfy) is infeasible. Instead of tabular (or look-up table) methods, in the RL literature, we modestly approximate the exact value function by another function belonging to a parameterized family of functions. A simple linear parameterization consists in looking for the approximate value function as a linear combination of some predetermined basis (or feature) functions. For every θ ∈ R d , the state-value function V π θ is then approximated for every state s ∈ S by a linear function of carefully chosen feature vectors as follows:

V π θ (s) ≈ V ω (s) = ω T φ(s) = m i=1 ω i φ i (s) , (1.7)
where ω = (ω 1 , • • • , ω m ) T ∈ R m for some integer m |S| and φ(s) = (φ 1 (s), • • • , φ m (s)) T is the feature vector of the state s ∈ S. We highlight though that the best practical performances are obtained by using nonlinear parameterizations based on deep neural networks. Typically, a neural network takes the state-action pair as an input and outputs the desired approximated value function V ω . In the case of actor-critic methods, the critic updates the parameters of this neural network whereas a similar neural network parameterization is used for the actor. The textbooks of [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]; [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF]; [START_REF] Szepesvári | Algorithms for reinforcement learning[END_REF] provide nice introductions to RL theory with further details.

Actor-critic methods integrating target networks have exhibited a stupendous empirical success in deep reinforcement learning [START_REF] Heess | Memory-based control with recurrent neural networks[END_REF][START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF][START_REF] Fujimoto | Addressing function approximation error in actorcritic methods[END_REF][START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. However, if standard actor-critic methods have been well-studied in the literature (see, for e.g., [START_REF] Konda | Actor-critic-type learning algorithms for markov decision processes[END_REF]; [START_REF] Konda | On actor-critic algorithms[END_REF]; [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF]), a theoretical understanding of the use of target networks in actor-critic methods is largely missing in the literature. We will devote the last part of this thesis to the study of an actor-critic method incorporating a target network in the linear function approximation setting. We will briefly introduce our actor-critic algorithm together with our contributions in Section 1.3 below and we will defer its precise mathematical presentation to Section 5.4 in which the reader can also find an exposition of the idea of target network from the RL literature.

Theoretical context: Stochastic Approximation

The stochastic algorithms considered in this thesis fall under the umbrella of general stochastic approximation algorithms of the form:

x n+1 = x n + γ n+1 h(x n , ξ n+1 ) , x 0 ∈ R d , (1.8)
where h is a map from R d × R k to R d , (ξ n ) is a sequence of r.v.s taking their values in R k and (γ n ) is a vanishing sequence of positive reals s.t. n γ n = +∞ and say n γ 2 n < +∞. Here, the sequence of r.v.s (ξ n ) evolves as follows. Given a probability space (Ω, A, P) and considering the σ-field F n generated by the r.v.s x 0 , ξ 1 , • • • , ξ n , we assume that there exist a family of transition probabilities {Π x : R k × B(R k ) → [0, 1] , x ∈ R d }10 on R k s.t. for all n ∈ N, for all B ∈ B(R k ), almost surely,

P[ξ n+1 ∈ B|F n ] = Π xn (ξ n ; B) , (1.9)
where P[ξ n+1 ∈ B|F n ] is the conditional probability of the event {ξ n+1 ∈ B} given F n . This property means that the sequence (ξ n ) is a Markov chain controlled by the parameter x. Suppose now that for every x, the Markov chain given by Π x admits a unique invariant probability µ x (this is common in applications) and define the function h : R d → R d for every x ∈ R d by h(x) := h(x, s)µ x (ds) .

(1.10)

We can then rewrite the recursion (1.8) under the following form:

x n+1 = x n + γ n+1 h(x n ) + γ n+1 (h(x n , ξ n+1 ) -h(x n )) .
(1.11)

Under a control of the small perturbation induced by the last term in the equation above, one can expect the asymptotic behavior of the algorithm to be closely related to that of the following ordinary differential equation governed by the mean field h: ẋ(t) = h(x(t)) .

(1.12)

Considering the discrete stochastic iterates (1.8) as a noisy discretization of the ODE (1.12), the iterates of the algorithm are shown to asymptotically track the trajectory of the solution to the ODE. An analysis of this ODE characterizing its equilibrium points provides the limit points of the algorithm at hand. This is the key idea of the so-called Ordinary Differential Equation (ODE) method from stochastic approximation theory which is at the heart of our analysis.

Initiated by the seminal work of [START_REF] Robbins | A stochastic approximation method[END_REF], stochastic approximation theory and the ODE method in particular have given rise to a vast literature that we cannot hope to do justice here. We refer though to the historical works of [START_REF] Ljung | Analysis of recursive stochastic algorithms[END_REF]; [START_REF] Kushner | Stochastic Approximation Methods for Constrained and Unconstrained Systems[END_REF], the contributions of [START_REF] Métivier | Applications of a kushner and clark lemma to general classes of stochastic algorithms[END_REF]; [START_REF] Métivier | Théorèmes de convergence presque sûre pour une classe d'algorithmes stochastiques à pas décroissant[END_REF] studying the general scheme (1.8) presented in this section, [START_REF] Benaïm | A dynamical system approach to stochastic approximations[END_REF]; [START_REF] Benaïm | Dynamics of stochastic approximation algorithms[END_REF] and the monographs [START_REF] Benveniste | Adaptive algorithms and stochastic approximations[END_REF]; [START_REF] Duflo | Random iterative models[END_REF]; [START_REF] Kushner | Stochastic approximation and recursive algorithms and applications[END_REF]; [START_REF] Borkar | Stochastic approximation. A dynamical systems viewpoint[END_REF] for comprehensive treatments of the subject.

Although this thesis is motivated by optimization and RL applications, we briefly mention that the powerful theory of stochastic approximation finds applications in many diverse domains such as adaptive signal processing (see for e.g., [START_REF] Benveniste | Adaptive algorithms and stochastic approximations[END_REF]), communication networks and economics (as referred to for instance in (Borkar, 2008, Chap. 1)), to name a few. To list a couple of recent applications, we highlight that stochastic approximation theory has also been fruitfully exploited for estimating entropically regularized Wasserstein distances between two probability measures in (semi-discrete) optimal transport (Bercu and Bigot, 2021;Bercu et al., 2021) and for solving min-max problems such as for Generative Adversarial Networks [START_REF] Hsieh | The limits of min-max optimization algorithms: Convergence to spurious non-critical sets[END_REF].

Back to the general scheme (1.8), and reusing the notations from Eq. (1.1), we mention that the canonical example of such algorithm is that of SGD which corresponds to the case where h(x n , ξ n+1 ) = ∇f (x n , ξ n+1 ) and (ξ n ) is a sequence of iid r.v.s. In this particular case, the transition kernel Π coincides with the unknown probability distribution of ξ 0 . The corresponding ODE is the so-called gradient flow ẋ(t) = -∇F (x(t)) .

In the next two sections, we will independently describe our contributions in both nonconvex stochastic optimization and reinforcement learning. With respect to the general stochastic approximation framework of (1.8)-(1.9), the adaptive gradient algorithms we will study will involve a more complex structure of the function h than that of SGD whereas the noise induced by the stochastic algorithms will take the form of a martingale difference sequence. Then, in the last part of this thesis where we tackle a RL problem, the theoretical framework relevant to this setting corresponds to the noise r.v. (ξ n ) being a Markov chain controlled by the sequence of interest (x n ). In this setting, the noise dynamics is more complex than simply iid r.v.s and controlling the Markov noise requires additional theoretical tools such as a decomposition of the perturbation based on the Poisson equation as proposed in [START_REF] Métivier | Applications of a kushner and clark lemma to general classes of stochastic algorithms[END_REF].

The actor-critic algorithm we will study involves multiple timescales. Accordingly, we will use the multiple timescales stochastic approximation theory developed by Borkar (see for e.g., (Borkar, 2008, Chap. 6))

Stochastic momentum algorithms for non-convex optimization

If SGD has been well studied in the literature (see for e.g., [START_REF] Benaïm | A dynamical system approach to stochastic approximations[END_REF]; [START_REF] Delyon | General results on the convergence of stochastic algorithms[END_REF]; [START_REF] Bertsekas | Gradient convergence in gradient methods with errors[END_REF]; Moulines and Bach (2011); Bach and Moulines (2013)) and is still the subject of active research [START_REF] Bottou | Optimization methods for large-scale machine learning[END_REF]; [START_REF] Gower | Sgd: General analysis and improved rates[END_REF]; [START_REF] Alacaoglu | A new regret analysis for Adam-type algorithms[END_REF]; [START_REF] Sebbouh | Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball[END_REF], adaptive gradient methods were comparatively much less studied prior to this thesis. Previously known results are either regret bounds in the online convex optimization framework or bounds on the expected gradient norm of the objective function for variants of famous algorithms such as Adam.

In particular, these results do not address the question of the convergence of the iterates of the algorithms themselves. We refer the reader to Sections 2.6,3.3.4,4.3 and 4.7 in the subsequent chapters for further details on related works and comparison to the literature.

About Adam (Chapter 2)

In this subsection, we describe our contributions related to the study of the Adam algorithm to solve the non-convex stochastic optimization problem we introduced in Section 1.1.1 above. Rigorous statements with precise assumptions can be found in Chapter 2.

Continuous-time system: ODE analysis. We introduce a continuous-time version of the Adam algorithm (see (1.2a)-(1.2e)) under the form of a non-autonomous ODE which can be written as follows:

ż(t) = h(z(t), t) , with h(z, t) =     -(1-e -at ) -1 m ε+ √ (1-e -bt ) -1 v a(∇F (x) -m) b(S(x) -v)     (1.13) for every t > 0, z = (x, m, v) in R d × R d × R d + , where z(t) = (x(t), m(t), v(t)), F (x) := E[f (x, ξ)] and S(x) := E[∇f (x, ξ) 2 ] for every x ∈ R d 11 and a, b are positive constants defined in Assumption 2.2.4.
Building on the existence of an explicit Lyapunov function for the ODE, we show the existence of a unique global solution to the ODE. This first result turns out to be non-trivial due to the irregularity of the vector field. We then establish the convergence of the continuous-time Adam trajectory to the set of critical points of the objective function F . The proposed continuous-time version of Adam provides useful insights on the effect of the bias correction step. It is shown that, close to the origin, the objective function F is non-increasing along the Adam trajectory, suggesting that early iterations of Adam can only improve the initial guess.

Convergence rates in continuous-time. Under a Łojasiewicz-type condition, we prove that the solution to the ODE converges to a single critical point x * of the objective function F . In this case, we provide convergence rates in terms of the distance to the critical point x(t)x * as a function of the so-called Łojasiewicz exponent of F at x * (see Section 2.3.3 of Chapter 2 for a definition).

Constant stepsize Adam. In discrete time, we first analyze the Adam iterates in the constant stepsize regime as originally introduced in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. Under a stability condition, we prove the asymptotic ergodic convergence of the probability of the discrete-time Adam iterates to approach the set of critical points of the objective function in the doubly asymptotic regime where n → ∞ then γ → 0. This long run convergence result stipulates that for every δ > 0 ,

lim γ↓0 lim sup n→∞ 1 n n k=1 P(d(x γ k , ∇F -1 ({0})) > δ) = 0 , (1.14)
where the iterates x γ k of Adam are indexed by the constant stepsize γ and d is the euclidean distance to a set. Note here that the iterates with constant stepsize γ cannot converge in the almost sure sense when only the number of iterations n → ∞.

A new decreasing stepsizes version of Adam. Beyond the original constant stepsize Adam, we propose a decreasing stepsize version of the algorithm. From the initial point z 0 = (x 0 , 0, 0) where x 0 ∈ R d , the algorithm generates a sequence z n = (x n , m n , v n ) as follows:

                 m n+1 = α n m n + (1 -α n )∇f (x n , ξ n+1 ) (1.15a) v n+1 = β n v n + (1 -β n )∇f (x n , ξ n+1 ) 2 (1.15b) mn+1 = m n+1 /(1 -Π n+1 i=1 α i ) (1.15c) vn+1 = v n+1 /(1 -Π n+1 i=1 β i ) (1.15d) x n+1 = x n -γ n+1 mn+1 /(ε + vn+1 ) , (1.15e)
First, compared to the original version of [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] introduced above in (1.2a), the hyperparameters (γ n , α n , β n ) now depend on n. Second, the debiasing steps rescaling the iterates m n and v n are different due to the use of time-dependent hyperparameters α n , β n . Further details can be found in Chapter 2.

For this decreasing stepsizes version, we provide sufficient conditions ensuring the stability and the almost sure convergence of the iterates towards the critical points of the objective function F as predicted by our continuous-time analysis.

Fluctuations. We establish a convergence rate of the stochastic iterates of the decreasing stepsize algorithm under the form of a conditional central limit theorem. More precisely, for some decreasing stepsizes γ n , we show that the vector √ γ n -1 (x nx * ) converges in distribution to a zero mean Gaussian distribution with a covariance matrix Σ 1 that we explicitely compute thanks to the Lyapunov equation verified by the covariance matrix.

Generalized momentum algorithm (Chapter 3)

After Chapter 2 which is focused on the specific Adam algorithm, in Chapter 3, we go one step further to study a general stochastic optimization procedure, unifying several variants of stochastic gradient descent. Among others, these variants include the Stochastic Heavy Ball (SHB) method, the stochastic version of Nesterov's Accelerated Gradient method (S-NAG), and the large class of adaptive gradient algorithms, among which Adam is perhaps the most used in practice. We thereby extend the results of Chapter 2 and the work of [START_REF] Gadat | Stochastic heavy ball[END_REF] focused on SHB to a general setting. Continuous-time system. The algorithm we consider is seen as a noisy Euler discretization of a non-autonomous ODE extending (1.13) (which is analyzed in Chapter 2) and concomitantly introduced by Belotto da [START_REF] Belotto Da Silva | A general system of differential equations to model first-order adaptive algorithms[END_REF]. This ODE can be written as follows:

ż(t) = g(z(t), t) , with g(z, t) =    p(t)S(x) -q(t)v h(t)∇F (x) -r(t)m -m/ √ v + ε    (1.16) for every t > 0, z = (v, m, x) in R d + × R d × R d , where F : R d → R is a continuously differentiable function, S : R d → R d
+ is a continuous function, h, r, p, q : (0, ∞) → R + are four continuous functions, and ε > 0. Moreover, the ODE is initialized by z(0

) = z 0 = (v 0 , m 0 , x 0 ) for some v 0 ∈ R d + , x 0 , m 0 ∈ R d , and z(t) = (v(t), m(t), x(t)) ∈ R d × R d × R d + for t ∈ R + .
A proper choice of the functions h, r, p, q followed by an Euler discretization of the ODE leads to numerous algorithms used in Machine Learning (see Chapter 3 for details). In view of applying the ODE method to study the corresponding stochastic algorithm, we analyze this non-autonomous ODE representing the continuous-time versions of the aforementioned set of algorithms. Under some conditions on the functions h, r, p, q (satisfied for instance for Adam) together with some regularity assumptions on the objective function F and some mild technical assumptions, we show that there exist a unique bounded global solution to ODE (1.16) and the x component of the latter converges towards the set of critical points of the objective function F .

Almost sure convergence and fluctuations. Then, we introduce the novel general algorithm of interest as a noisy discretization of ODE (1.16). This algorithm uses a sequence of decreasing stepsizes (γ n ) verifying the Robbins Monro conditions. As for the discretization, we define for every integer n ≥ 1,

τ n = n k=1 γ k , h n = h(τ n ), r n = r(τ n ), p n = p(τ n )
, and q n = q(τ n ).

(1.17)

Initialized with z 0 = (v 0 , m 0 , x 0 ) ∈ R d + ×R d ×R d
and h 0 , r 0 , p 0 , q 0 ∈ (0, ∞), the algorithm is written as follows12 :

       v n+1 = (1 -γ n+1 q n )v n + γ n+1 p n ∇f (x n , ξ n+1 ) 2 (1.18a) m n+1 = (1 -γ n+1 r n )m n + γ n+1 h n ∇f (x n , ξ n+1 ) (1.18b) x n+1 = x n -γ n+1 m n+1 / √ v n+1 + ε , (1.18c)
Similarly to Chapter 2, the stability and the almost sure convergence of the iterates of our general algorithm towards the set of critical points are established. Compared to Chapter 2, the algorithm above offers some freedom in the choice of the functions h, r, p, q beyond the specific case of Adam which corresponds to specific functions h ≡ r13 and p ≡ q provided in Chapter 3 (see also Belotto da [START_REF] Belotto Da Silva | A general system of differential equations to model first-order adaptive algorithms[END_REF]). This generalization needs some adaptations of the proofs even if they are very similar in spirit.

For instance, the Lyapunov function we consider is different from the one considered in Chapter 2. Beyond this generalization, we also note two additional improvements. First, for our almost sure convergence result, we provide noise conditions allowing to choose larger stepsizes. Second, regarding the stability result, we relax an assumption related to the sequence of hyperparameters (α n ) and stepsizes (γ n ) (Assumption 2.5.2-iii)) which is no more needed thanks to a slight modification of the discretized Lyapunov function used in the proof. A noteworthy special case is the convergence proof of S-NAG in a non-convex setting which needs a specific proof. Under some assumptions, the convergence rate is also provided under the form of a Central Limit Theorem for the general algorithm (1.18a)-(1.18c).

Avoidance of traps. In Chapter 3, we also tackle the important question of the avoidance of traps. As stated in the previous paragraph, the iterates of the algorithm are shown to converge almost surely to the set of critical points of the objective function F . However, in a non-convex setting, the set of critical points of the function F is generally larger than the set of local minimizers. A "trap" stands for a critical point at which the Hessian matrix of F has negative eigenvalues, namely, it is a local maximum or saddle point. We establish that the iterates cannot converge to such an undesired point, if the noise is exciting in some directions.

Since the contributions of [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF] and [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF], the numerous so-called avoidance of traps results that can be found in the literature (see also for e.g., (Benaïm, 1999, Section 9), (Borkar, 2008, Section 4.3)) deal with the case where the ODE that underlies the stochastic algorithm is an autonomous ODE. Nevertheless, this is not the case of the non-autonomous ODE (1.16). To address this issue, we first state a general avoidance of traps result that extends [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF] to a non-autonomous setting, and which is of independent interest. We loosely describe this result in the rest of this paragraph and refer to Section 3.4 for rigorous statements and Section 3.7 for its proof.

To state our result, consider the stochastic approximation algorithm in R d initialized at some z 0 ∈ R d :

z n+1 = z n + γ n+1 b(z n , τ n ) + γ n+1 η n+1 + γ n+1 ρ n+1 (1.19)
where b : R d × R + → R d is a continuous function, the sequence (γ n ) of nonnegative deterministic stepsizes satisfies the Robbins Monro conditions (i.e., n γ n = +∞, n γ 2 n < +∞), and τ n = n k=1 γ k . Given a filtration (F n ), assume that the sequence (η n ) is a martingale difference sequence (adapted to F n ) and that the sequence (ρ n ) which is also supposed to be adapted to F n has a square summable euclidean norm. Let z ∈ R d , and assume that on V × R + where V is a certain neighborhood of z , the function b can be developed as b(z, t) = D(zz ) + e(z, t), (1. [START_REF] Benaïm | Stochastic approximations and differential inclusions[END_REF] where e(z , •) ≡ 0, and where the matrix D ∈ R d×d is assumed to have at least one eigenvalue with positive real part. As a consequence, the point z is an unstable equilibrium point of the ODE ż(t) = b(z(t), t), in the sense that the ODE solution will only be able to converge to z along a specific so-called invariant manifold which precise characterization will be deferred to Chapter 3. In other words, the point z is a trap that the algorithm should desirably avoid.

As the reader can observe in Eq. (1.19), the stochastic algorithm is built around the non-autonomous ODE ż(t) = b(z(t), t). The function e can be seen as a non-autonomous perturbation of the autonomous linear ODE ż(t) = D(z(t)z ). Regularity assumptions made on the function e guarantee the existence of a local (around the unstable equilibrium z ) non-autonomous invariant manifold of the non-autonomous ODE ż(t) = b(z(t), t) with enough regularity properties. For proving this result, we use a non-autonomous version of Poincaré's invariant manifold theorem [START_REF] Daleckii | Stability of solutions of differential equations in Banach space[END_REF][START_REF] Kloeden | Nonautonomous dynamical systems[END_REF] instead of its classical autonomous version.

Under regularity conditions on the function e and mild technical assumptions, we show that the event {z n → z } is of probability zero if the noise sequence (η n ) is sufficiently exciting in directions of the eigenspace associated with eigenvalues of D with positive real part. Thanks to this exciting noise, the algorithm trajectories will move away from the invariant manifold mentioned above.

We apply this result to both our general stochastic algorithm and S-NAG. Our general avoidance of traps result extends previous works of [START_REF] Gadat | Stochastic heavy ball[END_REF] obtained in the context of SHB. This result not only allows to study a broader class of algorithms but also significantly weakens the assumptions (see Section 3.4 in Chapter 3).

Convergence rates of a momentum algorithm with bounded adaptive stepsize (Chapter 4)

In this subsection which summarizes our contributions in Chapter 4 of this thesis, we provide non-asymptotic results complementing our asymptotic analysis in Chapters 2 and 3.

We establish a convergence rate for Adam in the deterministic case for non-convex optimization under a bounded learning rate and constant stepsize. This algorithm can be seen as a deterministic clipped version of Adam, which guarantees safe theoretical stepsizes. More precisely, if n is the number of iterations of the algorithm, we show a O(1/n) convergence rate of the minimum of the squared gradients norms by introducing a suitable Lyapunov function. Then, we show a similar convergence result for non-convex stochastic optimization up to the limit of the variance of stochastic gradients under an almost surely bounded learning rate.

Finally, using the Kurdyka-Łojasiewicz (KŁ) property, we propose a convergence rate analysis of the objective function values along the iterates of the algorithm in the deterministic setting.

1.3 Actor-critic with target network for Reinforcement Learning (Chapter 5)

In Chapter 5 of this thesis, we deal with the RL problem we have introduced in subsection 1.1.3 using an algorithm which we will briefly present in what follows using the notations of the aforementioned subsection. Starting from θ 0 , ω 0 , ω0 ∈ R d and given three different sequences of positive stepsizes (α t ), (β t ), (ξ t ), the following recursions generate the sequence (θ t ) of interest (parameterizing the sought policy) online as follows:

                 δ t+1 = R t+1 + γ φ(S t+1 ) T ω t -φ( St ) T ω t (1.21a) δt+1 = R t+1 + γ φ(S t+1 ) T ωt -φ( St ) T ω t (1.21b) θ t+1 = θ t + α t 1 1-γ δ t+1 ∇ ln π θt ( Ãt | St ) (1.21c) ω t+1 = ω t + β t δt+1 φ( St ) (1.21d) ωt+1 = ωt + ξ t (ω t+1 -ωt ) , (1.21e)
Here, at each timestep t ∈ N, the state S t+1 is generated following the distribution p(•| St , Ãt ) and the action Ãt according to π θt (•| St ). In the light of our brief description of actor-critic methods in Section 1.1.3, Eq. (1.21c) is used to update the actor parameter whereas (1.21d) describes the evolution of the so-called critic approximating the unknown value function. Concerning the actor, the temporal difference error δ t+1 estimates the unknown advantage function involved in the policy gradient theorem (see Eq. (1.4) and Eq. (1.6)). The sequence (ω t ) corresponds to the critic: as previously mentioned in Eq. (1.7), the quantity φ( St ) T ω t is a linear approximation of V π θ t ( St ). Regarding the critic, if the sequences (ω t ) and (ω t ) coincide (i.e., Eq. (1.21e) is modified accordingly), then the algorithm coincides with a standard actor-critic algorithm where the critic uses a standard value-based method called TD-learning [START_REF] Sutton | Learning to predict by the methods of temporal differences[END_REF]). Instead, the algorithm uses a sequence (ω t ) defining the target network and updated using Eq. (1.21e). We refer the reader to Section 5.4 in Chapter 5 for a detailed presentation of the algorithm together with an explanation of the idea of target network.

Structure of the thesis and publications

Theoretical contributions investigating the use of a target network are very recent and limited to temporal difference (TD) learning for policy evaluation [START_REF] Lee | Target-based temporal-difference learning[END_REF] and critic-only methods such as Q-learning for control (Zhang et al., 2021b). In particular, these works are not concerned with actor-critic algorithms and leave the question of the finite-time analysis open.

We propose the first theoretical analysis of an online target-based actor-critic algorithm (see Eqs. (1.21a) to (1.21e) above) in the discounted reward setting. We consider the linear function approximation setting where a linear combination of pre-selected feature (or basis) functions estimates the value function in the critic as can be apppreciated in Eqs.

(1.21d) and (1.21b). An analysis of this setting is an insightful first step before tackling the more challenging nonlinear function approximation setting aligned with the use of neural networks. Our algorithm uses three different timescales: one for the actor using the stepsizes (α t ) and two for the critic using the stepsizes (β t ) and (ξ t ) s.t. α t /ξ t → 0 and ξ t /β t → 0 as t → +∞. Instead of using the standard single timescale TD learning algorithm as a critic, as can be seen in Eqs. (1.21b), (1.21d) and (1.21e), we use a two timescales target-based version of TD learning closely inspired from practical actor-critic algorithms implementing target networks.

First, we establish asymptotic convergence results for both the critic and the actor under Markovian sampling. More precisely, as the actor parameter changes slowly compared to the critic one, we show that the critic which uses a target variable tracks a slowly moving target. Then, we show that the actor parameter visits infinitely often a region of the parameter space where the norm of the policy gradient is dominated by a bias induced by linear function approximation.

Second, we conduct a finite-time analysis of our actor-critic algorithm which shows the impact of using a target variable on the convergence rates and the sample complexity. This non-asymptotic analysis provides a more quantitative result bounding the average expected squared gradient norm 1

T T t=1 E[ ∇J(θ t ) 2 ]
for a given positive time horizon T ∈ N. Here, the sequence of interest (θ t ) is produced by our target-based actor-critic algorithm to solve Problem (1.3) with a parameterized family of policies {π θ : θ ∈ R d } up to the linear function approximation error.

Structure of the thesis and publications

Our contributions in this thesis resulted in the following publications and preprints listed in chronological order: [START_REF] Bianchi | Stochastic subgradient descent escapes active strict saddles[END_REF] • Anas Barakat, Pascal Bianchi & Julien Lehmann (2021). Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation. ArXiv Preprint: arXiv:2106.07472.

• Anas Barakat & Pascal
This manuscript is organized as follows. Each chapter corresponds to one of the aforementioned publications and preprints. Especially devoted to RL, Chapter 5 is independent from the rest of the thesis. The last chapter of this thesis (Chapter 6) contains concluding remarks and some directions of future research.

Introduction

Consider the problem of finding a local minimizer of the expectation F (x) := E(f (x, ξ)) w.r.t. x ∈ R d , where f ( . , ξ) is a possibly non-convex function depending on some random variable ξ. The distribution of ξ is assumed unknown, but revealed online by the observation of iid copies (ξ n : n ≥ 1) of the r.v. ξ. Stochastic gradient descent (SGD) is the most classical algorithm to search for such a minimizer. Variants of SGD which include an inertial term have also become very popular. In these methods, the update rule depends on a parameter called the learning rate, which is generally assumed constant or vanishing. These algorithms, although widely used, have at least two limitations. First, the choice of the learning rate is generally difficult; large learning rates result in large fluctuations of the estimate, whereas small learning rates induce slow convergence. Second, a common learning rate is used for every coordinate despite the possible discrepancies in the values of the gradient vector's coordinates.

To alleviate these limitations, the popular Adam algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] adjusts the learning rate coordinate-wise, as a function of the past values of the squared gradient vectors' coordinates. The algorithm thus combines the assets of inertial methods with an adaptive per-coordinate learning rate selection. Finally, the algorithm includes a so-called bias correction step. Acting on the current estimate of the gradient vector, this step is especially useful during the early iterations.

Despite the growing popularity of the algorithm, only few works investigate its behavior from a theoretical point of view (see the discussion in Section 2.6). The present chapter studies the convergence of Adam from a dynamical system viewpoint.

Contributions

• We introduce a continuous-time version of the Adam algorithm under the form of a non-autonomous ordinary differential equation (ODE). Building on the existence of an explicit Lyapunov function for the ODE, we show the existence of a unique global solution to the ODE. This first result turns out to be non-trivial due to the irregularity of the vector field. We then establish the convergence of the continuous-time Adam trajectory to the set of critical points of the objective function F . The proposed continuous-time version of Adam provides useful insights on the effect of the bias correction step. It is shown that, close to the origin, the objective function F is non-increasing along the Adam trajectory, suggesting that early iterations of Adam can only improve the initial guess.

• Under a Łojasiewicz-type condition, we prove that the solution to the ODE converges to a single critical point of the objective function F . We provide convergence rates in this case.

• In discrete time, we first analyze the Adam iterates in the constant stepsize regime as originally introduced in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. In this work, it is shown that the discrete-time Adam iterates shadow the behavior of the non-autonomous ODE in the asymptotic regime where the stepsize parameter γ of Adam is small. More precisely, we consider the interpolated process z γ (t) which consists of a piecewise linear interpolation of the Adam iterates. The random process z γ is indexed by the parameter γ, which is assumed constant during the whole run of the algorithm. In the space of continuous functions on [0, +∞) equipped with the topology of uniform convergence on compact sets, we establish that z γ converges in probability to the solution to the non-autonomous ODE when γ tends to zero.

• Under a stability condition, we prove the asymptotic ergodic convergence of the probability of the discrete-time Adam iterates to approach the set of critical points of the objective function in the doubly asymptotic regime where n → ∞ then γ → 0.

• Beyond the original constant stepsize Adam, we propose a decreasing stepsize version of the algorithm. We provide sufficient conditions ensuring the stability and the almost sure convergence of the iterates towards the critical points of the objective function.

• We establish a convergence rate of the stochastic iterates of the decreasing stepsize algorithm under the form of a conditional central limit theorem.

We claim that our analysis can be easily extended to other adaptive algorithms such as e.g. RmsProp or AdaGrad [START_REF] Tieleman | Lecture 6.e-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF][START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] and AmsGrad (see Section 2.6).

Chapter organization. In Section 2.2, we present the Adam algorithm and the main assumptions. Our main results are stated in Sections 2.3 to 2.5. We provide a review of related works in Section 2.6. The rest of the chapter addresses the proofs of our results (Sections 2.7 to 2.9).

Notations. If x, y are two vectors on R d for some d ≥ 1, we denote by x y, x 2 , x/y, |x|, |x| the vectors on R d whose i-th coordinates are respectively given by x i y i ,

x 2 i , x i /y i , |x i |, |x i |.
Inequalities of the form x ≤ y are read componentwise. Denote by Algorithm 2.1 Adam(γ,α,β,ε).

Initialization: x 0 ∈ R d , m 0 = 0, v 0 = 0. for n = 1 to n iter do m n = αm n-1 + (1 -α)∇f (x n-1 , ξ n ) v n = βv n-1 + (1 -β)∇f (x n-1 , ξ n ) 2 mn = m n /(1 -α n ) {bias correction step} vn = v n /(1 -β n ) {bias correction step} x n = x n-1 -γ mn /(ε + √ vn ) . end for
• the standard Euclidean norm. For any vector v ∈ (0, +∞) d , write

x 2 v = i v i x 2 i . Notation A T represents the transpose of a matrix A. If z ∈ R d and A is a non-empty subset of R d , we use the notation d(z, A) := inf{ z -z : z ∈ A}. If A is a set, we
denote by 1 A the function equal to one on that set and to zero elsewhere. We denote by C([0, +∞), R d ) the space of continuous functions from [0, +∞) to R d endowed with the topology of uniform convergence on compact intervals.

The Adam algorithm

Algorithm and Assumptions

Let (Ω, F, P) be a probability space, and let (Ξ, S) denote another measurable space. Consider a measurable map f : R d × Ξ → R, where d is an integer. For a fixed value of ξ, the mapping x → f (x, ξ) is supposed to be differentiable, and its gradient w.r.t.

x is denoted by ∇f (x, ξ).

Define Z := R d × R d × R d , Z + := R d × R d × [0, +∞) d and Z * + := R d × R d × (0, +∞) d .
Adam generates a sequence z n := (x n , m n , v n ) on Z + given by Algorithm 2.1. It satisfies: z n = T γ,α,β (n, z n-1 , ξ n ) , for every n ≥ 1, where for every

z = (x, m, v) in Z + , ξ ∈ Ξ, T γ,α,β (n, z, ξ) :=     x - γ(1-α n ) -1 (αm+(1-α)∇f (x,ξ)) ε+(1-β n ) -1/2 (βv+(1-β)∇f (x,ξ) 2 ) 1/2 αm + (1 -α)∇f (x, ξ) βv + (1 -β)∇f (x, ξ) 2     .
(2.1) Remark 1. The iterates z n form a non-homogeneous Markov chain, because T γ,α,β (n, z, ξ) depends on n. This is due to the so-called debiasing step, which consists of replacing m n , v n in Algorithm 2.1 by their "debiased" versions mn , vn . The motivation becomes clear when expanding the expression:

mn = m n 1 -α n = 1 -α 1 -α n n-1 k=0 α k ∇f (x k , ξ k+1 ) .
From this equation, it is observed that, mn forms a convex combination of the past gradients. This is unlike m n , which may be small during the first iterations.

Assumption 2.2.1. The mapping f : R d × Ξ → R satisfies the following.

i) For every x ∈ R d , f (x, . ) is S-measurable.
ii) For almost every ξ, the map f ( . , ξ) is continuously differentiable.

iii) There exists

x * ∈ R d s.t. E(|f (x * , ξ)|) < ∞ and E( ∇f (x * , ξ) 2 ) < ∞. iv) For every compact subset K ⊂ R d , there exists L K > 0 such that for every (x, y) ∈ K 2 , E( ∇f (x, ξ) -∇f (y, ξ) 2 ) ≤ L 2 K x -y 2 .
Under Assumption 2.2.1, it is an easy exercise to show that the mappings F : R d → R and S : R d → R d , given by: It follows from our assumptions that the set of critical points of F , denoted by

F (x) := E(f (x, ξ)) and S(x) := E(∇f (x, ξ) 2 ) (2.
S := ∇F -1 ({0}),
is non-empty. Assumption 2.2.3 means that there is no point x ∈ R d satisfying ∇f (x, ξ) = 0 with probability one (w.p.1). This is a mild hypothesis in practice.

Asymptotic regime

We address the constant stepsize regime, where γ is fixed along the iterations (the default value recommended in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is γ = 0.001). As opposed to the decreasing stepsize context, the sequence z γ n := z n cannot in general converge as n tends to infinity, in an almost sure sense. Instead, we investigate the asymptotic behavior of the family of processes (n → z γ n ) γ>0 indexed by γ, in the regime where γ → 0. We use the ODE method (see e.g., [START_REF] Benaïm | Dynamics of stochastic approximation algorithms[END_REF]; [START_REF] Benaïm | Ergodic properties of weak asymptotic pseudotrajectories for semiflows[END_REF]). The interpolated process z γ is the piecewise linear function defined on [0, +∞) → Z + for all t ∈ [nγ, (n + 1)γ) by:

z γ (t) := z γ n + (z γ n+1 -z γ n ) t -nγ γ . (2.3)
We establish the convergence in probability of the family of random processes (z γ ) γ>0 as γ tends to zero, towards a deterministic continuous-time system defined by an ODE. The latter ODE, which we provide below at Eq. (ODE), will be referred to as the continuous-time version of Adam.

Before describing the ODE, we need to be more specific about our asymptotic regime. As opposed to SGD, Adam depends on two parameters α, β, in addition to the stepsize γ. [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] recommend choosing the constants α and β close to one (the default values α = 0.9 and β = 0.999 are suggested). It is thus legitimate to assume that α and β tend to one, as γ tends to zero. We set α := ᾱ(γ) and β := β(γ), where ᾱ(γ) and β(γ) converge to one as γ → 0.

Assumption 2.2.4. The functions ᾱ : R + → [0, 1) and β : R + → [0, 1) are s.t. the following limits exist:

a := lim γ ↓ 0 1 -ᾱ(γ) γ , b := lim γ ↓ 0 1 -β(γ) γ . (2.4)
Moreover, a > 0, b > 0, and the following condition holds: b ≤ 4a .

Note that the condition b ≤ 4a is compatible with the default settings recommended by [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]. In our model, we shall now replace the map T γ,α,β by T γ, ᾱ(γ), β(γ) . Let x 0 ∈ R d be fixed. For any fixed γ > 0, we define the sequence (z γ n ) generated by Adam with a fixed stepsize γ > 0:

z γ n := T γ, ᾱ(γ), β(γ) (n, z γ n-1 , ξ n ) , (2.5)
the initialization being chosen as z γ 0 = (x 0 , 0, 0).

Continuous-time system

Ordinary differential equation

In order to gain insight into the behavior of the sequence (z γ n ) defined by (2.5), it is convenient to rewrite the Adam iterations under the following equivalent form, for every n ≥ 1:

z γ n = z γ n-1 + γh γ (n, z γ n-1 ) + γ∆ γ n , (2.6)
where we define for every γ > 0, z ∈ Z + ,

h γ (n, z) := γ -1 E(T γ, ᾱ(γ), β(γ) (n, z, ξ) -z) , (2.7)
and where

∆ γ n := γ -1 (z γ n -z γ n-1 )-h γ (n, z γ n-1 ). Note that (∆ γ n
) is a martingale increment noise sequence in the sense that E(∆ γ n |F n-1 ) = 0 for all n ≥ 1, where F n stands for the σ-algebra generated by the r.v. ξ 1 , . . . , ξ n . Define the map h : (0, +∞) × Z + → Z for all t > 0, all z = (x, m, v) in Z + by:

h(t, z) =     -(1-e -at ) -1 m ε+ √ (1-e -bt ) -1 v a(∇F (x) -m) b(S(x) -v)     , (2.8)
where a, b are the constants defined in Assumption 2.2.4. We prove that, for any fixed (t, z), the quantity h(t, z) coincides with the limit of h γ ( t/γ , z) as γ ↓ 0. This remark along with Eq. (2.6) suggests that, as γ ↓ 0, the interpolated process z γ shadows the non-autonomous differential equation

ż(t) = h(t, z(t)) . (ODE)

Existence, uniqueness, convergence

Since h( . , z) is non-continuous at point zero for a fixed z ∈ Z + , and since h(t, . ) is not locally Lipschitz continuous for a fixed t > 0, the existence and uniqueness of the solution to (ODE) do not stem directly from off-the-shelf theorems.

Let x 0 be fixed. A continuous map z : [0, +∞) → Z + is said to be a global solution to (ODE) with initial condition (x 0 , 0, 0) if z is continuously differentiable on (0, +∞), if Eq. (ODE) holds for all t > 0, and if z(0) = (x 0 , 0, 0). There exists a unique global solution z : [0, +∞) → Z + to (ODE) with initial condition (x 0 , 0, 0). Moreover, z([0, +∞)) is a bounded subset of Z + .

On the other hand, we note that a solution may not exist for an initial point(x 0 , m 0 , v 0 ) with arbitrary (non-zero) values of m 0 , v 0 .

Theorem 2.2 (Convergence). Let Assumptions 2.2.1 to 2.2.4 hold true. Assume that F (S) has an empty interior. Let z : t → (x(t), m(t), v(t)) be the global solution to (ODE) with the initial condition (x 0 , 0, 0). Then, the set S is non-empty and lim t→∞ d(x(t), S) = 0, lim t→∞ m(t) = 0, lim t→∞ S(x(t))v(t) = 0.

Lyapunov function. The proof of Th. 2.1 relies on the existence of a Lyapunov function for the non-autonomous equation (ODE). Define V : (0, +∞) × Z + → R by

V (t, z) := F (x) + 1 2 m 2 U (t,v) -1 , (2.9)
for every t > 0 and every z = (x, m, v) in Z + , where U : (0, +∞) × [0, +∞) d → R d is the map given by:

U (t, v) := a(1 -e -at ) ε + v 1 -e -bt .
(2.10)

Then, t → V (t, z(t)) is decreasing if z(•) is the global solution to (ODE).
Cost decrease at the origin. As F itself is not a Lyapunov function for (ODE), there is no guarantee that F (x(t)) is decreasing w.r.t. t. Nevertheless, the statement holds at the origin. Indeed, it can be shown that .11) In other words, the (continuous-time) Adam procedure can only improve the initial guess x 0 . This is the consequence of the so-called bias correction steps in Adam (see Algorithm 2.1). If these debiasing steps were deleted in the Adam iterations, the early stages of the algorithm could degrade the initial estimate x 0 .

lim t↓0 V (t, z(t)) = F (x 0 ) (see Prop. 2.12). As a consequence, ∀t ≥ 0, F (x(t)) ≤ F (x 0 ) . ( 2
Derivatives at the origin. The proof of Th. 2.1 reveals that the initial derivative is given by ẋ(0) = -∇F (x 0 )/(ε + S(x 0 )) (see Lem. 2.9). In the absence of debiasing steps, the initial derivative ẋ(0) would be a function of the initial parameters m 0 , v 0 , and the user would be required to tune these hyperparameters. No such tuning is required thanks to the debiasing step. When ε is small and when the variance of ∇f (x 0 , ξ) is small (i.e., S(x 0 ) ∇F (x 0 ) 2 ), the initial derivative ẋ(0) is approximately equal to -∇F (x 0 )/|∇F (x 0 )|. This suggests that in the early stages of the algorithm, the Adam iterations are comparable to the sign variant of the gradient descent, the properties of which were discussed in previous works, see [START_REF] Balles | Dissecting adam: The sign, magnitude and variance of stochastic gradients[END_REF].

Convergence rates

In this paragraph, we establish the convergence to a single critical point of F and quantify the convergence rate, using the following assumption (Łojasiewicz, 1963).

Assumption 2.3.1 (Łojasiewicz property). For any x * ∈ S, there exist c > 0 , σ > 0 and θ ∈ (0, 1 2 ] s.t.

∀x ∈ R d s.t x -x * ≤ σ , ∇F (x) ≥ c|F (x) -F (x * )| 1-θ .
(2.12) Assumption 2.3.1 holds for real-analytic functions and semialgebraic functions. We refer to [START_REF] Haraux | The convergence problem for dissipative autonomous systems[END_REF]; [START_REF] Attouch | On the convergence of the proximal algorithm for nonsmooth functions involving analytic features[END_REF]; [START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF] for a discussion and a review of applications. We will call any θ satisfying (2.12) for some c, σ > 0, as a Moreover, if θ ∈ (0, 1 2 ] is a Łojasiewicz exponent of F at x * , there exists a constant C > 0 s.t. for all t ≥ 0,

x(t) -x * ≤ Ct -θ 1-2θ , if 0 < θ < 1 2 , x(t) -x * ≤ Ce -δt , for some δ > 0 if θ = 1 2 .
2.4 Discrete-time system: convergence of Adam Assumption 2.4.1. The sequence (ξ n : n ≥ 1) is iid, with the same distribution as ξ.

Assumption 2.4.2. Let p > 0. Assume either one of the following conditions.

i) For every compact set

K ⊂ R d , sup x∈K E( ∇f (x, ξ) p ) < ∞ . ii) For every compact set K ⊂ R d , ∃ p K > p, sup x∈K E( ∇f (x, ξ) p K ) < ∞.
The value of p will be specified in the sequel, in the statement of the results. Clearly, Assumption 2.4.2 ii) is stronger than Assumption 2.4.2 i). We shall use either the latter or the former in our statements.

Theorem 2.4. Let Assumptions 2.2.1 to 2.2.4 and 2.4.1 hold true. Let Assumption 2.4.2 ii) hold with p = 2. Consider x 0 ∈ R d . For every γ > 0, let (z γ n : n ∈ N) be the random sequence defined by the Adam iterations (2.5) and z γ 0 = (x 0 , 0, 0). Let z γ be the corresponding interpolated process defined by Eq. (2.3). Finally, let z denote the unique global solution to (ODE) issued from (x 0 , 0, 0). Then,

∀ T > 0, ∀ δ > 0, lim γ↓0 P   sup t∈[0,T ] z γ (t) -z(t) > δ   = 0 .
Recall that a family of r.v. (X α ) α∈I is called bounded in probability, or tight, if for every δ > 0, there exists a compact set K s.t. P(X α ∈ K) ≥ 1δ for every α ∈ I.

Assumption 2.4.3. There exists γ0 > 0 s.t. the family of r.v. (z γ n : n ∈ N, 0 < γ < γ0 ) is bounded in probability.

Theorem 2.5. Consider x 0 ∈ R d . For every γ > 0, let (z γ n : n ∈ N) be the random sequence defined by the Adam iterations (2.5) and z γ 0 = (x 0 , 0, 0

) : n ∈ N * ), ε). Initialization: x 0 ∈ R d , m 0 = 0, v 0 = 0, r 0 = r0 = 0. for n = 1 to n iter do m n = α n m n-1 + (1 -α n )∇f (x n-1 , ξ n ) v n = β n v n-1 + (1 -β n )∇f (x n-1 , ξ n ) 2 r n = α n r n-1 + (1 -α n ) rn = β n rn-1 + (1 -β n ) mn = m n /r n {bias correction step} vn = v n /r n {bias correction step} x n = x n-1 -γ n mn /(ε + √ vn ) . end for every δ > 0, lim γ↓0 lim sup n→∞ 1 n n k=1 P(d(x γ k , S) > δ) = 0 . (2.13)
Convergence in the long run. When the stepsize γ is constant, the sequence (x γ n ) cannot converge in the almost sure sense as n → ∞. Convergence may only hold in the doubly asymptotic regime where n → ∞ then γ → 0.

Randomization. For every n, consider a r.v. N n uniformly distributed on {1, . . . , n}. Define xγ n = x γ Nn . We obtain from Th. 2.5 that for every δ > 0, lim sup

n→∞ P(d(x γ n , S) > δ) --→ γ↓0 0 .
Relationship between discrete and continuous time Adam. Th. 2.4 means that the family of random processes (z γ : γ > 0) converges in probability as γ ↓ 0 towards the unique solution to (ODE) issued from (x 0 , 0, 0). This motivates the fact that the non-autonomous system (ODE) is a relevant approximation to the behavior of the iterates (z γ n : n ∈ N) for a small value of the stepsize γ. Stability. Assumption 2.4.3 ensures that the iterates z γ n do not explode in the long run. A sufficient condition is for instance that sup n,γ E z γ n < ∞ . In theory, this assumption can be difficult to verify. Nevertheless, in practice, a projection step on a compact set can be introduced to ensure the boundedness of the estimates.

A decreasing stepsize Adam algorithm

Algorithm

Adam inherently uses constant stepsizes. Consequently, the iterates (2.5) do not converge in the almost sure sense. In order to achieve convergence, we introduce in this section a decreasing stepsize version of Adam. The iterations are given in Algorithm 2.2. The algorithm generates a sequence z n = (x n , m n , v n ) with initial point z 0 = (x 0 , 0, 0), where x 0 ∈ R d . Apart from the fact that the hyperparameters (γ n , α n , β n ) now depend on n, the main difference w.r.t Algorithm 2.1 lies in the expression of the debiasing step. As noted in Remark 1, the aim is to rescale m n (resp. v n ) in such a way that the rescaled version mn (resp. vn ) is a convex combination of past stochastic gradients (resp. squared gradients). While in the constant step case the rescaling coefficient is

(1 -α n ) -1 (resp. (1 -β n ) -1
), the decreasing step case requires dividing m n by the coefficient r n = 1 -n i=1 α i (resp. v n by rn = 1 -n i=1 β i ), which keeps track of the previous weights:

mn = m n r n = n k=1 ρ n,k ∇f (x k-1 , ξ k ) n k=1 ρ n,k
,

where for every n, k, ρ n,k = α n • • • α k+1 (1 -α k ).
A similar equation holds for vn .

Almost sure convergence

Assumption 2.5.1 (Stepsizes). The following holds.

i) For all n ∈ N, γ n > 0 and

γ n+1 /γ n → 1, ii) n γ n = +∞ and n γ 2 n < +∞, iii) For all n ∈ N, 0 ≤ α n ≤ 1 and 0 ≤ β n ≤ 1, iv) There exist a, b s.t. 0 < b < 4a, γ -1 n (1 -α n ) → a and γ -1 n (1 -β n ) → b . Theorem 2.
∈ R d , E[ ∇f (x, ξ) 2 ] ≤ C(1 + F (x)).
iii) We assume the condition: lim sup n→∞

1 γn -1-α n+2 1-α n+1 1 γ n+1 < 2(a -b 4 ) ,
which is satisfied for instance if b < 4a and 1α n+1 = aγ n .

Theorem

F (x * ) of F at x * is positive definite. ii) S is continuously differentiable on V. Define D := diag (ε + S 1 (x * )) -1 , • • • , (ε + S d (x * )) -1
. Let P be an orthogonal matrix s.t. the following spectral decomposition holds: 2.14) where I d represents the d × d identity matrix and ∇S(x * ) is the Jacobian matrix of S at x * . The largest real part of the eigenvalues of H coincides with -L, where

D 1/2 ∇ 2 F (x * )D 1/2 = P diag(λ 1 , • • • , λ d )P -1 , where λ 1 , • • • , λ d are the (positive) eigenvalues of D 1/2 ∇ 2 F (x * )D 1/2 . Define H :=    0 -D 0 a∇ 2 F (x * ) -aI d 0 b∇S(x * ) 0 -bI d    (
L := b ∧ a 2   1 - 1 - 4λ 1 a ∨ 0    > 0 . (2.15)
Finally, define the 3d × 3d matrix

Q :=       0 0 0 0 0 E    a∇f (x * , ξ) b(∇f (x * , ξ) 2 -S(x * )) a∇f (x * , ξ) b(∇f (x * , ξ) 2 -S(x * )) T          . (2.16)
Assumption 2.5.4. The following holds.

i) There exist κ ∈ (0, 1], γ 0 > 0, s.t. the sequence (γ n ) satisfies γ n = γ 0 /(n + 1) κ for all n. If κ = 1, we assume moreover that γ 0 > 1 2L . ii) The sequences 1 γn ( 1-αn γna) and 1 γn ( 1-βn γnb) are bounded.

For an arbitrary sequence (X n ) of random variables on some Euclidean space, a probability measure µ on that space and an event Γ s.t. P(Γ) > 0, we say that X n converges in distribution to µ given Γ if the measures P(X n ∈ • |Γ) converge weakly to µ.

= (x n , m n , v n ) given by Algorithm 2.2. Set z * = (x * , 0, S(x * )). Set ζ := 0 if 0 < κ < 1 and ζ := 1 2γ 0 if κ = 1. Assume P(z n → z * ) > 0.
Then, given the event {z n → z * }, the rescaled vector √ γ n -1 (z nz * ) converges in distribution to a zero mean Gaussian distribution on R 3d with a covariance matrix Σ which is solution to the Lyapunov equation:

H + ζI 3d Σ + Σ H T + ζI 3d = -Q. In particular, given {z n → z * }, the vector √ γ n -1 (x n -x * )
converges in distribution to a zero mean Gaussian distribution with a covariance matrix Σ 1 given by:

Σ 1 = D 1/2 P   C k, (1 -2ζ a )(λ k + λ -2ζ + 2 a ζ 2 ) + 1 2(a-2ζ) (λ k -λ ) 2   k, =1...d P -1 D 1/2
(2.17) where C := P -1 D 1/2 E ∇f (x * , ξ)∇f (x * , ξ) T D 1/2 P .

The following remarks are useful.

• The variable v n has an impact on the limiting covariance Σ 1 through its limit S(x * ) (used to define D), but the fluctuations of v n and the parameter b have no effect on Σ 1 . As a matter of fact, Σ 1 coincides with the limiting covariance matrix that would have been obtained by considering iterates of the form

   x n+1 = x n -γ n+1 p n+1 p n+1 = p n + aγ n+1 (D∇f (x n , ξ n+1 ) -p n ) ,
which can be interpreted as a preconditioned version of the stochastic heavy ball algorithm [START_REF] Gadat | Stochastic heavy ball[END_REF]. Of course, the above iterates are not implementable because the preconditioning matrix D is unknown.

• When a is large, Σ 1 is close to the matrix Σ (0)

1 obtained when letting a → +∞ in Eq. (3.10). The matrix Σ (0) 1 is the solution to the Lyapunov equation

(D∇ 2 F (x * ) -ζI d )Σ (0) 1 + Σ (0) 1 (∇ 2 F (x * )D -ζI d ) = DE ∇f (x * , ξ)∇f (x * , ξ) T D .
The matrix Σ (0) 1 can be interpreted as the asymptotic covariance matrix of the xvariable in the absence of the inertial term (that is, when one considers RmsProp instead of Adam). The matrix

Σ (0) 1 approximates Σ 1 in the sense that Σ 1 = Σ (0) 1 + 1 a ∆ + O( 1 a 2
) for some symmetric matrix ∆ which can be explicited. The matrix ∆ is neither positive nor negative definite in general. This suggests that the question of the potential benefit of the presence of an inertial term is in general problem dependent.

• In the statement of Th. 2.8, the conditioning event {z n → z * } can be replaced by the event {x n → x * } under the additional assumption that n γ 2 n < +∞.

Related works

Although the idea of adapting the (per-coordinate) learning rates as a function of past gradient values is not new (see e.g. variable metric methods such as the BFGS algorithms), AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] led the way to a new class of algorithms that are sometimes referred to as adaptive gradient methods. AdaGrad consists of dividing the learning rate by the square root of the sum of previous gradients squared componentwise. The idea was to give larger learning rates to highly informative but infrequent features instead of using a fixed predetermined schedule. However, in practice, the division by the cumulative sum of squared gradients may generate small learning rates, thus freezing the iterates too early. Several works proposed heuristical ways to set the learning rates using a less aggressive policy. [START_REF] Tieleman | Lecture 6.e-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF] introduced an unpublished, yet popular, algorithm referred to as RmsProp where the cumulative sum used in AdaGrad is replaced by a moving average of squared gradients. Adam combines the advantages of both AdaGrad, RmsProp and inertial methods.

As opposed to AdaGrad, for which theoretical convergence guarantees exist [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF][START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF][START_REF] Zhou | On the convergence of adaptive gradient methods for nonconvex optimization[END_REF]Ward et al., 2019a;[START_REF] Traoré | Sequential convergence of adagrad algorithm for smooth convex optimization[END_REF], Adam is comparatively less studied. The initial paper of [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] suggests a O( 1 √ T ) average regret bound in the convex setting, but [START_REF] Reddi | On the convergence of adam and beyond[END_REF] exhibit a counterexample in contradiction with this statement. The latter counterexample implies that the average regret bound of Adam does not converge to zero. A first way to overcome the problem is to modify the Adam iterations themselves in order to obtain a vanishing average regret. This led [START_REF] Reddi | On the convergence of adam and beyond[END_REF] to propose a variant called AmsGrad with the aim to recover, at least in the convex case, the sought guarantees. [START_REF] Balles | Dissecting adam: The sign, magnitude and variance of stochastic gradients[END_REF] interpret Adam as a variance-adapted sign descent combining an update direction given by the sign and a magnitude controlled by a variance adaptation principle. A "noiseless" version of Adam is considered in [START_REF] Basu | Convergence guarantees for rmsprop and adam in non-convex optimization and their comparison to nesterov acceleration on autoencoders[END_REF]. Under quite specific values of the Adam-hyperparameters, it is shown that for every δ > 0, there exists some time instant for which the norm of the gradient of the objective at the current iterate is no larger than δ. The recent work of [START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF] provides a similar result for AmsGrad and AdaGrad, but the generalization to Adam is subject to conditions which are not easily verifiable. [START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF] provide a convergence result for RmsProp using the objective function F as a Lyapunov function. However, our work suggests that unlike RmsProp, Adam does not admit F as a Lyapunov function. This makes the approach of [START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF] hardly generalizable to Adam. Moreover, [START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF] consider biased gradient estimates instead of the debiased estimates used in Adam.

In the present work, we study the behavior of an ODE, interpreted as the limit in probability of the (interpolated) Adam iterates as the stepsize tends to zero. Closely related continuous-time dynamical systems are also studied in [START_REF] Attouch | The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system[END_REF]; [START_REF] Cabot | On the long time behavior of second order differential equations with asymptotically small dissipation[END_REF]. We leverage the idea of approximating a discrete time stochastic system by a deterministic continuous one, often referred to as the ODE method. The recent work of [START_REF] Gadat | Stochastic heavy ball[END_REF] fruitfully exploits this method to study a stochastic version of the celebrated heavy ball algorithm. We refer to [START_REF] Davis | Stochastic subgradient method converges on tame functions[END_REF] for the reader interested in the non-differentiable setting with an analysis of the stochastic subgradient algorithm for non-smooth non-convex objective functions.

Concomitant to the paper on which this chapter is based, Belotto da Silva and Gazeau (31 Oct 2018) (posted only four weeks after the first version of the present work) study the asymptotic behavior of a similar dynamical system as the one introduced here. They establish several results in continuous time, such as avoidance of traps as well as convergence rates in the convex case; such aspects are out of the scope of this chapter. However, the question of the convergence of the (discrete-time) iterates is left open. In the current chapter, we also exhibit a Lyapunov function which allows, amongst others, to draw useful conclusions on the effect of the debiasing step of Adam. Finally, Belotto da Silva and Gazeau (31 Oct 2018) study a slightly modified version of Adam allowing to recover an ODE with a locally Lipschitz continuous vector field, whereas the original Adam algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] leads to an ODE with an irregular vector field. This technical issue is tackled in the present chapter.

2.7 Proofs for Section 2.3

Preliminaries

The results in this section are not specific to the case where F and S are defined as in Eq. (2.2): they are stated for any mappings F , S satisfying the following hypotheses. In the sequel, we consider the following generalization of Eq. (ODE) for any η > 0:

ż(t) = h(t + η, z(t)) .
(ODE η )

When η = 0, Eq. (ODE η ) boils down to the equation of interest (ODE). The choice η ∈ (0, +∞) will be revealed useful to prove Th. 2.1. Indeed, for η > 0, a solution to Eq. (ODE η ) can be shown to exist (on some interval) due to the continuity of the map h( . + η, . ). Considering a family of such solutions indexed by η ∈ (0, 1], the idea is to prove the existence of a solution to (ODE) as a cluster point of the latter family when η ↓ 0. Indeed, as the family is shown to be equicontinuous, such a cluster point does exist thanks to the Arzelà-Ascoli theorem. When η = +∞, Eq. (ODE η ) rewrites

ż(t) = h ∞ (z(t)) , (ODE ∞ )
where h ∞ (z) := lim t→∞ h(t, z). It is useful to note that for

(x, m, v) ∈ Z + , h ∞ ((x, m, v)) = -m/(ε + √ v) , a(∇F (x) -m) , b(S(x) -v) .
(2.18) Contrary to Eq. (ODE), Eq. (ODE ∞ ) defines an autonomous ODE. The latter admits a unique global solution for any initial condition in Z + , and defines a dynamical system D.

We shall exhibit a strict Lyapunov function for this dynamical system D, and deduce that any solution to (ODE ∞ ) converges to the set of equilibria of D as t → ∞. On the otherhand, we will prove that the solution to (ODE) with a proper initial condition is a so-called asymptotic pseudotrajectory (APT) of D. Due to the existence of a strict Lyapunov function, the APT shall inherit the convergence behavior of the autonomous system as t → ∞, which will prove Th. 2.2.

It is convenient to extend the map h : (0, +∞)×Z + → Z on (0, +∞)×Z → Z by setting h(t, (x, m, v)) := h(t, (x, m, |v|)) for every t > 0, (x, m, v) ∈ Z. Similarly, we extend h ∞ as h ∞ ((x, m, v)) := h ∞ ((x, m, |v|)). For any T ∈ (0, +∞] and any η ∈ [0, +∞], we say that a map z : [0, T ) → Z is a solution to (ODE η ) on [0, T ) with initial condition z 0 ∈ Z + , if z is continuous on [0, T ), continuously differentiable on (0, T ), and if (ODE η ) holds for all t ∈ (0, T ). When T = +∞, we say that the solution is global. We denote by Z η T (z 0 ) the subset of C([0, T ), Z) formed by the solutions to (ODE η ) on [0, T ) with initial condition z 0 . For any K ⊂ Z + , we define Z η T (K) := z∈K Z η T (z). Lemma 2.9. Let Assumptions 2.7.1 and 2.7.2 hold. Consider x 0 ∈ R d , T ∈ (0, +∞] and let z ∈ Z 0 T ((x 0 , 0, 0)), which we write

z(t) = (x(t), m(t), v(t)). Then, z is continuously differentiable on [0, T ), ṁ(0) = a∇F (x 0 ), v(0) = bS(x 0 ) and ẋ(0) = -∇F (x 0 ) ε+ √ S(x 0 ) .
Proof. By definition of z( . ), m(t) = t 0 a(∇F (x(s))m(s))ds for all t ∈ [0, T ) (and a similar relation holds for v(t)). The integrand being continuous, it holds that m and v are differentiable at zero and ṁ(0) = a∇F (x 0 ), v(0) = bS(x 0 ). Similarly, x(t) = x 0 + t 0 h x (s, z(s))ds, where h x (s, z(s)) := -(1e -as ) -1 m(s)/(ε + (1e -bs ) -1 v(s)) . Note that m(s)/s → ṁ(0) = a∇F (x 0 ) as s ↓ 0. Thus, (1e -as ) -1 m(s) → ∇F (x 0 ) as s → 0. Similarly, (1e -bs ) -1 v(s) → S(x 0 ). It follows that h x (s, z(s)) → -(ε + S(x 0 )) -1 ∇F (x 0 ). Thus, s → h x (s, z(s)) can be extended to a continuous map on [0, T ) → R d and the differentiability of x at zero follows.

Lemma 2.10. Let Assumptions 2.2.3, 2.7.1 and 2.7.2 hold. For every η ∈ [0, +∞], T ∈ (0, +∞], z 0 ∈ Z + , z ∈ Z η T (z 0 ), it holds that z((0, T )) ⊂ Z * + .

Proof. Set z(t) = (x(t), m(t), v(t)) for all t. Consider i ∈ {1, . . . , d}. Assume by contradiction that there exists t 0 ∈ (0, T )

s.t. v i (t 0 ) < 0. Set τ := sup{t ∈ [0, t 0 ] : v i (t) ≥ 0}.
Clearly, τ < t 0 and v i (τ ) = 0 by the continuity of v i . Since v i (t) ≤ 0 for all t ∈ (τ, t 0 ], it holds that vi (t) = b(S i (x(t))v i (t)) is nonnegative for all t ∈ (τ, t 0 ]. This contradicts the fact that v i (τ ) > v i (t 0 ). Thus, v i (t) ≥ 0 for all t ∈ [0, T ). Now assume by contradiction that there exists t ∈ (0, T ) s.t. v i (t) = 0. Then, vi (t) = bS i (x(t)) > 0. Thus,

lim δ↓0 v i (t-δ) -δ
= bS i (x(t)) . In particular, there exists δ > 0 s.t. v i (tδ) ≤ -δb 2 S i (x(t)) . This contradicts the first point.

Recall the definitions of V and U from Eqs. (2.9) and (2.10)

. Clearly, U ∞ (v) := lim t→∞ U (t, v) = a(ε + √ v) is well defined for every v ∈ [0, +∞) d .
Hence, we can also define V ∞ (z) := lim t→∞ V (t, z) for every z ∈ Z + .

Lemma 2.11. Let Assumptions 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. Consider (t, z) ∈ (0, +∞) × Z * + and set z = (x, m, v). Then, V and V ∞ are differentiable at points (t, z) and z respectively. Moreover,

∇V ∞ (z), h ∞ (z) ≤ -ε am U∞(v) 2 and ∇V (t, z), (1, h(t, z)) ≤ - ε 2 a m U (t, v) 2 .
Proof.

We only prove the second point, the proof of the first point follows the same line. Consider (t, z) ∈ (0, +∞) × Z * + . We decompose ∇V (t, z), (1, h(t, z)) = ∂ t V (t, z) + ∇ z V (t, z), h(t, z) . After tedious but straightforward derivations, we get:

∂ t V (t, z) = -d i=1 a 2 m 2 i U (t,v i ) 2 e -at ε 2 + e -at 2 -be -bt (1-e -at )
4a(1-e -bt )

v i 1-e -bt , (2.19) where U (t, v i ) = a(1 -e -at ) ε + v i 1-e -bt and ∇ z V (t, z), h(t, z) is equal to: d i=1 -a 2 m 2 i (1 -e -at ) U (t, v i ) 2   ε + (1 - b 4a ) v i 1 -e -bt + bS i (x) 4a v i (1 -e -bt )   .
Using that S i (x) ≥ 0, we obtain: (2.20) where c a,b (t

∇V (t, z), (1, h(t, z)) ≤ - d i=1 a 2 m 2 i U (t, v i ) 2 (1 - e -at 2 )ε + c a,b (t) v i 1 -e -bt ,
) := 1 -e -at 2 -b 4a 1-e -at
1-e -bt . Using inequality 1e -at /2 ≥ 1/2 in (2.20), the inequality (2.20) proves the lemma, provided that one is able to show that c a,b (t) ≥ 0, for all t > 0 and all a, b satisfying 0 < b ≤ 4a. We prove this last statement. It can be shown that the function b → c a,b (t) is decreasing on [0, +∞). Hence, c a,b (t) ≥ c a,4a (t). Now, c a,4a (t) = q(e -at ) where q : [0, 1) → R is the function defined for all y ∈ [0, 1) by q(y) = y y 4 -2y 3 + 1 /(2(1y 4 )) . Hence q ≥ 0. Thus, c a,b (t) ≥ q(e -at ) ≥ 0.

Define Z 0 := {(x, 0, 0) : x ∈ R d }.
Let ē : (0, +∞) × Z + → Z + be defined for every t > 0 and every z = (x, m, v) in Z + by: ē(t, z) := (x, m/(1e -at ), v/(1e -bt )) . Proposition 2.12. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. For every z 0 ∈ Z 0 , there exists a compact set K ⊂ Z + s.t. for all η ∈ [0, +∞), all T ∈ (0, +∞] and all z ∈ Z η T (z 0 ), ē(t + η, z(t)) : t ∈ (0, T ) ⊂ K . Moreover, choosing z 0 of the form z 0 = (x 0 , 0, 0) and z(t) = (x(t), m(t), v(t)), it holds that F (x(t)) ≤ F (x 0 ) for all t ∈ [0, T ).

Proof.

Let η ∈ [0, +∞). Consider a solution z η (t) = (x η (t), m η (t), v η (t)) as in the statement, defined on some interval [0, T ). Define mη (t) = m η (t)/(1e -a(t+η) ), vη (t) = v η (t)/(1e -b(t+η) ). By Lem. 2.10, t → V (t + η, z(t)) is continuous on [0, T ), and continuously differentiable on (0, T ). By Lem. 2.11, V (t + η, z η (t)) ≤ 0 for all t > 0. As a consequence, t → V (t + η, z η (t)) is non-increasing on [0, T ). Thus, for all t ≥ 0, F (x η (t)) ≤ lim t ↓0 V (t + η, z η (t )). Note that:

V (t + η, z η (t)) ≤ F (x η (t)) + 1 2 d i=1 m η,i (t) 2 a(1 -e -a(t+η) )ε .
If η > 0, every term in the sum in the righthand side tends to zero, upon noting that m η (t) → 0 as t → 0. The statement still holds if η = 0. Indeed, by Lem. 2.9, for a given i ∈ {1, . . . , d}, there exists δ > 0 s.t. for all 0 < t < δ, m η,i (t) 2 ≤ 2a 2 (∂ i F (x 0 )) 2 t 2 and 1e -at ≥ (at)/2. As a consequence, each term of the sum is no larger than 4(∂ i F (x 0 )) 2 t/ε, which tends to zero as t → 0. We conclude that for all t ≥ 0, F (x η (t)) ≤ F (x 0 ). In particular, {x η (t) : t ∈ [0, T )} ⊂ {F ≤ F (x 0 )}, the latter set being bounded by Assumption 2.2.2.

We prove that v i,η (t) is (upper)bounded. Define R i := sup S i ({F ≤ F (x 0 )}), which is finite by continuity of S. Assume by contradiction that the set {t ∈ [0, T ) : v η,i (t) ≥ R i +1} is non-empty, and denote its infimum by τ . By continuity of v η,i , one has v η,i (τ ) = R i + 1. This by the way implies that τ > 0. Hence, vη,i (τ

) = b(S i (x η (τ )) -v η,i (τ )) ≤ -b.
This means that there exists τ < τ s.t. v η,i (τ ) > v η,i (τ ), which contradicts the definition of τ . We have shown that v η,i (t) ≤ R i + 1 for all t ∈ (0, T ). In particular, when

t ≥ 1, vη,i (t) = v η,i (t)/(1-e -bt ) ≤ (R i +1)/(1-e -b
) . Consider t ∈ (0, 1∧T ). By the mean value theorem, there exists tη

∈ [0, t] s.t. v η,i (t) = vη,i ( tη )t. Thus, v η,i (t) ≤ bS i (x( tη ))t ≤ bR i t.
Using that the map y → y/(1e -y ) is increasing on (0, +∞), it holds that for all t ∈ (0, 1 ∧ T ), vη,i (t) ≤ bR i /(1e -b ) . We have shown that, for all t ∈ (0, T ) and all i ∈ {1, . . . , d}, 0 ≤ vη,i (t) ≤ M , where

M := (1 -e -b ) -1 (1 + b)(1 + max{R : ∈ {1, . . . , d}).
As V (t+η, z η (t)) ≤ F (x 0 ), we obtain:

F (x 0 ) ≥ F (x η (t))+ 1 2 m η (t) 2 U (t+η,vη(t)) -1 . Thus, F (x 0 ) ≥ inf F + 1 2a(ε+ √ M ) m η (t) 2 .
Therefore, m η ( . ) is bounded on [0, T ), uniformly in η. The same holds for mη by using the mean value theorem in the same way as for vη . The proof is complete. Proposition 2.13. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. Let K be a compact subset of Z + . Then, there exists another compact set K ⊂ Z + s.t. for every T ∈ (0, +∞] and every z ∈ Z ∞ T (K), z([0, T )) ⊂ K .

Proof. The proof follows the same line as Prop. 2.12 and is omitted.

For any K ⊂ Z + , define v min (K) := inf{v k : (x, m, v) ∈ K, i ∈ {1, . . . ,
z ∈ Z ∞ ∞ (K), of the form z(t) = (x(t), m(t), v(t)), v i (t) ≥ c min 1, v min (K) 2c + t (∀t ≥ 0, ∀i ∈ {1, . . . , d}) .
ii) For every z 0 ∈ Z 0 , there exists c > 0 s.t. for every η ∈ [0, +∞) and every

z ∈ Z η ∞ (z 0 ), v i (t) ≥ c min(1, t) (∀t ≥ 0, ∀i ∈ {1, . . . , d}) .
Proof. We prove the first point. Consider a compact set K ⊂ Z + . By Prop. 2.13, one can find a compact set

K ⊂ Z + s.t. for every z ∈ Z ∞ ∞ (K), it holds that {z(t) : t ≥ 0} ⊂ K . Denote by L S the Lipschitz constant of S on the compact set {x : (x, m, v) ∈ K }. Introduce the constants M 1 := sup{ m/(ε + √ v) ∞ : (x, m, v) ∈ K }, M 2 := sup{ S(x) ∞ : (x, m, v) ∈ K }. The constants L S , M 1 , M 2 are finite. Now consider a global solution z(t) = (x(t), m(t), v(t)) in Z ∞ ∞ (K). Choose i ∈ {1, .
. . , d} and consider t ≥ 0. By the mean value theorem, there exists t ∈

[0, t] s.t. v i (t) = v i (0) + vi (t )t. Thus, v i (t) = v i (0) + vi (0)t + b(S i (x(t )) -v i (t ) -S i (x(0)) + v i (0))t, which in turn implies v i (t) ≥ v i (0) + vi (0)t -bL S x(t ) -x(0) t -b|v i (t ) -v i (0)|t.
Using again the mean value theorem, for every ∈ {1, . . . , d}, there exists t ∈ [0, t ] s.t.

|x (t ) -x (0)| = t | ẋ (t )| ≤ tM 1 . Therefore, x(t ) -x(0) ≤ √ dM 1 t. Similarly, there exists t s.t.: |v i (t ) -v i (0)| = t | vi ( t)| ≤ t bS i (x( t)) ≤ tbM 2 . Putting together the above inequalities, v i (t) ≥ v i (0)(1 -bt) + bS i (x(0))t -bCt 2 , where C := (M 2 + L S √ dM 1 ). For every t ≤ 1/(2b), v i (t) ≥ v min 2 + tbC S min C -t ,
where we defined S min := inf{S i (x) : i ∈ {1, . . . , d}, (x, m, v) ∈ K}. Setting τ := 0.5 min(1/b, S min /C),

∀t ∈ [0, τ ], v i (t) ≥ v min 2 + bS min t 2 .
(2.21)

Set κ 1 := 0.5(v min + bS min τ ). Note that v i (τ ) ≥ κ 1 . Define S min := inf{S i (x) : i ∈ {1, . . . , d}, (x, m, v) ∈ K } . Note that S min > 0 by Assumptions 2.7.2 and 2.2.3. Finally, define κ = 0.5 min(κ 1 , S min ). By contradiction, assume that the set {t ≥ τ : v i (t) < κ} is non-empty, and denote by τ its infimum. It is clear that τ > τ and v i (τ ) = κ. Thus, b -1 vi (τ ) = S i (x(τ ))κ. We obtain that b -1 vi (τ ) ≥ 0.5S min > 0. As a consequence, there exists t ∈ (τ, τ ) s.t. v i (t) < v i (τ ). This contradicts the definition of τ . We have shown that for all t ≥ τ , v i (t) ≥ κ. Putting this together with Eq. (2.21) and using that κ ≤ v min + bS min τ , we conclude that: ∀t ≥ 0, v i (t) ≥ min κ , v min 2 + bS min t e -b(t+h) ). Denote by L S the Lipschitz constant of S on the set {x : (x, m, v) ∈ K}. Introduce the constants

M 1 := sup{ m/(ε + √ v) ∞ : (x, m, v) ∈ K}, M 2 := sup{ S(x) ∞ : (x, m, v) ∈ K }.
These constants being introduced, the rest of the proof follows the same line as the proof of the first point.

∈ Z + ,Z ∞ ∞ (z 0 ) = ∅. For every (z 0 , η) ∈ Z 0 × (0, +∞),Z η ∞ (z 0 ) = ∅.
Proof.

We prove the first point (the proof of the second point follows the same line). Under Assumptions 2.7.1 and 2.7.2, h ∞ is continuous. Therefore, Cauchy-Peano's theorem guarantees the existence of a solution to the (ODE) issued from z 0 , which we can extend over a maximal interval of existence [0, T max ).We conclude that the solution is global (T max = +∞) using the boundedness of the solution given by Prop.

< b ≤ 4a. Consider z 0 ∈ Z 0 . Denote by (z η : η ∈ (0, +∞)) a family of functions on [0, +∞) → Z + s.t. for every η > 0, z η ∈ Z η ∞ (z 0 ). Then, (z η ) η>0 is equicontinuous.
Proof. For every such solution z η , we set z η (t) = (x η (t), m η (t), v η (t)) for all t ≥ 0, and define mη and vη as in Prop. 2.12. By Prop. 2.12, there exists a constant M 1 s.t. for all η > 0 and all t ≥ 0, max( x η (t) , mη (t) ∞ , vη (t) ) ≤ M 1 . Using the continuity of ∇F and S, there exists another finite constant

M 2 s.t. M 2 ≥ sup{ ∇F (x) ∞ : x ∈ R d , x ≤ M 1 } and M 2 ≥ sup{ S(x) ∞ : x ∈ R d , x ≤ M 1 }. For every (s, t) ∈ [0, +∞) 2 ,
we have for all i ∈ {1, . . . , d},

|x η,i (t) -x η,i (s)| ≤ t s mη,i (u) ε + vη,i (u) du ≤ M 1 ε |t -s| , |m η,i (t) -m η,i (s)| ≤ t s a ∂ i F (x η (u)) -m η,i (u) du ≤ a(M 1 + M 2 )|t -s| , |v η,i (t) -v η,i (s)| ≤ t s b S i (x η (u)) -v η,i (u) du ≤ b(M 1 + M 2 )|t -s| .
Therefore, there exists a constant M 3 , independent from η, s.t. for all η > 0 and all

(s, t) ∈ [0, +∞) 2 , z η (t) -z η (s) ≤ M 3 |t -s|.
Proposition 2.17. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. For every z 0 ∈ Z 0 , Z 0 ∞ (z 0 ) = ∅ i.e., (ODE) admits a global solution issued from z 0 .

Proof. By Cor. 2.15, there exists a family (z η ) η>0 of functions on [0, +∞) → Z s.t. for every η > 0, z η ∈ Z η ∞ (z 0 ). We set as usual z η (t) = (x η (t), m η (t), v η (t)). By Lem. 2.16, and the Arzelà-Ascoli theorem, there exists a map z : [0, +∞) → Z and a sequence η n ↓ 0 s.t. z ηn converges to z uniformly on compact sets, as n → ∞. Considering some fixed scalars t > s > 0, z(t) = z(s) + lim n→∞ t s h(u + η n , z ηn (u))du . By Prop. 2.12, there exists a compact set K ⊂ Z + s.t. {z ηn (t) : t ≥ 0} ⊂ K for all n. Moreover, by Lem. 2.14, there exists a constant c > 0 s.t. for all n and all u ≥ 0, v ηn,k (u) ≥ c min(1, u). Denote by

K := K ∩ (R d × R d × [c min(1, s), +∞) d ). It is clear that K is a compact subset of Z * + .
Since h is continuously differentiable on the set [s, t] × K, it is Lipschitz continuous on that set. Denote by L h the corresponding Lipschitz constant. We obtain:

t s h(u + η n , z ηn (u)) -h(u, z(u)) du ≤ L h   η n + sup u∈[s,t] z ηn (u) -z(u)   (t -s) ,
and the righthand side converges to zero. As a consequence, for all t > s, z(t) = z(s) + t s h(u, z(u))du . Moreover, z(0) = z 0 . This proves that z ∈ Z 0 ∞ (z 0 ). i) For every z 0 ∈ Z 0 , Z 0 ∞ (z 0 ) is a singleton i.e., there exists a unique global solution to (ODE) with initial condition z 0 . ii) For every compact subset K of Z + , there exist nonnegative constants c 1 , c 2 s.t. for every

(z, z ) ∈ Z ∞ ∞ (K) 2 , ∀t ≥ 0, z(t) -z (t) 2 ≤ z(0) -z (0) 2 exp(c 1 + c 2 t) .
Proof. i) Consider solutions z and z in Z 0 ∞ (z 0 ). We denote by (x(t), m(t), v(t)) the blocks of z(t), and we define (x (t), m (t), v (t)) similarly. For all t > 0, we define m(t) := m(t)/(1e -at ), v(t) := v(t)/(1e -bt ), and we define m (t) and v (t) similarly. By Prop. 2.12, there exists a compact set K ⊂ Z + s.t. (x(t), m(t), v(t)) and (x (t), m (t), v (t)) are both in K for all t > 0. We denote by L S and L ∇F the Lipschitz constants of S and ∇F on the compact set {x : (x, m, v) ∈ K}. These constants are finite by Assumptions 2.7.1 and 2.7.2. We define

M := sup{ m ∞ : (x, m, v) ∈ K}. Define u x (t) := x(t) -x (t) 2 , u m (t) := m(t) -m (t) 2 and u v (t) := v(t) -v (t) 2 . Let δ > 0. Define: u (δ) (t) := u x (t) + δu m (t) + δu v (t)
. By the chain rule and the Cauchy-Schwarz inequality,

ux (t) ≤ 2 x(t) -x (t) m(t) ε + v(t) - m (t) ε + v (t) ≤ 2 x(t) -x (t) ε -1 m(t) -m (t) + M ε -2 v(t) -v (t) .
For every i ∈ {1, . . . , d},

vi (t) -v i (t) = |v i (t)-v i (t)| | √ vi (t)+ √ v i (t)|
. By Lem. 2.14, there exists c > 0 s.t. for all t ≥ 0, for every i ∈ {1, . . . , d}, vi (t) ∧ v i (t) ≥ c min(1, t). Thus,

ux (t) ≤ 2 x(t) -x (t)   ε -1 m(t) -m (t) + M 2ε 2 c min(1, t) v(t) -v (t)   .
For any δ > 0, 2 .22) We now study u m (t). For all t > 0, we obtain after some algebra: d dt m(t) = a(∇F (x(t))m(t))/(1e -at ) . Therefore,

x(t) -x (t) m(t) -m (t) ≤ δ -1/2 (u x (t) + δu m (t)) ≤ δ -1/2 u (δ) (t). Similarly, 2 x(t) -x (t) v(t) -v (t) ≤ δ -1/2 u (δ) (t). Thus, for any δ > 0, ux (t) ≤   1 ε √ δ + M 2ε 2 δc min(1, t)   u (δ) (t) . ( 2
um (t) = 2a 1 -e -at m(t) -m (t), ∇F (x(t)) -m(t) -∇F (x (t)) + m (t) ≤ 2aL ∇F 1 -e -at m(t) -m (t) x(t) -x (t) .
For any θ > 0, it holds that 2 m(t)m (t)

x(t) -x (t) ≤ θu x (t) + θ -1 u m (t).
In particular, letting θ := 2L ∇F , we obtain that for all δ > 0,

δ um (t) ≤ a 2(1-e -at ) 4δL 2 ∇F u x (t) + δu m (t) ≤ a 2 + 1 2t 4δL 2 ∇F u x (t) + δu m (t) , (2.23
) where the last inequality is due to the fact that y/(1e -y ) ≤ 1 + y for all y > 0. Using the exact same arguments, we also obtain that

δ uv (t) ≤ b 2 + 1 2t 4δL 2 S u x (t) + δu m (t) . (2.24)
We now choose any δ s.t. 4δ ≤ 1/ max(L 2 S , L 2 ∇F ). Then, Eq. ( 2.23) and ( 2.24) respectively imply that δ um (t) ≤ 0.5(a + t -1 )u (δ) (t) and δ uv (t) ≤ 0.5(b + t -1 )u (δ) (t). Summing these inequalities along with Eq. ( 2.22), we obtain that for every t > 0, u(δ) (t) ≤ ψ(t)u (δ) (t) , where:

ψ(t) := a+b 2 + 1 ε √ δ + M 2ε 2 √ δc min(1,t) + 1 t . From Grönwall's inequality, it holds that for every t > s > 0, u (δ) (t) ≤ u (δ) (s) exp t s ψ(s )ds .
We first consider the case where t ≤ 1. We set

c 1 := (a + b)/2 + (ε √ δ) -1 and c 2 := M/(ε 2 √ δc). With these nota- tions, t s ψ(s )ds ≤ c 1 t + c 2 √ t + ln t s . Therefore, u (δ) (t) ≤ u (δ) (s) s exp c 1 t + c 2 √ t + ln t .
By Lem. 2.9, recall that ẋ(0) and ẋ (0) are both well defined (and coincide). Thus,

u x (s) = x(s) -x (s) 2 ≤ 2 x(s) -x(0) -ẋ(0)s 2 + 2 x (s) -x (0) -ẋ (0)s 2 .
It follows that u x (s)/s 2 converges to zero as s ↓ 0. We now show the same kind of result for u m (s) and u v (s). Consider i ∈ {1, . . . , d}. By the mean value theorem, there exists s

(resp. s ) in the interval [0, t] s.t. m i (s) = ṁi (s)s (resp. m i (s) = ṁ i (s )s).
Thus, mi (s) = as 1-e -as ∂ i F (x(s))m i (s) , and a similar equality holds for m i (s). As a consequence,

| mi (s) -m i (s)| ≤ as 1 -e -as |∂ i F (x(s)) -∂ i F (x (s ))| + |m i (s) -m i (s )| ≤ as 1 -e -as L ∇F x(s) -x (s ) + |m i (s) -m i (s )| ≤ 2a(L ∇F ∨ 1)s 1 -e -as z(s) -z (s ) , where we used x(s) -x (s ) ≤ z(s) -z (s ) and |m i (s) -m i (s )| ≤ z(s) -z (s )
to obtain the last inequality. Using that s ≤ s and s ≤ s, it follows that:

| mi (s) -m i (s)| s ≤ 2a(L ∇F ∨ 1)s 1 -e -as z(s) -z(0) s + z (s ) -z (0) s .
By Lem. 2.9, z and z are differentiable at point zero. Then, the above inequality gives lim sup s↓0

| mi (s)-m i (s)| s ≤ 4(L ∇F ∨ 1) ż(0) . Thus, lim sup s↓0 u m (s) s 2 ≤ 16d(L 2 ∇F ∨ 1) ż(0) 2 .
Therefore, u m (s)/s converges to zero as s ↓ 0. By similar arguments, it can be shown that lim sup s↓0 u v (s)/s 2 ≤ 16d(L 2 S ∨ 1) ż(0) 2 , thus lim u v (s)/s = 0. Finally, we obtain that u (δ) (s)/s converges to zero as s ↓ 0. Letting s tend to zero, we obtain that for every t ≤ 1, u (δ) (t) = 0. Setting s = 1 and t > 1, and noting that ψ is integrable on [1, t], it follows that u (δ) (t) = 0 for all t > 1. This proves that z = z .

ii) Consider the compact set K, and introduce the compact set K ⊂ Z + as in Prop. 2.13, and the constant c > 0 defined in Lem. 2.14. Define

K x = {x : (x, m, v) ∈ K }. The set is compact in R d . Respectively denote by L S and L ∇F the Lipschitz constants of S and ∇F on K x . Introduce the constant M := sup{ m ∞ : (x, m, v) ∈ K }. Consider (z 0 , z 0 ) ∈ K 2 and
two global solutions z( . ) and z ( . ) starting at z 0 and z 0 respectively. We denote by (x(t), m(t), v(t)) the blocks of z(t), and we define

(x (t), m (t), v (t)) similarly. Set u(t) := z(t) -z (t) 2 . Set also u x (t) := x(t) -x (t) 2 and define u m (t) and u v (t) similarly, hence, u(t) = u x (t) + u m (t) + u v (t).
Using the same derivations as above, we establish for all t ≥ 0 that:

um (t) ≤ aL ∇F u x (t) + a(L ∇F + 2)u m (t) . Similarly, uv (t) ≤ bL S u x (t) + b(L S + 2)u v (t). Moreover, ux (t) ≤ (ε -1 + ε -2 M C(t))u x (t) + ε -1 u m (t) + ε -2 M C(t)u v (t) where we set C(t) := ( v(t) + v (t)) -1
∞ . Putting all pieces together, we obtain that there exist nonnegative constants c 1 and c 2 , depending on K, s.t.

u(t) ≤ (c 1 + c 2 C(t))u(t) .
By Lem. 2.14, there exist two other nonnegative constants c 1 , c 2 depending on K, s.t. for all t > 0, u(t) ≤ (c 1 + c 2 max(1, t -1/2 ))u(t) . Using Grönwall's lemma, we obtain that for all t ≥ 0,

u(t) ≤ u(0) exp t 0 (c 1 + c 2 max(1, s -1/2 ))ds .
It is easy to show that the integral in the exponential is no larger than

2c 2 + (c 1 + c 2 )t.
This completes the proof.

We recall that a semiflow Φ on the space

(E, d) is a continuous map Φ from [0, +∞) × E to E defined by (t, x) → Φ(t, x) = Φ t (x) such that Φ 0 is the identity and Φ t+s = Φ t • Φ s for all (t, s) ∈ [0, +∞)
< b ≤ 4a. The map Z ∞ ∞ is single-valued on Z + → C([0, +∞), Z + ) i.e.
, there exists a unique global solution to (ODE ∞ ) starting from any given point in Z + . Moreover, the following map is a semiflow:

Φ : [0, +∞) × Z + → Z + (t, z) → Z ∞ ∞ (z)(t) (2.25)
Proof. The result is a direct consequence of Lem. 2.18. We first recall some useful definitions and results. Let Ψ represent any semiflow on an arbitrary metric space (Haraux, 1991, Th. 2.1.7). A continuous function z : [0, +∞) → E is said to be an asymptotic pseudotrajectory (APT, [START_REF] Benaïm | Asymptotic pseudotrajectories and chain recurrent flows, with applications[END_REF]) for the semiflow Ψ if for every

(E, d). A point z ∈ E is called an equilibrium point of the semiflow Ψ if Ψ t (z) = z for all t ≥ 0. We denote by Λ Ψ the set of equilibrium points of Ψ. A continuous function V : E → R is called a Lyapunov function for the semiflow Ψ if V(Ψ t (z)) ≤ V(z) for all z ∈ E and all t ≥ 0. It is called a strict Lyapunov function if, moreover, {z ∈ E : ∀t ≥ 0, V(Ψ t (z)) = V(z)} = Λ Ψ . If V is a strict Lyapunov function for Ψ and if z ∈ E is a point s.t. {Ψ t (z) : t ≥ 0} is relatively compact, then it holds that Λ Ψ = ∅ and d(Ψ t (z), Λ Ψ ) → 0, see
T ∈ (0, +∞), lim t→+∞ sup s∈[0,T ] d(z(t + s), Ψ s (z(t))) = 0 .
The following result follows from (Benaïm, 1999, Th. 5.7) and (Benaïm, 1999, Prop. 6.4).

Proposition 2.20 [START_REF] Benaïm | Dynamics of stochastic approximation algorithms[END_REF]). Consider a semiflow Ψ on (E, d) and a map z : [0, +∞) → E. Assume the following:

i) Ψ admits a strict Lyapunov function V. ii) The set Λ Ψ of equilibrium points of Ψ is compact. iii) V(Λ Ψ ) has an empty interior. iv) z is an APT of Ψ. v) z([0, ∞)) is relatively compact. Then, t≥0 z([t, ∞)) is a compact connected subset of Λ Ψ .
For every δ > 0 and every z = (x, m, v) ∈ Z + , define:

W δ (x, m, v) := V ∞ (x, m, v) -δ ∇F (x), m + δ S(x) -v 2 , (2.26)
where we recall that V ∞ (z) := lim t→∞ V (t, z) for every z ∈ Z + and V is defined by Eq.(2.9). Consider the set

E := h -1 ∞ ({0}) of all equilibrium points of (ODE ∞ ), namely: E = {(x, m, v) ∈ Z + : ∇F (x) = 0, m = 0, v = S(x)} . The set E is non-empty by Assumption 2.2.2. Proposition 2.21. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. Let K ⊂ Z + be a compact set. Define K := {Φ(t, z) : t ≥ 0, z ∈ K}. Let Φ : [0, +∞) × K → K be the restriction of the semiflow Φ to K i.e., Φ(t, z) = Φ(t, z) for all t ≥ 0, z ∈ K . Then, i) K is compact. ii) Φ is well defined and is a semiflow on K .
iii) The set of equilibrium points of Φ is equal to E ∩ K . iv) There exists δ > 0 s.t. W δ is a strict Lyapunov function for the semiflow Φ.

Proof. The first point is a consequence of Prop. 2.13. The second point stems from Prop. 2.19. The third point is immediate from the definition of E and the fact that Φ is valued in K . We now prove the last point. Consider z ∈ K and write Φ t (z) under the form Φ t (z) = (x(t), m(t), v(t)). For any map W :

Z + → R, define for all t > 0, L W (t) := lim sup s→0 s -1 (W(Φ t+s (z)) -W(Φ t (z))) . Introduce G(z) := -∇F (x), m
and H(z) := S(x)v 2 for every z = (x, m, v). Consider δ > 0 (to be specified later on). We study

L W δ = L V + δL G + δL H . Note that Φ t (z) ∈ K ∩ Z * + for all t > 0 by Lem. 2.10. Thus, t → V ∞ (Φ t (z)) is differentiable at any point t > 0 and the derivative coincides with L V (t) = V∞ (Φ t (z)). Define C 1 := sup{ v ∞ : (x, m, v) ∈ K }. Then, by Lem. 2.11, L V (t) ≤ -ε(ε + √ C 1 ) -2 m(t) 2 . Let L ∇F be the Lipschitz constant of ∇F on {x : (x, m, v) ∈ K }. For every t > 0, L G (t) ≤ lim sup s→0 s -1 ∇F (x(t)) -∇F (x(t + s)) m(t + s) -∇F (x(t)), ṁ(t) ≤ L ∇F ε -1 m(t) 2 -a ∇F (x(t)) 2 + a ∇F (x(t)), m(t) ≤ - a 2 ∇F (x(t)) 2 + a 2 + L ∇F ε m(t) 2 .
Denote by L S the Lipschitz constant of S on {x : (x, m, v) ∈ K }. For every t > 0,

L H (t) = lim sup s→0 s -1 ( S(x(t + s)) -S(x(t)) + S(x(t)) -v(t + s) 2 -S(x(t)) -v(t) 2 ) = -2 S(x(t)) -v(t), v(t) + lim sup s→0 2s -1 S(x(t + s)) -S(x(t)), S(x(t)) -v(t + s) ≤ -2b S(x(t)) -v(t) 2 + 2L S ε -1 m(t) S(x(t)) -v(t) . Using that 2 m(t) S(x(t)) -v(t) ≤ L S bε m(t) 2 + bε L S S(x(t)) -v(t) 2 , we obtain L H (t) ≤ -b S(x(t)) -v(t) 2 + L 2 S bε 2 m(t) 2 .
Hence, for every t > 0,

L W δ (t) ≤ -M (δ) m(t) 2 - aδ 2 ∇F (x(t)) 2 -δb S(x(t)) -v(t) 2 .
where

M (δ) := ε(ε + √ C 1 ) -2 - δL 2 S bε 2 -δ a 2 + L ∇F ε . Choosing δ s.t. M (δ) > 0, ∀t > 0, L W δ (t) ≤ -c m(t) 2 + ∇F (x(t)) 2 + S(x(t)) -v(t) 2 , (2.27)
where c := min{M (δ), aδ 2 , δb}. It can easily be seen that for every z ∈ K , t → W δ (Φ t (z)) is Lipschitz continuous, hence absolutely continuous. Its derivative almost everywhere coincides with L W δ , which is non-positive. Thus, W δ is a Lyapunov function for Φ. We prove that the Lyapunov function is strict.

Consider z ∈ K s.t. W δ (Φ t (z)) = W δ (z)
for all t > 0. The derivative almost everywhere of t → W δ (Φ t (z)) is identically zero, and by Eq. (2.27), this implies that

-c m t 2 + ∇F (x t ) 2 + S(x t ) -v t 2 is equal
to zero for every t a.e. (hence, for every t, by continuity of Φ). In particular for t = 0, m = ∇F (x) = 0 and S(x Assume that 0 < b ≤ 4a. Then, for every z 0 ∈ Z 0 , Z 0 ∞ (z 0 ) is an asymptotic pseudotrajectory of the semiflow Φ given by (2.25).

) -v = 0. Hence, z ∈ h -1 ∞ ({0}

Proof.

Consider z 0 ∈ Z 0 , T ∈ (0, +∞) and define z := Z 0 ∞ (z 0 ). Consider t ≥ 1. For every s ≥ 0, define ∆ t (s) := z(t + s) -Φ(z(t))(s) . The aim is to prove that sup s∈[0,T ] ∆ t (s) tends to zero as t → ∞. Putting together Prop. 2.12 and Lem. 2.14, the set

K := {z(t) : t ≥ 1} is a compact subset of Z * + . Define C(t) := sup s≥0 sup z ∈K h(t + s, z ) -h ∞ (z ) . It can be shown that lim t→∞ C(t) = 0. We ob- tain that for every s ∈ [0, T ], ∆ t (s) ≤ T C(t) + s 0 h ∞ (z(t + s )) -h ∞ (Φ(z(t))(s )) ds . By Lem. 2.14, K := z ∈Φ(K) z ([0, ∞)) is a compact subset of Z * + .
It is immediately seen from the definition that h ∞ is Lipschitz continuous on every compact subset of Z * + , hence on K ∪ K . Therefore, there exists a constant L, independent from t, s, s.t.

∆ t (s) ≤ T C(t)+ s 0 L∆ t (s )ds (∀t ≥ 1, ∀s ∈ [0, T ]) . Using Grönwall's lemma, it holds that for all s ∈ [0, T ], ∆ t (s) ≤ T C(t)e Ls . As a consequence, sup s∈[0,T ] ∆ t (s) ≤ T C(t)e LT
and the righthand side converges to zero as t → ∞.

End of the proof of Th. 2.2 By Prop. 2.12, the set

K := Z 0 ∞ (z 0 )([0, ∞)) is a compact subset of Z + . Define K := {Φ(t, z) : t ≥ 0, z ∈ K} ,
and let Φ : [0, +∞) × K → K be the restriction Φ to K . By Prop. 3.16, there exists δ > 0 s.t. W δ is a strict Lyapunov function for the semiflow Φ. Moreover, the set of equilibrium points coincides with E ∩ K . In particular, the equilibrium points of Φ form a compact set. By Prop. 2.23, Z 0 ∞ (z 0 ) is an APT of Φ. Note that every z ∈ E can be written under the form z = (x, 0, S(x)) for some x ∈ S. From the definition of W δ in (3.33)

, W δ (z) = W δ (x, 0, S(x)) = V ∞ (x, 0, S(x)) = F (x).
Since F (S) is assumed to have an empty interior, the same holds for

W δ (E ∩ K ). By Prop. 2.20, t≥0 Z 0 ∞ (z 0 )([t, ∞)) ⊂ E ∩ K .
The set in the righthand side coincides with the set of limits of convergent sequences of the form

Z 0 ∞ (z 0 )(t n ) for t n → ∞. As Z 0 ∞ (z 0 )([0, ∞)) is a bounded set, d(Z 0 ∞ (z 0 )(t), E) tends to zero.

Proof of Th. 2.3

The proof follows the path of (Haraux and Jendoubi, 2015, Th. 10.1.6, Th. 10.2.3), but requires specific adaptations to deal with the dynamical system at hand. Define for all δ > 0, t > 0, and z = (x, m, v),

Wδ (t, (x, m, v)) := V (t, (x, m, v)) -δ ∇F (x), m + δ S(x) -v 2 .
(2.28)

The function Wδ is the non-autonomous version of the function (3.33). Consider a fixed x 0 ∈ R d , and define w δ (t) := Wδ (t, z(t)) where z(t) = (x(t), m(t), v(t)) is the solution to (ODE) with initial condition (x 0 , 0, 0). The proof uses the following steps.

i) Upper-bound on w δ (t). From Eq. (2.9), we obtain that for every t ≥ 1,

V (t, z(t)) ≤ |F (x(t))| + m(t) 2 2aε(1-e -a ) . Using ∇F (x), m ≤ ( ∇F (x) 2 + m 2 )/2
, we obtain that there exists a constant c 1 (depending on δ) s.t. for every t ≥ 1,

w δ (t) ≤ c 1 |F (x(t))| + m(t) 2 + ∇F (x(t)) 2 + S(x(t)) -v(t) 2 .
(2.29)

ii) Upper-bound on d dt w δ (t). The function w δ is absolutely continuous on [1, +∞). Moreover, there exist δ > 0, c 2 > 0 (both depending on x 0 ) s.t. for every t ≥ 1 a.e.,

d dt w δ (t) ≤ -c 2 m(t) 2 + ∇F (x(t)) 2 + S(x(t)) -v(t) 2 .
(2.30)

The proof of Eq. ( 2.30) uses arguments that are similar to the ones used in the proof of Prop. 3.16 (just use Lem. 2.11 to bound the derivative of the first term in Eq. (2.28)).

For this reason, it is omitted.

iii) Positivity of w δ (t). By Lem. 2.11, the function t → V (t, z(t)) is decreasing. As it is lower bounded, := lim t→∞ V (t, z(t)) exists. By Th. 2.2, m(t) tends to zero, hence this limit coincides with lim t→∞ F (x(t)). Replacing F with F -, one can assume without loss of generality that = 0. By Eq. ( 2.30), w δ is non-increasing on [1, +∞), hence converging to some limit. Using again Th. 2.2, ∇F (x(t)), m(t) → 0 and S(x(t))v(t) → 0. Thus, lim t→∞ w δ (t) = = 0. Assume that there exists +∞). By Eq. ( 2.30), this implies that m(t) = 0 on this interval. Hence, dx(t)/dt = 0. This means that x(t) = x(t 0 ) for all t ≥ t 0 . By Th. 2.2, it follows that x(t 0 ) ∈ S. In that case, the final result is shown. Therefore, one can assume that w δ (t) > 0 for all t ≥ 1.

t 0 ≥ 1 s.t. w δ (t 0 ) = 0. Then, w δ is constant on [t 0 ,
iv) Putting together (2.29) and (2.30) using the Łojasiewicz condition. By Prop. 2.20 and 2.23, the set

L := s≥0 {z(t) : t ≥ s} is a compact connected subset of E = {(x, 0, S(x)) : ∇F (x) = 0}.
The set U := {x : (x, 0, S(x)) ∈ L} is a compact and connected subset of S. Using Assumption 2.3.1 and (Haraux and Jendoubi, 2015, Lem. 2.1.6), there exist constants σ, c > 0 and θ ∈ (0,

1 2 ], s.t. ∇F (x) ≥ c|F (x)| 1-θ for all x s.t. d(x, U) < σ . As d(x(t), U) → 0, there exists T ≥ 1 s.t. for all t ≥ T , ∇F (x(t)) ≥ c|F (x(t))| 1-θ .
Thus, we may replace the term ∇F (x(t)) 2 in the righthand side of Eq. (2.30) using -θ) . Upon noting that 2(1θ) ≥ 1, we thus obtain that there exists a constant c 3 and some T ≥ 1 s.t. for t ≥ T a.e.,

∇F (x(t)) 2 ≥ 1 2 ∇F (x(t)) 2 + 1 2 |F (x(t))| 2(1
d dt w δ (t) ≤ -c 3 m(t) 2 + ∇F (x(t)) 2 + |F (x(t))| + S(x(t)) -v(t) 2 2(1-θ) .
Putting this inequality together with Eq. ( 2.29), we obtain that for some constant c 4 > 0 and for all t ≥ T a.e.,

d dt w δ (t) ≤ -c 4 w δ (t) 2(1-θ) .
v) End of the proof. Following the arguments of (Haraux and Jendoubi, 2015, Th. 10.1.6), by integrating the preceding inequality, over [T , t], we obtain w δ (t) ≤ c 5 t -1 1-2θ for t ≥ T in the case where θ < 1 2 , whereas w δ (t) decays exponentially if θ = 1 2 . From now on, we focus on the case θ < 1 2 . By definition of (ODE), ẋ(t) 2 ≤ m(t) 2 /((1-e -aT ) 2 ε 2 ) for all t ≥ T . Since Eq. (2.30) implies m(t) 2 ≤ẇδ (t)/c 2 , we deduce that there exists c, c > 0 s.t. for all t ≥ T , and Jendoubi, 2015, Lem. 2.1.5), it follows that

2t t ẋ(s) 2 ds ≤ cw δ (t) ≤ c t -1 1-2θ .Applying (Haraux
∞ t ẋ(s) 2 ds ≤ ct -θ 1-2θ
for some other constant c. Therefore x * := lim t→+∞ x(t) exists by Cauchy's criterion and for all t ≥ T , x(t)x * ≤ ct -θ 1-2θ . Finally, since x(t) → a, we remark that, using the same arguments, the global Łojasiewicz exponent θ can be replaced by any Łojasiewicz exponent of f at x * . When θ = 1 2 , the proof follows the same line.

2.8 Proofs for Section 2.4

2.8.1 Proof of Th. 2.4

Given an initial point x 0 ∈ R d and a stepsize γ > 0, we consider the iterates z γ n given by (2.5) and z γ 0 := (x 0 , 0, 0). For every n ∈ N * and every z ∈ Z + , we define

H γ (n, z, ξ) := γ -1 (T γ, ᾱ(γ), β(γ) (n, z, ξ) -z) .
Thus,

z γ n = z γ n-1 + γH γ (n, z γ n-1 , ξ n ) for every n ∈ N * . For every n ∈ N * and every z ∈ Z of the form z = (x, m, v), we define e γ (n, z) := (x, (1 -ᾱ(γ) n ) -1 m, (1 -β(γ) n ) -1 v),
and set e γ (0, z) := z. Lemma 2.24. Let Assumptions 2.2.1, 2.2.4 and 2.4.2 hold true. There exists γ0 > 0 s.t. for every R > 0, there exists s > 0, sup

   E H γ (n + 1, z, ξ) 1+s : γ ∈ (0, γ0 ], n ∈ N, z ∈ Z + s.t. e γ (n, z) ≤ R    < ∞ .
(2.31)

Proof. Let R > 0. We denote by (H γ,x , H γ,m , H γ,v ) the block components of H γ . There exists a constant C s depending only on s s.t. H γ 1+s ≤ C s ( H γ,x 1+s + H γ,m 1+s + H γ,v 1+s
). Hence, it is sufficient to prove that Eq. ( 2.31) holds respectively when replacing H γ with each of

H γ,x , H γ,m , H γ,v . Consider z = (x, m, v) in Z + . We write: H γ,x (n + 1, z, ξ) ≤ ε -1 ( m 1-ᾱ(γ) n + ∇f (x, ξ) )
. Thus, for every z s.t. e γ (n, z) ≤ R, there exists a constant C depending only on ε, R and s s.t. H γ,x (n + 1, z, ξ) 1+s ≤ C(1 + ∇f (x, ξ) 1+s ). By Assumption 2.4.2, (2.31) holds for H γ,x instead of H γ . Similar arguments hold for H γ,m and H γ,v upon noting that the functions γ → (1-ᾱ(γ))/γ and γ → (1-β(γ))/γ are bounded under Assumption 2.2.4.

For every R > 0, and every arbitrary sequence z = (z n : n ∈ N) on Z + , we define τ R (z) := inf{n ∈ N : e γ (n, z n ) > R} with the convention that τ R (z) = +∞ when the set is empty. We define the map B R :

Z N + → Z N + given for any arbitrary sequence z = (z n : n ∈ N) on Z + by B R (z)(n) = z n 1 n<τ R (z) + z τ R (z) 1 n≥τ R (z) . We define the random sequence z γ,R := B R (z γ ). Recall that a family (X i : i ∈ I) of random variables on some Euclidean space is called uniformly integrable if lim A→+∞ sup i∈I E( X i 1 X i >A ) = 0.
Lemma 2.25. Let Assumptions 2.2.1, 2.2.4, 2.4.2 and 2.4.1 hold true. There exists γ0 > 0 s.t. for every R > 0, the family of r.v.

(γ -1 (z γ,R n+1 -z γ,R n ) : n ∈ N, γ ∈ (0, γ0 ]) is uniformly integrable. Proof. Let R > 0. As the event {n < τ R (z γ )} coincides with n k=0 { e γ (k, z γ k ) ≤ R}, it holds that for every n ∈ N, z γ,R n+1 -z γ,R n γ = z γ n+1 -z γ n γ 1 n<τ R (z γ ) = H γ (n + 1, z γ n , ξ n+1 ) n k=0 1 eγ (k,z γ k ) ≤R .
Choose γ0 > 0 and s > 0 as in Lem. 2.24. For every γ ≤ γ0 ,

E γ -1 (z γ,R n+1 -z γ,R n ) 1+s ≤ sup    E H γ ( + 1, z, ξ) 1+s : γ ∈ (0, γ0 ], ∈ N, z ∈ Z + , e γ ( , z) ≤ R    .
By Lem. 2.24, the righthand side is finite and does not depend on (n, γ).

For a fixed γ > 0, we define the interpolation map X γ : Z N → C([0, +∞), Z) as follows for every sequence z = (z n : n ∈ N) on Z:

X γ (z) : t → z t γ + (t/γ -t/γ )(z t γ +1 -z t γ
) .

For every γ, R > 0, we define

z γ,R := X γ (z γ,R ) = X γ • B R (z γ ). Namely, z γ,R is the interpolated process associated with the sequence (z γ,R n ). It is a random variable on C([0, +∞), Z). We recall that F n is the σ-algebra generated by the r.v. (ξ k : 1 ≤ k ≤ n).
For every γ, n, R, we use the notation: ∆ γ,R 0 := 0 and

∆ γ,R n+1 := γ -1 (z γ,R n+1 -z γ,R n ) -E(γ -1 (z γ,R n+1 -z γ,R n )|F n ) .
Lemma 2.26. Let Assumptions 2.2.1, 2.2.4, 2.4.2 and 2.4.1 hold true. There exists γ0 > 0 s.t. for every R > 0, the family of r.v.

(z γ,R : γ ∈ (0, γ0 ]) is tight. Moreover, for every δ > 0, P max 0≤n≤ T γ γ n k=0 ∆ γ,R k+1 > δ γ→0 ---→ 0 .
Proof. It is an immediate consequence of Lem. 2.25 and (Bianchi et al., 2019, Lem. 6.2).

The proof of the following lemma is omitted.

) = ē(t, z).
End of the Proof of Th. 2.4 Consider x 0 ∈ R d and set z 0 = (x 0 , 0, 0). Define

R 0 := sup ē(t, Z 0 ∞ (z 0 )(t)) : t > 0 . By Prop. 2.12, R 0 < +∞. We select an arbitrary R s.t. R ≥ R 0 + 1. For every n ≥ 0, z ∈ Z + , z γ,R n+1 = z γ,R n + γH γ (n + 1, z γ,R n , ξ n+1 )1 eγ (n,z γ,R n ) ≤R . Define for every n ≥ 1, z ∈ Z + , h γ,R (n, z) := h γ (n, z)1 eγ (n-1,z) ≤R . Then, ∆ γ,R n+1 = γ -1 (z γ,R n+1 -z γ,R n ) -h γ,R (n + 1, z γ,R n ) . Define also for every n ≥ 0, M γ,R n := n k=1 ∆ γ,R k = γ -1 (z γ,R n -z 0 ) -n-1 k=0 h γ,R (k + 1, z γ,R k )
. Consider t ≥ 0 and set n := t/γ . For any T > 0, it holds that :

sup t∈[0,T ] z γ,R (t) -z 0 - t 0 h γ,R ( s/γ + 1, z γ,R (γ s/γ ))ds ≤ max 0≤n≤ T /γ +1 γ M γ,R n .
By Lem. 2.26,

P   sup t∈[0,T ] z γ,R (t) -z 0 - t 0 h γ,R s/γ + 1, z γ,R (γ s/γ ) ds > δ   γ→0 ---→ 0 . (2.32)
As a second consequence of Lem. 2.26, the family of r.v. (z γ,R : 0 < γ ≤ γ0 ) is tight, where γ0 is chosen as in Lem. 2.26 (it does not depend on R). By Prokhorov's theorem, there exists a sequence

(γ k : k ∈ N) s.t. γ k → 0 and s.t. (z γ k ,R : k ∈ N) converges
in distribution to some probability measure ν on C([0, +∞), Z + ). By Skorohod's representation theorem, there exists a r.v. z on some probability space (Ω , F , P ), with distribution ν, and a sequence of r.v. (z (k) : k ∈ N) on that same probability space where for each k ∈ N, the r.v. z (k) has the same distribution as the r.v. z γ k ,R , and s.t. for every ω ∈ Ω , z (k) (ω) converges to z(ω) uniformly on compact sets. Now select a fixed T > 0. According to Eq. ( 2.32), the sequence

sup t∈[0,T ] z (k) (t) -z 0 - t 0 h γ k ,R s/γ k + 1, z (k) (γ k s/γ k ) ds ,
indexed by k ∈ N, converges in probability to zero as k → ∞. One can therefore extract a further subsequence z (ϕ k ) , s.t. the above sequence converges to zero almost surely. In particular, since z (k) (t) → z(t) for every t, we obtain that

z(t) = z 0 + lim k→∞ t 0 h γϕ k ,R s/γ ϕ k + 1, z (ϕ k ) (γ ϕ k s/γ ϕ k ) ds (∀t ∈ [0, T ]) . (2.33)
Consider ω ∈ Ω s.t. the r.v. z satisfies (2.33) at point ω. From now on, we consider that ω is fixed, and we handle z as an element of C([0, +∞), Z + ), and no longer as a random variable. Define

τ := inf{t ∈ [0, T ] : ē(t, z(t)) > R 0 + 1 2 }
if the latter set is non-empty, and τ := T otherwise. Since z(0) = z 0 and z 0 < R 0 , it holds that τ > 0 using the continuity of z. Choose any (s, t)

s.t. 0 < s < t < τ . Note that z (k) (γ k s/γ k ) → z(s) and γ k ( s/γ k + 1) → s. Thus, by Lem. 2.27, h γ k s/γ k + 1, z (k) (γ k s/γ k ) converges to h(s, z(s)) and e γ k ( s/γ k , z (k) (γ k s/γ k )) converges to ē(s, z(z)) . Since s < τ , ē(s, z(z)) ≤ R 0 + 1 2 . As R ≥ R 0 + 1, there exists a certain K(s) s.t. for every k ≥ K(s), 1 eγ k ( s/γ k ,z (k) (γ k s/γ k )) ≤R = 1 . As a consequence, h γ k ,R ( s/γ k + 1, z (k) (γ k s/γ k )) converges to h(s, z(s)) as k → ∞. Using Lebesgue's dominated convergence theorem, we obtain, for all t ∈ [0, τ ]: z(t) = z 0 + t 0 h s, z(s)) ds . Therefore z(t) = Z 0 ∞ (x 0 )(t) for every t ∈ [0, τ ].
In particular, z(τ ) ≤ R 0 and this means that τ = T . Thus, z(t) = Z 0 ∞ (x 0 )(t) for every t ∈ [0, T ] (and consequently for every t ≥ 0). We have shown that for every R ≥ R 0 + 1, the sequence of r.v. (z γ,R : γ ∈ (0, γ0 ]) is tight and converges in probability to Z 0 ∞ (z 0 ) as γ → 0. Therefore, for every T > 0,

∀δ > 0, lim γ→0 P   sup t∈[0,T ] z γ,R (t) -Z 0 ∞ (x 0 )(t) > δ   = 0 . (2.34)
In order to complete the proof, we show that

P sup t∈[0,T ] z γ,R (t) -z γ (t) > δ → 0 as γ → 0, for all δ > 0.
where we recall that

z γ = X γ (z γ ). Note that z γ,R (t) ≤ z γ,R (t) -Z 0 ∞ (z 0 )(t)
+ R 0 by the triangular inequality. Therefore, for every T, δ > 0,

P   sup t∈[0,T ] z γ,R (t) -z γ (t) > δ   ≤ P   sup t∈[0,T ] z γ,R (t) ≥ R   ≤ P   sup t∈[0,T ] z γ,R (t) -Z 0 ∞ (z 0 )(t) ≥ R -R 0   .
By Eq. ( 2.34), the RHS of the above inequality tends to zero as γ → 0. The proof is complete.

Proof of Th. 2.5

We start by stating a general result. Consider a Euclidean space X equipped with its Borel σ-field X . Let γ0 > 0, and consider two families (P γ,n : 0 < γ < γ0 , n ∈ N * ) and ( Pγ : 0 < γ < γ0 ) of Markov transition kernels on X. Denote by P(X) the set of probability measures on X. Let X = (X n : n ∈ N) be the canonical process on X. Let (P γ,ν : 0 < γ < γ0 , ν ∈ P(X)) and ( Pγ,ν : 0 < γ < γ0 , ν ∈ P(X)) be two families of measures on the canonical space (X N , X ⊗N ) such that the following holds:

• Under P γ,ν , X is a non-homogeneous Markov chain with transition kernels (P γ,n :

n ∈ N * ) and initial distribution ν, that is, for each n ∈ N * , P γ,ν (X n ∈ dx|X n-1 ) = P γ,n (X n-1 , dx) .
• Under Pγ,ν , X is an homogeneous Markov chain with transition kernel Pγ and initial distribution ν.

In the sequel, we will use the notation P γ,x as a shorthand notation for P γ,δx where δ x is the Dirac measure at some point x ∈ X. Finally, let Ψ be a semiflow on X. A Markov kernel P is Feller if P f is continuous for every bounded continuous f .

Assumption 2.8.1. Let ν ∈ P(X). i) For every γ, Pγ is Feller. ii) (P γ,ν X -1 n : n ∈ N, 0 < γ < γ0
) is a tight family of measures. iii) For every γ ∈ (0, γ0 ) and every bounded Lipschitz continuous function f : X → R, P γ,n f converges to Pγ f as n → ∞, uniformly on compact sets. iv) For every δ > 0, for every compact set K ⊂ X, for every t > 0, lim γ→0 sup x∈K P γ,x X t/γ -Ψ t (x) > δ = 0 .

Let BC Ψ be the Birkhof center of Ψ i.e., the closure of the set of recurrent points.

Theorem 2.28. Consider ν ∈ P(X) s.t. Assumption 2.8.1 holds true. Then, for every δ > 0, lim γ→0 lim sup n→∞

1 n+1 n k=0 P γ,ν d(X k , BC Ψ ) > δ = 0 . Proof. For every γ, n, define µ γ,n := νP γ,1 • • • P γ,n with the convention that µ γ,0 = ν. Otherwise stated, µ γ,n = P γ,ν X -1 n . Define Π γ,n := 1 n+1 n k=0 µ γ,k
for every n ∈ N. Assumption 2.8.1 implies that for any fixed γ, (Π γ,n : n ∈ N) is tight. By Prokhorov's theorem, it admits a cluster point π γ . For such a cluster point, consider a subsequence ϕ n s.t. Π γ,ϕn ⇒ π γ , where ⇒ stands for the weak convergence of probability measures. Consider a bounded Lipschitz continuous function f :

X → R. It holds that Π γ,n (f ) and Π γ,n ( Pγ f ) respectively converge to π γ (f ) and π γ ( Pγ f ) along the subsequence, because Pγ is Feller. We observe that Π γ,n Pγ f -Π γ,n f ≤ 1 n + 1 n k=0 |µ γ,k ( Pγ f -P γ,k+1 f )| + 2 f ∞ n + 1 . Choose δ > 0 and a compact set K ⊂ X s.t. sup k µ γ,k (K c ) < δ. For every k, |µ γ,k ( Pγ f - P γ,k+1 f )| ≤ sup x∈K | Pγ f (x) -P γ,k+1 f (x)| + 2 f ∞ δ. By Assumption 2.8.1iii), it holds that lim sup n Π γ,n Pγ f -Π γ,n f ≤ 2 f ∞ δ. As δ is arbitrary, Π γ,n Pγ f -Π γ,n f → 0, which shows that π γ Pγ f -π γ f = 0.
We have shown that every cluster point of (Π γ,n : n ∈ N) is an invariant measure of Pγ .

Consider an arbitrary sequence γ j ↓ 0 as j → ∞, and let π j be an invariant measure of Pγ j for every j. It is not difficult to show that the sequence (π j ) is also tight, hence converging to some π * as j → ∞, along some subsequence. We now prove that such a cluster point π * is an invariant measure for the semiflow Ψ i.e., π * Ψ -1 t = π * for every t > 0. Such a proof can be found for instance in [START_REF] Fort | Asymptotic behavior of a Markovian stochastic algorithm with constant step[END_REF], we reproduce it here for completeness. Denote by Ēγ,ν the expectation associated with Pγ,ν and by L the Lipschitz constant of f . For an arbitrary δ > 0, consider a compact set K s.t.

sup j π j (K c ) < δ. For every j and every t > 0, using that π j = π j P t γ j γ j , we obtain, by following the same approach as [START_REF] Fort | Asymptotic behavior of a Markovian stochastic algorithm with constant step[END_REF],

f • Ψ t dπ j -f dπ j = E γ j ,π j (f (Ψ t (X 0 )) -f (X t γ j )) ≤ E γ j ,π j |f (Ψ t (X 0 )) -f (X t γ j )|1 K (X 0 ) + 2 f ∞ δ ≤ E γ j ,π j   2 f ∞ ∧ L Ψ t (X 0 ) -X t γ j 1 K (X 0 )   + 2 f ∞ δ ≤ E γ j ,π j 2 f ∞ 1 K (X 0 )1 Ψt(X 0 )-X t γ j >δ + Lδ + 2 f ∞ δ ≤ 2 f ∞ sup x∈K P γ j ,x Ψ t (x) -X t γ j > δ + Lδ + 2 f ∞ δ . Thus, lim sup j f • Ψ t dπ j -f dπ j ≤ (L + 2 f ∞ )δ
, and since δ is arbitrary, the lim sup is equal to zero. Considering the limit along the converging subsequence, it follows that f

• Ψ t dπ * -f dπ * = 0. Hence, π * is invariant for Ψ. By Poincaré's recurrence theorem, π * (BC Ψ ) = 1.
We now conclude the proof of Th. 2.28. For every δ > 0, set A δ := {x : d(x, BC Ψ ) ≥ δ} By contradiction, assume that there exists δ > 0, a sequence γ j ↓ 0, and, for every j, a sequence (ϕ

j n : n ∈ N) s.t. for every n, Π γ j ,ϕ j n (A δ ) > δ. For every j, as (Π γ j ,ϕ j n : n ∈ N) is tight, one can extract a subsequence (Π γ j , φj n : n ∈ N)
converging weakly to some measure π j which is invariant for Pγ j . By the portmanteau theorem, π j (A δ ) > δ. As (π j ) is tight, it converges weakly along some subsequence to some π * satisfying π * (BC Ψ ) = 1. As π * (A δ ) > δ, this leads to a contradiction.

End of the Proof of Th. 2.5. We apply Th. 2.28 in the case where P γ,n is the kernel of the non-homogeneous Markov chain (z γ n ) defined by (2.5) and Pγ is the kernel of the homogeneous Markov chain (z γ n ) given by zγ

n = zγ n-1 + γH γ (∞, zγ n-1 , ξ n ) for every n ∈ N * and z0 ∈ Z + where H γ (∞, zγ n-1 , ξ n ) := lim k→∞ H γ (k, zγ n-1 , ξ n ).
The task is merely to verify Assumption 2.8.1iii), the other assumptions being easily verifiable using Th. 2.4, Consider γ ∈ (0, γ0 ). Let f : Z → R be a bounded M -Lipschitz continuous function and K a compact. For all z = (x, m, v) ∈ K:

|P γ,n (f )(z) -Pγ (f )(z)| ≤ M γE (1 -α n ) -1 mξ ε + (1 -β n ) -1 2 ṽ1/2 ξ - mξ ε + ṽ1/2 ξ ≤ M γα n ε(1-α n ) sup x,m α||m|| + (1 -α)E||∇f (x, ξ)|| + M γE|| mξ ṽ1/2 ξ || ε 2 1 - 1 (1-β n ) 1/2
where we write α = ᾱ(γ), β = β(γ), mξ := αm + (1α)∇f (x, ξ) and ṽξ := βv + (1β)∇f (x, ξ) 2 . Thus, condition 2.8.1iii) follows. Finally, Cor. 2.22 implies BC Φ = E.

Proofs for Section 2.5

In this section, we denote by

E n = E(•|F n ) the conditional expectation w.r.t. F n . We also use the notation ∇f n+1 := ∇f (x n , ξ n+1 ).
The following lemma will be useful in the proofs.

Lemma 2.29. Let the sequence (r n ) be defined as in Algorithm 2.2. Assume that 0 ≤ α n ≤ 1 for all n and that (1

-α n )/γ n → a > 0 as n → +∞. Then, i) ∀n ∈ N, r n = 1 -n i=1 α i , ii)
The sequence (r n ) is nondecreasing and converges to 1. iii) Under Assumption 2.5.4 i), for every > 0, for sufficiently large n, we have

r n -1 ≤ e - aγ 0 2(1-κ) n 1-κ if κ ∈ (0, 1) and r n -1 ≤ n -aγ 0 /(1+ ) if κ = 1.
A similar lemma holds for the sequence (r n ).

Proof. i) stems from observing that r n+1 -1 = α n+1 (r n -1) for every n ∈ N and iterating this relation (r 0 = 0). As a consequence, the sequence (r n ) is nondecreasing. We can write :

0 ≤ 1 -r n ≤ exp(-n i=1 (1 -α i )) . iii) As n≥1 γ n = +∞ and (1 -α n ) ∼ aγ n , we deduce that n i=1 (1 -α i ) ∼ n i=1 aγ i .
The results follow from the fact that n i=1 γ i ∼ γ 0 1-κ n 1-κ when κ ∈ (0, 1) and n i=1 γ i ∼ γ 0 ln n for κ = 1.

Proof of Th. 2.6

We define zn = (x n-1 , m n , v n ) (note the shift in the index of the variable x). We have

zn+1 = zn + γ n+1 h ∞ (z n ) + γ n+1 χ n+1 + γ n+1 ς n+1 ,
where h ∞ is defined in Eq. ( 2.18) and where we set

χ n+1 = 0, γ -1 n+1 (1 -α n+1 )(∇f n+1 -∇F (x n )), γ -1 n+1 (1 -β n+1 )(∇f 2 n+1 -S(x n ))
and

ς n+1 = (ς x n+1 , ς m n+1 , ς v n+1
) with the components defined by: ς

x n+1 = mn ε+ √ vn - γn γ n+1 mn ε+ √ vn , ς m n+1 = 1-α n+1 γ n+1 -a (∇F (x n ) -m n ) + a(∇F (x n ) -∇F (x n-1 )) and ς v n+1 = 1-β n+1 γ n+1 -b (S(x n ) -v n ) + b(S(x n ) -S(x n-1 ))
. Therefore, our algorithm has two perturbations: the first one is a martingale increment χ n+1 , while the second one is a negligible perturbation ς n+1 converging a.s. to zero. We first prove that ς n → 0 a.s. Using the triangular inequality,

ς x n ≤ m n ε + √ v n - m n r1/2 n ε + √ v n + 1 - γ n r -1 n γ n+1 r-1/2 n m n r1/2 n ε + √ v n ≤ ε -1 |1 -r-1/2 n | m n + ε -1 r-1/2 n - γ n r -1 n γ n+1 m n ,
which converges a.s. to zero because of the boundedness of (z n ) combined with Assumption 2.5.1 and Lem. 2.29 for (r n ). The components ς m n+1 and ς v n+1 converge a.s. to zero, as products of a bounded term and a term converging to zero. Indeed, note that ∇F and S are locally Lipschitz continuous under Assumption 2.2.1. Hence, there

exists a constant C s.t. ∇F (x n ) -∇F (x n-1 ) ≤ C x n -x n-1 ≤ C ε γ n m n .
The same inequality holds when replacing ∇F by S. Now consider the martingale increment sequence (χ n ), adapted to F n . Estimating the second order moments, it is easy to show using Assumption 2.4.2 i) that there exists a constant C s.

t. E n ( χ n+1 2 ) ≤ C . Using that k γ 2 k < ∞, it follows that n E n ( γ n+1 χ n+1
2 ) < ∞ a.s. By Doob's convergence theorem, lim n→∞ k≤n γ k χ k exists almost surely. Using this result along with the fact that ς n converges a.s. to zero, it follows from usual stochastic approximation arguments (Benaïm, 1999, Remark. 4.5) that the interpolated process z : [0, +∞) → Z + given by

z(t) = zn + (t -τ n ) zn+1 -zn γ n+1 ∀n ∈ N , ∀t ∈ [τ n , τ n+1 )
(where τ n = n k=0 γ k ), is almost surely a bounded APT of the semiflow Φ defined by (ODE ∞ ). The proof is concluded by applying Prop. 2.20 and Prop. 3.16.

Proof of Th. 2.7

As inf F > -∞, one can assume without loss of generality that F ≥ 0. In the sequel, C denotes some positive constant which may change from line to line. We define

a n := (1 -α n+1 )/γ n and P n := 1 2anrn m 2 n , 1 ε+ √ vn
. We have a n → a and r n → 1. By Assumption 2.5.2-i),

F (x n ) ≤ F (x n-1 ) -γ n ∇F (x n ), mn ε + √ vn + Cγ 2 n P n . (2.35)
We set u n := 1 -a n+1 an and

D n := r -1 n ε+ √ vn
, so that P n = 1 2an D n , m 2 n . We can write:

P n+1 -P n = u n P n+1 + D n+1 -D n 2a n , m 2 n+1 + D n 2a n , m 2 n+1 -m 2 n .
(2.36)

We estimate the vector

D n+1 -D n . Using that (r -1 n ) is non-increasing, D n+1 -D n ≤ r -1 n √ vn -vn+1 (ε + vn+1 ) (ε + √ vn ) .
Remarking that v n+1 ≥ β n+1 v n , recalling that (r n ) is nondecreasing and using the update rules of v n and rn , we obtain after some algebra

vn -vn+1 = r-1 2 n+1 (1 -β n+1 ) v n -∇f 2 n+1 √ v n + √ v n+1 + rn+1 -rn √ rn ( √ rn + √ rn+1 ) v n rn+1 ≤ c n+1 vn+1 where c n+1 := 1 -β n+1 β n+1   1 1 + β n+1 + 1 -rn 2r n   .
(2.37)

It is easy to see that c n /γ n → b/2. Thus, for any δ > 0, c n+1 ≤ (b + 2δ)γ n /2 for all n large enough. Using also that vn+1 /(ε + vn+1 ) ≤ 1, we obtain that D n+1 -D n ≤ b+2δ 2 γ n D n .
Substituting this inequality in Eq. ( 3.43), we get

P n+1 -P n ≤ u n P n+1 + γ n b + 2δ 4a n D n , m 2 n+1 + D n 2a n , m 2 n+1 -m 2 n . Using m 2 n+1 -m 2 n = 2m n (m n+1 -m n ) + (m n+1 -m n ) 2 , and noting that E n (m n+1 -m n ) = a n γ n (∇F (x n ) -m n ), E n D n 2a n , m 2 n+1 -m 2 n = γ n ∇F (x n ), mn ε + √ vn -2a n γ n P n + D n 2a n , E n [(m n+1 -m n ) 2 ]
As a n → a, we have a n -b+2δ 4 ≥ a -b+δ 4 for all n large enough. Hence,

E n P n+1 -P n ≤ u n P n+1 -2(a - b + δ 4 )γ n P n + γ n ∇F (x n ), mn ε + √ vn + γ 2 n b + 2δ 2 ∇F (x n ), mn ε + √ vn + C D n 2a n , E n [(m n+1 -m n ) 2 ] .
Using the Cauchy-Schwartz inequality and Assumption 2.5.2 ii), it is easy to show the inequality ∇F (x n ), mn

ε+ √ vn ≤ C(1 + F (x n ) + P n ). Moreover, using the componentwise inequality (∇f n+1 -m n ) 2 ≤ 2∇f 2 n+1 + 2m 2 n along with Assumption 2.5.2 ii), we obtain Dn 2an , E n [(m n+1 -m n ) 2 ] ≤ 2(1 -α n+1 ) 2 Dn 2an , E n [∇f 2 n+1 ] + m 2 n ≤ Cγ 2 n (1 + F (x n ) + P n ) .
Putting all pieces together with Eq. (3.49),

E n (F (x n ) + P n+1 ) ≤ F (x n-1 ) + P n + u n P n+1 -2(a -b+δ 4 )γ n P n + Cγ 2 n (1 + F (x n ) + P n ) . (2.38) Define V n := (1 -Cγ 2 n-1 )F (x n-1 ) + (1 -u n-1 )P n
where the constant C is fixed so that Eq. ( 2.38) holds. Then,

E n (V n+1 ) ≤ V n -2a - b + δ 2 - u n-1 γ n γ n P n + Cγ 2 n (1 + P n ) + Cγ 2 n-1 F (x n-1 ) . (2.39)
By Assumption 2.5.2, lim sup n u n-1 /γ n < 2ab/2 and for δ small enough, we obtain

E n (V n+1 ) ≤ V n + Cγ 2 n (1 + P n ) + Cγ 2 n-1 F (x n-1 ) ≤ (1 + C γ 2 n )V n + Cγ 2 n .
By the Robbins-Siegmund's theorem [START_REF] Robbins | A convergence theorem for non negative almost supermartingales and some applications[END_REF], the sequence (V n ) converges almost surely to a finite random variable V ∞ ∈ R + . In turn, the coercivity of F implies that (x n ) is almost surely bounded. The Robbins-Siegmund's theorem is sometimes used to prove at the same time the stability of an algorithm and its convergence. Here, we only use it to establish our stability result. In particular, we do not exploit the "repelling" term in Eq. ( 2.39) which we upperbound by zero. We now establish the almost sure boundedness of (m n ). Consider the martingale difference sequence ∆ n+1 := ∇f n+1 -∇F (x n ). We decompose m n = mn + mn where mn+1 = α n+1 mn + (1α n+1 )∇F (x n ) and mn+1 = α n+1 mn + (1α n+1 )∆ n+1 , setting m0 = m0 = 0. We prove that both terms mn and mn are bounded. Consider the first term:

mn+1 ≤ α n+1 mn + (1 -α n+1 ) sup k ∇F (x k )
. By continuity of ∇F , the supremum in the above inequality is almost surely finite. Thus, for every n, the ratio mn / sup k ∇F (x k ) is upperbounded by the bounded sequence r n . Hence, ( mn ) is bounded w.p.1. Consider now the term mn :

E n ( mn+1 2 ) = α 2 n+1 mn 2 + (1 -α n+1 ) 2 E n ( ∆ n+1 2 ) ≤ (1 + (1 -α n+1 ) 2 ) mn 2 + (1 -α n+1 ) 2 C ,
where C is a finite random variable (independent of n)

s.t. E n ( ∇f n+1 2 ) ≤ C by Assumption 2.4.2 i). Here, we used α 2 n+1 ≤ (1 + (1 -α n+1 ) 2 ) and the inequality E n ( ∆ n+1 2 ) ≤ E n ( ∇f n+1 2 ). By Assumption 2.5.1, n (1 -α n+1 ) 2 < ∞.
By the Robbins-Siegmund theorem, it follows that sup n mn 2 < ∞ w.p.1. Finally, it can be shown that (v n ) is almost surely bounded using the same arguments.

Proof of Th. 2.8

We use (Pelletier, 1998, Th. 1). All the assumptions in the latter can be verified in our case, at the exception of a positive definiteness condition on the limiting covariance matrix, which corresponds, in our case, to the matrix Q given by Eq. (2.16). As Q is not positive definite, it is strictly speaking not possible to just cite and apply (Pelletier, 1998, Th. 1). Nevertheless, a detailed inspection of the proofs of [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] shows that only a minor adaptation is needed in order to cover the present case. Therefore, proving the convergence result of Pelletier (1998) from scratch is worthless. It is sufficient to verify the assumptions of (Pelletier, 1998, Th. 1) (except the definiteness of Q) and then to point out the specific part of the proof of [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] which requires some adaptation.

Let

z n = (x n , m n , v n ) be the output of Algorithm 2.2. Define z * = (x * , 0, S(x * )). Define η n+1 := (0, a(∇f n+1 -∇F (x n )), b(∇f 2 n+1 -S(x n ))).
We have

z n+1 = z n + γ n+1 h ∞ (z n ) + γ n+1 η n+1 + γ n+1 n+1 , (2.40)
where n+1 := ( 1 n+1 , 2 n+1 , 3 n+1 ), whose components are given by

1 n+1 = mn ε+ √ vn -mn+1 ε+ √ vn+1 ; 2 n+1 = 1-αn+1 γn+1 -a ∇f n+1 -m n ; 3 n+1 = 1-βn+1 γn+1 -b ∇f 2 n+1 -v n .
Here, η n+1 is a martingale increment noise and n+1 = ( 1 n+1 , 2 n+1 , 3 n+1 ) is a remainder term. The aim is to check the assumptions (A1.1) to (A1.3) of [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF], where the role of the quantities (h, ε n , r n , σ n , α, ρ, β) in [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] is respectively played by the quantities (h ∞ , η n , n , γ n , κ, 1, 1) of the present chapter.

Let us first consider Assumption (A1.1) for h ∞ . By construction, h ∞ (z * ) = 0. By Assumptions 2.5.3 and 2.2.3, h ∞ is continuously differentiable in the neighborhood of z * and its Jacobian at z * coincides with the matrix H given by Eq. (2.14). As already discussed, after some algebra, it can be shown that the largest real part of the eigenvalues of H coincides with -L where L > 0 is given by Eq. (3.9). Hence, Assumption (A1.1) of [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF] is satisfied for h ∞ . Assumption (A1.3) is trivially satisfied using Assumption 2.5.4. The crux is therefore to verify Assumption (A1.2).

Clearly, E(η n+1 |F n ) = 0. Using Assumption 2.4.2ii), it follows from straightforward manipulations based on Jensen's inequality that for any M > 0, there exists δ > 0 s.t.

sup n≥0 E n η n+1 2+δ 1 { zn-z * ≤M } < ∞ . Next, we verify the condition lim n→∞ E γ -1 n+1 n+1 2 1 { zn-z * ≤M } = 0 . (2.41) It is sufficient to verify the latter for i n (i = 1, 2, 3) in place of n . The map (m, v) → m/(ε + √ v) is Lipschitz continuous in a neighborhood of (0, S(x * )) by Assumption 2.2.3. Thus, for M small enough, there exists a constant C s.t. if z n -z * ≤ M , then 1 n+1 ≤ C r -1 n+1 m n+1 -m n + C r-1 n+1 v n+1 -v n .
Using the triangular inequality and the fact that r n+1 , rn+1 are bounded sequences away from zero, there exists another constant C s.t.

1 n+1 ≤ C m n+1 -m n + C v n+1 -v n + C|r n+1 -1| + C|r n+1 -1| .
Using Lem. 2.29 under Assumption 2.5.4 (note that γ 0 > 1/2L ≥ 1/a when κ = 1), we obtain that the sequence

|r n -1|/γ n is bounded, thus |r n+1 -1| ≤ Cγ n+1 . The sequence (1 -α n )/γ n being also bounded, it holds that m n+1 -m n 2 1 { zn-z * ≤M } ≤ Cγ 2 n+1 (1 + ∇f n+1 2 )1 { zn-z * ≤M } . By Assumption 2.4.2 ii), E n ( ∇f n+1 2 |) is bounded by a deterministic constant on { z n -z * ≤ M }. Thus, E n ( m n+1 -m n 2 1 { zn-z * ≤M } ) ≤ Cγ 2 n+1 . A similar result holds for v n+1 -v n 2 .
We have thus shown that E n

1 n+1 2 1 { zn-z * ≤M } ≤ Cγ 2
n+1 . Hence, Eq. (2.41) is proved for 1 n+1 in place of n+1 . Under Assumption 2.5.4, the proof uses the same kind of arguments for 2 n+1 , 3 n+1 and is omitted. Finally, Eq. (2.41) is proved. Continuing the verification of Assumption (A1.2), we establish that

E n (η n+1 η T n+1 ) → Q a.s. on {z n → z * } . (2.42)
Denote by Q(x) the matrix given by the righthand side of Eq. (2.16) when x * is replaced by an arbitrary (2.42). Therefore, Assumption (A1.2) is fulfilled, except for the point mentioned at the beginning of this section : Pelletier (1998) puts the additional condition that the limit matrix in Eq. (2.42) is positive definite. This condition is not satisfied in our case, but the proof can still be adapted. The specific part of the proof where the positive definiteness comes into play is Th. 7 in [START_REF] Pelletier | Weak convergence rates for stochastic approximation with application to multiple targets and simulated annealing[END_REF]. The proof of (Pelletier, 1998, Th. 1) can therefore be adapted to the case of a positive semidefinite matrix. In the proof of (Pelletier, 1998, Th. 7), we only substitute the inverse of the square root of Q by the Moore-Penrose inverse. Finally, the uniqueness of the stationary distribution µ and its expression follow from (Karatzas and Shreve, 1991, Th. 6.7, p. 357).

x ∈ V. It is easily checked that E n (η n+1 η T n+1 ) = Q(x n ) and by continuity, Q(x n ) → Q a.s. on {z n → z * }, which proves
Proof of Eq. (3.10). We introduce the d × d blocks of the 3d × 3d matrix Σ = Σ i,j i,j=1,2,3 where Σ i,j is d×d. We denote by Σ the 2d×2d submatrix Σ := Σ i,j i,j=1,2 . By Th. 2.8, we have the subsystem:

H Σ + Σ HT = 0 0 0 -a 2 Q where H := ζI d -D a∇ 2 F (x * ) (ζ -a)I d (2.43)
and where Q := Cov ∇f (x * , ξ) . The next step is to triangularize the matrix H in order to decouple the blocks of Σ. For every k = 1, . . . , d, set ν ± k := -a 2 ± a 2 /4aλ k with the convention that √ -1 = ı (inspecting the characteristic polynomial of H, these are the eigenvalues of H).

Set M ± := diag (ν ± 1 , • • • , ν ± d ) and R ± := D -1/2 P M ± P T D -1/2 . Using the identities M + + M -= -aI d and M + M -= aΛ where Λ := diag (λ 1 , • • • , λ d ), it can be checked that R H = DR + + ζI d -D 0 R -D + ζI d R, where R := I d 0 R + I d .
Set Σ := R ΣR T . Denote by ( Σi,j ) i,j=1,2 the blocks of Σ. Note that Σ1,1 = Σ 1,1 . By left/right multiplication of Eq. ( 3.55) respectively with R and R T , we obtain

(DR + + ζI d )Σ 1,1 + Σ 1,1 (R + D + ζI d ) = Σ1,2 D + D ΣT 1,2
(2.44)

(DR + + ζI d ) Σ1,2 + Σ1,2 (DR -+ ζI d ) = D Σ2,2 (2.45) (R -D + ζI d ) Σ2,2 + Σ2,2 (DR -+ ζI d ) = -a 2 Q (2.46) Set Σ2,2 = P -1 D 1/2 Σ2,2 D 1/2 P . Define C := P -1 D 1/2 QD 1/2 P . Eq. (3.58) yields (M -+ ζI d ) Σ2,2 + Σ2,2 (M -+ ζI d ) = -a 2 C. Set Σ1,2 = P -1 D -1/2
Σ1,2 D 1/2 P . Eq. (3.57) rewrites

(M + + ζI d ) Σ1,2 + Σ1,2 (M -+ ζI d ) = Σ2,2 . We obtain that Σk, 1,2 = (ν + k + ν -+ 2ζ) -1 Σk, 2,2 = -a 2 C k, (ν + k +ν -+2ζ)(ν - k +ν -+2ζ) . Set Σ1,1 = P -1 D -1/2 Σ 1,1 D -1/2 P . Eq. (3.56) becomes (M + + ζI d ) Σ1,1 + Σ1,1 (M + + ζI d ) = Σ1,2 + ΣT 1,2 . Thus, Σk, 1,1 = Σk, 1,2 + Σ ,k 1,2 ν + k +ν + +2ζ = -a 2 C k, (ν + k +ν + +2ζ)(ν - k +ν -+2ζ) 1 ν + k +ν -+2ζ + 1 ν - k +ν + +2ζ = C k, (1 -2ζ a )(λ k + λ -2ζ + 2 a ζ 2 ) + 1 2(a-2ζ) (λ k -λ ) 2
, and the result is proved.

Stochastic Optimization with Momentum: Convergence, Fluctuations, and Traps Avoidance

Abstract In this chapter, a general stochastic optimization procedure is studied, unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm (S-NAG), and the widely used Adam algorithm. The algorithm is seen as a noisy Euler discretization of a non-autonomous ordinary differential equation, recently introduced by Belotto da Silva and Gazeau, which is analyzed in depth. Assuming that the objective function is non-convex and differentiable, the stability and the almost sure convergence of the iterates to the set of critical points are established. A noteworthy special case is the convergence proof of S-NAG in a non-convex setting. Under some assumptions, the convergence rate is provided under the form of a Central Limit Theorem. Finally, the non-convergence of the algorithm to undesired critical points, such as local maxima or saddle points, is established. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest.

Introduction

Given a probability space Ξ, an integer d > 0, and a function f : R d × Ξ → R, consider the problem of finding a local minimum of the function

F (x) := E ξ [f (x, ξ)] w.r.t. x ∈ R d ,
where E ξ represents the expectation w.r.t. the random variable ξ on Ξ. The chapter focuses on the case where F is possibly non-convex. It is assumed that the function F is unknown to the observer, either because the distribution of ξ is unknown, or because the expectaction cannot be evaluated. Instead, a sequence (ξ n : n ≥ 1) of i.i.d. copies of the random variable ξ is revealed online.

While the Stochastic Gradient Descent is the most classical algorithm that is used to solve such a problem, recently, several other algorithms became very popular. These include the Stochastic Heavy Ball (SHB), the stochastic version of Nesterov's Accelerated Gradient method (S-NAG) and the large class of the so-called adaptive gradient algorithms, among which Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is perhaps the most used in practice. As opposed to the vanilla Stochastic Gradient Descent, the study of such algorithms is more elaborate, for three reasons. First, the update of the iterates involves a so-called momentum term, or inertia, which has the effect of "smoothing" the increment between two consecutive iterates. Second, the update equation at the time index n is likely to depend on n, making these systems inherently non-autonomous. Third, as far as adaptive algorithms are concerned, the update also depends on some additional variable (a.k.a. the learning rate) computed online as a function of the history of the computed gradients.

In this work, we study in a unified way the asymptotic behavior of these algorithms in the situation where F is a differentiable function which is not necessarily convex, and where the stepsize of the algorithm is decreasing.

Our starting point is a generic non-autonomous Ordinary Differential Equation (ODE) introduced by Belotto da Silva and Gazeau (2020) (see also Chapter 2 for Adam ), depicting the continuous-time versions of the aforementioned florilegium of algorithms. The solutions to the ODE are shown to converge to the set of critical points of F . This suggests that a general provably convergent algorithm can be obtained by means of an Euler discretization of the ODE, including possible stochastic perturbations. Special cases of our general algorithm include SHB, Adam and S-NAG. We establish the almost sure boundedness and the convergence to critical points. Under additional assumptions, we obtain convergence rates, under the form of a central limit theorem. These results are new. They extend the work of [START_REF] Gadat | Stochastic heavy ball[END_REF] and Chapter 2 to a general setting. In particular, we highlight the almost sure convergence result of S-NAG in a non-convex setting, which is new to the best of our knowledge.

Next, we address the question of the avoidance of "traps". In a non-convex setting, the set of critical points of a function F is generally larger than the set of local minimizers. A "trap" stands for a critical point at which the Hessian matrix of F has negative eigenvalues, namely, it is a local maximum or saddle point. We establish that the iterates cannot converge to such a point, if the noise is exciting in some directions. The result extends the previous work of [START_REF] Gadat | Stochastic heavy ball[END_REF] obtained in the context of SHB. This result not only allows to study a broader class of algorithms but also significantly weakens the assumptions. In particular, [START_REF] Gadat | Stochastic heavy ball[END_REF] use a sub-Gaussian assumption on the noise and a rather stringent assumption on the stepsizes. The main difficulty in the approach of [START_REF] Gadat | Stochastic heavy ball[END_REF] lies in the use of the classical autonomous version of Poincaré's invariant manifold theorem. The key ingredient of our proof is a general avoidance of traps result, adapted to non-autonomous settings, which we believe to be of independent interest. It extends usual avoidance of traps results to a non-autonomous setting, by making use of a non-autonomous version of Poincaré's theorem [START_REF] Daleckii | Stability of solutions of differential equations in Banach space[END_REF][START_REF] Kloeden | Nonautonomous dynamical systems[END_REF].

Chapter organization. In Section 3.2, we introduce and study the ODE's governing our general stochastic algorithm. We establish the existence and uniqueness of the solutions, as well as the convergence to the set of critical points. In Section 3.3, we introduce the main algorithm. We provide sufficient conditions under which the iterates are bounded and converge to the set of critical points. A central limit theorem is stated. Section 3.4 introduces a general avoidance of traps result for non-autonomous settings. Next, this result is applied to the proposed algorithm. Sections 3.5,3.6 and 3.7 are devoted to the proofs of the results of Sections 3.2, 3.3 and 3.4, respectively.

Notations. Given an integer d ≥ 1, two vectors x, y ∈ R d , and a real α, we denote by x y, x α , x/y, |x|, and |x| the vectors in R d whose i-th coordinates are respectively given by x i y i ,

x α i , x i /y i , |x i |, |x i |.
Inequalities of the form x ≤ y are to be read componentwise. The standard Euclidean norm is denoted • . Notation M T represents the transpose of a matrix M . For x ∈ R d and ρ > 0, the notation B(x, ρ) stands for the open ball of R d with center x and radius ρ. We also write

R + = [0, ∞). If z ∈ R d and A ⊂ R d , we write dist(z, A) := inf{ z -z : z ∈ A}. By 1 A (x)
, we refer to the function that is equal to one if x ∈ A and to zero elsewhere. The set of zeros of a function

h : R d → R d is zer h = {x : h(x) = 0}. Let D be a domain in R d . Given an integer k ≥ 0, the class C k (D, R) is the class of D → R
maps such that all their partial derivatives up to the order k exist and are continuous. For a function h ∈ C k (D, R) and for every i ∈ {1, . . . , d}, we denote as ∂ k i h(x 1 , . . . , x d ) the k th partial derivative of the function h with respect to x i . When k = 1, we just write ∂ i h(x 1 , . . . , x d ). The gradient of a function F : R d → R at a point x ∈ R d is denoted as ∇F (x), and its Hessian matrix at x is ∇ 2 F (x) as usual. For a function S : R d → R d , the notation ∇S(x) stands for the jacobian matrix of S at point x. In this chapter, we bring to the attention of the reader that few notations may slightly differ from Chapter 2: the variables (v, m, x) to be used below were treated as (x, m, v) in Chapter 2, the vector fields are also impacted by this permutation. Moreover, the time variable t occurs as a second variable of the vector field in the non-autonomous ODE of this chapter, whereas it was first in Chapter 2.

Ordinary differential equations

A general ODE

Our starting point will be a non-autonomous ODE slightly extending the one in Chapter 2 and which is almost identical to the one introduced in Belotto da Silva and Gazeau (2020). Let F be a function in

C 1 (R d , R), let S be a continuous R d → R d + function, let h, r, p, q : (0, ∞) → R + be four continuous functions, and let ε > 0. Let v 0 ∈ R d + and x 0 , m 0 ∈ R d . Starting at v(0) = v 0 , m(0) = m 0 , and x(0) = x 0 , our ODE on R + with trajectories in Z + := R d + × R d × R d reads        v(t) = p(t)S(x(t)) -q(t)v(t) ṁ(t) = h(t)∇F (x(t)) -r(t)m(t) ẋ(t) = -m(t)/ v(t) + ε (ODE-1)
This ODE can be rewritten compactly as follows. Write z 0 = (v 0 , m 0 , x 0 ), and let

z(t) = (v(t), m(t), x(t)) ∈ Z + for t ∈ R + . Let Z := R d × R d × R d
, and define the map

g : Z + × (0, ∞) → Z as g(z, t) =    p(t)S(x) -q(t)v h(t)∇F (x) -r(t)m -m/ √ v + ε    (3.1) for z = (v, m, x) ∈ Z + .
With these notations, we can rewrite (ODE-1) as

z(0) = z 0 , ż(t) = g(z(t), t) for t > 0.
By setting S(x) = ∇F (x) 2 when necessary and by properly choosing the functions h, r, p, and q, a large number of iterative algorithms used in Machine Learning can be obtained by an Euler's discretization of this ODE. For instance, choosing

h(t) = r(t) = a(t, λ, α 1 ) and p(t) = q(t) = a(t, λ, α 2 ) with a(t, λ, α) = λ -1 (1 -exp(-λα))/(1 -exp(-αt))
and λ, α 1 , α 2 > 0, one obtains a version of the Adam algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] (see (Belotto da Silva and Gazeau, 2020, Sections 2.4-4.2) for details). To give another less specific example, if we set p = q ≡ 0, then the resulting ODE covers a family of algorithms to which the well-known Heavy Ball with friction algorithm [START_REF] Attouch | The heavy ball with friction method, i. the continuous dynamical system: global exploration of the local minima of a real-valued function by asymptotic analysis of a dissipative dynamical system[END_REF] belongs. For a comprehensive and more precise view of the deterministic algorithms that can be deduced from (ODE-1) by an Euler's discretization, the reader is referred to (Belotto da Silva and Gazeau, 2020, Table 1).

In this chapter, since we are rather interested in stochastic versions of these algorithms, Eq. (ODE-1) will be the basic building block of the classical "ODE method" which is widely used in the field of stochastic approximation [START_REF] Benaïm | Dynamics of stochastic approximation algorithms[END_REF]. In order to analyze the behavior of this equation in preparation of the stochastic analysis, we need the following assumptions.

i) h ∈ C 1 ((0, +∞), R + ), ḣ(t)
≤ 0 on (0, +∞) and the limit h ∞ := lim t→∞ h(t) is positive. ii) r and q are non-increasing and r ∞ := lim t→∞ r(t) , q ∞ := lim t→∞ q(t) are positive. iii) p converges towards p ∞ as t → ∞. iv) For all t ∈ (0, +∞), r(t) ≥ q(t)/4 and r ∞ > q ∞ /4. These assumptions are sufficient to prove the existence and the uniqueness of the solution to (ODE-1) starting at a time t 0 > 0. The following additional assumption extends the solution to t 0 = 0. Assumption 3.2.5. Either h, r, p, q ∈ C 1 ([0, +∞), R + ), or the following holds:

i) For every x ∈ R d , we have S(x) ≥ ∇F (x) 2 .
ii) The functions h p , h q-2r , t → th(t), t → tr(t), t → tp(t), t → tq(t) are bounded near zero.

iii) There exists t 0 > 0 such that for all t < t 0 , 2r(t)q(t) > 0 . iv) There exists δ > 0 such that h r ,

p q ∈ C 1 ([0, δ), R + ) . v) The initial condition z 0 = (v 0 , m 0 , x 0 ) ∈ Z + satisfies m 0 = ∇F (x 0 ) lim t↓0 h(t) r(t)
and v 0 = S(x 0 ) lim t↓0 p(t) q(t) .

Remark 2. The functions h, r, p, q corresponding to Adam satisfy these conditions. We leave the straightforward verifications to the reader. We just observe here that the function S that will correspond to our stochastic algorithm in Section 3.3 below will satisfy Assumption 3.2.5-i) by an immediate application of Jensen's inequality.

The following theorem slightly generalizes the results of (Belotto da Silva and Gazeau, 2020, Th. 3 and Th. 5).

Theorem 3.1. Let Assumptions 3.2.1 to 3.2.4 hold true. Consider z 0 ∈ Z + and t 0 > 0.

Then, there exists a unique global solution z : [t 0 , +∞) → Z + to (ODE-1) with initial condition z(t 0 ) = z 0 . Moreover, z([t 0 , +∞)) is a bounded subset of Z + . As t → +∞, z(t) converges towards the set

Υ := {z = (p ∞ S(x )/q ∞ , 0, x ) : x ∈ zer ∇F } . (3.2)
If, additionally, Assumption 3.2.5 holds, then we can take t 0 = 0.

Remark 3. Th. 3.1 only shows the convergence of the trajectory z(t) towards a set. Convergence of the trajectory towards a single point is not guaranteed when the set Υ is not countable.

Remark 4. A simpler version of (ODE-1) is obtained when omitting the momentum term. It reads:

   v(t) = p(t)S(x(t)) -q(t)v(t) ẋ(t) = -∇F (x(t))/ v(t) + ε . (ODE-1 )
This ODE encompasses the algorithms of the family of RMSProp [START_REF] Tieleman | Lecture 6.e-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF], as shown in Belotto da Silva and Gazeau (2020). The approach for proving the previous theorem can be adapted to (ODE-1 ) with only minor modifications. In the proofs below, we will point out the particularities of (ODE-1 ) when necessary.

The following paragraph is devoted to a particular case of (ODE-1), which does not satisfy Assumption 3.2.4, and which requires a more involved treatment than (ODE-1 ).

The Nesterov case

Cabot et al. ( 2009), [START_REF] Su | A differential equation for modeling Nesterov's accelerated gradient method: theory and insights[END_REF] and others studied the ODE

ẍ(t) + α t ẋ(t) + ∇F (x(t)) = 0, α > 0, F ∈ C 1 (R d , R),
which Euler's discretization generates the well-known Nesterov's accelerated gradient algorithm, see also [START_REF] Attouch | Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity[END_REF]; [START_REF] Aujol | Optimal convergence rates for nesterov acceleration[END_REF]. This ODE can be rewritten as

   ṁ(t) = ∇F (x(t)) -α t m(t) ẋ(t) = -m(t), (ODE-N)
which is formally the particular case of (ODE-1) that is taken for p(t) = q(t) = 0, h(t) = 1, and r(t) = α/t. Obviously, this case is not covered by Assumption 3.2.4. Moreover, it turns out that, contrary to the situation described in Remark 4 above, this case cannot be dealt with by a straightforward adaptation of the proof of Th. 3.1. The reason for this is as follows. Heuristically, the proof of Th. 3.1 is built around the fact that the solution of (ODE-1) "shadows" for large t the solution of the autonomous ODE

         v(t) = p ∞ S(x(t)) -q ∞ v(t) ṁ(t) = h ∞ ∇F (x(t)) -r ∞ m(t) ẋ(t) = -m(t) √ v(t)+ε
, and the latter can be shown to converge to the set Υ defined in Eq. (3.2), either under Assumption 3.2.4 or for the algorithms covered by Remark 4. This idea does not work anymore for (ODE-N), for its large-t autonomous counterpart

   ṁ(t) = ∇F (x(t)) ẋ(t) = -m(t).
can have solutions that do not converge to the critical points of F . As an example of such solutions, take d = 1 and F (x) = x 2 /2. Then, t → (cos(t), sin(t)) is an oscillating solution of the latter ODE.

Yet, we have the following result. Up to our knowledge, the proof of the convergence below as t → +∞ is new.

Related works

The continuous-time dynamical system (ODE-1) we consider was first introduced in (Belotto da Silva and Gazeau, 2020, Eq. ( 2.1)) with S = ∇F 2 . Th. 3.1 above is roughly the same as (Belotto da Silva and Gazeau, 2020, Ths. 3 and 5), with some slight differences regarding the assumptions on the function F , or Assumption 3.2.4-iv).

We point out that the main focus of Belotto da Silva and Gazeau ( 2020) is to study the properties of the deterministic continous-time dynamical system (ODE-1). In the present work, we highlight that the purpose of Th. 3.1 is to pave the way to our analysis of the corresponding stochastic algorithms in Section 3.3.

Concerning Th. 3.2, the existence and the uniqueness of a global solution to (ODE-N) has been previously shown in the literature, for instance in (Cabot et al., 2009, Prop. 2.1) or in (Su et al., 2016, Th. 1). The convergence statement in Th. 3.2 is new to the best of our knowledge. In particular, we stress that we do not make any convexity assumption on F . The closest result we are aware of is the one of [START_REF] Cabot | On the long time behavior of second order differential equations with asymptotically small dissipation[END_REF]. In (Cabot et al., 2009, Prop. 2.5), it is shown that if x(t) converges towards some point x, then necessarily x is a critical point of F . Our result in Th. 3.2 strengthens this statement, by establishing that x(t) actually converges to the set of critical points.

Stochastic algorithms

In this section, we discuss the asymptotic behavior of stochastic algorithms that consist in noisy Euler's discretizations of (ODE-1) and (ODE-N) studied in the previous section.

We first set the stage. Let (Ξ, T, µ) be a probability space. Denoting as

B(R d ) the Borel σ-algebra on R d , consider a B(R d ) ⊗ T-measurable function f : R d × Ξ → R that
satisfies the following assumption.

Assumption 3.3.1. The following conditions hold:

i) For every x ∈ R d , f (x, •) is µ-integrable. ii) For every s ∈ Ξ, the map f (•, s) is differentiable. Denoting as ∇f (x, s) its gradient w.r.t. x, the function ∇f (x, •) is integrable. iii) There exists a measurable map κ : R d × Ξ → R + s.t. for every x ∈ R d : a) The map κ(x, •) is µ-integrable, b) There exists ε > 0 s.t. for every s ∈ Ξ, ∀ u, v ∈ B(x, ε), ∇f (u, s) -∇f (v, s) ≤ κ(x, s) u -v .
Under Assumption 3.3.1, we can define the mapping F : R d → R as

F (x) = E ξ [f (x, ξ)] (3.4)
for all x ∈ R d , where we write E ξ ϕ(ξ) = ϕ(ξ)µ(dξ). It is easy to see that the mapping

F is differentiable, ∇F (x) = E ξ [∇f (x, ξ)]
for all x ∈ R d , and ∇F is locally Lipschitz.

Let (γ n ) n≥1 be a sequence of positive real numbers satisfying Assumption 3.3.2. γ n+1 /γ n → 1 and n γ n = +∞.

Define for every integer n ≥ 1

τ n = n k=1 γ k .
Let (Ω, F, P) be a probability space, and let (ξ n : n ≥ 1) be a sequence of iid random variables defined from (Ω, F, P) into (Ξ, T, µ) with the distribution µ.

General algorithm

Our first algorithm is a discrete and noisy version of (ODE-1).

Let z 0 = (v 0 , m 0 , x 0 ) ∈ Z + and h 0 , r 0 , p 0 , q 0 ∈ (0, ∞). Define for every n ≥ 1

h n = h(τ n ), r n = r(τ n ), p n = p(τ n ), and q n = q(τ n ). (3.5)
The algorithm is written as follows.

Algorithm 3.1 (general algorithm)

Initialization: z 0 ∈ Z + . for n = 1 to n iter do v n+1 = (1 -γ n+1 q n )v n + γ n+1 p n ∇f (x n , ξ n+1 ) 2 m n+1 = (1 -γ n+1 r n )m n + γ n+1 h n ∇f (x n , ξ n+1 ) x n+1 = x n -γ n+1 m n+1 / √ v n+1 + ε . end for
We suppose throughout the chapter that 1γ n+1 q n ≥ 0 for all n ∈ N. This will guarantee that the quantity √ v n + ε is always well-defined (see Algorithm 3.1). This mild assumption is satisfied as soon as q 0 ≤ 1 γ 1 since the sequence (q n ) is non-increasing and the sequence of stepsizes (γ n ) can also be supposed to be non-increasing.

Since this algorithm makes use of the function ∇f (x, ξ) 2 , a strengthening of Assumption 3.3.1 is required:

ii') For each x ∈ R d , the function ∇f (x, •) 2 is µ -integrable.
iii') There exists a measurable map κ :

R d × Ξ → R + s.t. for every x ∈ R d : a) The map κ(x, •) is µ-integrable. b) There exists ε > 0 s.t. for every u, v ∈ B(x, ε), ∇f (u, s) -∇f (v, s) ∨ ∇f (u, s) 2 -∇f (v, s) 2 ≤ κ(x, s) u -v .
Under Assumption 3.3.3, we can also define the mapping S : R d → R d as:

S(x) = E ξ [∇f (x, ξ) 2 ]
for all x ∈ R d . Notice that Assumptions 3.2.1 and 3.2.3 are satisfied for F and S. Assumption 3.3.4. Assume either of the following conditions. i) There exists q ≥ 2 s.t. for every compact set

K ⊂ R d , sup x∈K E ξ ∇f (x, ξ) 2q < ∞ and n γ 1+q/2 n < ∞ .
ii) For every compact set K ⊂ R d , there exists a real σ K = 0 s.t.

E ξ exp u, ∇f (x, ξ) -∇F (x) 1 x∈K ≤ exp σ 2 K u 2 /2 and E ξ exp u, ∇f (x, ξ) 2 -S(x) 1 x∈K ≤ exp σ 2 K u 2 /2 , for every x, u ∈ R d . Moreover, for every α > 0, n exp(-α/γ n ) < ∞ .
Remark 5. We make the following comments regarding Assumption 3.3.4.

• Assumption 3.3.4-i) allows to use larger stepsizes in comparison to the classical condition n γ 2 n < ∞ which corresponds to the particular case q = 2. • Recall that a random vector X is said to be subgaussian if there exists a real σ = 0 s.t. Ee u,X ≤ e σ 2 u 2 /2 for every constant vector u ∈ R d . In Assumption 3.3.4-ii), the subgaussian noise offers the possibility to use a sequence of stepsizes with an even slower decay rate than in Assumption 3.3.4-i).

Assumption 3.3.5. The set F ({x : ∇F (x) = 0}) has an empty interior.

Remark 6. Assumption 3.3.5 excludes a pathological behavior of the objective function F at critical points. It is satisfied when

F ∈ C k (R d , R) for k ≥ d. Indeed,
(z n = (v n , m n , x n ) : n ∈ N)
given by Algorithm 3.1 is bounded with probability one. Then, w.p.1, the sequence (z n ) converges towards the set Υ defined in Eq. (3.2). If, in addition, the set of critical points of the objective function F is finite or countable, then w.p.1, the sequence (z n ) converges to a single point of Υ.

We now deal with the boundedness problem of the sequence (z n ). We introduce an additional assumption for this purpose.

Assumption 3.3.6. The following conditions hold. 2, 3.2.4, 3.3.2, 3.3.3, 3.3.4-i) (with q = 2) and 3.3.6 hold. Then, the sequence (v n , m n , x n ) given by Algorithm 3.1 is bounded with probability one.

i) ∇F is (globally) Lipschitz continuous. ii) There exists C > 0 s.t. for all x ∈ R d , E ξ [ ∇f (x, ξ) 2 ] ≤ C(1 + F (x)) , iii) n γ 2 n < ∞ . Theorem 3.4. Let Assumptions 3.2.
Remark 7. The above stability result requires square summable stepsizes. Showing the same boundedness result under the Assumption 3.3.4 that allows for larger stepsizes is a challenging problem in the general case. In these situations, the boundedness of the iterates can be sometimes ensured by ad hoc means.

Remark 8. We can also consider the noisy discretization of (ODE-1 ) introduced in Remark 4 above. This algorithm reads

v n+1 = (1 -γ n+1 q n )v n + γ n+1 p n ∇f (x n , ξ n+1 ) 2 (3.6a) x n+1 = x n -γ n+1 ∇f (x n , ξ n+1 )/ √ v n+1 + ε (3.6b) for (v 0 , x 0 ) ∈ R d + × R d .
With only minor adaptations, Th. 3.3 and Th. 3.4 can be shown to hold as well for this algorithm. We refer to the concomitant paper (Gadat and Gavra, 2020, Sec. 2.2) for the link between this algorithm and the seminal algorithms AdaGrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] and RMSProp (Tieleman and Hinton, 2012).

Stochastic Nesterov's Accelerated Gradient (S-NAG)

S-NAG is the noisy Euler's discretization of (ODE-N). Given α > 0, it generates the sequence (m n , x n ) on R d × R d given by Algorithm 3.2.

Algorithm 3.2 (S-NAG with decreasing steps)

Initialization: m 0 = 0, x 0 ∈ R d . for n = 1 to n iter do m n+1 = (1 -αγ n+1 /τ n )m n + γ n+1 ∇f (x n , ξ n+1 ) x n+1 = x n -γ n+1 m n+1
. end for Assumption 3.3.7. Assume either of the following conditions. i) There exists q ≥ 2 s.t. for every compact set

K ⊂ R d , sup x∈K E ξ ∇f (x, ξ) q < ∞ and n γ 1+q/2 n < ∞ .
ii) For every compact set K ⊂ R d , there exists a real σ K = 0 s.t. 2, 3.3.1, 3.3.2, 3.3.5 and 3.3.7 hold true. Assume that the random sequence (y n = (m n , x n ) : n ∈ N) given by Algorithm 3.2 is bounded with probability one. Then, w.p.1, the sequence (y n ) converges towards the set Ῡ defined in Eq. (3.3). If, in addition, the set of critical points of the objective function F is finite or countable, then w.p.1, the sequence (y n ) converges to a single point of Ῡ.

E ξ exp u, ∇f (x, ξ) -∇F (x) 1 x∈K ≤ exp σ 2 K u 2 /2 , for every x, u ∈ R d . Moreover, for every α > 0, n exp(-α/γ n ) < ∞ . Theorem 3.5. Let Assumptions 3.2.
The almost sure boundedness of the sequence (y n ) is handled in what follows. Theorem 3.6. Let Assumptions 3.2.2,3.3.1,3.3.2 and 3.3.6 hold. Then, the sequence (y n = (m n , x n ) : n ∈ N) given by Algorithm 3.2 is bounded with probability one. Remark 9. Assumption 3.3.4-i) in Th. 3.4 is not needed for Th. 3.6.

Central limit theorem

In this section, we establish a conditional central limit theorem for Algorithm 3.1. Assumption 3.3.8. Let x ∈ zer ∇F . The following holds. i) F is twice continuously differentiable on a neighborhood of x and the Hessian ∇ 2 F (x ) is positive definite. ii) S is continuously differentiable on a neighborhood of x . iii) There exists M > 0 and b M > 4 s.t.

sup x∈B(x ,M ) E ξ [ ∇f (x, ξ) b M ] < ∞ . (3.7)
Under Assumptions 3.2.4-i) to iii), it follows from Eq. (3.5) that the sequences (h n ), (r n ), (p n ) and (q n ) of nonnegative reals converge respectively to h ∞ , r ∞ , p ∞ and q ∞ where h ∞ , r ∞ and q ∞ are supposed positive. Define v := q -1 ∞ p ∞ S(x ). Consider the matrix

V := diag (ε + v ) -1 2 .
(3.8)

Let P be an orthogonal matrix s.t. the following spectral decomposition holds:

V 1 2 ∇ 2 F (x )V 1 2 = P diag(π 1 , • • • , π d )P -1 ,
where

π 1 ≤ • • • ≤ π d are the (positive) eigenvalues of V 1 2 ∇ 2 F (x )V 1 2 . Define H := -r ∞ I d h ∞ ∇ 2 F (x ) -V 0
where I d is the d × d identity matrix. Then the matrix H is Hurwitz. Indeed, it can be shown that the largest real part of the eigenvalues of H coincides with -L, where

L := r ∞ 2   1 - 1 - 4h ∞ π 1 r 2 ∞ ∨ 0    > 0 . (3.9) Assumption 3.3.9. The sequence (γ n ) is given by γ n = γ 0 n α for some α ∈ (0, 1], γ 0 > 0. Moreover, if α = 1, we assume that γ 0 > 1 2(L∧q∞) .
Theorem 3.7. Let Assumptions 3.2.4-i) to iii), 3.3.3, 3.3.8 and 3.3.9 hold. Consider the iterates z n = (v n , m n , x n ) given by Algorithm 3.1. Set θ := 0 if α < 1 and θ := 1/(2γ 0 ) if α = 1. Assume that the event {z n → z }, where z = (v , 0, x ), has a positive probability. Then, given that event,

1 √ γ n m n x n -x ⇒ N (0, Γ) ,
where ⇒ stands for the convergence in distribution and N (0, Γ) is a centered Gaussian distribution on R 2d with a covariance matrix Γ given by the unique solution to the Lyapunov equation

(H + θI 2d )Γ + Γ(H + θI 2d ) T = - Cov(h ∞ ∇f (x , ξ)) 0 0 0 .
In particular, given {z n → z }, the vector √ γ n -1 (x nx ) converges in distribution to a centered Gaussian distribution with a covariance matrix given by:

Γ 2 = V 1 2 P    C k, r∞-2θ h∞ (π k + π + 2θ(θ-r∞) h∞ ) + (π k -π ) 2 2(r∞-2θ)    k, =1...d P -1 V 1 2 (3.10)
where

C := P -1 V 1 2 E ξ ∇f (x , ξ)∇f (x , ξ) T V 1 2 P .
A few remarks are in order.

• The matrix Γ 2 coincides with the limiting covariance matrix associated to the iterates

   m n+1 = m n + γ n+1 (h ∞ V ∇f (x n , ξ n+1 ) -r ∞ m n ) x n+1 = x n -γ n+1 m n+1 .
This procedure can be seen as a preconditioned version of the stochastic heavy ball algorithm [START_REF] Gadat | Stochastic heavy ball[END_REF] although the iterates are not implementable because of the unknown matrix V . Notice also that the limiting covariance Γ 2 depends on v but does not depend on the fluctuations of the sequence (v n ).

• When h ∞ = r ∞ (which is the case for Adam ), we recover the expression of the asymptotic covariance matrix previously provided in Chapter 2 Section 2.5.3 and the remarks formulated therein.

• The assumption r ∞ > 0 is crucial to establish Th. 3.7. For this reason, Th. 3.7 does not generalize immediately to Algorithm 3.2. The study of the fluctuations of Algorithm 3.2 is left for future works.

Related works

Gadat et al. ( 2018) study the SHB algorithm, which is a noisy Euler's discretization of (ODE-1) in the situation where h = r and p = q ≡ 0 (i.e., there is no v variable). In this framework, if we set h = r ≡ r > 0 in Algorithm 3.1 above, then Th. 3.3 above recovers the analogous case in (Gadat et al., 2018, Th. 2.1), which is termed as the exponential memory case. The other important case treated in [START_REF] Gadat | Stochastic heavy ball[END_REF] is the case where h(t) = r(t) = r/t for some r > 0, referred to as the polynomially memory case. Actually, it is known that the ODE obtained for h(t) = r(t) = r/t and p = q ≡ 0 boils down to (ODE-N) after a time variable change (see, e.g., Lem. 3.14 below). Nevertheless, we highlight that the stochastic algorithm that stems from this ODE and that is studied in [START_REF] Gadat | Stochastic heavy ball[END_REF] is different from the S-NAG algorithm introduced above which stems from a different ODE (ODE-N). Hence, the convergence result of Th. 3.5 for the S-NAG algorithm we consider is not covered by the analysis of [START_REF] Gadat | Stochastic heavy ball[END_REF].

The specific case of the Adam algorithm is analyzed in Chapter 2 in both the constant and vanishing stepsize settings (see Ths. 2.6-2.7 which are the analogues of our Ths. 3.3-3.4). Note that we deal with a more general algorithm in the present chapter. Indeed, Algorithm 3.1 offers some freedom in the choice of the functions h, r, p, q satisfying Assumption 3.2.4 beyond the specific case of the Adam algorithm studied in Chapter 2. Apart from this generalization, we also emphasize some small improvements.

Regarding Th. In most generality, the almost sure convergence result of the iterates of Algorithm 3.1 using vanishing stepsizes (Ths. 3.3-3.4) is new to the best of our knowledge. Moreover, while some recent results exist for S-NAG in the constant stepsize and for convex objective functions (see for e.g., Assran and Rabbat ( 2020)), Ths. 3.5 and 3.6 which tackle the possibly non-convex setting are also new to the best of our knowledge.

In the work of [START_REF] Gadat | Asymptotic study of stochastic adaptive algorithm in non-convex landscape[END_REF] that was posted on the arXiv repository a few days after our submission, Gadat and Gavra study the specific case of the algorithm described in Eq. (3.6) encompassing both Adagrad and RMSProp, with the possibility to use mini-batches. For this specific algorithm, the authors establish a similar almost sure convergence result to ours (Gadat and Gavra, 2020, Th. 1) for decreasing stepsizes and derive some quantitative results bounding in expectation the gradient of the objective function along the iterations for constant stepsizes (Gadat and Gavra, 2020, Th. 2). We highlight though that they do not consider the presence of momentum in the algorithm. Therefore, their analysis does not cover neither Algorithm 3.1 nor Algorithm 3.2.

In contrast to our analysis, some works in the literature explore the constant stepsize regime for some stochastic momentum methods either for smooth [START_REF] Yan | A unified analysis of stochastic momentum methods for deep learning[END_REF] or weakly convex objective functions [START_REF] Mai | Convergence of a stochastic gradient method with momentum for nonsmooth nonconvex optimization[END_REF]. Furthermore, concerning Adam-like algorithms, several recent works control the minimum of the norms of the gradients of the objective function evaluated at the iterates of the algorithm over N iterations in expectation or with high probability [START_REF] Basu | Convergence guarantees for rmsprop and adam in non-convex optimization and their comparison to nesterov acceleration on autoencoders[END_REF][START_REF] Zhou | On the convergence of adaptive gradient methods for nonconvex optimization[END_REF][START_REF] Chen | Closing the generalization gap of adaptive gradient methods in training deep neural networks[END_REF]Zou et al., 2019a;[START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF][START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF]Alacaoglu et al., 2020a;[START_REF] Défossez | A simple convergence proof of adam and adagrad[END_REF]Alacaoglu et al., 2020b) and establish regret bounds in the convex setting (Alacaoglu et al., 2020b).

Similar central limit theorems to Th. 3.7 are established in the cases of the stochastic heavy ball algorithm with exponential memory (Gadat et al., 2018, Th. 2.4) and Adam (see Th. 2.8 in Chapter 2). In comparison to [START_REF] Gadat | Stochastic heavy ball[END_REF], we precise that our theorem recovers their result and provides a closed formula for the asymptotic covariance matrix Γ 2 . Our proof of Th. 3.7 differs from the strategies adopted in [START_REF] Gadat | Stochastic heavy ball[END_REF] and Chapter 2.

Avoidance of traps

In Th. 3.3 and Th. 3.5 above, we established the almost sure convergence of the iterates x n towards the set of critical points of the objective function F for both Algorithms 3.1 and 3.2. However, the landscape of F can contain what is known as "traps" for the algorithm, namely, critical points where the Hessian matrix of F has negative eigenvalues, making these critical points local maxima or saddle points. In this section, we show that the convergence of the iterates to these traps does not take place if the noise is exciting in some directions.

Starting with the contributions of [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF] and [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF], the numerous so-called avoidance of traps results that can be found in the literature deal with the case where the ODE that underlies the stochastic algorithm is an autonomous ODE. Obviously, this is neither the case of (ODE-1), nor of (ODE-N). To deal with this issue, we first state a general avoidance of traps result that extends [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF] to a non-autonomous setting, and that has an interest of its own. We then apply this result to Algorithms 3.1 and 3.2.

A general avoidance-of-traps result in a non-autonomous setting

The notations in this subsection and in Sections 3.7.1-3.7.2 are independent from the rest of the chapter. We recall that for a function h : R d → R d , we denote by ∂ k i h(x 1 , . . . , x d ) the k th partial derivative of the function h with respect to x i .

The setting of our problem is as follows. Given an integer d > 0 and a continuous function b : R d × R + → R d , we consider a stochastic algorithm built around the non-autonomous ODE ż(t) = b(z(t), t). Let z ∈ R d , and assume that on V × R + where V is a certain neighborhood of z , the function b can be developed as

b(z, t) = D(z -z ) + e(z, t), (3.11)
where e(z , •) ≡ 0, and where the matrix D ∈ R d×d is assumed to admit the following spectral factorization: Given 0 ≤ d -< d and 0 < d + ≤ d with d -+ d + = d, we can write

D = QΛQ -1 , Λ = Λ - Λ + , (3.12)
where the Jordan blocks that constitute Λ -∈ R d -×d -(respectively Λ + ∈ R d + ×d + ) are those that contain the eigenvalues λ i of D for which λ i ≤ 0 (respectively λ i > 0). Since d + > 0, the point z is an unstable equilibrium point of the ODE ż(t) = b(z(t), t), in the sense that the ODE solution will only be able to converge to z along a specific so-called invariant manifold which precise characterization will be given in Section 3.7.1 below.

We now consider a stochastic algorithm that is built around this ODE. The condition d + > 0 makes that z is a trap that the algorithm should desirably avoid. The following theorem states that this will be the case if the noise term of the algorithm is omnidirectional enough. The idea is to show that the case being, the algorithm trajectories will move away from the invariant manifold mentioned above.

Theorem 3.8. Given a sequence (γ n ) of nonnegative deterministic stepsizes such that n γ n = +∞, n γ 2 n < +∞, and a filtration (F n ), consider the stochastic approximation algorithm in R d

z n+1 = z n + γ n+1 b(z n , τ n ) + γ n+1 η n+1 + γ n+1 ρ n+1
where τ n = n k=1 γ k . Assume that the sequences (η n ) and (ρ n ) are adapted to F n , and that z 0 is F 0 -measurable. Assume that there exists z ∈ R d such that Eq. (3.11) holds true on V × R + , where V is a neighborhood of z . Consider the spectral factorization (3.12), and assume that d + > 0. Assume moreover that the function e at the right hand side of Eq. (3.11) satisfies the conditions:

i) e(z , •) ≡ 0. ii) On V × R + , the functions ∂ n 2 ∂ k 1 e(z, t) exist and are continuous for 0 ≤ n < 2 and 0 ≤ k + n ≤ 2.
iii) The following convergence holds :

lim (z,t)→(z ,∞)
∂ 1 e(z, t) = 0 .

(3.13) iv) There exist t 0 > 0 and a neighborhood W ⊂ R d of z s.t.

sup z∈W,t≥t 0 ∂ 2 e(z, t) < + ∞ and sup z∈W,t≥t 0 ∂ 2 1 e(z, t) < + ∞ .
Moreover, suppose that :

v) n ρ n+1 2 1 zn∈W < ∞ almost surely. vi) lim sup E[ η n+1 4 | F n ]1 zn∈W < ∞, and E[η n+1 | F n ]1 zn∈W = 0. vii) Writing ηn = Q -1 η n = (η - n , η+ n ) with η± n ∈ R d ± , for some c 2 > 0, it holds that lim inf E[ η+ n+1 2 | F n ]1 zn∈W ≥ c 2 1 zn∈W . Then, P([z n → z ]) = 0.
Remark 10. Assumptions i) to iv) of Th. In Th. 3.3 above, we showed that the sequence (z n ) generated by Algorithm 3.1 converges almost surely towards the set Υ defined in Eq. (3.2). Our purpose now is to show that the traps in Υ (to be characterized below) are avoided by the stochastic algorithm 3.1 under a proper omnidirectionality assumption on the noise.

Our first task is to write Algorithm 3.1 in a manner compatible with the statement of Th. 3.8. The following decomposition holds for the sequence

(z n = (v n , m n , x n ), n ∈ N)
generated by this algorithm:

z n+1 = z n + γ n+1 g(z n , τ n ) + γ n+1 η n+1 + γ n+1 ρn+1
,

where ρn+1 = 0 , 0 , mn √ vn+ε -m n+1 √ v n+1 +ε
, and where η n+1 is the martingale increment with respect to the filtration (F n ) which is defined by Eq. (3.28).

Observe from Eq. (3.2) that each z ∈ Υ is written as z = (v , 0, x ) where x ∈ zer ∇F , and v = q -1 ∞ p ∞ S(x ) (in particular, x and z are in a one-to-one correspondence). We need to linearize the function g(•, t) around z . The following assumptions will be required.

Assumption 3.4.1. The functions F and S belong respectively to

C 3 (R d , R) and C 2 (R d , R d + ).
Assumption 3.4.2. The functions h, r, p, q belong to C 1 ((0, ∞), R + ) and have bounded derivatives on [t 0 , +∞) for some t 0 > 0.

Lemma 3.9. Let Assumptions 3.2.4-i) to iii), 3.4.1 and 3.4.2 hold. Let z = (v , 0, x ) ∈ Υ. Then, for every z ∈ Z + and every t > 0, the following decomposition holds true:

g(z, t) = D(z -z ) + e(z, t) + c(t),
where

D =    -q ∞ I d 0 p ∞ ∇S(x ) 0 -r ∞ I d h ∞ ∇ 2 F (x ) 0 -V 0    , c(t) =    p(t)S(x ) -q(t)v 0 0    ,
and the function e(z, t) (defined in Section 3.7.3.1 below for conciseness) has the same properties as its analogue in the statement of Th. 3.8.

Using this lemma, the algorithm iterate z n+1 can be rewritten as an instance of the algorithm in the statement of Th. 3.8, namely,

z n+1 = z n + γ n+1 b(z n , τ n ) + γ n+1 η n+1 + γ n+1 ρ n+1 , (3.14)
where in our present setting, b(z, t) = g(z, t)c(t) = D(zz ) + e(z, t) and ρ n = c(τ n-1 ) + ρn . In the following assumption, we use the well-known fact that a symmetric matrix H has the same inertia as AHA T for an arbitrary invertible matrix A.

Assumption 3.4.3. Let x ∈ zer ∇F , let v = q -1 ∞ p ∞ S(x )
, and define the diagonal matrix V = diag((v + ε) -1

2 ) as in (3.8). Assume the following conditions:

i) n q ∞ p n -p ∞ q n 2 < ∞ ,
ii) The Hessian matrix ∇ 2 F (x ) has a negative eigenvalue.

iii) There exists δ > 0 such that

sup x∈B(x ,δ) E ξ [ ∇f (x, ξ) 8 ] < ∞ .
iv) Defining Π u as the orthogonal projector on the eigenspace of V

1 2 ∇ 2 F (x )V 1
2 that is associated with the negative eigenvalues of this matrix, it holds that

Π u V 1 2 E ξ (∇f (x , ξ) -∇F (x ))(∇f (x , ξ) -∇F (x )) T V 1 2 Π u = 0.
Theorem 3.10. Let Assumptions 3.2.4,3.3.3,and 3.4.1,3.4.2 hold true. Let z ∈ Υ be such that Assumption 3.4.3 holds true for this z . Then, the eigenspace associated with the eigenvalues of D with positive real parts has the same dimension as the eigenspace of ∇ 2 F (x ) associated with the negative eigenvalues of this matrix. Let (z n = (v n , m n , x n ) : n ∈ N) be the random sequence generated by Algorithm 3.1 with stepsizes satisfying

n γ n = +∞ and n γ 2 n < +∞. Then, P([z n → z ]) = 0.
The assumptions and the result call for some comments.

Remark 11. The definition of a trap as regards the general algorithm in the statement of Th. 3.8 is that the matrix D in Eq. ( 3.11) has eigenvalues with positive real parts. Th. 3.10 states that this condition is equivalent to ∇ 2 F (x ) having negative eigenvalues. What's more, the dimension of the invariant subspace of D corresponding to the eigenvalues with positive real parts is equal to the dimension of the negative eigenvalue subspace of ∇ 2 F (x ). Thus, Assumption 3.4.3-iv) provides the "largest" subspace where the noise energy must be non zero for the purpose of avoiding the trap.

Remark 12. Assumptions 3.4.2 and 3.4.3-i) are satisfied by many widely studied algorithms, among which RMSProp and Adam.

Remark 13. The results of Th. 3.10 can be straightforwardly adapted to the case of (ODE-1 ). Assumption 3.4.3-iv) on the noise is unchanged.

In the case of the S-NAG algorithm, the assumptions become particularly simple. We state the afferent result separately. i) The Hessian matrix ∇ 2 F (x ) has a negative eigenvalue.

ii) There exists δ > 0 such that

sup x∈B(x ,δ) E ξ [ ∇f (x, ξ) 4 ] < ∞ . iii) Πu E ξ (∇f (x , ξ) -∇F (x ))(∇f (x , ξ) -∇F (x )) T Πu = 0,
where Πu is the orthogonal projector on the eigenspace of ∇ 2 F (x ) associated with its negative eigenvalues. Theorem 3.11. Let Assumptions 3.2.4,3.3.1,3.4.1 and 3.4.4 hold. Define y = (0, x ). Let (y n = (m n , x n ) : n ∈ N) be the random sequence given by Algorithm 3.2 with stepsizes satisfying n γ n = +∞ and n γ 2 n < +∞. Then, P([y n → y ]) = 0 .

Related works

Up to our knowledge, all the avoidance of traps results that can be found in the literature, starting from [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF], refer to stochastic algorithms that are discretizations of autonomous ODE's (see for e.g., (Benaïm, 1999, Sec. 9) for general Robbins Monro algorithms and (Mertikopoulos et al., 2020, Sec. 4.3) for SGD). In this line of research, a powerful class of techniques relies on Poincaré's invariant manifold theorem for an autonomous ODE in a neighborhood of some unstable equilibrium point.

In our work, we extend the avoidance of traps results to a non-autonomous setting, by borrowing a non-autonomous version of Poincaré's theorem from the rich literature that exists on the subject [START_REF] Daleckii | Stability of solutions of differential equations in Banach space[END_REF][START_REF] Kloeden | Nonautonomous dynamical systems[END_REF].

In [START_REF] Gadat | Stochastic heavy ball[END_REF], the authors succeeded in establishing an avoidance of traps result for their non-autonomous stochastic algorithm which is close to our S-NAG algorithm (see the discussion at the end of Section 3.3.4 above), at the expense of a sub-Gaussian assumption on the noise and a rather stringent assumption on the stepsizes. The main difficulty in the approach of [START_REF] Gadat | Stochastic heavy ball[END_REF] lies in the use of the classical autonomous version of Poincaré's theorem (see (Gadat et al., 2018, Remark 2.1)). This kind of difficulty is avoided by our approach, which allows to obtain avoidance of traps results with close to minimal assumptions. More recently, in the contribution of Gadat and Gavra (2020) discussed in Sec. 3.3.4, the authors establish an avoidance of traps result ((Gadat and Gavra, 2020, Th. 3)) for the algorithm described in Eq. (3.6) using techniques inspired from [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Benaïm | Dynamics of stochastic approximation algorithms[END_REF]. As previously mentioned, this recent work does not handle momentum and hence neither Algorithm 3.1 nor Algorithm 3.2. Moreover, as indicated in our discussion of [START_REF] Gadat | Stochastic heavy ball[END_REF], our strategy of proof is different.

Taking another point of view as concerns the trap avoidance, some recent works (Lee et al., 2019;Du et al., 2017;[START_REF] Jin | How to escape saddle points efficiently[END_REF][START_REF] Panageas | Gradient descent only converges to minimizers: Nonisolated critical points and invariant regions[END_REF][START_REF] Panageas | First-order methods almost always avoid saddle points: The case of vanishing step-sizes[END_REF] address the problem of escaping saddle points when the algorithm is deterministic but when the initialization point is random. In contrast to this line of research, our work considers a stochastic algorithm for which randomness enters into play at each iteration of the algorithm via noisy gradients.

3.5 Proofs for Section 3. The arguments of the proof of this theorem that we provide here follow the approach of Belotto da Silva and Gazeau (2020) with some small differences. Close arguments can be found in Chapter 2. We provide the proof here for completeness and in preparation of the proofs that will be related with the stochastic algorithms.

Existence and uniqueness

The following lemma guarantees that the term v(t) + ε in (ODE-1) is well-defined.

Lemma 3.12. Let t 0 ∈ R + and T ∈ (t 0 , ∞]. Assume that there exists a solution

z(t) = (v(t), m(t), x(t)) to (ODE-1) on [t 0 , T ) for which v(t 0 ) ≥ 0. Then, for all t ∈ [t 0 , T ), v(t) ≥ 0. Proof. Assume that ν := inf{t ∈ [t 0 , T ), v(t) < 0} satisfies ν < T . If v(t 0 ) > 0, Gronwall's lemma implies that v(t) ≥ v(t 0 ) exp(- t t 0 q(t)) on [t 0 , ν]
which is in contradiction with the fact that v(ν) = 0. If v(t 0 ) = 0, since ν < T , there exists t 1 ∈ (t 0 , ν) s.t. v(t 1 ) < 0. Hence, using the first equation from (ODE-1), we obtain v(t 1 ) > 0. This brings us back to the first case, replacing t 0 by t 1 .

Recall that F = inf F is finite by Assumption 3.2.2. Of prime importance in the proof will be the energy (Lyapunov) function E : R + × Z + → R, defined as

E(h, z) = h(F (x) -F ) + 1 2 m (v + ε) 1 4 2 , (3.15)
for every h ≥ 0 and every z = (v, m, x) ∈ Z + . This function is slightly different from its analogues that were used in Chapter 2 or in [START_REF] Alvarez | On the minimizing property of a second order dissipative system in Hilbert spaces[END_REF]; Belotto da Silva and Gazeau (2020).

Consider (t, z) ∈ (0, +∞) × Z + and set z = (v, m, x). Then, using Assumption 3.2.1, we can write

∂ t E(h(t), z) + ∇ z E(h(t), z), g(z, t) = ḣ(t)(F (x) -F ) - 1 4 m 2 (v + ε) 3 2
, p(t)S(x)q(t)v

+ m (v + ε) 1 2 , h(t)∇F (x) -r(t)m - m (v + ε) 1 2 , h(t)∇F (x) ≤ -r(t) - q(t) 4 m (v + ε) 1 4 2 + ḣ(t)(F (x) -F ) - p(t) 4 S(x), m 2 (v + ε) 3 2
. (3.16) With the help of this function, we can now establish the existence, the uniqueness and the boundedness of the solution of (ODE-1) on [t 0 , ∞) for an arbitrary t 0 > 0.

Lemma 3.13. For each t 0 > 0 and z 0 ∈ Z + , (ODE-1) has a unique solution on [t 0 , ∞) starting at z(t 0 ) = z 0 . Moreover, the orbit {z(t) : t ≥ t 0 } is bounded.

Proof. Let t 0 > 0, and fix z 0 ∈ Z + . On each set of the type [t 0 , t 0 +A]× B(z 0 , R) where A, R > 0 and B(z Using the continuity of the functions ∇F , h and r along with Gronwall's lemma, we get that {m(t)} is bounded if T < ∞. We can show a similar result for {v(t)}. Thus, {z(t)} is bounded on [t 0 , T ) if T < ∞ which is a contradiction, see, e.g., (Hartman, 2002, Cor.3.2).

0 , R) ⊂ (-ε, ∞) d × R d × R d ,
It remains to show that the trajectory {z(t)} is bounded. To that end, let us apply the variation of constants method to the equation ṁ

(t) = h(t)∇F (x(t)) -r(t)m(t). Writing R(t) = t t 0 r(u) du, we get that d dt e R(t) m(t) = e R(t) h(t)∇F (x(t)).
Therefore, for every t ≥ t 0 ,

m(t) = e -R(t) m(t 0 ) + t t 0 e R(u)-R(t) h(u)∇F (x(u))du .
Using the continuity of ∇F together with the boundedness of x, Assumption 3.2.4 and the triangle inequality, we obtain the existence of a constant C > 0 independent of t s.t.

m(t) -m(t 0 ) -m(t 0 ) ≤ Ch(t 0 ) t t 0 e -t u r(s) ds du ≤ Ch(t 0 ) t t 0 e -r∞(t-u) du ≤ Ch(t 0 ) r ∞ .
The same reasoning applies to v(t) using the continuity of S and Assumption 3.2.4. This completes the proof.

We can now extend this solution to t 0 = 0 along the approach of Belotto da Silva and Gazeau (2020), where the detailed derivations can be found. The idea is to replace h(t) with h(max(η, t)) for some η > 0 and to do the same for p, q, and r. It is then easy to see that the ODE that is obtained by doing these replacements has a unique global solution on R + . By making η → 0 and by using the Arzelà-Ascoli theorem along with Assumption 3.2.5, we obtain that (ODE-1) has a unique solution on R + .

Convergence

The first step in this part consists in transforming (ODE-1) into an autonomous ODE by including the time variable into the state vector. More specifically, we start with the following ODE:

ż(t) u(t) = g(z(t), u(t)) 1 with z(0) u(0) = z 0 t 0 ,
then, we perform the following change of variable in time

z u → z s = 1/u
allowing the solution to lie in a compact set. We initialize the above ODE at a time instant t 0 > 0. Define the functions H, R, P, Q :

R + → R + by setting H(s) = h(1/s), R(s) = r(1/s), P(s) = p(1/s); Q(s) = q(1/s) for s > 0; H(0) = h ∞ , R(0) = r ∞ , P(0) = p ∞ and Q(0) = q ∞ .
Our autonomous dynamical system can then be described by the following system of equations:

               v(t) = P(s(t))S(x(t)) -Q(s(t))v(t) ṁ(t) = H(s(t))∇F (x(t)) -R(s(t))m(t) ẋ(t) = -m(t) √ v(t)+ε ṡ(t) = -s(t) 2
(3.17)

Since the solution of the ODE ṡ(t) = -s(t) 2 for which s(t 0 ) = 1/t 0 is s(t) = 1/t, the trajectory {s(t)} is bounded. The three remaining equations are a reformulation of (ODE-1) for which the trajectories have already been shown to exist and to be bounded in Lem. 3.13. In the sequel, we denote by Φ : Z + × R + → Z + × R + the semiflow induced by the autonomous ODE (3.17), i.e., for every u = (z, s)

∈ Z + × R + , Φ(u, •)
is the unique global solution to the autonomous ODE (3.17) initialized at u. Observe that the orbits of this semiflow are precompact. Moreover, the function Φ((z, 0), •) is perfectly defined for each z ∈ Z + since the associated solution satisfies the ODE (3.19) defined below, which three first equations satisfy the hypotheses of Lem. 3.13.

Consider now a continuous function V : Z + × R + → R defined by:

V (u) = E H(s), z , u = (z, s) ∈ Z + × (0, ∞).
As for Ineq. (3.16) above, we have here that

d dt V Φ(u, t) ≤ -r(t) - q(t) 4 m(t) (v(t) + ε) 1 4 2 + ḣ(t)(F (x(t)) -F ) - p(t) 4 S(x(t)), m(t) 2 (v(t) + ε) 3 2
if s > 0, and the same inequality with ( ḣ(t), p(t), r(t), q(t)) being replaced with (0, p ∞ , r ∞ , q ∞ ) otherwise.

Since V •Φ(u, •) is non-increasing and nonnegative, we can define V ∞ := lim t→∞ V (Φ(u, t)).

Let ω(u) := s>0 t≥s Φ(u, t) be the ω-limit set of the semiflow Φ issued from u. Recall that ω(u) is an invariant set for the flow Φ(u, •), and that

dist(Φ(u, t), ω(u)) ---→ t→∞ 0,
see, e.g., (Haraux, 1991, Th. 1.1.8)). In order to finish the proof of Th. 3.1, we need to make explicit the structure of ω(u).

We know from La Salle's invariance principle that ω(u) ⊂ V -1 (V ∞ ). In particular,

∀y ∈ ω(u), ∀t ≥ 0, V (Φ(y, t)) = V (y) = V ∞ (3.18)
by the invariance of ω(u).

From ODE (3.17), we have that any y ∈ ω(u) is of the form y = (z, 0) since s(t) → 0.

As a consequence, Φ(y, •) is a solution to the autonomous ODE

               v(t) = p ∞ S(x(t)) -q ∞ v(t) ṁ(t) = h ∞ ∇F (x(t)) -r ∞ m(t) ẋ(t) = -m(t) √ v(t)+ε ṡ(t) = 0 . (3.19)
The three first equations can be written in a more compact form :

ż(t) = g ∞ (z(t)) (3.20)
where z(t) = (v(t), m(t), x(t)), and

g ∞ (z) = lim t→∞ g(z, t) =    p ∞ S(x) -q ∞ v h ∞ ∇F (x) -r ∞ m -m/ √ v + ε   
for each z ∈ Z + . Consider y = (v, m, x, 0) ∈ ω(u). Using Eq. (3.18), we obtain that dV (Φ(y, t))/dt = 0, which implies that

r ∞ - q ∞ 4 m(t) (v(t) + ε) 1 4 2 + p ∞ 4 S(x(t)), m(t) 2 (v(t) + ε) 3 2 = 0
for all (v(t), m(t), x(t), 0) = Φ(y, t). As a consequence, Assumption 3.2.4-iv) gives m(t) = m = 0, and then, x(t) = x for some x s.t. ∇F (x) = 0 using ODE (3.19). We now turn to showing that v(t) = v = p ∞ S(x)/q ∞ . We have proved so far that any element y ∈ ω(u) is written y = (v, 0, x, 0) where

∇F (x) = 0. The component v(•) of Φ(y, •) is a solution to the ODE v(t) = p ∞ S(x) -q ∞ v(t)
and is thus written

v(t) = p ∞ S(x) q ∞ + e -q∞t v - p ∞ S(x) q ∞ . (3.21)
Fixing x, let S x be the section of ω(u) defined by:

S x ω(u) = y ∈ ω(u) : y = (ṽ, 0, x, 0) , ṽ ∈ R d + .
As ω(u) is invariant, we have S x ω(u) = S x Φ(ω(u), t) for all t ≥ 0. Since the set {ṽ ∈ R d + s.t. (ṽ, 0, x, 0) ∈ S x ω(u)} lies in a compact, we deduce from Eq. ( 3.21) that this set is reduced to the singleton {p ∞ S(x)/q ∞ } and in particular v = p ∞ S(x)/q ∞ . Therefore, the union of ω-limit sets of the semiflow Φ induced by ODE (3.17) coincides with the set of equilibrium points of this semiflow. The latter set itself corresponds to the set of points (z, 0) s.t. z ∈ zer g ∞ . It remains to notice that Υ = zer g ∞ to finish the proof.

Remark 14. Commenting on Remark 4, the same proof works for (ODE-1 ) by using the function F -F as a Lyapunov function. The corresponding limit set (as t → +∞) is then of the form

{z ∞ = (ṽ ∞ , x∞ ) ∈ R d + × R d : ∇F (x ∞ ) = 0 , ṽ∞ = p ∞ S(x ∞ )/q ∞ }.
Similarly, if we set p = q ≡ 0 in (ODE-1) and we keep what remains in Assumption 3.2.4, the function h(t)(F (x) -F ) + 1 2 m 2 works as a Lyapunov function, and the limit set has the form {(0, x) : ∇F (x) = 0}.

Proof of Th. 3.2

The existence and the uniqueness of the solution to (ODE-N) have been shown in the literature. We refer to (Cabot et al., 2009, Prop. 2.1-2.2.c)) for an identical statement of this result and (Su et al., 2016, Th. 1, Appendix A) for a complete proof. The boundedness of the solution follows immediately from the coercivity of F together with the fact that the function t → F (x(t)) + 1 2 m(t) 2 is nonincreasing. Concerning the convergence statement, our proof is based on comparing the solutions of (ODE-N) to the solutions of the ODE in (Gadat et al., 2018, Eq. (2.3)). We first note that under a change of variable, a solution to (ODE-N) gives a solution to (Gadat et al., 2018, Eq. (2.3)).

Lemma 3.14. Let (m, x) be a solution to (ODE-N). Define

y(t) = κm κ √ t 2 √ t , u(t) = x κ √ t , with κ = √ 2α + 2 and β = κ 2 4 . Then, (y, u) verifies    ẏ(t) = β t (∇F (u(t))) -y(t)) u(t) = -y(t) . (3.22)
Proof. By simple differentiation, we get:

ẏ(t) = β t ∇F x(κ √ t) - α κ √ t m κ √ t - κ 4t 3 2 m κ √ t = β t ∇F (u(t)) -y(t) , u(t) = - κ 2 √ t m κ √ t = -y(t) .
Consider a solution (m, x) of (ODE-N) starting at (m 0 , x 0 ) ∈ R d × R d . As in Section 3.5.1.2, for every t 0 > 0, on [t 0 , +∞), we have that (m, x, s) is a solution to the autonomous ODE

       ṁ(t) = ∇F (x(t)) -αs(t)m(t) ẋ(t) = -m(t) ṡ(t) = -s(t) 2 , (3.23)
starting at (m 0 , x 0 , 1/t 0 ). Denote by

Φ N = (Φ m N , Φ x N , Φ s N )
the semiflow induced by ODE (3.23) and ω N ((m 0 , x 0 , 1/t 0 )) its limit set.

Define (y, u) as in Lem. 3.14. Starting at (y(t 0 ), u(t 0 ), 1/t 0 ), we also have that (y, u, s) is a solution on [t 0 , +∞) to the "autonomized" Heavy-Ball ODE

       ẏ(t) = βs(t)(∇F (u(t))) -y(t)) u(t) = -y(t) ṡ(t) = -s(t) 2 . (3.24) Denote by Φ H = (Φ y H , Φ u H , Φ s H
) the semiflow induced by ODE (3.24) and its limit set ω H ((y(t 0 ), u(t 0 ), 1/t 0 )). Lemma 3.15. For any compact set K ⊂ R 2d+1 and any T > 0, the family of functions Let (m, x, 0) ∈ ω N ((m 0 , x 0 , 1/t 0 )). There exists a sequence (t k ) of nonnegative reals such that (m, x, 0) = lim k→∞ (m(t k ), x(t k ), 1/t k ). For any T > 0 , using Lem. 3.15, up to an extraction, we can say that the sequence of functions

Φ(z, •) : [0, T ] → R 2d+1 z∈K , where Φ is either Φ H or Φ N , is relatively compact in (C 0 ([0, T ], R 2d+1 ), • ∞ ).
{Φ N ((m(t k ), x(t k ), 1/t k ), •)} k converges towards ( m, x, 0) in C 0 ([0, T ], R d ), where ( m, x) is a solution to    ṁ(t) = ∇F (x(t)) ẋ(t) = -m(t) , (3.25)
with ( m(0), x(0)) = (m, x). Moreover, by Lem. 3.14, we also have that:

sup h∈[0,T 2 /κ 2 ] x(κ √ h) -Φ x N ((m(t k ), x(t k ), 1/t k ), κ √ h) = sup h∈[0,T 2 /κ 2 ] x(κ √ h) -Φ u H ((m(t k ), x(t k ), 1/t k ), h) ----→ k→+∞ 0 . (3.26)
Using Lem. 3.15, up to an additional extraction, we get on C 0 ([0,

T 2 /κ 2 ], R 2d+1 ) that {Φ H ((x(t k ), m(t k ), 1/t k ), •)} k converges to (u, y, 0), where (u, y) is a solution to    ẏ(t) = 0 u(t) = -y(t) . (3.27)
Therefore, u(t) = A + Bt for some A and B in R d . Imagine that B = 0. We previously proved that x (and therefore u) is bounded by some constant This implies that u is constant. Then, if we go back to Eqs. (3.26) and (3.25), we get that x is constant, hence m ≡ 0 and then ∇F (x) ≡ 0. In particular, this means that m = m(0) = 0 and ∇F (x) = ∇F (x(0)) = 0.

C > 0. Let T > C+ A B . Up to an extraction, we obtain that {Φ H ((x(t k ), m(t k ), 1/t k ), •)} k converges to u on C 0 ([0, T ], R 2d+1
3.6 Proofs for Section 3.3

Preliminaries

We first recall some useful definitions and results. Let Ψ represent any semiflow on an arbitrary metric space (E, d). As in the previous section, a point z ∈ E is called an equilibrium point of the semiflow Ψ if Ψ(z, t) = z for all t ≥ 0. We denote by Λ Ψ the set of equilibrium points of

Ψ. A continuous function V : E → R is called a Lyapunov function for the semiflow Ψ if V(Ψ(z, t)) ≤ V(z) for all z ∈ E and all t ≥ 0. It is called a strict Lyapunov function if, moreover, {z ∈ E : ∀t ≥ 0, V(Ψ(z, t)) = V(z)} = Λ Ψ . If V is a strict Lyapunov function for Ψ and if z ∈ E is a point s.t.
{Ψ(z, t) : t ≥ 0} is relatively compact, then it holds that Λ Ψ = ∅ and d(Ψ(z, t), Λ Ψ ) → 0, see (Haraux, 1991, Th. 2.1.7). A continuous function z : [0, +∞) → E is said to be an asymptotic pseudotrajectory (APT, [START_REF] Benaïm | Asymptotic pseudotrajectories and chain recurrent flows, with applications[END_REF]) for the semiflow Ψ if lim t→+∞ sup s∈[0,T ] d(z(t + s), Ψ(z(t), s)) = 0 for every T ∈ (0, +∞) .

Proof of Th. 3.3

Recall that Φ is the semiflow induced by the autonomous ODE (3.17) which is an "autonomized" version of our initial (ODE-1). In the remainder of this section, the proof will be divided into two main steps : (a) we show that a certain continuous-time linearly interpolated process constructed from the iterates of our algorithm 3.1 is an APT of Φ; (b) we exhibit a strict Lyapunov function for a restriction to a carefully chosen compact set of a well chosen semiflow related to Φ. Then, we characterize the limit set of the APT using (Benaïm, 1999, Th. 5.7) and (Benaïm, 1999, Prop. 6.4). The sequence (z n ) converges almost surely to this same limit set.

(a) APT. For every n ≥ 1, define zn = (v n , m n , x n-1 ) (note the shift in the index of the variable x). We have the decomposition

zn+1 = zn + γ n+1 g(z n , τ n ) + γ n+1 η n+1 + γ n+1 ς n+1 ,
where g is defined in Eq. (3.1),

η n+1 = p n (∇f (x n , ξ n+1 ) 2 -S(x n )), h n (∇f (x n , ξ n+1 ) -∇F (x n )), 0 , (3.28)
is a martingale increment and where we set ς n+1 = (ς v n+1 , ς m n+1 , ς x n+1 ) with the components defined by:

         ς v n+1 = p n (S(x n ) -S(x n-1 )) ς m n+1 = h n (∇F (x n ) -∇F (x n-1 )) ς x n+1 = ( γn γ n+1 -1) mn √ vn+ε .
We first prove that ς n → 0 a.s. by considering the components separately. The components ς m n+1 and ς v n+1 converge a.s. to zero by using Assumptions 3.2.1, 3.2.3, together with the boundedness of the sequences (p n ) and (h n ) (which are both convergent). Indeed, since ∇F is locally Lipschitz continuous and the sequence (z n ) is supposed to be almost surely bounded, there exists a constant C s.t.

∇F (x n ) -∇F (x n-1 ) ≤ C x n -x n-1 ≤ C ε γ n m n .
The same inequality holds when replacing ∇F by S which is also locally Lipschitz continuous. The component ς x n+1 also converges a.s. to zero by observing that ς x n+1 ≤ |1 -γn γ n+1 |. m n / √ ε and using Assumption 3.3.2 together with the a.s. boundedness of (z n ). Now consider the martingale increment sequence (η n ), adapted to F n . Take δ > 0. Since (z n ) is a.s bounded, there is a constant C > 0 such that P(sup x n > C ) ≤ δ. Denoting ηn := η n 1 xn ≤C and combining Assumptions 3.2.4 with 3.3.4-i) we can show using convexity inequalities that

sup n E ηn+1 q < ∞.
Then, we deduce from this result together with the corresponding stepsize assumption from 3.3.4-i) and (Benaïm, 1999, Prop. 4.2) (see also (Métivier and Priouret, 1987, Prop. 8)) the key property:

∀T > 0 , max L-1 k=n γ k+1 ηk+1 : L = n + 1, . . . , J(τ n + T ) a.s. ---→ n→∞ 0 (3.29)
where J(t) = max{n ≥ 0 : τ n ≤ t}. Hence, for all T > 0, with probability at least 1δ :

max L-1 k=n γ k+1 η k+1 : L = n + 1, . . . , J(τ n + T ) ---→ n→∞ 0 . (3.30)
Since δ can be chosen arbitrary small, Eq. (3.30) remains true with probability 1. This result also holds under Assumption 3.3.4-ii) (instead of 3.3.4-i)) by applying (Benaïm, 1999, Prop. 4.4).

Let z : [0, +∞) → Z + be the continous-time linearly interpolated process given by

z(t) = zn + (t -τ n ) zn+1 -zn γ n+1 ∀n ∈ N , ∀t ∈ [τ n , τ n+1 ) (where τ n = n k=1 γ k ). Let t 0 > 0. Define u : [t 0 , ∞) → Z × (0, 1/t 0 ] by u(t) = z(t) 1/t , for t ≥ t 0 > 0.
Using Eq. (3.30) and the almost sure boundedness of the sequence (z n ) along with the fact that ς n converges a.s. to zero, it follows from (Benaïm, 1999, Prop. 4.1, Remark 4.5) that u(t) is an APT of the already defined semiflow Φ induced by (3.17). Remark that it also holds that z(t) is an APT of the semiflow Φ ∞ induced by (3.20). As the trajectory of u(t) is precompact, the limit set

L(u) = t≥t 0 u([t, ∞))
is compact. Moreover, it has the form

L(u) = S 0
, where S :=

t≥t 0 z([t, ∞)) . (3.31)
Our objective now is to prove that

S ⊂ Λ Φ ∞ . (3.32)
In order to establish this inclusion, we study the behavior of the restriction Φ|L of the semiflow Φ to the set L (which is well-defined since L is Φ-invariant). Remark that

Φ|L = Φ ∞ |S 0 ,
where Φ ∞ is the semiflow associated to (3.20). In the second part of the proof, we establish Eq. ( 3.32) combining item (a) we just proved with (Benaïm, 1999, Th. 5.7) and (Benaïm, 1999, Prop. 6.4). In order to use the latter proposition, we prove a useful proposition in item (b).

(b) Strict Lyapunov function and convergence. For every δ > 0 and every z = (v, m, x) ∈ Z + , define:

W δ (v, m, x) := E ∞ (z) -δ ∇F (x), m + δ q ∞ v -p ∞ S(x) 2 , (3.33)
where, under Assumption 3.2.4-i), the function E ∞ is defined by

E ∞ (z) := lim t→+∞ E(t, z) = h ∞ (F (x) -F ) + 1 2 m (v + ε) 1 4 2 . ( 3
∞ : S ×[t 0 , +∞) → S be the restriction of the semiflow Φ ∞ to S i.e., Φ ∞ (z, t) = Φ ∞ (z, t) for all z ∈ S, t ≥ t 0 .Then, i) S is compact. ii) Φ ∞ is a well-defined semiflow on S.
iii) The set of equilibrium points of Φ ∞ is equal to Λ Φ ∞ ∩ S.

iv) There exists δ > 0 s.t. W δ is a strict Lyapunov function for the semiflow Φ ∞ .

Proof. The first point is a consequence of the definition of S and the boundedness of z. The second point stems from the definition of Φ ∞ . Observing that Φ ∞ is valued in S, the third point is immediate from the definition of Λ Φ ∞ . We now prove the last point. Consider z ∈ S and write Φ ∞ (z, t) under the form Φ ∞ (z, t) = (v(t), m(t), x(t)). Notice that this quantity is bounded as a function of the variable t. For any map W :

Z + → R, define for all t ≥ t 0 , L W (t) := lim sup s→0 s -1 (W(Φ ∞ (z, t + s)) -W(Φ ∞ (z, t))) . Introduce G(z) := -∇F (x), m and H(z) := q ∞ v -p ∞ S(x) 2 for every z = (v, m, x) ∈ Z + .
Consider δ > 0 (to be specified later on). We study L W δ = L E∞ + δL G + δL H . Note that Φ ∞ (z, t) ∈ S ∩ Z + for all t ≥ t 0 by an analogous result to Lem. 3.12 for Φ ∞ . Thus,

t → E ∞ (Φ ∞ (z, t)) is differentiable at any point t ≥ t 0 and L E∞ (t) = d dt E ∞ (Φ ∞ (z, t)).
Using similar derivations to Ineq. (3.16), we obtain that .35) We now study L G . For every t ≥ t 0 ,

L E∞ (t) ≤ -r ∞ - q ∞ 4 m(t) (v(t) + ε) 1 4 2 . ( 3
L G (t) = lim sup s→0 s -1 (-∇F (x(t + s)), m(t + s) + ∇F (x(t)), m(t) ) ≤ lim sup s→0 s -1 ∇F (x(t)) -∇F (x(t + s)) m(t + s) -∇F (x(t)), ṁ(t) .
Let L ∇F be the Lipschitz constant of ∇F on the bounded set {x : (v, m, x) ∈ S}. Define

C 1 := sup t v(t) + ε . Then, L G (t) ≤ L ∇F lim sup s→0 s -1 x(t) -x(t + s) m(t + s) -∇F (x(t)), ṁ(t) ≤ L ∇F ẋ(t) m(t) -∇F (x(t)), ṁ(t) ≤ L ∇F ẋ(t) m(t) -h ∞ ∇F (x(t)) 2 + r ∞ ∇F (x(t)), m(t) ≤    L ∇F C 1 2 1 ε 1 4 + r ∞ C 1 2u 2 1    m(t) (v(t) + ε) 1 4 2 -h ∞ - r ∞ u 2 1 2 ∇F (x(t)) 2 (3.36)
where we used the classical inequality | a, b | ≤ a 2 /(2u 2 ) + u 2 b 2 /2 for any non-zero real u to derive the last above inequality. We now study L H . For every t ≥ t 0 ,

L H (t) = lim sup s→0 s -1 ( q ∞ v(t + s) -p ∞ S(x(t + s)) 2 -q ∞ v(t) -p ∞ S(x(t)) 2 ) = lim sup s→0 s -1 (p 2 ∞ S(x(t)) -S(x(t + s)) 2 + 2p ∞ S(x(t)) -S(x(t + s)), q ∞ v(t + s) -p ∞ S(x(t)) ) + lim s→0 s -1 ( q ∞ v(t + s) -p ∞ S(x(t)) 2 -q ∞ v(t) -p ∞ S(x(t)) 2 ) .
The second term in the righthand side coincides with

-2q ∞ p ∞ S(x(t)) -q ∞ v(t), v(t) = -2q ∞ p ∞ S(x(t)) -q ∞ v(t) 2 .
Denote by L S the Lipschitz constant of S on the set {x : (v, m, x) ∈ S}. Note that s -1 ( S(x(t + s)) -S(x(t)) 2 ) ≤ L 2 S s s -1 (x(t + s)x(t)) 2 which converges to zero as s → 0. Thus,

L H (t) = -2q ∞ p ∞ S(x(t)) -q ∞ v(t) 2 + lim sup s→0 2p ∞ s -1 S(x(t)) -S(x(t + s)), q ∞ v(t + s) -p ∞ S(x(t)) ≤ -2q ∞ p ∞ S(x(t)) -q ∞ v(t) 2 + 2p ∞ ẋ(t) L S q ∞ v(t) -p ∞ S(x(t)) ≤ p ∞ ε 1 2 u 2 2 m(t) (v(t) + ε) 1 4 2 -(2q ∞ -p ∞ u 2 2 L 2 S ) p ∞ S(x(t)) -q ∞ v(t)
L W δ (t) ≤ -M (δ) m(t) (v(t) + ε) 1 4 2 -δ h ∞ - r ∞ u 2 1 2 ∇F (x(t)) 2 -δ 2q ∞ -p ∞ u 2 2 L 2 S p ∞ S(x(t)) -q ∞ v(t) 2 . (3.38)
where

M (δ) := r ∞ -q∞ 4 -δ   r∞C 1 2u 2 1 + L ∇F C 1 2 1 ε 1 4 + p∞ ε 1 2 u 2 2   . Now select u 1 , u 2 small enough s.t. h ∞ -r ∞ u 2 1 /2 > 0 and 2q ∞ -p ∞ u 2 2 L 2 S > 0.
Then, choose δ in such a way that M (δ) > 0. Thus, there exists a constant c depending on δ s.t. for every t ≥ t 0 ,

L W δ (t) ≤ -c     m(t) (v(t) + ε) 1 4 2 + ∇F (x(t)) 2 + p ∞ S(x(t)) -q ∞ v(t) 2     . (3.39)
It can easily be seen that for every z ∈ S, t → W δ (Φ ∞ (z, t)) is Lipschitz continuous, hence absolutely continuous. Its derivative almost everywhere coincides with L W δ , which is nonpositive. Thus, W δ is a Lyapunov function for Φ ∞ . We prove that the Lyapunov function is strict.

Consider z = (v, m, x) ∈ S s.t. W δ (Φ ∞ (z, t)) = W δ (z) for all t ≥ t 0 . The derivative almost everywhere of t → W δ (Φ ∞ (z, t)) is identically zero,
and by Eq. (3.39), this implies that

-c     m(t) (v(t) + ε) 1 4 2 + ∇F (x(t)) 2 + p ∞ S(x(t)) -q ∞ v(t) 2    
is equal to zero for every t ≥ t 0 a.e. (hence, for every t ≥ t 0 , by continuity of Φ ∞ ). In particular for t = t 0 , m = ∇F (x) = 0 and p ∞ S(x)q ∞ v = 0. Hence, z ∈ zer g ∞ ∩ S. This concludes the proof since Λ Φ ∞ = zer g ∞ .

End of the Proof of Th. (Benaïm, 1999, Prop. 6.4) that S ⊂ ΛΦ∞. This gives the desired result (Eq. (3.32)) given Prop. 3.16-iii).

The last assertion of Th. 3.3 is a consequence of (Benaïm, 1999, Cor. 6.6).

Proof of Th. 3.5

We can rewrite the iterates from Algorithm 3.2 as follows:

   m n+1 = m n + γ n+1 (∇F (x n ) -α τn m n ) + γ n+1 (∇f (x n , ξ n+1 ) -∇F (x n )) x n+1 = x n -γ n+1 m n+1 .
(3.40)

We prove that the sequence (y n = (m n , x n ) : n ∈ N) of iterates of this algorithm converges almost surely towards the set Ῡ defined in Eq. ( 3.3) if it is supposed to be bounded with probability one. The proof follows a similar path to the proof in Section 3.5.2.

Indeed, denote by X and M the linearly interpolated processes constructed respectively from the sequences (x n ) and (m n ) and let s(t) = 1/t. Recall that Φ N = (Φ m N , Φ x N , Φ s N ) is the semiflow induced by (3.23). As in Section 3.6.2, we have that Z := (M, X, s) is an APT of (3.23). In particular, this means that

∀T > 0 , sup h∈[0,T ] X(t + h) -Φ x N (Z(t), h) ---→ t→∞ 0 . (3.41)
By Lem. 3.14, we also have that

sup h∈[0,T 2 /κ 2 ] X(t + κ √ h) -Φ x N (Z(t), κ √ h) = sup h∈[0,T 2 /κ 2 ] X(t + κ √ h) -Φ u H (Z(t), h) ---→ t→∞ 0 . (3.42)
Let (m, x) be a limit point of the sequence (y n ) and let T > 0. Using Lem. 3.15, we can proceed in the same manner as in Section 3.5.2 and get a sequence (t k ) such that

(M(t k + •), X(t k + •)) → (m, x) and (Φ y H (Z(t k ), •), Φ u H (Z(t k ), •)) → (y, u) ,
where (m(0), x(0)) = (m, x) , and (m, x) and (x, u) are respectively solutions to (3.25) and (3.27). As in the end of Section 3.5.2, we obtain that u and x are constant, therefore m ≡ 0 and ∇F (x) ≡ 0 , which finishes the proof.

Proof of Th. 3.4

The idea of the proof is to apply Robbins-Siegmund's theorem [START_REF] Robbins | A convergence theorem for non negative almost supermartingales and some applications[END_REF] to

V n = h n-1 F (x n ) + 1 2 m 2 n , 1 √ v n + ε
(note the similarity of V n with the energy function (3.15)). Since inf F > -∞, we assume without loss of generality that F ≥ 0. In this subsection, we use the notation ∇f n+1 as a shorthand notation for ∇f (x n , ξ n+1 ) and C denotes some positive constant which may change from line to line. We write

E n = E[• | F n ]
for the conditional expectation w.r.t the σ-algebra F n . Define P n := 1 2 D n , m 2 n , with D n := 1 √ vn+ε . We have the decomposition:

P n+1 -P n = 1 2 D n+1 -D n , m 2 n+1 + 1 2 D n , m 2 n+1 -m 2 n . (3.43)
We estimate the vector

D n+1 -D n = √ v n + ε - √ v n+1 + ε √ v n+1 + ε √ v n + ε .
Remarking that v n+1 ≥ (1γ n+1 q n )v n and using the update rule of v n , we obtain for a sufficiently large n that

√ v n + ε - √ v n+1 + ε = γ n+1 q n v n -p n ∇f 2 n+1 √ v n + ε + √ v n+1 + ε ≤ γ n+1 q n v n (1 + √ 1 -γ n+1 q n ) √ v n + ε = γ n+1 q n 1 + √ 1 -γ n+1 q n √ v n √ v n √ v n + ε ≤ c n+1 √ v n+1 , (3.44)
where c n+1 :=

γ n+1 qn √ 1-γ n+1 qn(1+ √ 1-γ n+1 qn) .
It is easy to see that c n+1 /γ n → q ∞ /2. Thus, for any δ > 0, c n+1 ≤ (q ∞ + 2δ)γ n /2 for all n large enough. Using also that

√ v n+1 / √ v n+1 + ε ≤ 1, we obtain D n+1 -D n ≤ q ∞ + 2δ 2 γ n D n . (3.45)
Substituting the above inequality in Eq. ( 3.43), we obtain

P n+1 -P n ≤ q ∞ + 2δ 2 γ n 2 D n , m 2 n+1 + 1 2 D n , m 2 n+1 -m 2 n ≤ q ∞ + 2δ 2 γ n P n + 1 + q ∞ + 2δ 2 γ n 1 2 D n , m 2 n+1 -m 2 n . Using m 2 n+1 -m 2 n = 2m n (m n+1 -m n ) + (m n+1 -m n ) 2 , and noting that E n (m n+1 - m n ) = γ n+1 h n ∇F (x n ) -γ n+1 r n m n , E n 1 2 D n , m 2 n+1 -m 2 n = γ n+1 h n ∇F (x n ), m n √ v n + ε -2γ n+1 r n P n + 1 2 D n , E n [(m n+1 -m n ) 2 ] .
There exists δ > 0 such that r ∞ -q∞ 4 -δ 2 > 0 by Assumption 3.2.4-iv). As

γ n+1 γn r n -q∞ 4 → r ∞ -q∞ 4 , for all n large enough, γ n+1 γn r n -q∞ 4 > r ∞ -q∞ 4 -δ 2 > 0.
Hence, for all n large enough,

E n P n+1 -P n ≤ -2 r ∞ - q ∞ 4 - δ 2 γ n P n + γ n+1 h n ∇F (x n ), m n √ v n + ε + Cγ 2 n ∇F (x n ), m n √ v n + ε + C D n , E n [(m n+1 -m n ) 2 ] . (3.46)
Using the inequality u, v ≤ ( u 2 + v 2 )/2 and Assumption 3.3.6-ii), it is easy to show the inequality ∇F (x n ), mn

√ vn+ε ≤ C(1 + F (x n ) + P n ). Moreover, using the componentwise inequality (h n ∇f n+1 -r n m n ) 2 ≤ 2h 2 n ∇f 2 n+1 + 2r 2 n m 2
n along with Assumption 3.3.6-ii) and the boundedness of the sequences (h n ), (r n ) and (γ n+1 /γ n ), we obtain

D n , E n [(m n+1 -m n ) 2 ] ≤ Cγ 2 n (1 + F (x n ) + P n ) . (3.47)
Combining Eq. (3.46) and Eq. ( 3.47), we get

E n (P n+1 -P n ) ≤ γ n+1 h n ∇F (x n ), m n D n + Cγ 2 n (1 + F (x n ) + P n ) . (3.48)
Denoting by M the Lipschitz coefficient of ∇F , we also have

F (x n+1 ) ≤ F (x n ) -γ n+1 ∇F (x n ), m n+1 D n+1 + γ 2 n+1 M 2 m n+1 D n+1 2 .
(3.49)

Using (3.45) and the update rule of m n , we have

m n+1 D n+1 -m n D n 2 ≤ C (m n+1 -m n ) D n 2 + C m n+1 (D n+1 -D n ) 2 ≤ Cγ 2 n+1 ( ∇f n+1 2 + m n D n 2 ) + Cγ 2 n+1 m n+1 D n 2 ≤ Cγ 2 n+1 ( m n D n 2 + ∇f n+1 2
) .

(3.50)

Finally, recalling that 3.48),(3.49),(3.50), and using Assumption 3.3.6, we have

V n = h n-1 F (x n )+P n , (h n ) is decreasing, combining Eq. (
E n [V n+1 ] ≤ V n + γ n+1 h n ∇F (x n ), E n m n D n -m n+1 D n+1 + Cγ 2 n+1 1 + F (x n ) + P n + m n D n 2 + Cγ 2 n+1 E n [ m n D n -m n+1 D n+1 2 ] ≤ V n + Cγ 2 n   1 + F (x n ) + P n + m n D n 2 + E n ∇f n+1 2   ≤ V n + Cγ 2 n (1 + F (x n ) + P n ) ≤ (1 + Cγ 2 n )V n + Cγ 2 n ,
where we used Cauchy-Schwarz's inequality and the fact that m n D n 2 ≤ CP n . By the Robbins-Siegmund's theorem, the sequence (V n ) converges almost surely to a finite random variable V ∞ ∈ R + . Then, the coercivity of F implies that (x n ) is almost surely bounded.

We now establish the almost sure boundedness of (m n ). Assume in the sequel that n is large enough to have (1γ n+1 r n ) ≥ 0. Consider the martingale difference sequence ∆ n+1 := ∇f n+1 -∇F (x n ). We decompose m n = mn + mn where mn+1 = (1γ n+1 r n ) mn +γ n+1 h n ∇F (x n ) and mn+1 = (1-γ n+1 r n ) mn +γ n+1 h n ∆ n+1 , setting m0 = 0 and m0 = m 0 . We prove that both terms mn and mn are bounded. Consider the first term:

mn+1 ≤ (1 -γ n+1 r n ) mn + γ n+1 sup k h k ∇F (x k )
, where the supremum in the above inequality is almost surely finite by continuity of ∇F . We immediately get that if mn ≥ sup k h k ∇F (x k ) r∞ , then mn+1 ≤ mn . Thus

mn+1 ≤ sup k h k ∇F (x k ) r ∞ + sup k γ k+1 h k ∇F (x k ) ,
which implies that mn is bounded. Consider now the term mn :

E n [ mn+1 2 ] = (1 -γ n+1 r n ) 2 mn 2 + γ 2 n+1 h 2 n E n [ ∆ n+1 2 ] ≤ mn 2 + γ 2 n+1 h 2 n E n [ ∆ n+1 2 ] .
Then, the inequality

E n [ ∆ n+1 2 ] ≤ E n [ ∇f n+1 2 ]
combined with Assumption 3.3.4-i) and the a.s. boundedness of the sequence (x n ) imply that there exists a finite random variable

C K (independent of n) s.t. E n [ ∇f n+1 2 ] ≤ C K .
As a consequence, since n γ 2 n+1 < ∞ and the sequence (h n ) is bounded, we obtain that a.s.:

n≥0 γ 2 n+1 h 2 n E n [ ∆ n+1 2 ] ≤ CC K n≥0 γ 2 n+1 < +∞ .
Hence, we can apply the Robbins-Siegmund theorem to obtain that sup n mn 2 < ∞ w.p.1. Finally, it can be shown that (v n ) is almost surely bounded using the same arguments, decomposing v n into vn + ṽn as above. Indeed, first, we have:

E n [ ṽn+1 2 ] ≤ ṽn 2 + γ 2 n+1 p 2 n E n [ ∇f 2 n+1 -S(x n ) 2 ] .
Second, it also holds that:

E n [ ∇f 2 n+1 -S(x n ) 2 ] ≤ E n [ ∇f 2 n+1 2 ] ≤ E n [ ∇f n+1 4 ] .
Then, using Assumption 3.3.4-i) and the a.s. boundedness of the sequence (x n ), there exists a finite random variable

C K (independent of n) s.t. E n [ ∇f n+1 4 ] ≤ C K . Moreover, the sequence (p n ) is bounded and n γ 2 n+1 < ∞.
As a consequence, it holds that a.s:

n≥0 γ 2 n+1 p 2 n E n [ ∇f 2 n+1 -S(x n ) 2 ] ≤ CC K n≥0 γ 2 n+1 < +∞ .
It follows that the Robbins-Siegmund theorem can be applied to the sequence ṽn 2 as for the sequence mn 2 to obtain that sup n ṽn 2 < ∞ w.p.1.

Proof of Th. 3.6

The proof of Th. 3.4 easily adapts to Algorithm 3.2 by replacing V n by

Ṽn := F (x n ) + 1 2 m n 2 .
The boundedness of (m n ) is an immediate consequence of the convergence of Ṽn .

Proof of Th. 3.7

We shall use the following result.

Theorem 3.17 (adapted from Pelletier (1998), Th. 7). Let k ≥ 1. On some probability space equipped with a filtration F = (F n ) n∈N , consider a sequence of r.v. on R k given by

Z n+1 = (I + γ n+1 H)Z n + γ n+1 b n+1 + √ γ n+1 η n+1 and E[ Z 0 2 ] < ∞,
where H is a k × k Hurwitz matrix, (b n ) and (η n ) are random sequences, and γ n = γ 0 n -α for some γ 0 > 0 and α ∈ (0, 1]. Let Ω 0 ∈ F ∞ have a positive probability. Assume that the following holds almost surely on Ω 0 :

i) E[η n+1 |F n ] = 0. ii) There exists a constant b > 2 s.t. sup n≥0 E[ η n+1 b|F n ] < ∞. iii) E[η n+1 η T n+1 |F n ] = Σ + ∆ n where E[ ∆ n 1 Ω 0 ] → 0 and Σ is a positive semidefinite matrix.
iv) The sequence (b n ) is the sum of two sequences (b n,1 ) and (b n,2 ), adapted to F, s.t.

sup n≥0 E[ b n,1 2 ] < ∞, E[ b n,1 1 Ω 0 ] → 0 and b n,2 → 0 a.s. on Ω 0 .
Then, given Ω 0 , (Z n ) converges in distribution to the unique stationary distribution µ of the generalized Ornstein-Uhlenbeck process

dX t = HX t dt + √ ΣdB t
where (B t ) is the standard Brownian motion and √ Σ is the unique positive semidefinite square root of Σ. The distribution µ is the zero mean Gaussian distribution with covariance matrix Γ given as the solution to

( H + 1 α=1 2γ 0 I k )Γ + Γ( H + 1 α=1 2γ 0 I k ) T = -Σ.
Proof.

The proof is identical to the proof of (Pelletier, 1998, Th. 7), only substituting the inverse of the square root of Σ by the Moore-Penrose inverse. Finally, the uniqueness of the stationary distribution µ and its expression follow from (Karatzas and Shreve, 1991, Th. 6.7, p. 357).

We define v n = vn + δ n where δ 0 = 0, v0 = v 0 and

δ n+1 = (1 -γ n+1 q n )δ n + γ n+1 (p n -q n q -1 ∞ p ∞ )S(x n ) , vn+1 = (1 -γ n+1 q n )v n + γ n+1 q n q -1 ∞ p ∞ S(x n ) + γ n+1 p n (∇f (x n , ξ n+1 ) 2 -S(x n )) .
For every z = (v, m, x) ∈ Z + and δ ≥ 0, we define

r n (z, δ) :=     q n q -1 ∞ p ∞ (S(x -γ n m √ v+δ+ε ) -S(x)) h n (∇F (x -γ n m √ v+δ+ε ) -∇F (x)) γn γ n+1 ( 1 √ v+ε - 1 √ v+δ+ε ) m     .
Moreover, for every z = (v, m, x) ∈ Z + and every n ∈ N, we set

g n (z) =     q n q -1 ∞ p ∞ S(x) -q n v h n ∇F (x) -r n m -γn γ n+1 m √ v+ε     . Defining ζ n = (v n , m n , x n-1
) and recalling the definition of (η n ) from Eq. (3.28), we have the decomposition

ζ n+1 = ζ n + γ n+1 g n (ζ n ) + γ n+1 η n+1 + γ n+1 r n (ζ n , δ n ) .
Define z := (x , 0, v ). Note that g n (z ) = 0. Evaluating the Jacobian matrix G n of g n at z , we obtain that there exist constants C > 0, M > 0 and n 0 ∈ N s.t. for all n ≥ n 0 ,

g n (z) -G n (z -z ) ≤ C z -z 2 (∀z ∈ B(z , M )) , (3.51)
where G n is given by

G n :=    -q n I d 0 q n q -1 ∞ p ∞ ∇S(x ) 0 -r n I d h n ∇ 2 F (x ) 0 -γn γ n+1 V 0    ,
where ∇S is the Jacobian of S and the matrix V is defined in Eq. (3.8). We define

G ∞ := lim n G n =    -q ∞ I d 0 p ∞ ∇S(x ) 0 -r ∞ I d h ∞ ∇ 2 F (x ) 0 -V 0    .
One can verify that G ∞ is Hurwitz, and that the largest real part of its eigenvalues is -L , where L := L ∧ q ∞ and L is defined in Eq. (3.9).

We define Ω (0) := {z n → z }. We assume P(Ω (0) ) > 0. Using for instance (Delyon et al., 1999, Lem. 4 and Lem. 5), it holds that δ n (ω) → 0 for every ω ∈ Ω (0) , and since

x n (ω)x n-1 (ω) → 0 on that set, we obtain that Ω (0) = {ζ n → z }. Let M ∈ (0, M ) be a constant, whose value will be specified later on. For every N 0 ∈ N, define Ω (0)

N 0 := {ζ n → z and sup n≥N 0 ζ n -z ≤ M }. We seek to show that √ γ n -1 (ζ n -z ) ⇒ ν
given Ω (0) , for some Gaussian measure ν, using Th. 3.17. As Ω (0) 0) , it is sufficient to show that the latter convergence holds given Ω (0) N 0 , for every N 0 large enough. From now on, we consider that N 0 is fixed. We define the sequence ( ζn ) n≥N 0 as ζN 0 = ζ N 0 and for every n ≥ N 0 , ζn+1 = ζn + γ n+1 gn ( ζn ) + γ n+1 (η n+1 + r n ( ζn , δ n ))1 An where A n is the event defined by

N 0 ↑ Ω (
A n := n k=N 0 { x k -x ≤ M } ∩ { ζn -z ≤ M } and gn (z) := g n (z)1 z-z ≤M -K(z -z )1 z-z >M ,
where K > 0 is a large constant which will be specified later on. The sequences ( ζn ) n≥N 0 and (ζ n ) n≥N 0 coincide on Ω (0) N 0 . Thus, it is sufficient to study the weak convergence of ( ζn ) n≥N 0 .

An estimate of r n ( ζn , δ n ) 1 An . We start by studying the sequence ( δ n 1 An ). Unfolding the update rule defining δ n and using the fact that (q n ) is a sequence of positive reals converging to q ∞ > 0, we obtain that

δ n 1 An ≤ n k=1    n j=k+1 |1 -γ j q j-1 |    γ k |p k-1 -q k-1 q -1 ∞ p ∞ | S(x k-1 ) 1 An ≤ C n k=1 exp   -β n j=k+1 γ j    γ k |p k-1 -q k-1 q -1 ∞ p ∞ | := w n ,
for some β > 0. The sequence (w n ) is deterministic and converges to zero by (Delyon et al., 1999, Lem. 4). There exists n 1 ≥ n 0 s.t. w n ≤ M . As v → 1 √ v+ε is Lipschitz and ∇F and S are locally Lipschitz, for every z = (v, m, x) and δ s.t. zz ≤ M and δ ≤ M , we have

r n (z, δ) ≤ Cγ n+1 (v + δ + ε) -1 2 m + C (v + δ + ε) -1 2 -(v + ε) -1 2 m ≤ Cγ n+1 z -z + C δ z -z .
This implies that for every n ≥ n 1 ,

r n ( ζn , δ n ) 1 An ≤ C(γ n+1 + w n ) ζn -z . (3.52) Tightness of √ γ n -1 ( ζn -z ). We decompose ζn+1 -z = (I 3d + γ n+1 G n )( ζn -z ) + γ n+1 g n ( ζn ) -G n ( ζn -z ) 1 ζn-z ≤M -γ n+1 (K + G n )( ζn -z )1 ζn-z >M + γ n+1 (η n+1 + r n ( ζn , δ n ))1 An . (3.53)
For a given t > 0, we write G ∞ = B -1 t G t B t the Jordan-like decomposition of G ∞ , where the ones of the second diagonal of the usual Jordan decomposition are replaced by t, and where B t is some invertible matrix. We define

S n := B t ( ζn -z ). Setting G (t) n := B t G n B -1
t , we obtain

S n+1 = (I 3d + γ n+1 G (t) n )S n + γ n+1 B t g n ( ζn ) -G n ( ζn -z ) 1 ζn-z ≤M -γ n+1 (K + G (t) n )S n 1 ζn-z >M + γ n+1 B t (η n+1 + r n ( ζn , δ n ))1 An .
Choose A ∈ (0, 2L ) and A ∈ (A, 2L ). There exists γ and t > 0 s.t. for every γ < γ,

I + γG t 2 ≤ 1 -γ(A + 2L )/2, where • 2 is the spectral norm. As G (t)
n → G t , there exists n 2 ≥ n 1 , such that for all n ≥ n 2 , I + γG

(t) n 2 ≤ 1 - γA . Recall the notation E n = E[• | F n ]. We expand S n+1
2 and use the inequality

g n ( ζn ) -G n ( ζn -z ) 2 1 ζn-z ≤M ≤ C S n 2
to obtain after straightforward algebra

E n S n+1 2 ≤ (1 -γ n+1 A ) S n 2 + Cγ 2 n+1 S n 2 + Cγ 2 n+1 (E n η n+1 2 + r n ( ζn , δ n ) 2 )1 An + 2γ n+1 S * n B t g n ( ζn ) -G n ( ζn -z ) 1 ζn-z ≤M -2γ n+1 S * n (K + G (t) n )S n 1 ζn-z >M + 2γ n+1 S * n B t r n ( ζn , δ n )1 An . Choose c := (A -A)/2. If M is chosen small enough, g n ( ζn ) -G n ( ζn -z ) 1 ζn-z ≤M ≤ c 2 B t -1 B -1 t ζn -z . Moreover, choosing K > sup n G (t) n 2 , it holds that S * n (K + G (t) n )S n ≥ 0. Then, E n S n+1 2 ≤ (1 -γ n+1 (A -c)) S n 2 + Cγ 2 n+1 S n 2 + Cγ 2 n+1 (E n η n+1 2 + r n ( ζn , δ n ) 2 )1 An + 2γ n+1 B t S n r n ( ζn , δ n ) 1 An .
Using Eq. (3.52),

E n S n+1 2 ≤ (1 -γ n+1 (A -c -w n )) S n 2 + Cγ 2 n+1 (1 + w 2 n ) S n 2 + Cγ 2 n+1 E n η n+1 2 1 An .
Therefore, there exists n 3 ≥ n 2 s.t. for all n ≥ n 3 ,

E S n+1 2 ≤ (1 -γ n+1 A)E S n 2 + Cγ 2 n+1 E( η n+1 2 1 xn-x ≤M ) .
The second expectation in the righthand side is bounded uniformly in n by the condition (3.7). Using (Delyon et al., 1999, Lem. 4 and Lem. 5), we conclude that

sup n γ -1 n E S n 2 < ∞. Therefore, sup n γ -1 n E ζn -z 2 < ∞, which in turn implies sup n γ -1 n E( ζ n -z 2 1 Ω (0) N 0 ) < ∞.
Strongly perturbed iterations. We define ỹn = √ γ n -1 ( ζnz ). Define

Ḡn := γ -1 n+1 γ n γ n+1 -1 I 3d + γ n γ n+1 G n .
The sequence Ḡn converges to Ḡ∞ := G ∞ + 1 α=1 2γ 0 I 3d . Recalling Eq. ( 3.53), we can write

ỹn+1 = (I 3d + γ n+1 Ḡ∞ )ỹ n + γ n+1 rn + √ γ n+1 ηn+1
where ηn+1 = η n+1 1 An and rn = rn,1 + rn,2 + rn,3 , where

rn,1 := √ γ n+1 -1 r n ( ζn , δ n )1 An + ( Ḡn -Ḡ∞ )ỹ n rn,2 := √ γ n+1 -1 g n ( ζn ) -G n ( ζn -z ) 1 ζn-z ≤M rn,3 := - √ γ n+1 -1 (K + G n )( ζn -z )1 ζn-z >M .
We now check that the assumptions of Th. 3.17 are fulfilled. On the event Ω (0)

N 0 , we recall that ζn = ζ n , hence rn,3 is identically zero. Moreover, using Eq. (3.52)

M > 4 s.t. E n [ ηn+1 b M /2 ] ≤ E n [ η n+1 b M /2 ]1 xn-x ≤M ≤ CE n [ ∇f (x n , ξ n+1 ) b M ]1 xn-x ≤M ≤ C .
Moreover, E n [η n+1 ] = 0 and finally, almost surely on Ω (0)

N , E n [η n+1 ηT n+1 ] converges to Σ :=       E ξ    p ∞ (∇f (x , ξ) 2 -S(x )) h ∞ ∇f (x , ξ) p ∞ (∇f (x , ξ) 2 -S(x )) h ∞ ∇f (x , ξ) T    0 0 0 0 0       .
(3.54) Therefore, the assumptions of Th. 3.17 are fulfilled for the sequence ỹn . We obtain the desired result for the sequence (m n , x n-1 ). We now show that the same result also holds for the sequence (m n , x n ). For this purpose, observe that

1 √ γ n m n x n -x = 1 √ γ n m n x n-1 -x + 0 1 √ γn (x n -x n-1
) .

Then, notice that xn-x n-1

√ γn = √ γ n mn √ vn+ε ≤ γn ε m n → 0 as n → ∞
since it is assumed that z n → z (which implies in particular that m n → 0). Hence, it holds that √ γ n -1 (x nx n-1 ) converges a.s. to 0. We conclude by invoking Slutsky's lemma.

Proof of Eq. (3.10). We have the subsystem:

HΓ + Γ HT = -h 2 ∞ Q 0 0 0 where H := (θ -r ∞ )I d h ∞ ∇ 2 F (x ) -V θI d (3.55)
and where Q := Cov ∇f (x , ξ) . The next step is to triangularize the matrix H in order to decouple the blocks of Γ. For every k = 1, . . . , d, set ν ± k := -r∞ 2 ± r 2 ∞ /4h ∞ π k with the convention that √ -1 = ı (inspecting the characteristic polynomial of H, these are the eigenvalues of H). Set M

± := diag (ν ± 1 , • • • , ν ± d ) and R ± := V -1 2 P M ± P T V -1 2 . Using the identities M + + M -= -r ∞ I d and M + M -= h ∞ diag (π 1 , • • • , π d ), it can be checked that R H = R -V + θI d 0 -V V R + + θI d R, where R := I d R + 0 I d .
Set Γ := RΓR T . Denote by ( Γi,j ) i,j=1,2 the blocks of Γ. Note that Γ2,2 = Γ 2,2 . By left/right multiplication of Eq. ( 3.55) respectively by R and R T , we obtain

(R -V + θI d ) Γ1,1 + Γ1,1 (V R -+ θI d ) = -h 2 ∞ Q (3.56) (R -V + θI d ) Γ1,2 + Γ1,2 (R + V + θI d ) = Γ1,1 V (3.57) (V R + + θI d ) Γ2,2 + Γ2,2 (R + V + θI d ) = V Γ1,2 + ΓT 1,2 V . (3.58) Set Γ1,1 = P -1 V 1 2 Γ1,1 V 1 2 P . Define C := P -1 V 1 2 QV 1 2 P . Eq. (3.56) yields (M -+ θI d ) Γ1,1 + Γ1,1 (M -+ θI d ) = -h 2 ∞ C . Set Γ1,2 = P -1 V 1 2 Γ1,2 V -1 2 P . Eq. (3.57) is rewritten (M -+ θI d ) Γ1,2 + Γ1,2 (M + + θI d ) = Γ1,1 . The component (k, ) is given by Γk, 1,2 = (ν - k + ν + + 2θ) -1 Γk, 1,1 = -h 2 ∞ C k, (ν - k + ν + + 2θ)(ν - k + ν -+ 2θ)
.

Set finally Γ2,2 = P -1 V -1 2 Γ 2,2 V -1 2 P . Eq. (3.58) becomes

(M + + θI d ) Γ2,2 + Γ2,2 (M + + θI d ) = Γ1,2 + ΓT 1,2 .
Thus,

Γk, 2,2 = Γk, 1,2 + Γ ,k 1,2 ν + k + ν + + 2θ = -h 2 ∞ C k, (ν + k + ν + + 2θ)(ν - k + ν -+ 2θ) 1 ν - k + ν + + 2θ + 1 ν + k + ν -+ 2θ
.

After tedious but straightforward computations, we obtain

Γk, 2,2 = h 2 ∞ C k, (r ∞ -2θ)(h ∞ (π k + π ) + 2θ(θ -r ∞ )) + h 2 ∞ (π k -π ) 2 2(r∞-2θ)
, and the result is proved.

3.7 Proofs for Section 3.4

Preliminaries

Most of the avoidance of traps results in the stochastic approximation literature deal with the case where the ODE that underlies the stochastic algorithm under study is an autonomous ODE ż = h(z). In this setting, a point z ∈ zer h is called a trap if h(z) admits an expansion around z of the type h(z) = D(zz ) + o( zz ), where the matrix D has at least one eigenvalue which real part is (strictly) positive. Initiated by [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF] and by [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF], the most powerful class of techniques for establishing avoidance of traps results makes use of Poincaré's invariant manifold theorem for the ODE ż = h(z) in a neighborhood of some point z ∈ zer h. The idea is to show that with probability 1, the stochastic algorithm strays away from the maximal invariant manifold of the ODE where the convergence to z of the ODE flow can take place. As previously mentioned, since we are dealing with algorithms derived from non-autonomous ODEs, we extend the results of [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF] to this setting. The proof of Th. 3.8 relies on a non-autonomous version of Poincaré's theorem. We borrow this result from the rich literature that exists on the subject [START_REF] Daleckii | Stability of solutions of differential equations in Banach space[END_REF][START_REF] Kloeden | Nonautonomous dynamical systems[END_REF].

Let us start by setting the context for the non-autonomous version that we shall need for the invariant manifold theorem. Given an integer d > 0 and a matrix D ∈ R d×d , consider the linear autonomous differential equation ż(t) = Dz(t), (3.59) which solution is obviously z(t) = e Dt z(0) for t ∈ R. Let us factorize D as in (3.12), and write D = QΛQ -1 with Λ = Λ - Λ + where we recall that the Jordan blocks that constitute Λ -∈ R d -×d -(respectively Λ + ∈ R d + ×d + ) are those that contain the eigenvalues λ i of D such that λ i ≤ 0 (respectively λ i > 0). Let us assume here that both d -and d + are positive. It will be convenient to work in the basis of the columns of Q by making the variable change

z → y = y - y + = Q -1 z,
where y ± ∈ R d ± . In this new basis, the ODE (3.59) is written as

ẏ- ẏ+ = Λ - Λ + y - y + , (3.60)
which solution is y ± (t) = exp(tΛ ± )y ± (0). One can readily check that for each couple of real numbers α + and α -that satisfy

0 < α -< α + < min{ λ i : λ i > 0}, (3.61)
there exists a so-called exponential dichotomy of the ODE solutions, which amounts in our case to the existence of two constants K -, K + ≥ 1 such that exp(tΛ -) ≤ K -e α -t for t ≥ 0, exp(tΛ + ) ≤ K + e α + t for t ≤ 0, see, e.g., [START_REF] Horn | Topics in matrix analysis[END_REF].

We now consider a non-autonomous perturbation of this ODE, which is represented in the basis of the columns of Q as ẏ(t) = h(y(t), t) with h(y, t) = Λ - Λ + y + ε(y, t), (3.62) and ε : R d × R → R d is a continuous function. In the sequel, we shall be interested in the asymptotic behavior of this equation for the large values of t, and therefore, restrict our study to the interval I = [t 0 , ∞) for some given t 0 ≥ 0 that we shall fix later. We assume that ε(0, •) = 0 on I. We denote as φ : I × I × R d → R d the so-called general solution of (3.62), which is defined by the fact that φ(•, t, x) is the unique noncontinuable solution of (3.62) such that φ(t, t, x) = x for t ∈ I and x ∈ R d , assuming this solution exists and is unique for each (x, t) ∈ R d × I.

In the linear autonomous case provided by the ODE (3.60), the subspace

G =        t, y - 0   ∈ R × R d : y -∈ R d -     
is trivially invariant in the sense that if (t, y) ∈ G, then, (s, φ(s, t, y)) ∈ G for each s ∈ R. This concept can be generalized to the non-linear and non-autonomous case. We say that the C 1 function w : R d -× I → R d + defines a global non-autonomous invariant manifold for the ODE (3.62) if w(0, t) = 0 for all t ∈ I, and, furthermore, if for each t ∈ I and each y -∈ R d -, writing y = (y -, w(y -, t)), the general solution φ(s, t, y) = (φ -(s, t, y), φ + (s, t, y)) with φ ± (s, t, y) ∈ R d ± verifies φ + (s, t, y) = w(φ -(s, t, y), s) for each s ∈ I. The non-autonomous invariant manifold is the set

G =        t, y - w(y -, t)   ∈ I × R d : y -∈ R d -     
, which obviously satisfies (t, y) ∈ G ⇒ (s, φ(s, t, y)) ∈ G for each s ∈ I.

These invariant manifolds are described by the following proposition, which is a straightforward application of (Pötzsche and Rasmussen, 2006, Th. A.1) (see also (Kloeden and Rasmussen, 2011, Th. 6.3 p. 106, Rem. 6.6 p. 111)). It is useful to note that under the conditions provided in the statement of this proposition, the existence of the general solution φ of the ODE (3.62) is ensured by Picard's theorem. Proposition 3.18. Let I = [t 0 , ∞) for some t 0 ≥ 0. Assume that the function ε(y, t) is such that ε(0, •) ≡ 0 on I, the function ε(•, t) is continuously differentiable for each t ∈ I, and furthermore, the Jacobian matrix ∂ 1 ε(y, t) satisfies

|ε| 1 := sup (y,t)∈R d ×I ∂ 1 ε(y, t) < α + -α - 4K (3.63) with K = K -+ K + + K -K + (K -∨ K + )

Let us partition the function h(y, t) as

h(y, t) = h -(y, t) h + (y, t) = Λ -y -+ ε -(y, t) Λ + y + + ε + (y, t) , (3.67)
where

h ± : R d × I → R d ± , y ± ∈ R d ± and ε ± : R d × I → R d ± .
With these notations, the previous proposition leads to the following lemma.

Lemma 3.19. In the setting of Prop. 3.18, for each t in the interior of I and each vector y = (y -, y + ) such that y ± ∈ R d ± and y + = w(y -, t), it holds that

h + (y, t) = ∂ 1 w(y -, t)h -(y, t) + ∂ 2 w(y -, t) . (3.68)
Assume that α -is small enough so that Ineq. (3.65) Equating the derivatives with respect to s of the two members of this equation and taking s = t, we get the first equation.

Writing g : R d-× I → R d , (y -, t) → (y -, w(y -, t)), Eq. ( 3.68) can be rewritten as

∂ 2 w(y -, t) = h + (g(y -, t), t) -∂ 1 w(y -, t)h -(g(y -, t), t). (3.72)
By Prop. 3.18, the function w is twice differentiable, and we can write

∂ 2 2 w(y -, t) = ∂ 1 h + ∂ 2 g + ∂ 2 h + -(∂ 1 ∂ 2 w)h --(∂ 1 w)(∂ 1 h -∂ 2 g + ∂ 2 h -), (3.73)
where, e.g., h + is a shorthand notation for h + (g(y -, t), t). It holds from Eq. ( 3.67) and the assumptions of Prop. 3.18 that for each (y, t) ∈ R d × I,

∂ 1 h(y, t) ≤ Λ + ∂ 1 ε(y, t) ≤ C, (3.74)
where the constant C > 0 is independent of (y, t) and can change from an inequality to another in the remainder of the proof. By the mean value inequality and Prop. 3.18, we also get that

w(y -, t) = w(y -, t) -w(0, t) ≤ sup (u,s) ∂ 1 w(u, s) y -≤ C y -,
thus, g(y -, t) ≤ C y -. By the mean value inequality again,

h(g(y -, t), t) = h(g(y -, t), t) -h(0, t) ≤ sup (u,t) ∂ 1 h(u, t) g(y -, t) ≤ C g(y -, t) ≤ C y -.
By Eq. (3.72) and Prop. 3.18, this implies that and (3.75)

∂ 2 g(y -, t) = ∂ 2 w(y -, t) = h + -(∂ 1 w)h -≤ C y -,
∂ 1 ∂ 2 w(y -, t) = ∂ 1 h + ∂ 1 g -(∂ 2 1 w)h --(∂ 1 w)(∂ 1 h -∂ 1 g) ≤ C( y -+ 1). (3.76)
Let V -⊂ R dbe a small enough neighborhood of zero so that g(y -, t) ∈ V for each y -∈ V -, which is possible by the inequality g(y -, t) ≤ C y -. By the assumption on ∂ 2 ε(y, t) in the statement of Lem. 3.19, we have

∀y -∈ V -, ∂ 2 h(g(y -, t), t) = ∂ 2 ε(g(y -, t), t) ≤ C. (3.77)
The bound (3.70) is an immediate consequence of Eq. (3.76). Getting back to Eq. ( 3.73), the bound (3.71) follows from the inequalities (3.74)-(3.77).

Prop. 3.18 deals with the case where the function ε is globally Lipschitz continuous.

In practical cases, such a strong assumption is not necessarily verified. In particular, for the ODEs we consider for our application, it is not satisfied (see the function e defined in Subsec. 3.7.3.1 below). Nonetheless, recall that we only need the existence of a local non-autonomous invariant manifold, i.e. defined in the vicinity of an arbitrary solution such as the trivial zero solution (since we suppose here ε(0, •) = 0) whereas the aforementioned strong assumption provides a global non-autonomous invariant manifold. Indeed, as for the avoidance of traps result we intend to show, we will only need to look at the behavior of our ODE in the neighborhood of a trap z . Therefore, in prevision of the proof of Th. 3.8, we localize the ODE (3.62) in the neighborhood of zero. This is the purpose of the next proposition.

Proposition 3.20. Let I = [t 0 , +∞) for some t 0 ≥ 0 and let h : R d × I → R d be defined as in Eq. (3.62). Assume that ε(0, •) ≡ 0 on I, that the function ε(•, t) is continuously differentiable for every t ∈ I and that lim (y,t)→(0,+∞)

∂ 1 ε(y, t) = 0 . (3.78)
Then, there exist σ > 0, t 1 > 0, a function ε :

R d × I 1 → R d where I 1 := [t 1 , +∞) and a function h : R d × I 1 → R d defined for every y ∈ R d , t ∈ I 1 by h(y, t) = Λy + ε(y, t) s.t.
h and ε verify the assumptions of Prop. 3.18 and for every (y, t) ∈ B(0, σ) × I 1 , we have that h(y, t) = h(y, t) and ε(y, t) = ε(y, t). Moreover, for any δ > 0, we can choose σ, t 1 respectively small and large enough s.t. the mapping w : R d -× I 1 → R d + obtained from Prop. 3.18 (applied to h and ε) satisfies

|w| 1 = sup (y,t)∈R d -×I 1 ∂ 1 w(y, t) < δ . (3.79)
Furthermore, Eq. (3.68) holds for h and w for all (y, t) ∈ B(0, σ) × I 1 . If, additionally, Eq. (3.69) holds for ε, then there exists σ 1 ≤ σ such that

sup (y -,t)∈B(0,σ 1 )×I 1 ∂ 1 ∂ 2 w(y -, t) < +∞ , (3.80) sup (y -,t)∈B(0,σ 1 )×I 1 ∂ 2 2 w(y -, t) < +∞ . (3.81)
Proof. The idea of the proof is to localize the function h(y, t) to a neighborhood of zero in the variable y for the purpose of applying Prop. 3.18. This cut-off technique is known in the non-autonomous ODE literature, see, e.g., (Kloeden and Rasmussen, 2011, Th. 6.10). Let ψ : R d → [0, 1] be a smooth function such that ψ(y) = 1 if y ≤ 1, and ψ(y) = 0 if y ≥ 2. Let C = max y ∇ψ(y) where ∇ψ is the Jacobian matrix of ψ.

Thanks to the convergence (3.78), we can choose t 1 > 0 large enough and σ > 0 small enough so that

sup (t,y)∈[t 1 ,∞)×B(0,2σ) ∂ 1 ε(y, t) < α + -α - 4K(1 + 2C) ,
and we set I 1 = [t 1 , ∞). Writing ε(y, t) = ψ(y/σ)ε(y, t), it holds that for each (t, y) ∈

I 1 × R d , ∂ 1 ε(y, t) ≤ σ -1 C1 y ≤2σ ε(y, t) + 1 y ≤2σ ∂ 1 ε(y, t) ≤ max y ≤2σ ∂ 1 ε(y, t) σ -1 C y + 1 1 y ≤2σ ≤ α + -α - 4K ,
where we used the mean value inequality along with ε(0, t) = 0 to obtain the second inequality. Thus, the function h(y, t) = Λy + ε(y, t) satisfies all the assumptions of Prop. 3.18. In addition, the function ε coincides with the function ε on B(0, σ 1 ) × I 1 , and so it is for the functions h and h. Finally, it follows from (Kloeden and Rasmussen, 2011, Th. 6.3) that

|w| 1 ≤ 2K 2 α + -α --4K|ε| 1 |ε| 1
(note that L in (Kloeden and Rasmussen, 2011, Th. 6.3) corresponds to |ε| 1 with our notations). Using Eq. ( 3.78), we can make |ε| 1 as small as needed by choosing σ, t 1 respectively small and large enough, which gives us Eq. (3.79). The proof of the last two equations follows from the application of Lem. 3.19 to h and w. The result is immediate after noticing that for (y,

t) ∈ R d × I 1 , we have ∂ 2 ε(y, t) ≤ ∂ 2 ε(y, t) .

Proof of Th. 3.8

We shall rely on the following result of Brandière and Duflo. Recall that (Ω, F, P) is a probability space equipped with a filtration (F n ) n∈N .

Proposition 3.21. ((Brandière and Duflo, 1996, Prop. 4)) Given a sequence (γ n ) of deterministic nonnegative stepsizes such that k γ k = +∞ and k γ 2 k < +∞, consider the R d -valued stochastic process (z n ) n∈N given by

z n+1 = (I + γ n+1 H n )z n + γ n+1 η n+1 + γ n+1 ρ n+1 .
Assume that z 0 is F 0 -measurable and that the sequences (η n ), (ρ n ) together with the sequence of random matrices (H n ) are (F n )-adapted. Moreover, on a given event A ∈ F, assume the following facts:

i) n ρ n 2 < ∞. ii) lim sup E[ η n+1 2+a | F n ] < ∞ for some a > 0, and E[η n+1 | F n ] = 0. iii) lim inf E[ η n+1 2 | F n ] > 0.
Let H ∈ R d×d be a deterministic matrix such that the real parts of its eigenvalues are all positive. Then,

P A ∩ [z n → 0] ∩ [H n → H] = 0.
We now enter the proof of Th. 3.8. Recall the development (3.11) of b(z, t) near z and the spectral factorization (3.12) of the matrix D. To begin with, it will be convenient to make the variable change y = Q -1 (zz ), and set

h(y, t) = Q -1 b(Qy + z , t) = Λy + ẽ(y, t),
with ẽ(y, t) = Q -1 e(Qy + z , t), in such a way that our stochastic algorithm is rewritten as

y n+1 = y n + γ n+1 h(y n , τ n ) + γ n+1 ηn+1 + γ n+1 ρn+1
where ηn is as in the statement of the theorem and ρn = Q -1 ρ n . Observe that the assumptions on the function e in the statement of the theorem remain true for ẽ with z replaced by zero.

If the matrix Λ has only eigenvalues with (strictly) positive real parts, i.e., d -= 0, then we can apply Prop. 3.21 to the sequence (z n ). Henceforth, we deal with the more complicated case where d -> 0.

Apply Prop. 3.20 to h to obtain h and σ, t 1 respectively small and large enough and w : R +∞). By Assumption iv) of Th. 3.8 and Prop. 3.20 we can choose σ 1 ≤ σ such that Eq. ( 3.80) and Eq. ( 3.81) hold. Now, given p ∈ N, let us define the event

d -× I 1 → R d + where I 1 := [t 1 ,
E p = ∀n ≥ p, y n < σ 1 , τ n ∈ I 1 . On E p , it holds that h(y n , τ n ) = h(y n , τ n ) and ∀n ≥ p, y n+1 = y n + γ n+1 h(y n , τ n ) + γ n+1 ηn+1 + γ n+1 ρn+1 = y - n y + n + γ n+1 h -(y n , τ n ) h + (y n , τ n ) + γ n+1 η- n+1 η+ n+1 + γ n+1 ρ- n+1 ρ+ n+1 (3.82)
where h is partitioned as in (3.67), and where η± n , ρ± n ∈ R d ± . Note that, by Prop. 3.20 and Assumptions vi) and vii) on the sequence (η n ), we can choose σ, t 1 respectively small and large enough s.t.

lim inf E[ η+ n+1 2 |F n ]1 Ep (y n ) -2 lim sup E[ ∂ 1 w(y - n , τ n )η - n+1 2 |F n ]1 Ep (y n ) > c 2 2 . (3.83)
This inequality will be important in the end of our proof. Let t be in the interior of I 1 , and let y = (y -, y + ) be in a neighborhood of 0. Make the variable change (y -, y + ) → (u -, u + ) with

u + = y + -w(y -, t), u -= y -,
where w is the function defined in the statement of Prop. 3.20, and let

W (u -, u + , t) = h + (y, t) -∂ 1 w(y -, t)h -(y, t) -∂ 2 w(y -, t) = h + ((u -, u + + w(u -, t)), t) -∂ 1 w(u -, t)h -((u -, u + + w(u -, t)), t) -∂ 2 w(u -, t).
By Prop. 3.20 and Lem. 3.19, it holds that W (u -, 0, t) = 0. Moreover, W (u -, •, t) ∈ C 1 by the assumptions on h. Therefore, writing y(r) = (u -, ru + + w(u -, t)) for r ∈ [0, 1], and using the decomposition (3.67), we get that

W (u -, u + , t) = 1 0 ∂ 2 W (u -, ru + , t)u + dr = Λ + u + + 1 0   ∂ 1 ε + (y(r), t) 0 I d + -∂ 1 w(u -, t)∂ 1 ε -(y(r), t) 0 I d +   u + dr.
We can also write y(r) = (y -, ry + + (1r)w(y -, t)). Recalling that w(0, t) = 0 and that ∂ 1 w(y -, t) is bounded on R d -× I, we get by the mean value inequality that w(y -, t) ≤ C y -where C > 0 is a constant. Thus, y(r) ≤ (1 + C) y . Moreover, ε(y, t) = Q -1 e(Qy, t) for y < σ. Thus, we get by (3.13) that ∂ 1 ε(y(r), t) → 0 as (y, t) → (0, ∞) uniformly in r ∈ [0, 1]. Using again the boundedness of ∂ 1 w(•, •) , we eventually obtain that

W (u -, u + , t) = Λ + + ∆(y, t) u + , with lim (y,t)→(0,∞) ∆(y, t) = 0.
On the event E p , assume that n ≥ p, and write

u + n = y + n -w(y - n , τ n ), u - n = y - n ,
(see Eq. (3.82)). Choosing α -> 0 small enough so that the gap condition (3.65) is satisfied with m = 2, we have by Taylor's expansion

w(y - n+1 , τ n+1 ) -w(y - n , τ n ) = w(y - n+1 , τ n+1 ) -w(y - n , τ n+1 ) + w(y - n , τ n+1 ) -w(y - n , τ n ) = ∂ 1 w(y - n , τ n+1 )(y - n+1 -y - n ) + γ n+1 ∂ 2 w(y - n , τ n ) + n+1 + γ n+1 , with n+1 ≤ sup y -∈[y - n ,y - n+1 ] ∂ 2 1 w(y -, τ n+1 ) y - n+1 -y - n 2 ,
and

γ n+1 ≤ sup τ ∈[τn,τ n+1 ] ∂ 2 2 w(y - n , τ ) γ 2 n+1 .
Using this equation, we obtain

u + n+1 -u + n = γ n+1 W (u - n , u + n , τ n ) + γ n+1 η+ n+1 -∂ 1 w(y - n , τ n+1 )η - n+1 + γ n+1 ρ+ n+1 -∂ 1 w(y - n , τ n+1 )ρ - n+1 -n+1 -γ n+1 + γ n+1 ∂ 1 w(y - n , τ n ) -∂ 1 w(y - n , τ n+1 ) h -(y n , τ n ) ,
which leads to

u + n+1 = u + n + γ n+1 Λ + + ∆(y n , τ n ) u + n + γ n+1 ηn+1 + γ n+1 ρn+1 , (3.84) with ηn+1 = η+ n+1 -∂ 1 w(y - n , τ n )η - n+1 and ρn+1 = ρ+ n+1 -∂ 1 w(y - n , τ n )ρ - n+1 -1 γ n+1 >0 n+1 + γ n+1 γ n+1 + ∂ 1 w(y - n , τ n ) -∂ 1 w(y - n , τ n+1 ) h -(y n , τ n ) . (3.85)
To finish the proof, it remains to check that the noise sequence satisfies the assumptions of Prop.

1 γ n+1 >0 n+1 γ n+1 2 ≤ 1 γ n+1 >0 C γ 2 n+1 y n+1 -y n 4 ≤ C γ 2 n+1 ( h(y n , τ n ) 4 + ηn+1 4 + ρn+1 4
) .

A coupling argument (see (Brandière and Duflo, 1996, p. 401)) shows that we can simplify the condition lim sup

E[ η n+1 4 | F n ]1 zn∈W < ∞ to E[ η n+1 4 | F n ]1 zn∈W < C . The latter condition implies that E[1 Ap n γ 2 n+1 η n+1 4 ] ≤ n C γ 2 n+1
, and therefore n γ 2 n+1 η n+1 4 1 Ap < +∞ a.s. As a consequence, noticing also the boundedness of (h(y n , τ n )) and (ρ n ) on A p , we deduce that n 1 γ n+1 >0 n+1 γ n+1

2

< +∞ on A p . We now briefly control the last term of ρn . By the mean value inequality, we obtain that

∂ 1 w(y - n , τ n ) -∂ 1 w(y - n , τ n+1 ) h -(y n , τ n ) ≤ γ n+1 sup (y -,t) ∂ 2 ∂ 1 w(y -, t) h -(y n , τ n ) ≤ C γ n+1 ,
where the last inequality stems from Prop. 3.20-Eq. ( 3.80) together with the boundedness of the sequence (h(y n , τ n )). In view of Eq. ( 3.85) and the above estimates, we deduce that n ρn+1

2 1 Ap < +∞ a.s. on A p .
We verify the remaining conditions on the noise sequence (η n ). We can easily remark that

E[η n+1 |F n ] = 0 and ηn+1 ≤ C η n+1 on A p . Hence, lim sup E[ ηn+1 4 | F n ]1 zn∈W < ∞.
The last condition, meaning that the noise is exciting enough, stems from noting that

2 lim inf E[ ηn+1 2 |F n ]1 Ap ≥ lim inf E[ η+ n+1 2 |F n ]1 Ap -2 lim sup E[ ∂ 1 w(y - n , τ n )η - n+1 2 |F n ]1 Ap > c 2 2 ,
where we used our choice of σ, t 1 and Eq. (3.83).

Noticing that [y n → 0] ⊂ [∆(y n , τ n ) → 0], we can now apply Prop. 3.21 to the sequence (u + n ) (see Eq. ( 3.84)) with A = A p to obtain

P A p ∩ [u + n → 0] = P A p ∩ [u + n → 0] ∩ [∆(y n , τ n ) → 0] = 0 .
We now show that [y n → 0] ⊂ [u + n → 0] , which amounts to prove that w(y - n , τ n ) → 0 given y n → 0. To that end, upon noting that w(0, •) ≡ 0 and that ∂ 1 w(•, •) is bounded, it suffices to apply the mean value inequality, writing:

w(y - n , τ n ) = w(y - n , τ n ) -w(0, τ n ) ≤ sup (y -,t) ∂ 1 w(y -, t) y - n ≤ K y - n .
We have shown so far that P(A p ) = 0. Since y n = Q -1 z n and [y n → 0] ⊂ p∈N E p , we finally obtain that

P[z n → 0] = P[y n → 0] = P    p∈N ([y n → 0] ∩ E p )    = P    p∈N A p    = 0.
Th. 3.8 is proven. The matrix D coincides with ∇g ∞ (z ), where the function g ∞ is defined in (3.20). As such, its expression is immediate. Recalling that p ∞ S(x )q ∞ v = 0, we get

g(z, t) -D(z -z ) =      p(t)S(x) -q(t)v -p ∞ ∇S(x )(x -x ) + q ∞ (v -v ) h(t)∇F (x) -r(t)m -h ∞ ∇ 2 F (x )(x -x ) + r ∞ m -m (v + ε) -1 2 -(v + ε) -1 2      =     -q(t) + q ∞ 0 (p(t) -p ∞ )∇S(x ) 0 r ∞ -r(t) (h(t) -h ∞ )∇ 2 F (x ) m 2(v +ε) 3 2 0 0        v -v m x -x    +       p(t)(S(x) -S(x ) -∇S(x )(x -x )) h(t)(∇F (x) -∇ 2 F (x )(x -x )) -m 1 √ v+ε -1 √ v +ε + v-v 2(v +ε) 3 2       +    p(t)S(x ) -q(t)v 0 0    := e(z, t) + c(t).
Under the assumptions made, it is easy to see that the function e(z, t) has the properties required in the statement of Th. 3.8.

Proof of Th. 3.10

Consider the matrix D defined in the statement of Lem. 3.9. A spectral analysis of this matrix as regards its eigenvalues with positive real parts is done in the following lemma.

Lemma 3.22. Let D be the matrix provided in the statement of Lem. 3.9. Each eigenvalue ζ of the matrix D such that ζ > 0 is real, and its algebraic and geometric multiplicities are equal. Moreover, there is a one-to-one correspondence ϕ between these eigenvalues and the negative eigenvalues of V

1 2 ∇ 2 F (x )V 1 2 . Let d + be the dimension of the eigenspace of V 1 2 ∇ 2 F (x )V 1 2
that is associated with its negative eigenvalues, let

W =     w 1
. . .

w d +     ∈ R d + ×d
be a matrix which rows are independent eigenvectors of V

1 2 ∇ 2 F (x )V 1 2
that generate this eigenspace, and denote as β k < 0 the eigenvalue associated with w k . Then, the rows of the rank d + -matrix

A + = 0 d + ×d , W V 1 2 , -diag(r ∞ + ϕ -1 (β k ))W V -1 2 ∈ R d + ×3d
generate the left eigenspace of D, the row k being an eigenvector for the eigenvalue ϕ -1 (β k ).

Proof. It is obvious that the block lower-triangular matrix D has d eigenvalues equal to -q ∞ and 2d eigenvalues which are those of the sub-matrix

D = -r ∞ I d h ∞ ∇ 2 F (x ) -V 0 .
Given λ ∈ C, we obtain by standard manipulations involving determinants that

det( D -λ) = det(λ(r ∞ + λ) + h ∞ V ∇ 2 F (x )) = det(λ(r ∞ + λ) + h ∞ V 1 2 ∇ 2 F (x )V 1 2 ).
Denoting as

{β k } d k=1 the eigenvalues of h ∞ V 1 2 ∇ 2 F (x )V 1 2
counting the multiplicities, we obtain from the last equation that the eigenvalues of D are the solutions of the second order equations

λ 2 + r ∞ λ + β k = 0, k = 1, . . . , d.
The product of the roots of such an equation is β k , and their sum is -r ∞ ≤ 0. Thus, denoting as ζ k,1 and ζ k,2 these roots, it is easy to see that if

β k ≥ 0, then ζ k,1 , ζ k,2 ≤ 0, while if β k < 0, then both ζ k,i
are real, and only one of them is positive. Thus, we have so far shown that the eigenvalues of D which real parts are positive are themselves real, and there is a one-to-one map ϕ from the set of positive eigenvalues of D to the set of negative eigenvalues of V

1 2 ∇ 2 F (x )V 1 2
. Moreover, the algebraic multiplicity of the eigenvalue ζ > 0 of D is equal to the multiplicity of ϕ(ζ).

Let us now turn to the left (row) eigenvectors of D that correspond to these eigenvalues. To that end, we shall solve the equation (3.86) for a given eigenvalue ζ > 0 of D. Developing this equation, we get

uD = ζu with u = [0, u 1 , u 2 ], u 1,2 ∈ R 1×d ,
-r ∞ u 1 -u 2 V = ζu 1 , h ∞ u 1 ∇ 2 F (x ) = ζu 2 .
If we now write

ũ1 = u 1 V -1 2 and ũ2 = u 2 V 1 2 , this system becomes -r ∞ ũ1 -ũ2 = ζ ũ1 , h ∞ ũ1 V 1 2 ∇ 2 F (x )V 1 2 = ζ ũ2 , or, equivalently, ũ2 = -(r ∞ + ζ)ũ 1 , ũ1 ζ 2 + r ∞ ζ + h ∞ V 1 2 ∇ 2 F (x )V 1 2 = 0, which shows that ũ1 is a left eigenvector of V 1 2 ∇ 2 F (x )V 1
2 associated with the eigenvalue ϕ(ζ). What's more, assume that r is the multiplicity of ϕ(ζ), and, without generality loss, that the submatrix W r,• made of the first r rows of W generates the left eigenspace of ϕ(ζ). Then, the matrix

0 r×d W r• V 1 2 -(r ∞ + ζ)W r• V -1 2
is a r-rank matrix which rows are independent left eigenvectors that generate the left eigenspace of D for the eigenvalue ζ. In particular, the algebraic and geometric multiplicities of this eigenvalue are equal. The same argument can be applied to the other positive eigenvalues of D.

We now have all the elements to prove Th. 3.10. Recall Eq. (3.14):

z n+1 = z n + γ n+1 b(z n , τ n ) + γ n+1 η n+1 + γ n+1 ρ n+1 ,
where b(z, t) = g(z, t)c(t) = D(zz ) + e(z, t) and ρ n = c(τ n-1 ) + ρn . With these same notations, we check that Assumptions i)-vi) in the statement of Th. 3.8 are satisfied. The function e(z, t) satisfies Assumptions i)-iv) by Lem. 3.9. We now verify that the sequence (ρ n ) fulfills Assumption v). First, observe that n c(τ n ) 2 < ∞ under . Then, we control the second term (ρ n ). After straightforward derivations, one can show the existence of a positive constant C (depending only on ε and a neighborhood W of z ) such that

ρn+1 2 1 zn∈W ≤ C( m n -m n+1 2 + v n+1 -v n 2 )1 zn∈W . (3.87)
Using the boundedness of the sequences (h n ) and (r n ) together with the update rule of m n and Assumption 3.4.3-iii), there exists a positive constant C independent of n (which may change from an inequality to another) such that

E m n -m n+1 2 1 zn∈W ≤ γ 2 n+1 C E (1 + E ξ ∇f (x n , ξ) 2 )1 zn∈W ≤ C γ 2 n+1 . ( 3
Q -1 = B - B +
where B ± has d ± rows, Lem. 3.22 shows that the row spaces of B + and A + are the same, which implies that Assumption vii) can be rewritten equivalently as

E[ A + η n+1 2 | F n ]1 zn∈W ≥ c 2 1 zn∈W .
By inspecting the form of η n provided by Eq. (3.28) (written as a column vector), one can readily check that implies Assumption vii) of Th. 3.8 for a small enough neighborhood W, using the continuity of the covariance matrix V

1 2 E ξ (∇f (x, ξ) -∇F (x))(∇f (x, ξ) -∇F (x)) T V 1 2
when x is near x .

Proof of Th. 3.11

As mentioned in Section 3.4.2.2, the proof of Th. 3.11 is almost identical to the one of Th. 3.10. We point out the main differences here. In Lem. 3.9, replace D by

D = 0 h ∞ ∇ 2 F (x ) -I d 0
and set c(t) = 0. Then, in Lem. 3.22, replace the matrix

V 1/2 ∇ 2 F (x )V 1/2 by the Hessian ∇ 2 F (x ).
Convergence Rates of a Momentum Algorithm with Bounded Adaptive Stepsize for Non-convex Optimization Abstract In this chapter, we study the Adam algorithm for smooth non-convex optimization under a boundedness assumption on the adaptive learning rate. The bound on the adaptive stepsize depends on the Lipschitz constant of the gradient of the objective function and provides safe theoretical adaptive stepsizes. Under this boundedness assumption, we show a novel first order convergence rate result in both deterministic and stochastic contexts. Compared to the previous chapters, the results are of a different flavour. Instead of asymptotic results, we focus on "convergence rates" which are more common in the machine learning community. Furthermore, we establish convergence rates of the function value sequence using the Kurdyka-Łojasiewicz property, borrowing results for gradient-like sequences from the optimization community.

Contributions

Consider the unconstrained optimization problem min x∈R d f (x), where f : R d → R is a differentiable map and d is an integer. In this chapter, we focus on the class of adaptive algorithms among which Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is probably the most popular algorithm for optimizing the weights of neural networks. Recently, [START_REF] Reddi | On the convergence of adam and beyond[END_REF] exhibited a simple convex stochastic optimization problem over a compact set where Adam fails to converge because of its short-term gradient memory for specific values of its hyperparameters. Moreover, they proposed an algorithm called Amsgrad to fix this issue. This work opened the way to the emergence of other variants of Adam (see Section 4.3 for a detailed review). In this chapter, under a bounded stepsize assumption, we propose a convergence rate analysis of Adam-like algorithms for non-convex optimization.

Our contributions are as follows.

• We establish a convergence rate for Adam in the deterministic case for nonconvex optimization under a bounded stepsize. This algorithm can be seen as a deterministic clipped version of Adam which guarantees safe theoretical stepsizes. More precisely, if n is the number of iterations of the algorithm, we show a O(1/n) convergence rate of the minimum of the squared gradients norms by introducing a suitable Lyapunov function.

• We show a similar convergence result for non-convex stochastic optimization up to the limit of the variance of stochastic gradients under an almost surely bounded stepsize. In comparison to the literature, the hypothesis of the boundedness of the gradients is relaxed and the convergence result is independent of the dimension d of the parameters.

• We propose a convergence rate analysis of the objective function of the algorithm using the Kurdyka-Łojasiewicz (KŁ) property. To the best of our knowledge, this is the first time such a result is established for an adaptive optimization algorithm.

Table 4.1 -Some famous algorithms.

Algorithm

Effective stepsize a n+1 Momentum SGD [START_REF] Robbins | A stochastic approximation method[END_REF] a n+1 ≡ a b = 1 (no momentum) Adagrad [START_REF] Duchi | Adaptive subgradient methods for online learning and stochastic optimization[END_REF])

a n+1 = a n i=0 g 2 i -1/2 b = 1
Rmsprop [START_REF] Tieleman | Lecture 6.e-rmsprop: Divide the gradient by a running average of its recent magnitude[END_REF])

a n+1 = a + c n i=0 (1 -c) n-i g 2 i 1/2 -1 b = 1
Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF])

a n+1 = a + c n i=0 (1 -c) n-i g 2 i 1/2 -1 0 ≤ b ≤ 1 (close to 0)
Chapter organization. Section 5.4 introduces the algorithm we analyze. Section 4.3 considers some related works. Section 4.4 establishes first order convergence rates in terms of the minimum of the gradients norms in both deterministic and stochastic settings. Finally, Section 4.5 derives function value convergence rates under the KŁ property. All the proofs are deferred to the last sections of this chapter.

A momentum algorithm with adaptive stepsize

Notations. All operations between vectors of R d are to read coordinatewise. In particular, for two vectors x, y in R d and α ∈ Z, we denote by xy, x/y, x α the vectors on R d whose k-th coordinates are respectively given by x k y k , x k /y k , x α k . The vector of ones of R d is denoted by 1. When a scalar is added to a vector, it is added to each one of its coordinates. Inequalities are also to be read coordinatewise. If x ∈ R d , x ≤ λ ∈ R means that each coordinate of x is smaller than λ.

We investigate the following algorithm defined by two sequences (x n ) and (p n ) in R d :

   x n+1 = x n -a n+1 p n+1 p n+1 = p n + b (∇f (x n ) -p n ) (4.1)
where ∇f (x) is the gradient of f at point x, (a n ) is a sequence of vectors in R d with positive coordinates, b is a positive real constant and x 0 , p 0 ∈ R d .

Algorithm 5.1 includes the classical Heavy-ball method as a special case, but is much more general. Indeed, we allow the sequence of stepsizes (a n ) to be adaptive: a n ∈ R d may depend on the past gradients g k := ∇f (x k ) and the iterates x k for k ≤ n. We stress that the stepsize a n is a vector of R d and that the product a n+1 p n+1 in (5.1) is read componentwise (this is equivalent to the formulation with a diagonal matrix preconditioner applied to the gradient [START_REF] Mcmahan | Adaptive bound optimization for online convex optimization[END_REF][START_REF] Gupta | A unified approach to adaptive regularization in online and stochastic optimization[END_REF][START_REF] Agarwal | Efficient full-matrix adaptive regularization[END_REF][START_REF] Staib | Escaping saddle points with adaptive gradient methods[END_REF]).

We present in Table 4.1 how to recover some of the famous algorithms with a vector stepsize formulation. In particular, Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] defined by the iterates:

         x n+1 = x n - a + √ v n+1 p n+1 p n+1 = p n + b (∇f (x n ) -p n ) v n+1 = v n + c (∇f (x n ) 2 -v n ) (4.2)
for constants1 a ∈ R + , b, c ∈ [0, 1], can be seen as an instance of this algorithm by setting a n = a + √ vn where the vector v n , as defined above, is an exponential moving average of the gradient squared. For simplification, we omit bias correction steps for p n+1 and v n+1 . Their effect vanishes quickly along the iterations.

We introduce the main assumption on the objective function which is standard in gradient-based algorithms analysis.

Assumption 4.2.1. The mapping f : R d → R is:

(i) continuously differentiable and its gradient ∇f is L-Lipschitz continuous, (ii) bounded from below, i.e., inf x∈R d f (x) > -∞ .

Related works

Adaptive algorithms as Heavy Ball. Thanks to its small per-iteration cost and its acceleration properties (at least in the strongly convex case), the Heavy-ball method, also called gradient descent with momentum, recently regained popularity in large-scale optimization [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF]. This speeding up idea dates back to the sixties with the seminal work of [START_REF] Polyak | Some methods of speeding up the convergence of iteration methods[END_REF]. In order to tackle non-convex optimization problems, [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] proposed iPiano, a generalization of the well known heavy-ball in the form of a forward-backward splitting algorithm with an inertial force for the sum of a smooth possibly non-convex and a convex function. In the particular case of the Heavy-ball method, this algorithm writes for two sequences of reals (α n ) and (β n ):

x n+1 = x n -α n ∇f (x n ) + β n (x n -x n-1 ) . (4.3)
We remark that Algorithm 5.1 can be written in a similar fashion by choosing stepsizes α n = ba n+1 and inertial parameters β n = (1b)a n+1 /a n . [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] only consider the case where α n and β n are real-valued. Moreover, the latter does not consider adaptive stepsizes, i.e stepsizes depending on past gradient information. We can show some improvement with respect to [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] with weaker convergence conditions in terms of the stepsize of the algorithm (see Section 4.8.5) while allowing adaptive vector-valued stepsizes a n (see Prop. 4.9).

It is shown in [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] that the sequence of function values converges and that every limit point is a critical point of the objective function. Moreover, supposing that the Lyapunov function has the KŁ property at a cluster point, they show the finite length of the sequence of iterates and its global convergence to a critical point of the objective function. Similar results are shown in [START_REF] Wu | General inertial proximal gradient method for a class of nonconvex nonsmooth optimization problems[END_REF] for a more general version than iPiano [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF]) computing gradients at an extrapolated iterate like in Nesterov's acceleration.

Convergence rate. [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] determines a O(1/n) convergence rate (where n is the number of iterations of the algorithm) with respect to the proximal residual which boils down to the gradient for noncomposite optimization. Furthermore, a recent work introduces a generalization of the Heavy-ball method (and Nesterov's acceleration) to constrained convex optimization in Banach spaces and provides a non-asymptotic hamiltonian based analysis with O(1/n) convergence rate [START_REF] Diakonikolas | Generalized momentum-based methods: A hamiltonian perspective[END_REF]. In the same vein, in Section 4.4, we establish a similar convergence result for an adaptive stepsize instead of a fixed predetermined stepsize policy like in the Heavy-ball algorithm (see Th. 4.2).

Convergence rates under the KŁ property. The KŁ property is a powerful tool to analyze gradient-like methods. We elaborate on this property in Section 4.5. Assuming that the objective function satisfies this geometric property, it is possible to derive convergence rates. Indeed, some recent progress has been made to study convergence rates of the Heavy-ball algorithm in the non-convex setting. Ochs ( 2018) establishes local convergence rates for the iterates and the function values sequences under the KŁ property. The convergence proof follows a general method that is often used in non-convex optimization convergence theory. This framework was used for gradient descent [START_REF] Absil | Convergence of the iterates of descent methods for analytic cost functions[END_REF], for proximal gradient descent (see [START_REF] Attouch | On the convergence of the proximal algorithm for nonsmooth functions involving analytic features[END_REF] for an analysis with the Łojasiewicz inequality) and further generalized to a class of descent methods called gradient-like descent algorithms.

KŁ-based asymptotic convergence rates were established for constant Heavy-ball parameters [START_REF] Ochs | Local convergence of the heavy-ball method and ipiano for non-convex optimization[END_REF]. Asymptotic convergence rates based on the KŁ property were also shown [START_REF] Johnstone | Convergence rates of inertial splitting schemes for nonconvex composite optimization[END_REF] for a general algorithm solving non-convex nonsmooth optimization problems called Multi-step Inertial Forward-Backward splitting [START_REF] Liang | A multi-step inertial forward-backward splitting method for non-convex optimization[END_REF] which has iPiano and Heavy-ball methods as special cases. In this work, stepsizes and momentum parameter vary along the algorithm run and are not supposed constant. However, specific values are chosen and consequently, their analysis does not encompass adaptive stepsizes i.e. stepsizes that can possibly depend on past gradient information. In the present work, we establish similar convergence rates for methods such as Adam under a bounded stepsize assumption (see Th. 4.6). We also mention [START_REF] Li | Convergence analysis of proximal gradient with momentum for nonconvex optimization[END_REF] which analyzes the accelerated proximal gradient method for non-convex programming (APGnc) and establishes convergence rates of the function value sequence by exploiting the KŁ property. This algorithm is a descent method i.e. the function value sequence is shown to decrease over time. In the present work, we analyze adaptive algorithms which are not descent methods. Note that even Heavy-ball is not a descent method. Hence, our analysis requires additional treatments to exploit the KŁ property: we introduce a suitable Lyapunov function which is not the objective function. We also point out the recent work of [START_REF] Xie | Linear convergence of adaptive stochastic gradient descent[END_REF] which analyzes the AdaGrad-Norm algorithm under the global Polyak-Łojasiewicz condition. This condition is a particular case of the KŁ property (see Section 4.5).

Theoretical guarantees for Adam-like algorithms. The recent literature on adaptive optimization algorithms is vast. For instance, for AdaGrad-like algorithms, several works cover the non-convex setting [START_REF] Wu | Wngrad: Learn the learning rate in gradient descent[END_REF]Ward et al., 2019b;[START_REF] Xie | Linear convergence of adaptive stochastic gradient descent[END_REF][START_REF] Li | On the convergence of stochastic gradient descent with adaptive stepsizes[END_REF]. In the following, we almost exclusively focus on Adam-like algorithms which are different because of the momentum. The first type of convergence results uses the online optimization framework which controls the convergence rate of the average regret. This framework was adopted for AmsGrad, AdamNC [START_REF] Reddi | On the convergence of adam and beyond[END_REF], AdaBound and AmsBound [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF]. In this setting, it is assumed that the feasible set containing the iterates is bounded by adding a projection step to the algorithm if needed. We do not make such an assumption in our analysis. [START_REF] Reddi | On the convergence of adam and beyond[END_REF] establishes a regret bound in the convex setting.

The second type of theoretical results is based on the control of the norm of the (stochastic) gradients. We remark that some of these results depend on the dimension of the parameters. [START_REF] Zhou | On the convergence of adaptive gradient methods for nonconvex optimization[END_REF] improve this dependency in comparison to [START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF]. The convergence result in [START_REF] Basu | Convergence guarantees for rmsprop and adam in non-convex optimization and their comparison to nesterov acceleration on autoencoders[END_REF] is established under quite specific values of a n+1 , b n and . [START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF] show a O(1/n) convergence rate for an increasing mini-batch size. However, the proof is provided for RMSprop and seems difficult to adapt to Adam which involves a momentum term. Indeed, unlike RMSProp, Adam does not admit the objective function as a Lyapunov function.

We also remark that all the available theoretical results assume boundedness of the (stochastic) gradients. We do not make such an assumption. Furthermore, we do not add any decreasing 1/ √ n factor in front of the adaptive stepsize as it is considered in [START_REF] Reddi | On the convergence of adam and beyond[END_REF]; [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF] and [START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF]. Although constant hyperparameters b and c are used in practice, theoretical results are often established for non constant b n and c n [START_REF] Reddi | On the convergence of adam and beyond[END_REF][START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF]. We also mention that most of the theoretical bounds depend on the dimension of the parameter [START_REF] Reddi | On the convergence of adam and beyond[END_REF][START_REF] Zhou | On the convergence of adaptive gradient methods for nonconvex optimization[END_REF][START_REF] Chen | Closing the generalization gap of adaptive gradient methods in training deep neural networks[END_REF]Zou et al., 2019a;[START_REF] Chen | On the convergence of a class of adam-type algorithms for non-convex optimization[END_REF][START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF].

Other variants of Adam. Recently, several other algorithms were proposed in the literature to enhance Adam. Although these algorithms lack theoretical guarantees, they present interesting ideas and show good practical performance. For instance, AdaShift [START_REF] Zhou | Adashift: Decorrelation and convergence of adaptive learning rate methods[END_REF] argues that the convergence issue of Adam is due to its unbalanced stepsizes. To solve this issue, they propose to use temporally shifted gradients to compute the second moment estimate in order to decorrelate it from the first moment estimate. Nadam [START_REF] Dozat | Incorporating nesterov momentum into adam[END_REF]) incorporates Nesterov's acceleration into Adam, in order to improve its speed of convergence. Moreover, originally motivated by variance reduction, QHAdam [START_REF] Ma | Quasi-hyperbolic momentum and adam for deep learning[END_REF] replaces both Adam's moment estimates by quasi-hyperbolic terms and recovers Adam, Rmsprop and Nadam as particular cases (modulo the bias correction). Guided by the same variance reduction principle, Radam [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] estimates the variance of the effective stepsize of the algorithm and proposes a multiplicative variance correction to the update rule.

Stepsize bound. Perhaps, the closest idea to our algorithm is the recent AdaBound [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF] which considers a dynamic learning rate bound. [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF] show that extremely small and large learning rates can cause convergence issues to Adam and exhibit empirical situations where such an issue shows up. Inspired by the gradient clipping strategy proposed in [START_REF] Pascanu | On the difficulty of training recurrent neural networks[END_REF] to tackle the problem of vanishing and exploding gradients in training recurrent neural networks (see [START_REF] Zhang | Why gradient clipping accelerates training: A theoretical justification for adaptivity[END_REF] for recent progress), [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF] apply clipping to the effective stepsize of the algorithm in order to circumvent stepsize instability. More precisely, authors propose dynamic bounds on the learning rate of adaptive methods such as Adam or AmsGrad to solve the problem of extreme learning rates which can lead to poor performance. Initialized respectively at 0 and ∞, lower and upper bounds both converge smoothly to a constant final stepsize following a predetermined formula defined by the user. Consequently, the algorithm resembles an adaptive algorithm in the first iterations and becomes progressively similar to a standard SGD algorithm. Our approach is different: we propose a static bound on the adaptive learning rate which depends on the Lipschitz constant of the objective function. This bound stems naturally from our theoretical derivations.

First order convergence rate

Deterministic setting

Let (H n ) n≥0 be a sequence defined for all n ∈ N by H n := f (x n ) + 1 2b a n , p 2 n . We further assume the following stepsize growth condition.

Assumption 4.4.1. There exists α > 0 s.t. a n+1 ≤ an α .

Note that this assumption is satisfied for Adam with α = √ 1c where c is the parameter in (4.2). Unlike in AmsGrad [START_REF] Reddi | On the convergence of adam and beyond[END_REF], the stepsize a n is not necessarily nonincreasing. Indeed, α can be strictly smaller than 1 in Assumption 4.4.1 as it is the case for Adam.

We provide a proof of the following key lemma in Section 4.8.1.

Lemma 4.1. Let Assumptions 4.2.1 and 4.4.1 hold true. Then, for all n ∈ N, for all u ∈ R + ,

H n+1 ≤ H n -a n+1 p 2 n+1 , A n+1 - b 2 a n+1 (∇f (x n ) -p n ) 2 , B1 , (4.4)
where

A n+1 := 1 - a n+1 L 2 - |b -(1 -α)| 2u - 1 -α 2b , and
B := 1 - |b -(1 -α)|u b -(1 -α) .
We now state one of the principal convergence results about Algorithm 5.1. In particular, we establish a sublinear convergence rate for the minimum of the gradients norms until time n.

Theorem 4.2. Let Assumptions 4.2.1 and 4.4.1 hold true. Suppose that 1α < b ≤ 1.

Let ε > 0 s.t. a sup := 2 L 1 -(b-(1-α)) 2 2bα -1-α 2b -ε is nonnegative. Let δ > 0 s.t. for all n ∈ N, δ ≤ a n+1 ≤ min a sup , a n α . (4.5)
Then, the sequence (H n ) is nonincreasing and n p n 2 < ∞. In particular, lim x n+1 -

x n → 0 and lim ∇f (x n ) → 0 as n → +∞. Moreover, for all n ≥ 1, min 0≤k≤n-1 ∇f (x k ) 2 ≤ 4 nb 2 H 0 -inf f δε + p 0 2 .
Sketch of the proof. The key element of the proof is Lem. 4.1 which is a descent lemma on the function H. Indeed, the assumptions of the theorem guarantee that A n+1 ≥ ε and B ≥ 0. Then, the result stems from summing the inequalities of Lem. 4.1. The proof can be found in Section 4.8.3.

We provide some comments on this result.

Dimension dependence. Unlike most of the theoretical results for variants of Adam as gathered in Section 4.7, we remark that the bound does not depend on the dimension d of the parameter x k .

Comparison to gradient descent. A similar result holds for deterministic gradient descent (see Nesterov (2004, p.28)). If γ is a fix stepsize for gradient descent and there exist δ > 0, ε > 0 s.t. γ > δ and 1 -γL 2 > ε, then (see Section 4.8.6) for all n ≥ 1:

min 0≤k≤n-1 ∇f (x k ) 2 ≤ f (x 0 ) -inf f nγ(1 -γL 2 ) ≤ f (x 0 ) -inf f nδε .
When p 0 = 0 (this is the case for Adam), the bound in Th. 4.2 coincides with the gradient descent bound, up to the constant 4/b 2 . We mention however that ε for Algorithm 5.1 is defined by a slightly more restrictive condition than for gradient descent: when b = 1, there is no momentum and a sup = 1 L (1 -2ε) < 2/L. Hence, under the boundedness of the effective stepsize, the algorithm has a similar convergence guarantee to gradient descent. Remark that the stepsize bound almost matches the classical 2/L upperbound on the stepsize of gradient descent (see for example Nesterov (2004, Th. 2.1.14)).

Stepsize bound. Condition 4.5 should be seen as a clipping step of the algorithm. Indeed, the lower bound on the effective stepsize has not to be verified a posteriori after running the algorithm. Instead, a clipping of the learning rate would ensure that this boundedness assumption holds. Furthermore, if we drop the lower bound assumption on the effective stepsize a n from Th. 4.2, we still get the following result (see Prop. 4.9), for all n ≥ 1,

1 n n-1 k=0 a k+1 , ∇f (x k ) 2 ≤ 2(1 + α) nb 2 α H 0 -inf f ε + a 0 , p 2 0 .
Influence of ε and δ. In the specific case of Adam, we obtain La sup /2 + ε = 0.93 with the recommended default parameters b = 0.1 and c = 0.001. Hence, we can choose ε of the order of 0.1 without exceeding 0.93. In view of Equation (4.6), the smaller is ε and the larger will be the stepsizes. However, a small ε deteriorates the bounds of Ths. 4.2 and 4.3. Once b, c (and then α) are fixed, ε can be seen as a constant. The clipping parameter δ can also be seen as constant once it is chosen.

Stochastic setting

We establish a similar bound in the stochastic setting. Note that the control of the minimum of the gradients norms is also standard in non-convex stochastic optimization literature (see for e.g., [START_REF] Ghadimi | Stochastic first-and zeroth-order methods for nonconvex stochastic programming[END_REF]). Let (Ξ, S) denote a measurable space and d ∈ N. Consider the problem of finding a local minimizer of the expectation

F (x) := E(f (x, ξ)) w.r.t. x ∈ R d , where f : R d × Ξ → R is a measurable map and f ( . , ξ
) is a possibly non-convex function depending on some random variable ξ. The distribution of ξ is assumed to be unknown, but revealed online by the observation of iid copies (ξ n : n ≥ 1) of the r.v. ξ. For a fixed value of ξ, the mapping x → f (x, ξ) is supposed to be differentiable, and its gradient w.r.t. x is denoted by ∇f (x, ξ). We study a stochastic version of Algorithm 5.1 by replacing the deterministic gradient ∇f (x n ) by ∇f (x n , ξ n+1 ).

Theorem 4.3. Let Assumption 4.2.1 (for F ) and Assumption 4.4.1 hold true. Assume the following bound on the variance in stochastic gradients:

E ∇f (x, ξ) -∇F (x) 2 ≤ σ 2 for all x ∈ R d . Suppose moreover that 1 -α < b ≤ 1. Let ε > 0 s.t. āsup := 2 L 3 4 -(b-(1-α)) 2 2bα -1-α 2b -ε is nonnegative. Let δ > 0 s.t. for all n ≥ 1, almost surely, δ ≤ a n+1 ≤ min āsup , a n α . (4.6)
Then,

E[ ∇F (x τ ) 2 ] ≤ 4 nb 2 H 0 -inf f δε + p 0 2 + 4ā sup δεb 2 σ 2 ,
where x τ is an iterate uniformly randomly chosen from {x 0 , • • • , x n-1 }.

Remark 15. We recover the deterministic bound of Th. 4.2 when the gradients are noiseless (σ = 0). The complete proof is deferred to Section 4.8.4.

Before proceeding, a few remarks are in order.

SGD as a particular case. By setting b = 1 (no momentum) and a n+1 = a n for all n which implies α = 1, we recover a known rate for non-convex SGD (Ghadimi and Lan, 2013) with a maximal stepsize here of āsup = 1 2L (1 -2ε) and note that the proof can be slightly modified to make āsup as close as possible to 1/L. We highlight though that the Lyapunov function H was especially tailored to handle a momentum algorithm and an analysis with f as a Lyapunov function is largely satisfying for SGD.

RMSProp. In the particular case where there is no momentum in the algorithm (i.e. RMSProp) and assuming that the gradients are bounded, a similar convergence rate is obtained in Zaheer et al. (2018, Thm. 1) (see Section 4.7). Furthermore, although we assume boundedness of the stepsize by Condition (4.6), we do not suppose that a 1 ≤ 2L (see table in Section 4.7). The latter assumption imposes a very small stepsize ( = 10 -8 in [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]) which may result in a slow convergence.

Stepsize lower bound. In the case of Adam (a n = a + √ vn ), the uniform lower bound a n+1 ≥ δ prevents the exponential moving average v n of the squared gradients from exploding. This can be guaranteed on the fly by a clipping of a n . If we drop the uniform lower bound on the effective stepsize, we still obtain the following result (see Remark 17)

E   n-1 k=0 a k+1 , ∇f (x k , ξ k+1 ) 2   ≤ 2(1 + α) b 2 α H 0 -inf f ε + a 0 , p 2 0 + nā sup σ 2 ε .
Influence of the momentum parameter. Note that ε depends on the momentum parameter b and consequently the bound does not decrease with b. The influence of this parameter is more complex.

Convergence analysis under the KŁ property

Historically introduced by the fundamental works of Łojasiewicz (1963) and [START_REF] Kurdyka | On gradients of functions definable in o-minimal structures[END_REF], the KŁ inequality is the key tool of our analysis. We refer to [START_REF] Bolte | Characterizations of łojasiewicz inequalities: subgradient flows, talweg, convexity[END_REF] for an in-depth presentation of this property. The KŁ inequality is satisfied by a broad class of functions including most nonsmooth deep neural networks. More precisely, as exposed in Davis et al. (2020, Section 5.2, Corollary 5.11) and Castera et al. (2019, Section 2.2), feedforward neural networks with arbitrary number of layers of arbitrary dimensions, with activations such as sigmoid, ReLU, leaky ReLU, tanh, softplus (and many others), with a loss function such as l p norm, hinge loss, logistic loss or cross entropy (and many others), belong to this class of so-called definable functions in an o-minimal structure [START_REF] Kurdyka | On gradients of functions definable in o-minimal structures[END_REF][START_REF] Attouch | Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdykałojasiewicz inequality[END_REF][START_REF] Davis | Stochastic subgradient method converges on tame functions[END_REF]. We refer the interested reader to Zeng et al. (2019, Section 3, Section C) for general conditions for which KŁ inequality holds in the context of deep neural networks training models. The class of definable functions is stable under all the typical functional operations in optimization (e.g. sums, compositions, inf-projections) and generalizes the class of semialgebraic functions including objective functions such as • p for p rational, real polynomials, rank, etc. (see Bolte et al. (2014, Appendix)).

The KŁ inequality has been used to show the convergence of several first-order optimization methods towards critical points [START_REF] Attouch | On the convergence of the proximal algorithm for nonsmooth functions involving analytic features[END_REF][START_REF] Attouch | Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the kurdykałojasiewicz inequality[END_REF][START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF][START_REF] Li | Convergence analysis of proximal gradient with momentum for nonconvex optimization[END_REF]. In this section, we use a methodology exposed in Bolte et al. (2018, Appendix) to show convergence rates based on the KŁ property.

Recently developed in [START_REF] Bolte | Proximal alternating linearized minimization for nonconvex and nonsmooth problems[END_REF], this abstract convergence mechanism can be used for any descent type algorithm. We modify it to encompass momentum methods. Note that although this modification was initiated in [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF]; [START_REF] Ochs | Local convergence of the heavy-ball method and ipiano for non-convex optimization[END_REF], we use a different separable Lyapunov function. The first part of the proof follows these approaches and the second part follows the proof of Johnstone and Moulin (2017, Th. 2).

Consider the function

H : R d × R d → R defined for all z = (x, y) ∈ R d × R d by H(z) = H(x, y) = f (x) + 1 2b y 2 . (4.7) Notice that H n = f (x n ) + 1 2b a n , p 2 n = H(x n , y n ) where (y n ) n∈N is defined for all n ∈ N by y n = √ a n p n .
Notations and definitions. If (E, d) is a metric space, z ∈ E and A is a non-empty subset of E, we use the notation d(z, A) := inf{d(z, z ) : z ∈ A} . The set of critical points of the function H is defined by crit

H := {z ∈ R 2d s.t. ∇H(z) = 0} . Assumption 4.5.1. f is coercive, that is f (x) → +∞ as x → +∞.
Assumption 4.5.1 will be particularly useful to ensure that the sequence of the iterates (z k ) k≥0 of Algorithm 5.1 is bounded. Indeed, a coercive function has compact level sets and Lem. 4.1 will guarantee that the iterates lie in a level set of the function H.

We now introduce the limit point set of the sequence (z k ) k≥0 and exhibit some of its properties.

Definition 4.5.1. (Limit point set) The set of all limit points of (z k ) k∈N initialized at z 0 is defined by

ω(z 0 ) := {z ∈ R 2d : ∃ an increasing sequence of integers (k j ) j∈N s.t z k j → z as j → ∞} .
(i) ω(z 0 ) is a nonempty compact set. (ii) ω(z 0 ) ⊂ critH = critf × {0} . (iii) lim k→+∞ d(z k , ω(z 0 )) = 0.
(iv) H is finite and constant on ω(z 0 ).

We introduce the KŁ inequality in the following. Define [α < H < β] := {z ∈ R 2d : α < H(z) < β} . Let η > 0 and define Φ η as the set of continuous functions ϕ on [0, η) which are also continuously differentiable on (0, η), concave and satisfy ϕ(0) = 0 and ϕ > 0.

z ∈ dom H if there exist η > 0, ϕ ∈ Φ η and a neighborhood U (z) s.t. for all z ∈ U (z) ∩ [H(z) < H < H(z) + η] : ϕ (H(z) -H(z)) ∇H(z) ≥ 1 . (4.8)
When H(z) = 0, we can rewrite Eq. (4.8) as: ∇(ϕ • H)(z) ≥ 1 for suitable z points. This means that H becomes sharp under a reparameterization of its values through the so-called desingularizing function ϕ.

The function H is said to be a KŁ function if it has the KŁ property at each point of the domain of its gradient. Note that this property can be defined for nonsmooth functions using the Clarke subdifferential in order to encompass nonsmooth neural networks. We limit ourserlves to the simpler differentiable setting. KŁ inequality holds at any non critical point (see Attouch et al. (2010, Remark 3.2 (b))). We introduce now a uniformized version of the KŁ property which will be useful for our analysis. Let Ω be a compact set and let H : R 2d → (-∞, +∞] be a proper l.s.c function. Assume that H is constant on Ω and satisfies the KŁ property at each point of Ω. Then, there exist ε > 0, η > 0 and ϕ ∈ Φ η such that for all z ∈ Ω, for all z ∈ {z ∈ R

d : d(z, Ω) < ε} ∩ [H(z) < H < H(z) + η], one has ϕ (H(z) -H(z)) ∇H(z) ≥ 1 (4.9)
Definition 4.5.3. (KŁ exponent) If ϕ can be chosen as ϕ(s) = c θ s θ for some c > 0 and θ ∈ (0, 1] in Def. 4.5.2, then we say that H has the KŁ property at z with an exponent of θ2 . We say that H is a KŁ function with an exponent θ if it has the same exponent θ at any z.

In the particular case when θ = 1/2, we recover the Polyak-Łojasiewicz condition (see for e.g., [START_REF] Karimi | Linear convergence of gradient and proximalgradient methods under the polyak-łojasiewicz condition[END_REF]) satisfied for strongly convex functions. Furthermore, if H is a proper closed semialgebraic function, then H is a KŁ function with a suitable exponent θ ∈ (0, 1] . The slope of ϕ around the origin informs about the "flatness" of a function around a point. Hence, the KŁ exponent allows to obtain convergence rates. In the light of this remark, we state one of the main results of this work.

(i) If θ = 1, then f (x k ) converges in a finite number of iterations. (ii) If 1/2 ≤ θ < 1, then f (x k ) converges to f (x * ) linearly i.e. there exist q ∈ (0, 1), C > 0 s.t. f (x k ) -f (x * ) ≤ C q k . (iii) If 0 < θ < 1/2 , then f (x k ) -f (x * ) = O(k 1 2θ-1 ) .
The exact same rates hold for gradient descent by supposing that f (instead of H) is KL with exponent θ. Assumption 4.4.1 and condition (4.5) are not needed in this case.

Sketch of the proof. The proof consists of two main steps. The first one is to show that the iterates enter and stay in a region where the KŁ inequality holds. This is achieved using the properties of the limit set (Lem. 4.4) and the uniformized KŁ property (Lem. 4.5). Then, the second step is to exploit this inequality to derive the sought convergence results. We defer the complete proof to Section 4.9.3.

We introduce a lemma in order to make the KŁ assumption on the objective function f instead of the function H. Lemma 4.7. Let f be a continuously differentiable function satisfying the KL property at x with an exponent of θ ∈ (0, 1/2]. Then the function H defined in Eq. (4.7) has also the KŁ property at (x, 0) with an exponent of θ .

The following result derives a convergence rate on the objective function values under a KŁ assumption on this same function instead of an assumption on the Lyapunov function H. The result is an immediate consequence of Lem. 4.7 and Th. 4.6. Suppose moreover that f is a KŁ function with KŁ exponent θ ∈ (0, 1/2). Then, the sequence (H(z k )) k∈N converges to f (x * ) where x * is a critical point of f and

f (x k ) -f (x * ) = O(k 1 2θ-1 ) .
Toy problem: KŁ rates for f (x) = x p .

KŁ rates are asymptotic rates in the sense that the constants cannot be explicited in the convergence rates. As a consequence, the rates can be hardly observable in practice from experiments. However, we can still illustrate these convergence results (Th. 4.6) in a simple toy example to give more insight. Consider the problem of minimizing the function f (x) = x p for a real p ∈ [1, 7]. One can easily show that f is a KŁ function with KŁ exponent θ = 1 p . Note that the KŁ exponent is difficult to compute in general. This justifies the choice of this toy problem. Moreover, even if the function f is indeed convex, we recall the reader that the KŁ property is a local geometric property of the function that is only interesting at its critical points (since it is automatically verified at any non critical point). Notice that the KŁ analysis is valid in the general non-convex case. The present toy example remains relevant if we modify the objective function f to be non-convex and still keep a x p shape in a neighborhood of the point zero which is the unique critical point in this example.

The KŁ exponent as shown in the first plot in Figure 4.1 encodes information about the flatness of the function f . Indeed, as p increases, the function f gets flatter around the origin x = 0. We run the clipped version of Adam (see Algorithm 5.1), the Adam algorithm and gradient descent on the functions f corresponding to different values of the exponent θ, from the same initialization point x = 1. As expected from Th. 4.6 for the clipped Adam, we observe in Figure 4.1 that f (x k ) converges linearly or even in a finite number of iterations for p ∈ {1, 1.3, 1.5, 2}. Notice that the linear rate is clearly observable for p = 2 corresponding to θ = 1 2 . Even if we did not establish KŁ rates for original Adam, Figure 4.1 shows that it presents a very similar behavior to the clipped version of Adam in terms of KŁ convergence rates in this simple problem. We also represent gradient descent iterates for comparison. Note that KŁ rates are known to hold for gradient descent. Moreover, for p > 2, we also observe a slower rate corresponding to the sublinear rate of the function values.

Conclusion

In this chapter, we provided convergence rates for a clipped version of Adam which stems from a boundedness assumption on the effective stepsize of the original Adam. More precisely, similarly to gradient descent, we established a O(1/n) convergence rate of the minimum of the squared gradient norms in the deterministic case. Furthermore, we showed a similar convergence result in the stochastic setting up to the variance of the noisy gradients. Finally, we established function value convergence rates under the same boundedness assumption on the effective stepsizes together with the KŁ geometric property. This property is a powerful tool allowing to address non-convex nonsmooth optimization and covers most deep neural networks.

About theoretical guarantees of variants of Adam.

We list most of the existing variants of the Adam algorithm together with their theoretical convergence guarantees in Table 4.2.

Remark 16. The average regret bound result in the last line of Table 4.2 figures in [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF]. Actually, according to [START_REF] Savarese | On the convergence of adabound and its connection to sgd[END_REF], slightly different assumptions on the bound functions should be considered to guarantee this regret rate.

,i = [g 1,i , g 2,i , • • • , g T,i ] T . Algorithm Effective stepsize a n+1 b n c n Assumptions Convergence Result AmsGrad (1) , AdamNC (2)
( [START_REF] Reddi | On the convergence of adam and beyond[END_REF] a0

√ n 1 √ vn (1) vn+1 = max(vn, (1 -cn)vn + cng 2 n ) (2) vn+1 = (1 -cn)vn + cng 2 n 1 -b 1 λ n-1 or 1 -b1 n c n ≡ c 1 c1 n (for AdamNC) • convex functions • bounded gradients • bounded feasible set • d i=1 v1/2 T,i ≤ d (AmsGrad) • d i=1 g 1:T,i 2 ≤ √ dT • b 1 < √ c 1 (AdamNC) R T /T = O( logT /T ) RT T = O(1/ √ T ) (AdamNC)
Adam [START_REF] Basu | Convergence guarantees for rmsprop and adam in non-convex optimization and their comparison to nesterov acceleration on autoencoders[END_REF])

4 gn 2 η 3L(1-(1-b) n ) 2 (η+2σ) 2 1 + √ vn v n+1 = (1 -c 1 )v n + c 1 g 2 n b n ≡ b 1 = 1 -η η+2σ c n ≡ c 1 • σ-bounded gradients • = 2σ ∀η > 0 ∃n ≤ 9Lσ 2 (f (x2)-f (x * )) η 6 s.t. g n ≤ η
Padam,AmsGrad [START_REF] Zhou | On the convergence of adaptive gradient methods for nonconvex optimization[END_REF])

1 √ N 1 vp n 1 √ dN 1 v 1 2 n (AmsGrad) vn = max(v n-1 , (1 -c)v n-1 + cg 2 n ) b n ≡ b c n ≡ c • bounded gradients For Padam: • p ∈ [0, 1 4 ] • 1 -b < (1 -c) 2p • d i=1 g 1:N,i 2 ≤ √ dN AmsGrad: p = 1 2 and 1 -b < 1 -c E[ g τ 2 ] = O( 1+ √ d √ N + d N ) = O( d N + d N ) (AmsGrad) τ uniform r.v in {1, • • • , N } RmsProp (1) , Yogi (2)
( [START_REF] Zaheer | Adaptive methods for nonconvex optimization[END_REF] a1

+ √ vn (1) vn+1 = (1 -c)vn + cg 2 n (2) vn = vn-1 -csign(vn-1 -g 2 n ) b n ≡ b c n ≡ c • G-bounded gradients • a 1 ≤ √ 1-c 2L (Yogi) • a 1 ≤ 2L • c ≤ 2 16G 2 • σ 2 -bounded variance E[ g τ 2 ] = O( 1 N + σ 2 ) τ uniform r.v in {1, • • • , N } O( 1 N ) if minibatch Θ(N ) AmsGrad (1) , AdaFom (2)
(Chen et al., 2019)

1 √ n 1 √ vn (1) vn+1 = max(vn, (1 -cn)vn + cng 2 n ) (2) vn+1 = (1 -1 n )vn + 1 n g 2 n )
non-increasing

c n ≡ c 1 • bounded gradients • ∃c > 0 s.t. |g 1,i | ≥ c min n∈[0,N ] E[ g n 2 ] = O( logN +d 2 √ N )
Generic Adam (Zou et al., 2019a) αn

√ vn v n+1 = (1 -c n )v n + c n g 2 n α n = α √ 1-(1-c) n 1-(1-b) n b n ≥ b > 0 0 < c n < 1 non-increasing lim cn = c > b 2 • bounded gradients in expectation • d n ≤ αn √ cn ≤ c 0 d n d n non-increasing E[ g τ 4 3 ] 3 2 ≤ C+C N n=1 αn √ cn N αN τ uniform r.v in {1, • • • , N } AdaBound (1) , AMSBound (2)
( [START_REF] Luo | Adaptive gradient methods with dynamic bound of learning rate[END_REF])

1 √ n clip( α √ vn , η l (n), η u (n)) η l (n) non-decreasing to α * η u (n) non-increasing to α * (1) vn+1 = (1 -c)vn + cg 2 n (2) vn+1 = max(vn, (1 -c)vn + cg 2 n ) 1 -(1 -b)λ n-1 or 1 -1-b n b n ≥ b c n ≡ c • bounded gradients • closed convex bounded feasible set • 1 -b < √ 1 -c R T /T = O(1/ √ T )
4.8 Proofs for Section 4.4

4.8.1 Proof of Lem. 4.1

Supposing that ∇f is L-Lipschitz, using Taylor's expansion and the expression of p n in the algorithm, we obtain the following inequality:

f (x n+1 ) ≤ f (x n ) -∇f (x n ), a n+1 p n+1 + L 2 a n+1 p n+1 2 (4.10)
Moreover,

1 2b a n+1 , p 2 n+1 - 1 2b a n , p 2 n = 1 2b a n+1 , p 2 n+1 -p 2 n + 1 2b a n+1 -a n , p 2 n . (4.11) Observing that p 2 n+1 -p 2 n = -b 2 (∇f (x n ) -p n ) 2 + 2bp n+1 (∇f (x n ) -p n )
, we obtain after simplification:

H n+1 ≤ H n + L 2 a n+1 p n+1 2 - b 2 a n+1 , (∇f (x n )-p n ) 2 -a n+1 p n+1 , p n + 1 2b a n+1 -a n , p 2 n . (4.12) Using again p n = p n+1 -b(∇f (x n ) -p n ), we replace p n : H n+1 ≤ H n + L 2 a n+1 p n+1 2 - b 2 a n+1 , (∇f (x n ) -p n ) 2 -a n+1 , p 2 n+1 + b a n+1 p n+1 , ∇f (x n ) -p n + 1 2b a n+1 -a n , p 2 n .
Under Assumption 4.4.1, we write: a n+1a n , p 2 n ≤ (1α) a n+1 , p 2 n and using

p 2 n = p 2 n+1 + b 2 (∇f (x n ) -p n ) 2 -2bp n+1 (∇f (x n ) -p n )
, it holds that:

H n+1 ≤ H n -a n+1 , p 2 n+1 - b 2 a n+1 , (∇f (x n ) -p n ) 2 + L 2 a n+1 p n+1 2 + (b -(1 -α)) a n+1 p n+1 , ∇f (x n ) -p n + 1 -α 2b a n+1 , p 2 n+1 + b(1 -α) 2 a n+1 , (∇f (x n ) -p n ) 2 .
Using the classical inequality xy ≤ x 2 2u + uy 2 2 , we have:

(b-(1-α))a n+1 p n+1 (∇f (x n )-p n ) ≤ |b -(1 -α)| 2u a n+1 , p 2 n+1 + |b -(1 -α)|u 2 a n+1 , (∇f (x n )-p n ) 2 .
(4.13) Hence, after using this inequality and rearranging the terms, we derive the following inequality:

H n+1 ≤ H n -a n+1 p 2 n+1 , 1 - a n+1 L 2 - |b -(1 -α)| 2u - 1 -α 2b - b 2 a n+1 (∇f (x n ) -p n ) 2 , 1 - |b -(1 -α)|u b -(1 -α) 1 .
This concludes the proof.

A first result under an upperbound of the stepsize

Proposition 4.9. Let Assumption 4.2.1 hold true. Suppose moreover that 1-α < b ≤ 1.

Let ε > 0 s.t. a sup := 2 L 1 -(b-(1-α)) 2 2bα -1-α 2b -ε is nonnegative. Assume for all n ∈ N, a n+1 ≤ min a sup , a n α .
Then, for all n ≥ 1,

n-1 k=0 a k+1 , ∇f (x k ) 2 ≤ 2(1 + α) b 2 α H 0 -inf f ε + a 0 , p 2 0
Proof. This is a consequence of Lem. 4.1. Conditions A n+1 ≥ ε and B ≥ 0 write as follow:

a n+1 ≤ 2 L 1 - b -(1 -α) 2u - 1 -α 2b -ε and u ≤ αb b -(1 -α) .
We get the assumption made in the proposition by injecting the second condition into the first one and adding the assumption a n+1 an ≤ 1 α made in the lemma. Under this assumption, we sum over 0 ≤ k ≤ n -1 Eq. (4.4), rearrange it and use A n+1 ≥ ε, B ≥ 0 to obtain:

n-1 k=0 ε a k+1 , p 2 k+1 ≤ H 0 -H n , Then, observe that H n ≥ f (x n ) ≥ inf f . Therefore, we derive: n-1 k=0 a k+1 , p 2 k+1 ≤ H 0 -inf f ε . (4.14)
Moreover, from the Algorithm 5.1 second update rule, we get ∇f

(x k ) = 1 b p k+1 -1-b b p k . Hence, we have for all k ≥ 0 : ∇f (x k ) 2 ≤ 2 1 b 2 p 2 k+1 + (1 -b) 2 b 2 p 2 k ≤ 2 b 2 (p 2 k+1 + p 2 k ) .
We deduce that:

n-1 k=0 a k+1 , ∇f (x k ) 2 ≤ 2 b 2 n-1 k=0 a k+1 , p 2 k+1 + p 2 k = 2 b 2 n-1 k=0 a k+1 , p 2 k+1 + 2 b 2 n-1 k=0 a k+1 , p 2 k ≤ 2 b 2 n-1 k=0 a k+1 , p 2 k+1 + 2 b 2 α n-1 k=0 a k , p 2 k ≤ 2 b 2 (1 + 1 α ) n k=0 a k , p 2 k ≤ 2(1 + α) b 2 α H 0 -inf f ε + a 0 , p 2 0 . 4.8.3 Proof of Th. 4.2
This is a consequence of Lem. 4.1. Conditions A n+1 ≥ ε and B ≥ 0 write as follow:

a n+1 ≤ 2 L 1 - b -(1 -α) 2u - 1 -α 2b -ε and u ≤ αb b -(1 -α) .
We get the assumption made in the proposition by injecting the second condition into the first one and adding the assumption a n+1 an ≤ α made in the lemma. Under this assumption, we sum over 0 ≤ k ≤ n -1 Eq. (4.4), rearrange it and use A n+1 ≥ ε, B ≥ 0 and a k+1 ≥ δ to obtain:

n-1 k=0 δ ε p k+1 2 ≤ H 0 -H n ,
Then, observe that H n ≥ f (x n ) ≥ inf f . Therefore, we derive:

n-1 k=0 p k+1 2 ≤ H 0 -inf f δε . (4.15)
Moreover, from the algorithm 5.1 second update rule, we get ∇f

(x k ) = 1 b p k+1 -1-b b p k . Hence, we have for all k ≥ 0 : ∇f (x k ) 2 ≤ 2 1 b 2 p k+1 2 + (1 -b) 2 b 2 p k 2 ≤ 2 b 2 ( p k+1 2 + p k 2 ) .
We deduce that:

n-1 k=0 ∇f (x k ) 2 ≤ 2 b 2 n-1 k=0 ( p k+1 2 + p k 2 ) = 2 b 2   2 n-1 k=1 p k 2 + p n 2 + p 0 2   ≤ 4 b 2 n k=0 p k 2 . (4.16)
Finally, using Eqs. (4.15)-(4.16), we have:

min 0≤k≤n-1 ∇f (x k ) 2 ≤ 1 n n-1 k=0 ∇f (x k ) 2 ≤ 4 nb 2 H 0 -inf f δε + p 0 2 .

Proof of Th. 4.3

The proof of this proposition mainly follows the same path as its deterministic counterpart. However, due to stochasticity, a residual term (the last term in Eq. (4.17)) quantifying the difference between the stochastic gradient estimate and the true gradient of the objective function (compare Eq. (4.17) to Lem. 4.1) remains. Following the exact same steps of Section 4.8.1, we obtain by replacing the deterministic gradient ∇f (x n ) by its stochastic estimate ∇f (x n , ξ n+1 ) :

H n+1 ≤ H n -a n+1 p 2 n+1 , 1 - a n+1 L 2 - |b -(1 -α)| 2u - 1 -α 2b - b 2 a n+1 (∇f (x n , ξ n+1 ) -p n ) 2 , 1 - |b -(1 -α)|u b -(1 -α) 1 + ∇f (x n , ξ n+1 ) -∇F (x n ), a n+1 p n+1 .
(4.17)

Using the classical inequality xy ≤ x 2 2η + ηy 2 2 with η = 1/2 and the almost sure boundedness of the stepsize a n+1 , we get:

∇f (x n , ξ n+1 ) -∇F (x n ), a n+1 p n+1 ≤ (∇f (x n , ξ n+1 ) -∇F (x n )) 2 + 1 4 p 2 n+1 , a n+1 ≤ āsup ∇f (x n , ξ n+1 ) -∇F (x n ) 2 + 1 4 a n+1 , p 2 n+1 .
Therefore, taking the expectation and using the boundedness of the variance, we obtain from Eq. (4.17):

E[H n+1 ] -E[H n ] ≤ -E a n+1 p 2 n+1 , 3 4 - a n+1 L 2 - |b -(1 -α)| 2u - 1 -α 2b + āsup σ 2 .
Then, the proof follows the lines of Section 4.8.2. Hence, we have

E[H n+1 ] -E[H n ] ≤ -E a n+1 p 2 n+1 , ε1 + āsup σ 2 .
We sum these inequalities for k = 0, • • • , n -1, inject the assumption a n+1 ≥ δ and rearrange the terms to obtain

δ E   n-1 k=0 p k+1 2   ≤ E   n-1 k=0 a k+1 , p 2 k+1   ≤ H 0 -inf f ε + nā sup σ 2 ε . (4.18)
Then, using ∇f (x k , ξ k+1 ) = 1 b p k+1 -1-b b p k and a similar upperbound to Eq. ( 4.16) we show that

n-1 k=0 ∇f (x k , ξ k+1 ) 2 ≤ 4 b 2 n k=0 p k 2 . (4.19)
Therefore, combining Eqs. (4.19)-(4.18), we establish the following inequality

E   n-1 k=0 ∇f (x k , ξ k+1 ) 2   ≤ 4 b 2 H 0 -inf f δε + p 0 2 + 4ā sup n δεb 2 σ 2 .
Finally, we apply Jensen's inequality to • 2 and divide the previous inequality by n to obtain the sought result

1 n n-1 k=0 E ∇F (x k ) 2 ≤ 4 nδb 2 H 0 -inf f δε + p 0 2 + 4ā sup δεb 2 σ 2 .
Remark 17. Following the derivations in Section 4.8.2, note that we also obtain the following result

E   n-1 k=0 a k+1 , ∇f (x k , ξ k+1 ) 2   ≤ 2(1 + α) b 2 α H 0 -inf f ε + a 0 , p 2 0 + nā sup σ 2 ε .

Comparison to Ochs et al. (2014)

We recall the conditions satisfied by α n and β n in [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] in order to traduce them in terms of the algorithm (5.1) at stake. Define:

δ n := 1 α n - L 2 - β n 2α n γ n := δ n - β n 2α n .
Conditions of [START_REF] Ochs | ipiano: Inertial proximal algorithm for nonconvex optimization[END_REF] write:

α n ≥ c 1 β n ≥ 0 δ n ≥ γ n ≥ c 2
where c 1 , c 2 are positive constants and (δ n ) is monotonically decreasing.

One can remark that Algorithm 5.1 can be written as (4.3) with stepsizes α n = ba n+1 and inertial parameters

β n = (1 -b) a n+1
an . Conditions on these parameters can be expressed in terms of a n . Supposing c 2 = 0, the condition γ n ≥ c 2 is equivalent to

a n+1 a n ≤ 2 2 -b(2 -a n L) . (4.20)
(ii) There exists a positive scalar ρ 2 s.t.:

∇H(z k+1 ) ≤ ρ 2 ( x k+1 -x k + x k -x k-1 ) ∀k ≥ 1. (iii) (continuity condition) If z is a limit point of a subsequence (z k j ) j∈N , then lim j→+∞ H(z k j ) = H(z).
Remark 18. Note that the conditions in Lem. 4.10 can be generalized to a nonsmooth objective function. Indeed, in Bolte et al. (2018, Appendix), the Fréchet subdifferential replaces the gradient.

Proof.

(i) From Lem. 4.1 and Th. 4.2, we get for all k ∈ N:

H(z k+1 )-H(z k ) ≤ -ε a k+1 , p 2 k+1 ≤ -ε a k+1 , x k+1 -x k -a k+1 2 ≤ - ε a sup x k+1 -x k 2 .
We set ρ 1 := ε asup .

(ii) First, observe that for all k ∈ N

∇H(z k+1 ) ≤ ∇f (x k+1 ) + 1 b y k+1 . (4.22)
Now, let us upperbound each one of these two terms. Recall that we can rewrite our algorithm under a "Heavy-ball"-like form as follows:

x k+1 = x k -α k ∇f (x k ) + β k (x k -x k-1 ) ∀k ≥ 1.
where α k := ba k+1 and

β k = (1 -b) a k+1
a k are vectors. On the one hand, using the L-Lipschitz continuity of the gradient, we obtain

∇f (x k+1 ) 2 ≤ 2 ∇f (x k+1 ) -∇f (x k ) 2 + ∇f (x k ) 2 ≤ 2 L 2 x k+1 -x k 2 + ∇f (x k ) 2 Moreover, ∇f (x k ) 2 = x k -x k+1 α k + β k α k (x k -x k-1 ) 2 ≤ 2 x k -x k+1 ba k+1 2 + 2 1 -b b 1 a k (x k -x k-1 ) 2 ≤ 2 b 2 δ 2 x k+1 -x k 2 + 2(1 -b) 2 b 2 δ 2 x k -x k-1 2 ≤ 2 b 2 δ 2 ( x k+1 -x k 2 + x k -x k-1 2 ). Hence, ∇f (x k+1 ) 2 ≤ 2 L 2 x k+1 -x k 2 + ∇f (x k ) 2 ≤ 2 L 2 + 2 b 2 δ 2 x k+1 -x k 2 + 4 b 2 δ 2 x k -x k-1 2 ≤ 2 L 2 + 2 b 2 δ 2 ( x k+1 -x k 2 + x k -x k-1 2 ) .
Therefore, the following inequality holds:

∇f (x k+1 ) ≤ 2 L 2 + 2 b 2 δ 2 ( x k+1 -x k + x k -x k-1 ) .
On the otherhand,

y k+1 = √ a k+1 p k+1 = x k+1 -x k √ a k+1 ≤ 1 √ δ x k+1 -x k .
Finally, combining the inequalities for both terms in Eq. ( 4.22), we obtain

∇H(z k+1 ) ≤ ρ 2 ( x k+1 -x k + x k -x k-1 ) ∀k ≥ 1 . with ρ 2 := 2 L 2 + 2 b 2 δ 2 + 1 b √ δ .
(iii) This is a consequence of the continuity of H.

4.9.2 Proof of Lem. 4.4

(i) By Th. 4.2, the sequence (H(z n )) n∈N is nonincreasing. Therefore, for all n ∈ N, H(z n ) ≤ H(z 0 ) and hence z n ∈ {z : H(z) ≤ H(z 0 )} . Since f is coercive,
H is also coercive and its level sets are bounded. As a consequence, (z n ) n∈N is bounded and there exist z * ∈ R d and a subsequence (z

k j ) j∈N s.t. z k j → z * as j → ∞. Hence, ω(z 0 ) = ∅ . Furthermore, ω(z 0 ) = q∈N k≥q {z k } is compact as an intersection of compact sets. (ii) First, critH = critf × {0} because ∇H(z) = (∇f (x), y/b) T . Let z * ∈ ω(z 0 ).
Recall that x k+1x k → 0 as k → ∞ by Th. 4.2. We deduce from the second assertion of Lem. 4.10 that ∇H(z k ) → 0 as k → ∞ . As z * ∈ ω(z 0 ), there exists a subsequence (z k j ) j∈N converging to z * . Then, by Lipschitz continuity of ∇H, we get that ∇H(z k j ) → ∇H(z * ) as j → ∞ . Finally, ∇H(z * ) = 0 since ∇H(z k ) → 0 and (∇H(z k j )) j∈N is a subsequence of (∇H(z n )) n∈N .

(iii) This point stems from the definition of limit points. Every subsequence of the sequence (d(z k , ω(z 0 ))) k∈N converges to zero as a consequence of the definition of ω(z 0 ).

(iv) The sequence (H(z n )) n∈N is nonincreasing by Th. 4.2. It is also bounded from below because H(z k ) ≥ f (x k ) ≥ inf f for all k ∈ N. Hence we can denote by l its limit. Let z ∈ ω(z 0 ). There there exists a subsequence (z k j ) j∈N converging to z as j → ∞ . By the third assertion of Lem. 4.10, lim j→+∞ H(z k j ) = H(z) . Hence this limit equals l since (H(z n )) n∈N converges towards l. Therefore, the restriction of H to ω(z 0 ) equals l .

4.9.3 Proof of Th. 4.6

The first step of this proof follows the same path as Bolte et al. (2018, Proof of Th. 6.2, Appendix). Since f is coercive, H is also coercive. The sequence (H(z k )) k∈N is nonincreasing. Hence, (z k ) is bounded and there exists a subsequence (z kq ) q∈N and z ∈ R 2d s.t. z kq → z as q → ∞ . Then, since (H(z k )) k∈N is nonincreasing and lowerbounded by inf f , it is convergent and we obtain by continuity of H,

lim k→+∞ H(z k ) = H(z) . (4.23)
Using Th. 4.2, observe that the sequence (y k ) converges to zero since (a k ) is bounded and

p k → 0. If there exists k ∈ N s.t. H(zk) = H(z) , then H(zk +1 ) = H(z)
and by the first point of Lem. 4.10, xk +1 = xk and then (x k ) k∈N is stationary and for all k ≥ k , H(z k ) = H(z) and the results of the theorem hold in this case (note that z ∈ critH by Lem. 4.4). Therefore, we can assume now that H(z) < H(z k )∀k > 0 since (H(z k )) k∈N is nonincreasing and Eq. (4.23) holds. One more time, from Eq. (4.23), we have that for all η > 0, there exists k 0 ∈ N s.t. H(z k ) < H(z) + η for all k > k 0 . From Lem. 4.4, we get d(z k , ω(z 0 )) → 0 as k → +∞ . Hence, for all ε > 0, there exists k 1 ∈ N s.t. d(z k , ω(z 0 )) < ε for all k > k 1 . Moreover, ω(z 0 ) is a nonempty compact set and H is finite and constant on it. Therefore, we can apply the uniformization Lem. 4.5 with Ω = ω(z 0 ). Hence, for any k > l := max(k 0 , k 1 ), we get

ϕ (H(z k ) -H(z)) 2 ∇H(z k ) 2 ≥ 1 . (4.24)
This completes the first step of the proof which is illustrated in Figure 4.2.

In the second step, we follow the proof of Johnstone and Moulin (2017, Th. 2). Using Lem. 4.10 .(i)-(ii), we can write for all k ≥ 1,

∇H(z k+1 ) 2 ≤ 2ρ 2 2 ( x k+1 -x k 2 + x k -x k-1 2 ) ≤ 2ρ 2 2 ρ 1 (H(z k-1 ) -H(z k+1 )) .
Injecting the last inequality in Eq. (4.24), we obtain for all k > k 2 := max(l, 2),

2ρ 2 2 ρ 1 ϕ (H(z k ) -H(z)) 2 (H(z k-2 ) -H(z k )) ≥ 1 .
Now, use ϕ (s) = cs θ-1 to derive the following for all k > k 2 : -θ) .

[H(z k-2 ) -H(z)] -[H(z k ) -H(z)] ≥ ρ 1 2ρ 2 2 c2 [H(z k ) -H(z)] 2(1
(4.25)

:= H(z k ) -H(z) and C 1 = ρ 1 2ρ 2 2 c2
. Then, we can rewrite Eq. (4.25) as

r k-2 -r k ≥ C 1 r 2(1-θ) k ∀k > k 2 .
(4.26)

We distinguish three different cases to obtain the sought results.

(i) θ = 1 : Suppose r k > 0 for all k > k 2 . Then, since we know that r k → 0 by Eq. (4.23), C 1 must be equal to 0. This is a contradiction. Therefore, there exist k 3 ∈ N s.t. r k = 0 for all k > k 3 (recall that (r k ) k∈N is nonincreasing).

(ii) θ ≥ 1 2 : As r k → 0, there exists k 4 ∈ N s.t. for all k ≥ k 4 , r k ≤ 1 . Observe that 2(1θ) ≤ 1 and hence r k-2r k ≥ C 1 r k for all k > k 2 and then

r k ≤ (1 + C 1 ) -1 r k-2 ≤ (1 + C 1 ) -p 1 r k 4 . (4.27)
where

p 1 := k-k 4 2 . Notice that p 1 > k-k 4 -2 2
. Thus, the linear convergence result follows. Note also that if θ = 1/2, 2(1θ) = 1 and Eq. (4.27) holds for all k > k 2 .

(iii) θ < 1 2 : Define the function h by h(t) = D 1-2θ t 2θ-1 where D > 0 is a constant. Then,

h(r k ) -h(r k-2 ) = r k r k-2 h (t)dt = D r k-2 r k t 2θ-2 dt ≥ D (r k-2 -r k ) r 2θ-2 k-2 .
We disentangle now two cases:

(a) Suppose 2r 2θ-2 k-2 ≥ r 2θ-2 k
. Then, by Eq. (4.26), we get . We can suppose without loss of generality that r k are all positive. Otherwise, if there exists p such that r p = 0, the sequence (r k ) k∈N will be stationary at 0 for all k ≥ p . Observe that 2θ -2 < 2θ -1 < 0, thus 2θ-1 2θ-2 > 0 . As a consequence, we can write in this case r 2θ-1 k > q r 2θ-1 k-2 where q := 2

h(r k ) -h(r k-2 ) = D (r k-2 -r k ) r 2θ-2 k-2 ≥ C 1 D 2 . ( 4
2θ-1 2θ-2 > 1 . Therefore, using moreover that the sequence (r k ) k∈N is nonincreasing and 2θ -1 < 0, we derive the following

h(r k ) -h(r k-2 ) = D 1 -2θ (r 2θ-1 k -r 2θ-1 k-2 ) > D 1 -2θ (q -1)r 2θ-1 k-2 > D 1 -2θ (q -1)r 2θ-1 k 2 := C 2 . (4.29)
Combining Eq. (4.28) and Eq. (4.29) yields h(r k ) ≥ h(r k-2 ) + C 3 where

C 3 := min(C 2 , C 1 D 2 ) . Consequently, h(r k ) ≥ h(r k-2 p 2 ) + p 2 C 3 where p 2 := k-k 2 2
. We deduce from this inequality that

h(r k ) ≥ h(r k ) -h(r k-2 p 2 ) ≥ p 2 C 3 .
Therefore, rearranging this inequality using the definition of h, we obtain r

1-2θ k ≤ D 1-2θ (C 3 p 2 ) -1 . Then, since p 2 > k-k 2 -2 2 , r k ≤ C 4 p 1 2θ-1 2 ≤ C 4 k -k 2 -2 2 1 2θ-1 . where C 4 := C 3 (1-2θ) D 1 2θ-1 .
We conclude the proof by observing that f (x k ) ≤ H(z k ) and recalling that z ∈ critH . 4.9.4 Proof of Lem. 4.7 Since f has the KŁ property at x with an exponent θ ∈ (0, 1/2], there exist c, ε and ν > 0 s.t.

∇f (x)

1 1-θ ≥ c(f (x) -f (x)) (4.30) for all x ∈ R d s.t. x -x ≤ ε and f (x) < f (x) + ν where condition f (x) -f (x)
is dropped because Eq. (4.30) holds trivially otherwise. Let z = (x, y) ∈ R 2d be s.t.

xx ≤ ε , y ≤ ε and H(x, 0) < H(x, y) < H(x, 0) + ν . We assume that ε < b (ε can be shrunk if needed). We have f (x) ≤ H(x, y) < H(x, 0) + ν = f (x) + ν . Hence Eq. (4.30) holds for these x.

By concavity of u → u 1 2(1-θ) , we obtain ∇H(x, y)

1 1-θ ≥ C 0   ∇f (x) 1 1-θ + y b 1 1-θ   where C 0 := 2 1 2(1-θ) -1 .
Hence, using Eq. (4.30), we get ∇H(x, y)

1 1-θ ≥ C 0   c (f (x) -f (x)) + y b 1 1-θ   . Observe now that 1 1-θ ≥ 2 and y b ≤ ε b ≤ 1. Therefore, y b 1 1-θ ≥ y/b 2 .
Finally,

∇H(x, y) 1 1-θ ≥ C 0 c (f (x) -f (x)) + 2 b 1 2b y 2 ≥ C 0 min c, 2 b f (x) -f (x) + 1 2b y 2 = C 0 min c, 2 b H(x, y) -H(x, 0) .
This completes the proof.

Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation

Abstract Actor-critic methods integrating target networks have exhibited a stupendous empirical success in deep reinforcement learning. However, a theoretical understanding of the use of target networks in actor-critic methods is largely missing in the literature. In this chapter, we bridge this gap between theory and practice by proposing the first theoretical analysis of an online target-based actor-critic algorithm with linear function approximation in the discounted reward setting. Our algorithm uses three different timescales: one for the actor and two for the critic. Instead of using the standard single timescale temporal difference (TD) learning algorithm as a critic, we use a two timescales target-based version of TD learning closely inspired from practical actor-critic algorithms implementing target networks. First, we establish asymptotic convergence results for both the critic and the actor under Markovian sampling. Then, we provide a finite-time analysis showing the impact of incorporating a target network into actor-critic methods.

Introduction

Actor-critic algorithms [START_REF] Barto | Neuronlike adaptive elements that can solve difficult learning control problems[END_REF][START_REF] Konda | Actor-critic-type learning algorithms for markov decision processes[END_REF][START_REF] Konda | On actor-critic algorithms[END_REF][START_REF] Peters | Natural actor-critic[END_REF][START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF] are a class of reinforcement learning (RL) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF][START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF] methods to find an optimal policy maximizing the total expected reward in a stochastic environment modelled by a Markov Decision Process (MDP) [START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF]. In this type of algorithms, two main processes interplay: the actor and the critic. The actor updates a parameterized policy in a direction of performance improvement whereas the critic estimates the current policy of the actor by estimating the unknown state-value function. In turn, the critic estimation is used to produce the update rule of the actor. Combined with deep neural networks as function approximators of the value function, actor-critic algorithms witnessed a tremendous success in a range of challenging tasks [START_REF] Heess | Memory-based control with recurrent neural networks[END_REF][START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF][START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF][START_REF] Fujimoto | Addressing function approximation error in actorcritic methods[END_REF][START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. Apart from using neural networks for function approximation (FA), one of the main features underlying their remarkable empirical achievements is the use of target networks for the critic estimation of the value function. Introduced by the seminal work of [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] to stabilize the training process, this target innovation consists in using two neural networks maintaining two copies of the estimated value function: A so-called target network tracking a main network with some delay computes the target values for the value function update.

Despite their resounding empirical success in deep RL, a theoretical understanding of the use of target networks in actor-critic methods is largely missing in the literature. Theoretical contributions investigating the use of a target network are very recent and limited to temporal difference (TD) learning for policy evaluation [START_REF] Lee | Target-based temporal-difference learning[END_REF] and critic-only methods such as Q-learning for control (Zhang et al., 2021b). In particular, these works are not concerned with actor-critic algorithms and leave the question of the finite-time analysis open.

In the present work, we bridge this gap between theory and practice by proposing the first theoretical analysis of an online target-based actor-critic algorithm in the discounted reward setting. We consider the linear FA setting where a linear combination of preselected feature (or basis) functions estimates the value function in the critic. An analysis of this setting is an insightful first step before tackling the more challenging nonlinear FA setting aligned with the use of neural networks. We conduct our study in the multiple timescales framework. In the standard two timescales actor-critic algorithms [START_REF] Konda | On actor-critic algorithms[END_REF][START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF], at each iteration, the actor and the critic are updated simultaneously but the actor evolves more slowly than the critic, using smaller stepsizes than the latter. We face two main challenges due to the integration of the target variable mechanism. First, in contrast to standard two timescales actor-critic algorithms, our algorithm uses three different timescales: one for the actor and two for the critic. Instead of using the single timescale TD learning algorithm as a critic, we use a two timescales target-based version of TD learning closely inspired from practical actor-critic algorithms implementing target networks. Second, incorporating a target variable into the critic results in the intricate interplay between three processes evolving on three different timescales. In particular, the use of a target variable significantly modifies the dynamics of the actor-critic algorithm and deserves a careful analysis accordingly.

Our main contributions are summarized as follows. First, we prove asymptotic convergence results for both the critic and the actor. More precisely, as the actor parameter changes slowly compared to the critic one, we show that the critic which uses a target variable tracks a slowly moving target corresponding to the well-known TD solution [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF]. Our development is based on the classical ordinary differential equation (ODE) method of stochastic approximation (see, for e.g., [START_REF] Benveniste | Adaptive algorithms and stochastic approximations[END_REF]; [START_REF] Borkar | Stochastic approximation. A dynamical systems viewpoint[END_REF]). Then, we show that the actor parameter visits infinitely often a region of the parameter space where the norm of the policy gradient is dominated by a bias due to linear FA. Second, we conduct a finite-time analysis of our actor-critic algorithm which shows the impact of using a target variable on the convergence rates and the sample complexity. Loosely speaking, up to a FA error, we show that our target-based algorithm converges in expectation to an -approximate stationary point of the non-concave performance function using at most O( -3 log 3 1 ) samples compared with O( -2 log( 1)) for the best known complexity for two timescales actor-critic algorithms without a target network. All the proofs are deferred to Sections 5.7 and 5.8.

Related works

In this section, we briefly discuss the most relevant related works to ours. Existing theoretical results in the literature can be divided into two classes.

Asymptotic results. Almost sure convergence results are referred to as asymptotic. [START_REF] Konda | On actor-critic algorithms[END_REF][START_REF] Konda | Actor-Critic Algorithms[END_REF] provided almost sure (with probability one) convergence results for a two timescales actor-critic algorithm in which the critic estimates the action-value function via linear FA. Our algorithm is closer to an actor-critic algorithm introduced by [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF] in the average reward setting. However, unlike [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF], we consider the discounted reward setting and integrate a target variable mechanism into our critic. Moreover, as previously mentioned, the target variable for the critic adds an additional timescale in comparison to [START_REF] Konda | On actor-critic algorithms[END_REF]; [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF] which only involve two different timescales. Regarding theoretical results considering target networks, [START_REF] Lee | Target-based temporal-difference learning[END_REF] proposed a family of single timescale target-based TD learning algorithms for policy evaluation. Our critic corresponds to a two timescales version of the single timescale target-based TD learning algorithm of (Lee and He, 2019, Algorithm 2) called Averaging TD. In (Lee and He, 2019, Th. 1), this single timescale algorithm is shown to converge with probability one (w.p.1) towards the standard TD solution solving the projected Bellman equation (see [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF] for a precise statement). Besides the timescales difference with [START_REF] Lee | Target-based temporal-difference learning[END_REF], in this article, we are concerned with a control setting in which the policy changes at each timestep via the actor update. [START_REF] Yang | Convergent reinforcement learning with function approximation: A bilevel optimization perspective[END_REF] proposed a bilevel optimization perspective to analyze Q-learning with a target network and an actor-critic algorithm without any target network. More recently, Zhang et al. (2021b) investigated the use of target networks in Q-learning with linear FA and a target variable with Ridge regularization. Their analysis covers both the average and discounted reward settings and establishes asymptotic convergence results for both policy evaluation and control. This recent work of Zhang et al. (2021b) focuses on the critic-only Q-learning method with a target network update rule, showing the role of the target network in the off-policy setting. In particular, this work is not concerned with actor-critic algorithms.

Finite-time analysis. The second type of results consists in establishing timedependent bounds on some error or performance quantities such as the average expected norm of the gradient of the performance function. These are referred to as finite-time analysis. In the last few years, several works proposed finite-time analysis for TD learning [START_REF] Bhandari | A finite time analysis of temporal difference learning with linear function approximation[END_REF][START_REF] Srikant | Finite-time error bounds for linear stochastic approximation and td learning[END_REF] for two timescales TD methods [START_REF] Xu | Two time-scale off-policy td learning: Non-asymptotic analysis over markovian samples[END_REF] and even more generally for two timescales linear stochastic approximation algorithms (Gupta et al., 2019;[START_REF] Dalal | Finite sample analysis of two-timescale stochastic approximation with applications to reinforcement learning[END_REF][START_REF] Kaledin | Finite time analysis of linear two-timescale stochastic approximation with markovian noise[END_REF]. These works opened the way to the recent development of a flurry of nonasymptotic results for actor-critic algorithms [START_REF] Yang | A finite sample analysis of the actorcritic algorithm[END_REF][START_REF] Qiu | On the finite-time convergence of actor-critic algorithm[END_REF][START_REF] Kumar | On the sample complexity of actor-critic method for reinforcement learning with function approximation[END_REF][START_REF] Hong | A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic[END_REF][START_REF] Xu | A finite-time analysis of q-learning with neural network function approximation[END_REF][START_REF] Wang | Neural policy gradient methods: Global optimality and rates of convergence[END_REF]Wu et al., 2020;[START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF]. Regarding online one-step actor-critic algorithms, Wu et al. (2020) provided a finite-time analysis of the standard two timescales actor-critic algorithm (Bhatnagar et al., 2009, Algorithm 1) in the average reward setting with linear FA. [START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF] conducted a similar study for a revisited version of the asynchronous advantage actor-critic (A3C) algorithm in the discounted setting. None of the mentioned works uses a target network. In this work, we conduct a finite-time analysis of our target-based actor-critic algorithm. Such new results are missing in all theoretical results investigating the use of a target network [START_REF] Lee | Target-based temporal-difference learning[END_REF]Zhang et al., 2021b).

The summary table 5.1 compiles some key features of our work to situate it in the literature and highlights our contributions with respect to (w.r.t.) the closest related works.

Preliminaries

Notation. For every finite set X, we use the notation P(X) for the set of probability measures on X. The cardinality of a finite set Y is denoted by |Y|. For two sequences of nonnegative reals (x n ) and (y n ), the notation x n = O(y n ) means that there exists a constant C independent of n such that x n ≤ Cy n for all n ∈ N . For any integer p, the 1 refers to the use of samples generated from the MDP and the acting policy, this excludes experience replay as in [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] and identically independently distributed (i.i.d.) samples used in theoretical analysis. 2 Lee & and He [START_REF] Lee | Target-based temporal-difference learning[END_REF] provide a finite-time analysis for a target-based TD-learning algorithm (for policy evaluation) based on the periodic update style of the target variable used in [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] involving two loops. They highlight that a finite-time analysis of the Polyak-averaging style update rule [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] is an open question. Here, we address this question in the control setting.

euclidean space R p is equipped with its usual inner product •, • and its corresponding 2-norm • . For any integer d and any matrix A ∈ R d×p , we use the notation A for the operator norm induced by the euclidean vector norm. For a symmetric positive semidefinite matrix B ∈ R p×p and a vector x ∈ R p , the notation x 2 B refers to the quantity x, Bx . The transpose of the vector x is denoted by x T and I p is the identity matrix.

Markov decision process and problem formulation

Consider the RL setting [START_REF] Sutton | Reinforcement learning: An introduction[END_REF][START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF][START_REF] Szepesvári | Algorithms for reinforcement learning[END_REF] where a learning agent interacts with an environment modeled as an infinite horizon discrete-time discounted MDP. We denote by S = {s 1 , • • • , s n } the finite set of states and A the finite set of actions. Let p : S × A → P(S) be the state transition probability kernel and R : S×A → R the immediate reward function. A randomized stationary policy, which we will simply call a policy in the rest of the chapter, is a mapping π : S → P(A) specifying for each s ∈ S, a ∈ A the probability π(a|s) of selecting action a in state s. At each time step t ∈ N, the RL agent in a state S t ∈ S executes an action A t ∈ A with probability π(A t |S t ), transitions into a state S t+1 ∈ S with probability p(S t+1 |S t , A t ) and observes a random reward R t+1 ∈ [-U R , U R ] where U R is a positive real. We denote by P ρ,π the probability distribution of the Markov chain (S t , A t ) issued from the MDP controlled by the policy π with initial state distribution ρ. The notation E ρ,π refers to the associated expectation. We will use E π whenever there is no dependence on ρ. The sequence (R t ) is such that (s.t.) E π [R t+1 |S t , A t ] = R(S t , A t ) . Let γ ∈ (0, 1) be a discount factor. Given a policy π, the long-term expected cumulative discounted reward is quantified by the state-value function V π : S → R and the action-value function

Q π : S × A → R defined for all s ∈ S, a ∈ A by V π (s) := E π [ ∞ t=0 γ t R t+1 |S 0 = s]and Q π (s, a) := E π [ ∞ t=0 γ t R t+1 |S 0 = s, A 0 = a] .
We also define the advantage function ∆ π : S × A → R by ∆ π (s, a) := Q π (s, a) -V π (s). Given an initial probability distribution ρ over states for the initial state S 0 , the goal of the agent is to find a policy π maximizing the expected long-term return J(π) := s∈S ρ(s)V π (s) . For this purpose, the agent has only access to realizations of the random variables S t , A t and R t whereas the state transition kernel p and the reward function R are unknown.

Fix θ ∈ R d . Let P θ be the transition matrix over the finite state space associated to the Markov chain (S t ), i.e., the matrix of size n × n defined for every s, s ∈ S by P θ (s |s) := a∈A p(s |s, a)π θ (a|s) .Consider the vector R θ = (R θ (s 1 ), • • • R θ (s n )) whose ith coordinate is provided by R θ (s i ) = a∈A π θ (a|s i )R(s i , a) . Let D ρ,θ be the diagonal matrix with elements d ρ,θ (s i ), i = 1, • • • n along its diagonal. Define also the Bellman operator T θ : R n → R n for every V ∈ R n by T θ V := R θ + γP θ V . The true value function V π θ satisfies the celebrated Bellman equation V π θ = T θ V π θ . This naturally leads to minimize the mean-square Bellman error (MSBE) (Sutton et al., 2009b, Section 3)

defined for every ω ∈ R m by E θ (ω) := 1 2 T θ V ω -V ω 2 D ρ,θ
where V ω = Φω . The gradient of the MSBE w.r.t. ω can be written as

∇ ω E θ (ω) = E S∼d ρ,θ [(T θ V ω ( S) - V ω ( S))(E S∼P θ (•| S) [γ∇ ω V ω (S)] -∇ ω V ω ( S))] .
As explained in (Bertsekas and Tsitsiklis, 1996, p. 369), omitting the gradient term

∇ ω T θ V ω ( S) = E S∼P θ (•| S) [γ∇ ω V ω (S)] in ∇ ω E θ (ω)
yields the standard TD learning update rule ω t+1 = ω t + δ t+1 φ( St ). The TD learning update does not coincide with a stochastic gradient descent on the MSBE or even any other objective function (see [START_REF] Barnard | Temporal-difference methods and markov models[END_REF], Appendix 1) for a proof). The idea of target-based TD learning is to consider a modified version of the MSBE Ẽθ (ω, ω) :

= 1 2 T θ V ω -V ω 2 D ρ,θ .
Observe that the term T θ V ω depending on ω in the MSBE is now freezed in Ẽθ (ω, ω) thanks to the target variable ω . We now need to introduce a new sequence ωt to define a sample-based version of T θ V ω -V ω which will be a modified version of the standard TD-error

δt+1 = R t+1 + γφ(S t+1 ) T ωt -φ( St ) T ω t .
(5.7)

Then, a stochastic gradient descent on Ẽ w.r.t. ω yields the critic update ω t+1 = ω t + β t δt+1 φ( St ) .

(5.8)

The target variable sequence ωt needs to be a slowed down version of the critic parameter ω t . For this purpose, instead of using a periodical synchronization of the target variable ωt with ω t through a copy as in DQN, we use the Polyak-averaging update rule proposed by [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] ωt+1 = ωt + ξ t (ω t+1ωt ) , (5.9) where ξ t is a positive stepsize chosen s.t. the sequence (ω t ) evolves on a slower timescale than the sequence (ω t ) to track it. The update rules of the actor and the critic collected together from Eqs. (5.5) to (5.8) give rise to Algorithm 5.1. We will use the shorthand notation δ t+1 := δ Vω t t+1 from now on.

Convergence analysis

In this section, we provide asymptotic convergence guarantees for the critic and the actor of Algorithm 5.1 successively . For every θ ∈ R d , let Kθ ∈ R |S||A|×|S||A| be the transition matrix over the state-action pairs defined for every (s, a), (s , a ) ∈ S × A by Kθ (s , a |s, a) = p(s |s, a)π θ (a |s ) . Let K := { Kθ : θ ∈ R d } and let K be its closure. Every element of K defines a Markov chain on the state-action space. We make the following assumption (see also Zhang et al. (2021b); [START_REF] Marbach | Simulation-based optimization of markov reward processes[END_REF]).

Assumption 5.5.1. For every K ∈ K, the Markov chain induced by K is ergodic.

their Section 6) propose for example to regularize the objective function J by adding a quadratic penalty θ 2 leading to an additional term θ t (for any positive ) in the actor update of the actor-critic algorithm 1 of [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF]. It is also worth mentioning that several works enforce the boundedness via a projection of the iterates on some compact set [START_REF] Bhandari | A finite time analysis of temporal difference learning with linear function approximation[END_REF]Wu et al., 2020;[START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF]. The drawback of this procedure is that it modifies the dynamics of the iterates and could possibly introduce spurious equilibria.

First, we will analyze the critic before investigating the convergence properties of the actor.

Critic analysis

The following assumption regarding the family of basis functions is a standard requirement [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF][START_REF] Konda | On actor-critic algorithms[END_REF][START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF].

Assumption 5.5.4 (critic features). The matrix Φ has full column rank.

We follow the strategy of (Borkar, 2008, Chap. 6, Lem. 1) for the analysis of multitimescale stochastic approximation schemes based on the ODE method. We start by analyzing the sequence (ω t ) evolving on the fastest timescale, i.e., with the slowly vanishing stepsizes β t (see Assumption 5.5.2). The main idea behind the proofs is that θ t , ωt can be considered as quasi-static in this timescale. Then, loosely speaking (see Section 5.7.1.1 for a rigorous statement and proof), we can show from its update rule Eq. ( 5 Recall that the matrices D ρ,θ , P θ and the vector R θ are defined in Section 5.4.2.

Remark 19. Under Assumptions 5.5.1 and 5.5.4 , the matrix -Ḡ(θ) is Hurwitz for every θ ∈ R d , i.e., all its eigenvalues have negative real parts. In particular, it is invertible.

The matrix -Ḡ(θ) being Hurwitz, it follows from (ODE-ω) that ω t tracks a slowly moving target ω * (θ t , ωt ) governed by the slower iterates θ t and ωt . The detailed proof in Section 5.7.1.1 makes use of a result from [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF] to handle the Markovian noise. In a second step, we analyze the target variable sequence (ω t ) which is evolving on a faster timescale than the sequence (θ t ) and slower than the sequence (ω t ). At the timescale ξ t , everything happens as if the quantity ω t in Eq. ( 5.9) could be replaced by ω * (θ t , ωt ) thanks to Prop. 5.1. Thus, in a sense that is made precise in Section 5.7.1.2, we can show from Eq. (5.9) that (ω t ) is related to the ODE (5.11) We show in Section 5.7.1.2 that the matrix -G(θ) is Hurwitz. This result differs from (Bertsekas and Tsitsiklis, 1996, Lem. 6.6. p.300) or (Tsitsiklis and Van Roy, 1997, Lem. 9) because the matrix D ρ,θ corresponds to the stationary distribution associated to the artificial kernel p and the policy π θ in lieu of the original transition kernel p. Then, we prove that -Ḡ(θ) -1 G(θ) is also stable, which suggests from (ODE-ω) that ωt tracks an other slowly moving target ω * (θ t ). This is established in the next proposition. Combining the results from Props. 5.1 and 5.2 , we prove that ω t tracks the same target ω * (θ t ). Moreover, this limit implies the following: lim t Π θt T θt (Φ ω t ) -Φ ω t = 0 w.p.1 .

Remark 20. When the actor parameter θ t is fixed (i.e., we are back to a policy evaluation problem), the second part of the above convergence result coincides with the widely known interpretation of the limit of the TD learning algorithm provided in [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF] (see also (Bertsekas and Tsitsiklis, 1996, p. 303-304)).

Actor analysis

Theorem 5.4. Let Assumptions 5.3.1 and 5.5.1 to 5.5.4 hold true. Then, w.p.1

lim inf t ∇J(θ t ) -b(θ t ) ≤ 0 ,
where for every

θ ∈ R d , (s, a) ∈ S × A , b(θ) = 1 1-γ E µ ρ,θ [ψ θ ( S, Ã)( Qθ ( S, Ã) -Q π θ ( S, Ã
))] and Qθ (s, a) = R(s, a) + γ s ∈S p(s |s, a)φ(s ) T ω * (θ) . Th. 5.4 is analog to (Konda, 2002, Th. 5.5) which is established for the standard on-policy actor-critic in the average reward setting and (Zhang et al., 2020b, Th. 3) for an off-policy actor-critic without any target network. The result states that the sequence (θ t ) generated by our actor-critic algorithm visits any neighborhood of the set {θ ∈ R d : ∇J(θ) ≤ b(θ) } infinitely often. The bias b(θ) corresponds to the difference between the gradient ∇J(θ) and the steady state expectation of the actor's update direction. The estimate used to update the actor in Eq. (5.6) is only a biased estimate of ∇J(θ) because of linear FA.

Remark 21. The bias b(θ) disappears in the tabular setting (m = |S| and the features spanning R |S| ) when we do not use FA and in the linear FA setting when the value function belongs to the class of linear functions spanned by the pre-selected feature (or basis) functions. Beyond these particular settings, considering compatible features as introduced in [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF]; [START_REF] Konda | On actor-critic algorithms[END_REF] can be a solution to cancel the bias b(θ) incurred by Algorithm 5.1. We do not investigate this direction in this work.

Finite-time analysis

Our analysis in this section should be valid for a continuous state space S (and still finite action space) upon supposing that the feature map φ defined in Section 5.4.2 has bounded norm (i.e., φ(•) ≤ 1) and slightly adapting our notations and definitions to this more general setting (see also for e.g., Wu et al. (2020)). To stay concise and consistent with the first part of our analysis in Section 5.5, we restrict ourselves to the finite state space setting.

Critic analysis

For every θ ∈ R d , we suppose that the Markov chain ( St ) induced by the policy π θ and the transition kernel p mixes at a geometric rate.

Assumption 5.6.1. There exist constants c > 0 and σ ∈ (0, 1) s.t.

sup s∈S d T V (P( St ∈ •| S0 = s, π θ ), d ρ,θ ) ≤ cσ t , ∀ t ∈ N , ∀ θ ∈ R d ,
where d T V (•, •) denotes the total-variation distance between two probability measures.

This assumption is used to control the Markovian noise induced by sampling transitions from the MDP under a dynamically changing policy1 . It was considered first in [START_REF] Bhandari | A finite time analysis of temporal difference learning with linear function approximation[END_REF] in a policy evaluation setting for the finite-time analysis of TD learning. It was later used for instance in Zou et al. (2019b); Wu et al. (2020); [START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF].

We have seen in Section 5.5.1 that the dynamics of the critic is driven by two key matrices -Ḡ(θ) and -Ḡ(θ) -1 G(θ). While we only need these matrices to be stable for our asymptotic results, we actually show in Section 5.7.1.1 that -Ḡ(θ) is even negative definite uniformly in θ. We suppose that the second matrix -Ḡ(θ) -1 G(θ) is also negative definite uniformly in θ.

Assumption 5.6.2. There exists ζ > 0 s.t. for every

θ ∈ R d , ω ∈ R m , ω T Ḡ(θ) -1 G(θ)ω ≥ ζ ω 2 .
We are now ready to state our critic convergence rate. Theorem 5.5. Let Assumptions 5.3.1,5.5.1 and 5.5.3 to 5.6.2

hold. Let c 1 , c 2 , c 3 , α, ξ, β be positive constants s.t. 0 < β < ξ < α < 1 . Set α t = c 1 (1+t) α , ξ t = c 2 (1+t) ξ and β t = c 3 (1+t) β .
Then, the sequences (ω t ) and (θ t ) generated by Algorithm 5.1 satisfy for every integer T ≥ 1 ,

1 T T t=1 E[ ω t -ω * (θ t ) 2 ] = O 1 T 1-ξ + O log T T β + O 1 T 2(α-ξ) + O 1 T 2(ξ-β) .
The bound of Th. 5.5 shows the impact of using a target variable. First, the last two terms impose the conditions α > ξ and ξ > β. At least with linear FA, this may provide a theoretical justification to the common practice of updating the target network at a slower rate compared to the main network for the critic. Second, compared to (Wu et al., 2020, Th. 4.7) which is concerned with the standard actor-critic in the average reward setting, we have the slower O(T ξ-1 ) instead of O(T β-1 ) and our bound comprises four error terms. These are also consequences of the use of a target variable.

Actor analysis

Assumption 5.6.3. There exists FA s.t. for every θ ∈ R d , V π θ -Φ ω * (θ) D ρ,θ ≤ FA . Theorem 5.6. Let Assumptions 5.3.1 ,5.5.1 ,5.5.3 to 5.6.1 and 5.6.3

hold. Let c 1 , c 2 , c 3 , α, ξ, β be positive constants s.t. 0 < β < ξ < α < 1 . Set α t = c 1 (1+t) α , ξ t = c 2 (1+t) ξ and β t = c 3 (1+t) β . Then, for every integer T ≥ 1 , 1 T T t=1 E[ ∇J(θ t ) 2 ] = O 1 T 1-α +O log 2 T T α +O   1 T T t=1 E[ ω t -ω * (θ t ) 2 ]   +O FA .
Combining Th. 5.5 and Th. 5.6, we obtain the following result.

Corollary 5.7. Under the setting and the assumptions of Ths. 5.5 and 5.6 , we have for every T ≥ 1 ,

1 T T t=1 E[ ∇J(θ t ) 2 ] = O 1 T 1-α +O log T T β +O 1 T 2(α-ξ) +O 1 T 2(ξ-β) +O FA .
Moreover, if we set α = 2 3 , ξ = 1 2 and β = 1 3 to define the stepsizes (α t ), (ξ t ) and (β t ), the actor parameter sequence (θ t ) generated by Algorithm 5.1 within T = O( -3 log 3 ( 1 )) steps, satisfies

min 0≤t≤T E[ ∇J(θ t ) 2 ] ≤ O( FA ) + .
As a consequence, since Algorithm 5.1 uses a single sample from the MDP per iteration, its sample complexity is O( -3 log 3 ( 1 )) . This is to compare with the best O( -2 log( 1 )) sample complexity known in the literature (to the best of our knowledge) for actor-critic algorithms up to the linear FA error (Xu et al., 2020, Th. 2). Although the use of a target variable seems to deteriorate the sample complexity w.r.t. the best known result for target-free actor-critic methods, note that it is still aligned with the complexity reported in [START_REF] Qiu | On the finite-time convergence of actor-critic algorithm[END_REF] (up to logarithmic factors), better than the O( -4 ) sample complexity obtained in [START_REF] Kumar | On the sample complexity of actor-critic method for reinforcement learning with function approximation[END_REF] with i.i.d. sampling and that we do not make use of mini-batching of samples (even from a single sample path) or nested loops as in [START_REF] Xu | A finite-time analysis of q-learning with neural network function approximation[END_REF]. We refer to (Wu et al., 2020, Section 4.4) and [START_REF] Xu | A finite-time analysis of q-learning with neural network function approximation[END_REF], Table 1) for further discussion. We briefly comment on the origin of this deteriorated sample complexity stemming from our finite-time bounds. Due to the use of a target variable, instead of the O(T 2(α-β) ) error term of the standard actor-critic (see (Wu et al., 2020, Cor. 4.9) or (Shen et al., 2020, Ths.3-4)), we have two error terms O(T 2(α-ξ) ) and O(T 2(ξ-β) ) slowing down the convergence because of the condition β < ξ < α. Interestingly, at least in the linear FA setting, this corroborates the practical intuition that the use of a target network may slow learning as formulated for instance in (Lillicrap et al., 2016, Section 3) (even if constant stepsizes are used in practice).

Remark 22. Remark 21 also applies to the function approximation error FA .

5.7 Proofs for Section 5.5

5.7.1 Proof of Th. 5.3

The objective of this section is to prove Th. 5.3. First, we recall the outline of the proof. Our actor-critic algorithm features three different timescales associated to three different stepsizes converging to zero with different rates, each one associated to one of the sequences (θ t ), (ω t ) and (ω t ). In spirit, we follow the strategy of (Borkar, 2008, Chap. 6, Lem. 1) for the analysis of two timescales stochastic approximation schemes. We make use of the results of [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF] which handles controlled Markov noise. The proof is divided into three main steps:

(i) We start by analyzing the sequence (ω t ) evolving on the fastest timescale, i.e., with the stepsizes β t which are converging the slowest to zero (see Assumption 5.5.2). We rewrite the slower sequences (θ t ), (ω t ) with the stepsizes β t . In this timescale, (θ t ), (ω t ) are quasi-static from the point of view of the evolution of the sequence (ω t ). We deduce from this first step that ω t tracks a slowly moving target ω * (θ t , ωt ) governed by the slower iterates θ t and ωt . This is the purpose of Prop. 5.1 which is proved in Section 5.7.1.1 below.

(ii) In a second step, we analyze the sequence (ω t ) which is evolving in a faster timescale than the sequence (θ t ) and slower than the sequence (ω t ). Similarly, we show that ωt tracks an other slowly moving target ω * (θ t ). This is established in the proof of Prop. 5.2 in Section 5.7.1.2.

(iii) We conclude in Section5.7.1.3 by combining the results from the first two steps, proving that the sequence ω t tracks the same target ω * (θ t ). Let F t be the σ-field generated by the random variables S l , Sl , Ãl , θ l , ωl , ω l for l ≤ t.

For each time step t, let Z t = ( St , Ãt ). Our objective here is to show that the critic sequence (ω t ) tracks the slowly moving target ω * (θ t , ωt ) defined in Prop. 5.1. From the update rule of the sequence (ω t ), we have (1)

ω t+1 = ω t + β t δt+1 φ( St ) = ω t + β t (R t+1 + γφ(S t+1 ) T ωt -φ( St ) T ω t )φ( St ) = ω t + β t w(ω t , ω t , Z t ) + β t η (
t+1 is a martingale difference sequence defined as .14) As can be seen in Eq. ( 5.12), the sequence (ω t ) can be written as a linear stochastic approximation scheme controlled by the slowly varying Markov chains (θ t ) and (ω t ).

η (1) t+1 = (R t+1 -R( St , Ãt ))φ( St ) + γ ωT t (φ(S t+1 ) -E[φ(S t+1 )|F t ]) φ( St ) . ( 5
In view of characterizing its asymptotic behavior, we compute for fixed ω, ω ∈ R m the expectation of the quantity w(ω, ω, Z) (see Eq. ( 5.13)) where Z = ( S, Ã) is a random variable (on S × A) following the stationary distribution µ ρ,θ (see Eq. ( 5.2)) of the Markov chain (Z t ). Recall the definitions of h : R d × R m → R m and Ḡ : R d → R m×m from Eq. ( 5.10), for every θ

∈ R d , ω ∈ R m h(θ, ω) := Φ T D ρ,θ (R θ + γP θ Φ ω) and Ḡ(θ) := Φ T D ρ,θ Φ .
Lemma 5.8. Under Assumption 5.5.1 , for every ω, ω ∈ R m , we have

E Z∼µ ρ,θ [w(ω, ω, Z)] = h(θ, ω) -Ḡ(θ)ω .
Proof. We obtain from the definitions of w in Eq. ( 5.13) and µ ρ,θ in Eq. ( 5.2) that

E Z∼µ ρ,θ [w(ω, ω, Z)] = E Z∼µ ρ,θ      R( S, Ã) + γ s ∈S p(s | S, Ã)φ(s ) T ω  φ( S) -φ( S)φ( S) T ω    = s∈S,a∈A µ ρ,θ (s, a)   R(s, a) + γ s ∈S p(s |s, a)φ(s ) T ω  φ(s) -φ(s)φ(s) T ω = s∈S d ρ,θ (s)   R θ (s)φ(s) + γ s ∈S p θ (s |s)φ(s ) T ωφ(s) -φ(s)φ(s) T ω   = h(θ, ω) -Ḡ(θ)ω ,
where the penultimate equation stems from recalling that R θ (s) = a∈A R(s, a)π θ (a|s) and p θ (s |s) = a∈A p(s |s, a)π θ (a|s) for every s ∈ S.

Defining χ t = (θ t , ωt ), we obtain from the update rules of (θ t ) and (ω t ) that 5.15) where

χ t+1 = χ t + β t ε t , (
ε t = αt βt 1 1-γ δ t+1 ψ θt (Z t ), ξt βt (ω t+1 -ωt ) . Notice that t → 0 as t → ∞.
ζ t = (χ t , ω t ), ζ = (θ, ω, ω) ∈ R d+2m , W (ζ, z) = (0, w(ω, ω, z)), ε t = (ε t , 0) and η(1) t+1 = (0, η (1)
t+1 ). Then, we can write Eqs. (5.15) and (5.12) in the framework of (Karmakar and Bhatnagar, 2018, Section 3, Eq.(14), Lem. 9), i.e., as a single timescale controlled Markov noise stochastic approximation scheme: 5.16) with t → 0 . Under the assumptions of [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF] that we will verify at the end of the proof, we obtain that the sequence (ζ t ) converges to an internally chain transitive set (i.e., a compact invariant set which has no proper attractor, see definition in (Karmakar and Bhatnagar, 2018, Section 2.1) or [START_REF] Benaïm | A dynamical system approach to stochastic approximations[END_REF], Section 1 p. 439)) of the ODE

ζ t+1 = ζ t + β t [W (ζ t , Z t ) + ε t + η(1) t+1 ] , (
d ds ζ(s) = W (ζ(s)) where W (ζ) = (0, h(χ) -Ḡ(θ)ω) , i.e.,    d ds χ(s) = 0 , d ds ω(s) = h(χ(s)) -Ḡ(θ(s))ω(s) .
(5.17)

As we will show that the second ODE governing ω has a unique asymptotically stable equilibrium ω * (θ, ω) for every constant function χ(t) = χ = (θ, ω), it follows that (χ t , ω t ) converges a.s. towards the set {(χ, ω * (χ)) : χ ∈ R d+m }. In other words, lim t ω tω * (θ t , ωt ) = 0, which is the desired result.

We now conclude the proof by verifying among (A1) to (A7) of [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF] the assumptions under which [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF], Lems. 9 and 10) hold.

(i) (A1): (Z t ) takes values in a compact metric space. Note that it is a finite stateaction Markov chain controlled by the sequence (θ t ).

(ii) (A2): It is easy to see from Eq. ( 5.13) that the drift function w is Lipschitz continuous w.r.t. the variables ω, ω uniformly w.r.t. the last variable z because p is a probability kernel and the set of states S is finite. (vi) (A6'): We first note that the inverse of the matrix Ḡ(θ) exists thanks to Assumptions 5.5.1 and 5.5.4 . For all χ = (θ, ω) ∈ R d+m , we now show that the ODE d ds ω(s) = h(χ) -Ḡ(θ)ω(s) has a unique globally asymptotically stable equilibrium ω * (χ) = Ḡ(θ) -1 h(χ). The aforementioned ODE is stable if and only if the matrix Ḡ(θ) is Hurwitz. We actually show that we have a stronger result in Lem. 5.9 under Assumptions 5.5.1 and 5.5.4 . We briefly explicit why the assumption as formulated in the rest of (A6') holds. Define the function L(χ, ω) = 1 2 Ḡ(θ)ωh(χ) 2 . For every χ = (θ, ω) ∈ R d+m , the function L(χ, •) is a Lyapunov function for ODE (5.17). Indeed, using Lem. 5.9 below, we can write

(iii) (A3): (η (1) t+1 ) is a martingale difference sequence w.r.t. the filtration (F t ) . Moreover, since (R t ) is bounded, there exists K > 0 s.t. E[ η(1) n+1 2 |F t ] ≤ K(1+ ω t 2 + ωt 2 ). (
d ds L(χ, ω(s)) = -h(χ)-Ḡ(θ)ω(s), Ḡ(θ)( h(χ)-Ḡ(θ)ω(s)) ≤ -ε Ḡ(θ)ω(s)-h(χ) 2 .
(vii) (A7): The stability Assumption 5.5.3 ensures that sup t ( ω t + θ t ) < +∞ w.p.1 .

As a consequence, it also follows from the update rule of (ω t ) that sup t ωt < +∞ .

Lemma 5.9. Under Assumptions 5.5.1 and 5.5.4 , there exists ε > 0 s.t. for all

θ ∈ R d , ω ∈ R m , ω T Ḡ(θ)ω ≥ ε ω 2 .
In particular, it holds that sup θ∈R d Ḡ(θ) -1 < ∞ .

Proof. Recall that K := { Kθ : θ ∈ R d } where for every θ ∈ R d , Kθ ∈ R |S||A|×|S||A| is the transition matrix over the state-action pairs defined for every (s, a), (s , a ) ∈ S × A by Kθ (s , a |s, a) = p(s |s, a)π θ (a |s ) . We also denoted by K the closure of K. Under Assumption 5.5.1 , there exists a unique stationary distribution µ K ∈ R S×A for every K ∈ K.

We first show that the map K → µ K is continuous over the set K . The proof of this fact is similar to the proofs of (Zhang et al., 2021b, Lem. 9) and (Marbach and Tsitsiklis, 2001, Lem. 1). We reproduce a similar argument here for completeness. Observe first that µ K satisfies:

M (K)µ K = 0 1 where M (K) := K T -I 1 .
As a consequence, since M (K) has full column rank thanks to Assumption 5.5.1 , the matrix M (K) T M (K) is invertible and we obtain a closed form expression for µ K given by:

µ K = (M (K) T M (K)) -1 M (K) T 0 1 = com(M (K) T M (K)) T det (M (K) T M (K)) M (K) T 0 1 ,
where com(A) stands for the comatrix of the matrix A. Then, it can be seen from this expression that the map K → µ K is continuous. Note for this that the entries of the comatrix are polynomial functions of the entries of M (K) T M (K), and the determinant operator is continuous.

It follows from Assumption 5.5.1 that for every K ∈ K and every (s, a) ∈ S × A, µ K (s, a) > 0 . We deduce from the continuity of the map K → µ K over the compact set K that inf K∈ K µ K (s, a) > 0 . Since Kθ ∈ K for every θ ∈ R d , we obtain that inf θ µ ρ,θ (s, a) > 0 where we recall that µ ρ,θ is the unique stationary distribution of the Markov chain induced by Kθ . As a consequence, since d ρ,θ (s) = a∈A µ ρ,θ (s, a), it also holds that inf

θ d ρ,θ (s) > 0 . Therefore, for every θ ∈ R d , ω ∈ R m : ω T Ḡ(θ)ω = (Φ ω) T D ρ,θ (Φ ω) ≥ min s∈S inf θ d ρ,θ (s) Φω 2 ≥ min s∈S inf θ d ρ,θ (s)λ min (Φ T Φ) ω 2 ,
where λ min (Φ T Φ) > 0 corresponds to the smallest eigenvalue of the symmetric positive definite matrix Φ T Φ which is invertible thanks to Assumption 5.5.4 . The proof is concluded by setting ε := λ min (Φ T Φ)•min s∈S inf θ d ρ,θ (s) > 0 which is independent of θ.

5.7.1.2 Proof of Prop. 5.2

Recall the definitions of the vector h(θ) and the matrix G(θ) from Eq. ( 5.11): .18) We begin the proof by showing the existence of a unique solution ω * (θ) to the linear system G(θ)ω = h(θ) . The following lemma establishes the uniform positive definiteness of the matrix G(θ) Note that we do not include symmetry in our definition of positive definiteness as in [START_REF] Bertsekas | Neuro-Dynamic Programming[END_REF]. As a matter of fact, the matrix G(θ) is not symmetric in general.

h(θ) := Φ T D ρ,θ R θ and G(θ) := Φ T D ρ,θ (I n -γP θ )Φ . ( 5
∈ R m , ω T G(θ)ω ≥ κ ω 2 .
In particular, the matrix G(θ) is invertible.

Proof. First, we have for every

θ ∈ R d , ω ∈ R m , ω T G(θ)ω = (Φω) T D ρ,θ (I n -γP θ )Φω = (Φω) T D ρ,θ (Φω) -γ(Φω) T D ρ,θ P θ (Φω) . (5.19)
Then, the Cauchy-Schwarz inequality yields .20) Notice now that we cannot use the classical result (Tsitsiklis and Van Roy, 1997, Lem. 1) to obtain that P θ V D ρ,θ ≤ V D ρ,θ for any V ∈ R n because D ρ,θ is not the stationary distribution of the kernel P θ but it is instead associated to the artificial kernel Pθ . Nevertheless, the following lemma provides an analogous result with a similar proof.

(Φω) T D ρ,θ P θ (Φω) = (Φω) T D 1 2 ρ,θ D 1 2 ρ,θ P θ (Φω) ≤ Φω D ρ,θ P θ Φω D ρ,θ . ( 5
Lemma 5.11. For every θ ∈ R d , V ∈ R n , we have

P θ V 2 D ρ,θ ≤ 1 γ V 2 D ρ,θ - 1 -γ γ V 2 ρ ≤ 1 γ V 2 D ρ,θ .
Proof. It follows from Jensen's inequality that

P θ V 2 D ρ,θ = n i=1 d ρ,θ (s i ) n j=1 P θ (s j |s i )V j 2 ≤ n i=1 d ρ,θ (s i ) n j=1 P θ (s j |s i )V 2 j .
Then, observe that Pθ = γP θ + (1γ)1ρ T as a consequence of Eq. (5.3). By plugging this formula and then using the fact that d T ρ,θ Pθ = d T ρ,θ , we obtain

n i=1 d ρ,θ (s i ) n j=1 P θ (s j |s i )V 2 j = 1 γ n j=1 n i=1 d ρ,θ (s i ) Pθ (s j |s i )V 2 j -(1 -γ) n j=1 ρ(s j )V 2 j = 1 γ n j=1 d ρ,θ (s j )V 2 j -(1 -γ) n j=1 ρ(s j )V 2 j = 1 γ V 2 D ρ,θ - 1 -γ γ V 2 ρ ,
which concludes the proof of Lem. 5.11 .

We now complete the proof of Lem. 5.10 . From Eq. (5.20), Lem. 5.11 with V = Φω yields

(Φω) T D ρ,θ P θ (Φω) ≤ 1 √ γ Φω 2 D ρ,θ = 1 √ γ (Φω) T D ρ,θ (Φω) .
Whence, we obtain from Eq. ( 5.19) that

ω T G(θ)ω ≥ (1 - √ γ)(Φω) T D ρ,θ (Φω) ≥ ε(1 - √ γ) ω 2 ,
where the last inequality stems from Lem. 5.9.

We now prove the remaining convergence results. We start with the first result showing that the sequence (ω t ) tracks ω * (θ t ) . From the update rules of the sequences (ω t ) and (ω t ) (Eqs. (5.8)-(5.9)), we can introduce the quantity ω * (θ t , ωt ) as defined in Prop. 5.1 to obtain

ωt+1 = ωt + ξ t (ω t+1 -ωt ) = ωt + ξ t (ω t + β t w(ω t , ω t , Z t ) + β t η (1) t+1 -ωt ) = ωt + ξ t (ω * (θ t , ωt ) -ωt ) + ξ t (ω t -ω * (θ t , ωt ) + β t w(ω t , ω t , Z t )) + ξ t β t η (1)
t+1 .

(5.21)

Then, using the expressions of h, Ḡ in Eq. ( 5.10) and h, G in Eq. ( 5.11), we can write

ω * (θ t , ωt ) -ωt = Ḡ(θ t ) -1 ( h(θ t , ωt ) -Ḡ(θ t )ω t ) = Ḡ(θ t ) -1 (h(θ t ) -G(θ t )ω t ) .
As a consequence,

ωt+1 = ωt + ξ t Ḡ(θ t ) -1 (h(θ t ) -G(θ t )ω t ) + ξ t (ω t -ω * (θ t , ωt ) + β t w(ω t , ω t , Z t )) + ξ t β t η (1)
t+1 . (5.22) Therefore, the sequence (ω t ) satisfies a linear stochastic approximation scheme driven by the slowly varying Markov chain (θ t ) evolving on a slower timescale than the iterates (ω t ). We proceed similarly to the proof of Prop. 5.1 . 5.23) where εt = ( αt ξt

Recall the notation χ

t = (θ t , ωt ). Let χ = (θ, ω) ∈ R d+m , U (χ) = (0, Ḡ(θ) -1 (h(θ) - G(θ)ω)). Then, χ t+1 = χ t + ξ t [U (χ t ) + εt ] , (
1 1-γ δ t+1 ψ θt ( St , Ãt ), ω t -ω * (θ t , ωt ) + β t w(ω t , ω t , Z t ) + β t η (1) t+1
) . It can be shown that εt → 0 as t → +∞. Note for this that α t /ξ t → 0 and β t → 0 by Assumption 5.5.2 , ω t -ω * (θ t , ωt ) → 0 as proved in Prop. 5.1 and δ t+1 ψ θt ( St , Ãt ), w(ω t , ω t , Z t ) are bounded by Assumptions 5.3.1-(c) , 5.5.3, the boundedness of the reward function R and the fact that the sets S, A are finite. Moreover, Assumption 5.5.2 ensures that t ξ t = +∞ and t ξ 2 t < +∞ . Furthermore, one can show that the function U is Lipschitz continuous. For this, remark that:

(a) The function U is affine in ω. (c) The function θ → D ρ,θ is Lipschitz continuous. We refer to (Zhang et al., 2021b, Lem. 9) for a proof.

(d) The function θ → Ḡ(θ) -1 is Lipschitz continuous. Observe for this that for every θ, θ ∈ R d , Ḡ(θ) -1 -Ḡ(θ ) -1 = Ḡ(θ) -1 ( Ḡ(θ ) -Ḡ(θ)) Ḡ(θ ) -1 and that sup θ Ḡ(θ) -1 < ∞ using Lem. 5.9.

(e) The reward function R is bounded and the entries of the matrices D ρ,θ and P θ are bounded by one.

Using classical stochastic approximation results (see, for e.g., (Benaïm, 1996, Th.1.2)), we obtain that the sequence (χ t ) converges a.s. towards an internally chain transitive set of the ODE d ds χ

(s) = U (χ(s)) , i.e.,    d ds θ(s) = 0 , d ds ω(s) = Ḡ(θ(s)) -1 (h(θ(s)) -G(θ(s))ω(s)) .
(5.24)

We conclude by showing that for every θ ∈ R d , the ODE d ds ω(s) = Ḡ(θ) -1 (h(θ) -G(θ)ω(s)) has a globally asymptotically stable equilibrium ω * (θ). This result holds if the matrix -Ḡ(θ) -1 G(θ) is Hurwitz, i.e., all its eigenvalues have negative real parts. We show this result in Lem. 5.12 below. Then, it follows that χ t = (θ t , ωt ) converges a.s. towards the set {(θ, ω * (θ)) : θ ∈ R d }. This yields the desired result lim t ωtω * (θ t ) = 0. Lemma 5.12. For every θ ∈ R d , the matrix -Ḡ(θ) -1 G(θ) is Hurwitz .

Proof. We first recall Lyapunov's theorem which characterizes Hurwitz matrices (see, for e.g., (Horn and Johnson, 1994, Th.2.2.1 p. 96)). A complex matrix A is Hurwitz if and only if there exists a positive definite matrix M = M * s.t. A * M + M A is negative definite, where M * and A * are the complex conjugate transposes of M and A . We use this theorem with A = -Ḡ(θ) -1 G(θ) and M = Ḡ(θ) which is symmetric by definition and positive definite thanks to Lem. 5.9. Then, we obtain that

A * M + M A = -G(θ) T Ḡ(θ) -1 Ḡ(θ) -Ḡ(θ) Ḡ(θ) -1 G(θ) = -(G(θ) T + G(θ)) .
We conclude the proof by showing that G(θ) T + G(θ) is a (symmetric) positive definite matrix. For that, observe that for every nonzero vector ω ∈ R m , it holds that ω T (G(θ) T + G(θ))ω = 2ω T G(θ)ω > 0 where the positivity stems from Lem. 5.10.

The last result states that for every θ ∈ R d , Φω * (θ) is a fixed point of the projected Bellman operator Π θ T θ . This is a consequence of the following derivations: 5.25) where the first equality uses the expression of the projection Π θ , the second one uses the definition of the Bellman operator T θ and the third one stems from the definitions of the matrices Ḡ(θ) and G(θ) (see Eqs. (5.10) and (5.11)).

Π θ T θ (Φω * (θ)) = Φ Ḡ(θ) -1 Φ T D ρ,θ T θ (Φω * (θ)) = Φ Ḡ(θ) -1 Φ T D ρ,θ (R θ + γP θ Φω * (θ)) = Φ Ḡ(θ) -1 h(θ) + Φ Ḡ(θ) -1 ( Ḡ(θ) -G(θ))G(θ) -1 h(θ) = Φ Ḡ(θ) -1 h(θ) + ΦG(θ) -1 h(θ) -Φ Ḡ(θ) -1 h(θ) = ΦG(θ) -1 h(θ) = Φω * (θ) , (
5.7.1.3 Proof of Th. 5.3

The proof of Th. 5.3 uses both Prop. 5.1 and Prop. 5.2 .

In order to show that lim t ω tω * (θ t ) = 0 w.p.1 ,we prove the two following results:

(a) lim t ω t -ω * (θ t , ω * (θ t )) = 0 w.p.1 . (b) ω * (θ, ω * (θ)) = ω * (θ) for all θ ∈ R d .
(a) We have the decomposition

ω t -ω * (θ t , ω * (θ t )) = [ω t -ω * (θ t , ωt )] + [ω * (θ t , ωt ) -ω * (θ t , ω * (θ t ))], = [ω t -ω * (θ t , ωt )] + Ḡ(θ t ) -1 ( h(θ t , ωt ) -h(θ t , ω * (θ t ))) = [ω t -ω * (θ t , ωt )] + Ḡ(θ t ) -1 Φ T D ρ,θt P θt Φ(ω t -ω * (θ t )) = [ω t -ω * (θ t , ωt )] + Ḡ(θ t ) -1 ( Ḡ(θ t ) -G(θ t ))(ω t -ω * (θ t )) = [ω t -ω * (θ t , ωt )] + (I m -Ḡ(θ t ) -1 G(θ t ))(ω t -ω * (θ t )) . (5.26)
It follows from Prop. 5.1 that the first term in the above decomposition goes to zero. Then, observe that sup θ Ḡ(θ) -1 < ∞ given Lem. 5.9 and sup θ G(θ) < ∞ thanks to the boundedness of the matrices P θ and D ρ,θ uniformly in θ. As a consequence, the second term also converges to zero using Prop. 5.2 .

(b) Using the definitions of the functions ω * and ω * , we can write for every θ

∈ R d , ω * (θ, ω * (θ)) = Ḡ(θ) -1 h(θ, ω * (θ)) = Ḡ(θ) -1 Φ T D ρ,θ (R θ + γP θ ΦG(θ) -1 h(θ)) = Ḡ(θ) -1 (h(θ) + γΦ T D ρ,θ P θ ΦG(θ) -1 h(θ)) = Ḡ(θ) -1 (I n + γΦ T D ρ,θ P θ ΦG(θ) -1 )h(θ) = Ḡ(θ) -1 (G(θ) + γΦ T D ρ,θ P θ Φ)G(θ) -1 h(θ) = Ḡ(θ) -1 Ḡ(θ)G(θ) -1 h(θ) = ω * (θ) .
For the last result, we write .27) Then, as previously mentioned in the proof, observe that sup θ Ḡ(θ) -1 < ∞ and sup θ G(θ) < ∞. Since ωtω * (θ t ) → 0 as t → ∞, the result follows.

Π θt T θt (Φω t ) -Φω t = Φ Ḡ(θ t ) -1 Φ T D ρ,θt T θt (Φω t ) -ω t = Φ Ḡ(θ t ) -1 Φ T D ρ,θt (T θt (Φω t ) -Φω t ) = Φ Ḡ(θ t ) -1 (h(θ t ) -G(θ t )ω t ) = Φ Ḡ(θ t ) -1 G(θ t )(ω t -ω * (θ t )) ≤ Φ Ḡ(θ t ) -1 G(θ t ) ω t -ω * (θ t ) . ( 5
Lemma 5.13. For every θ

∈ R d , b(θ) = γ 1 -γ s∈S,a∈A µ ρ,θ (s, a)ψ θ (s, a) s ∈S p(s |s, a)(φ(s ) T ω * (θ) -V π θ (s )) .
Proof. The expression follows from using the definition of b(θ) and computing both the function H defined in Eq. ( 5.29) and the gradient of the function J.

First, we explicit the function H, writing (5.36) where the last equality stems from remarking that a∈A µ ρ,θ (s, a)ψ θ (s, a) = 0.

H(θ) = E Z∼µ ρ,θ [H θ (Z)] = E Z∼µ ρ,θ     ψ θ ( S, Ã)   γ s ∈S p(s | S, Ã)φ(s ) -φ( S)   T     = s∈S,a∈A µ ρ,θ (s, a)ψ θ (s, a)   γ s ∈S p(s |s, a)φ(s ) T -φ(s) T   = γ s∈S,a∈A µ ρ,θ (s, a)ψ θ (s, a) s ∈S p(s |s, a)φ(s ) T ,
Then, the policy gradient theorem as formulated in Eq. ( 5.1) and the definition of the advantage function provide

(1 -γ)∇J(θ) = E Z∼µ ρ,θ [∆ π θ ( S, Ã)ψ θ ( S, Ã)] = E Z∼µ ρ,θ [(R( S, Ã) + γ s ∈S p(s | St , Ãt )V π θ (s ) -V π θ ( S))ψ θ ( S, Ã)] = s,a µ ρ,θ (s, a)(R(s, a) + γ s ∈S p(s |s, a)V π θ (s ) -V π θ (s))ψ θ (s, a) = u(θ) + γ s∈S,a∈A µ ρ,θ (s, a)ψ θ (s, a) s ∈S p(s |s, a)V π θ (s ) .
(5.37)

The result stems from using the definition of b(θ) together with Eqs. (5.36) and (5.37).

Using a second-order Taylor expansion of the L-Lipschitz function ∇J (again see (Zhang et al., 2020a, Lem. 4.2)) together with Eq. ( 5.31), we can derive the following inequalities

J(θ t+1 ) ≥ J(θ t ) + ∇J(θ t ), θ t+1 -θ t -L θ t+1 -θ t 2 , ≥ J(θ t ) + α t ∇J(θ t ), f (θ t ) + α t 1 -γ ∇J(θ t ), ηt+1 + e (1) t + e (2) t - L α 2 t (1 -γ) 2 δ t+1 ψ θt ( St , Ãt ) 2 . (5.38)
The above inequality consists of a main term involving the function f and noise terms.

The following lemma controls these noise terms which are shown to be negligible. The end of the proof follows the same line as [START_REF] Konda | Actor-Critic Algorithms[END_REF]Tsitsiklis, 2003, p. 1163) (see also (Konda, 2002, p. 86)). We reproduce the argument here for completeness. Let T > 0.

Define a sequence k t by

k 0 = 0 , k t+1 = min      k ≥ k t : k i=kt α i ≥ T      for t > 0 .
Using Eq. ( 5.38) together with the Cauchy-Schwartz inequality and Eq. ( 5.35), we can write

J(θ k t+1 ) ≥ J(θ kt ) + k t+1 -1 k=kt α k ( ∇J(θ k ) 2 -b(θ k ) • ∇J(θ k ) ) + υ t ,
where υ t is defined by

υ t = k t+1 -1 k=kt α k 1 -γ ∇J(θ k ), ηk+1 + e (1) k + e (2) k - L α 2 k (1 -γ) 2 δ k+1 ψ θ k ( Sk , Ãk ) 2 .
It stems from Lem. 5.14 that υ t → 0 as t → +∞ . By contradiction, if the result does not hold, the sequence J(θ k ) would increase indefinitely. This contradicts the boundedness of the function J (note that θ → V π θ is bounded since the rewards are bounded).

Proofs for Section 5.6

Throughout our finite-time analysis, we will not track the constants although these can be precisely determined. The universal constant C may change from line to line and from inequality to inequality. It may depend on constants of the problem s.t. the Lipschitz constants of the functions J, θ → ψ θ , θ → π θ , upperbounds of the rewards and the score function ψ θ or the cardinal |A| of the action space .

5.8.1 Proof of Th. 5.5

The proof is inspired from the recent works Wu et al. (2020); [START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF]. However, it significantly deviates from these works because of the use of a target variable ω in Algorithm 5.1 . In particular, as previously mentioned, Algorithm 5.1 involves three different timescales whereas the actor-critic algorithms considered in Wu et al. (2020); [START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF] only use two different timescales respectively associated to the critic and the actor.

We follow a similar strategy to our asymptotic analysis of the critic. Indeed, our nonasymptotic analysis consists of two main steps based on the following decomposition:

ω t -ω * (θ t ) = ω t -ω * (θ t , ωt ) + ω * (θ t , ωt ) -ω * (θ t ) = ω t -ω * (θ t , ωt ) + ω * (θ t , ωt ) -ω * (θ t , ω * (θ t )) = ω t -ω * (θ t , ωt ) + Ḡ(θ t ) -1 ( h(θ t , ωt ) -h(θ t , ω * (θ t ))) . (5.41)
Hence, it is sufficient to obtain a control of the convergence rates of the quantities ω tω * (θ t , ωt ) and ωtω * (θ t ) . We already know that these quantities converge a.s. to zero thanks to Props. 5.1 and 5.2 . We conduct a finite-time analysis of each of the terms separately in the subsections below and combine the obtained results to conclude the proof.

We start by introducing a few useful shorthand notations. Let xt := ( St , Ãt , S t+1 ) . Define for every x = (s, ã, s) ∈ S × A × S and every ω, ω ∈ R m :

δ(x, ω, ω) = R(s, ã) + γφ(s) T ω -φ(s) T ω , (5.42) g(x, ω, ω) = δ(x, ω, ω)φ(s) . (5.43)
Finally, define for every θ ∈ R d the steady-state expectation: We introduce an additional shorthand notation for brevity:

ḡ(θ, ω, ω) = E s∼d ρ,θ ,ã∼π θ ,s∼p(•|s,ã) [g(x, ω, ω)] . ( 5
ν t := ω t -ω * (θ t , ωt ) .
Decomposition of the error. Using the update rule of the critic gives

ν t+1 2 = ω t + β t g(x t , ωt , ω t ) -ω * (θ t+1 , ωt+1 ) 2 = ν t + β t g(x t , ωt , ω t ) + ω * (θ t , ωt ) -ω * (θ t+1 , ωt+1 ) 2 .
Then, we develop the squared norm and use the classical inequality a+b 2 ≤ 2 a +2 b 2 to obtain where we used the shorthand notation Λ(θ, ω, ω, x) := ωω * (θ, ω), g(x, ω, ω)ḡ(θ, ω, ω) .

ν t+1 2 ≤ ν t 2 + 2β t ν t , g(x t , ωt , ω t ) + 2 ν t , ω * (θ t , ωt ) -ω * (θ t+1 , ωt+1 ) + 2 ω * (θ t , ωt ) -ω * (θ t+1 , ωt+1 ) 2 + 2Cβ
(5.47)

We control the first term in Eq. ( 5.46) as follows

ν t , ḡ(θ t , ωt , ω t ) = ν t , ḡ(θ t , ωt , ω t ) -ḡ(θ t , ωt , ω * (θ t , ωt )) = -ν t , Ḡ(θ t )ν t ≤ -κ ν t 2 .
(5.48) We used the fact that ḡ(θ t , ωt , ω * (θ t , ωt )) = 0 for the first equality and Lem. 5.9 for the inequality. Then, it can be shown that

ω * (θ t , ωt ) -ω * (θ t+1 , ωt+1 ) ≤ C( θ t -θ t+1 + ωt -ωt+1 ) ≤ C(α t + ξ t ) . (5.49)
Combining Eqs. (5.45) to (5.49) .51) In the remainder of the proof, we will use the notation τ for τ T (interchangeably). In order to control the difference between the update rule of the critic and its steady-state expectation, we introduce an auxiliary chain which coincides with xt except for the τ last steps where the policy is fixed to π θ t-τ . The auxiliary chain will be denoted by xt := ( Št , Ǎt , S t+1 ) where S t+1 ∼ p(•| Št , Ǎt ) and ( Št , Ǎt ) is generated as follows:

T := min{t ∈ N, t ≥ 1 : cσ t-1 ≤ min{α T , ξ T , β T }} . ( 5
St-τ θ t-τ ---→ Ãt-τ p - → St-τ+1 θ t-τ ---→ Ǎt-τ+1 p - → Št-τ+2 θ t-τ ---→ Ǎt-τ+2 p - → • • • p - → Št θ t-τ ---→ Ǎt p - → Št+1 .
Compared to this chain, the original chain has a drifting policy, i.e., at each time step, the actor parameter θ t is updated and so is the policy π θt and we recall that it is given by:

St-τ θ t-τ ---→ Ãt-τ p - → St-τ+1 θ t-τ +1 ----→ Ãt-τ+1 p - → St-τ+2 θ t-τ +2 ----→ Ãt-τ+2 p - → • • • p - → St θt -→ Ãt p - → St+1 .
Using the shorthand notation z t := (ω t , ω t ), the Markov noise term can be decomposed as follows:

Λ(θ t , ωt , ω t , xt ) = (Λ(θ t , z t , xt )-Λ(θ t-τ , z t-τ , xt ))+(Λ(θ t-τ , z t-τ , xt )-Λ(θ t-τ , z t-τ , xt )) + Λ(θ t-τ , z t-τ , xt ) . (5.52)
We control each one of the terms successively.

(a) Control of Λ(θ t , z t , xt ) -Λ(θ t-τ , z t-τ , xt ): Using that ω * and ḡ are Lipschitz in all their arguments, g is Lipschitz in its two last arguments and ω t , ω * , g and ḡ are all bounded, one can show after tedious decompositions that

|Λ(θ t , z t , xt )-Λ(θ t-τ , z t-τ , xt )| ≤ C( θ t -θ t-τ + ωt -ω t-τ + ω t -ω t-τ ) . (5.53)
Then, recalling that the sequence (α t ) is nonincreasing, remark that

θ t -θ t-τ ≤ t-1 t-τ θ j+1 -θ j ≤ C t-1 t-τ α j ≤ Cτ α t-τ .
Similarly, we have ωtωt-τ ≤ Cτ ξ t-τ , ω tω t-τ ≤ Cτ β t-τ and we can therefore deduce from Eq. ( 5.53) that 5.56) where the first equality stems from the definition of Λ, the first inequality uses the definition of the total variation distance d T V between two probability measures and the last inequality is a consequence of (Wu et al., 2020, Lem. B.2, p.17) (see also (Shen et al., 2020, Lem. 2 p.12)).

|Λ(θ t , z t , xt ) -Λ(θ t-τ , z t-τ , xt )| ≤ Cτ (α t-τ + β t-τ + ξ t-τ ) . ( 5
E[Λ(θ t-τ , z t-τ , xt ) -Λ(θ t-τ , z t-τ , xt )| St-τ+1 , θ t-τ ] (5.55) = E[ ν t-τ , g(x t , z t-τ ) -g(x t , z t-τ ) | St-τ+1 , θ t-τ ] ≤ Cd T V (P(x t ∈ •| St-τ+1 , θ t-τ ), P(x t ∈ •| St-τ+1 , θ t-τ )) ≤ C 2 |A|L π t i=t-τ E[ θ i -θ t-τ | St-τ+1 , θ t-τ ] , (
Then, we have

t i=t-τ E[ θ i -θ t-τ | St-τ+1 , θ t-τ ] ≤ t i=t-τ i-1 j=t-τ E[ θ j+1 -θ j | St-τ+1 , θ t-τ ] ≤ C t i=t-τ i-1 j=t-τ α j ≤ Cα t-τ τ i=0 i ≤ Cα t-τ (τ + 1) 2 .
As a consequence of these derivations, Eq. ( 5.55) yields 5.58) where the first inequality stems again from the definition of the total variation norm and the last two ones follow from Assumption 5.6.1 and the definition of the mixing time τ = τ T (see Eq. (5.51)).

E[Λ(θ t-τ , z t-τ , xt ) -Λ(θ t-τ , z t-τ , xt )| St-τ+1 , θ t-τ ] ≤ Cα t-τ (τ + 1) 2 , (5.57) (c) Control of Λ(θ t-τ , z t-τ , xt ) : Define xt := ( St , Āt , S t+1 ) where St ∼ d ρ,θ t-τ , Āt ∼ π θ t-τ and S t+1 ∼ p(•| St , Āt ). Observing that E[Λ(θ t-τ , z t-τ , xt )| St-τ+1 , θ t-τ ] = 0, we obtain E[Λ(θ t-τ , z t-τ , xt )| St-τ+1 , θ t-τ ] = E[Λ(θ t-τ , z t-τ , xt ) -Λ(θ t-τ , z t-τ , xt )| St-τ+1 , θ t-τ ] = E[ ν t-τ , g(x t , z t-τ ) -g(x t , z t-τ ) | St-τ+1 , θ t-τ ] ≤ Cd T V (P(x t ∈ •| St-τ+1 , θ t-τ ), P(x t ∈ •| St-τ+1 , θ t-τ )) = Cd T V (P( St ∈ •| St-τ+1 , θ t-τ ), d ρ,θ t-τ ) ≤ Cσ τ -1 ≤ α T , (
Given the decomposition of Eq. ( 5.52), collecting Eqs.(5.54), (5.57), (5.58) and taking total expectation leads to the conclusion of this subsection

E[Λ(θ t , z t , xt )] ≤ C(τ (α t-τ + β t-τ + ξ t-τ ) + α t-τ (τ + 1) 2 + α T ) . ( 5

.59)

Derivation of the convergence rate of the mean error term 1 T T t=1 ν t 2 . We obtain from taking the total expectation in Eq. ( 5.50) together with Eq. ( 5.59) that

E[ ν t+1 2 ] ≤ (1 -2κβ t )E[ ν t 2 ] + 2Cβ t (τ (α t-τ + β t-τ + ξ t-τ ) + α t-τ (τ + 1) 2 + α T ) + C(α t + ξ t )E[ ν t ] + C(α 2 t + ξ 2 t + β 2 t ) . (5.60)
Rearranging the inequality and summing for t between τ T and T, we get

2κ T t=τ T E[ ν t 2 ] ≤ I 1 (T ) + I 2 (T ) + I 3 (T ) + I 4 (T ) , (5.61)
where

I 1 (T ) := T t=τ T 1 β t (E[ ν t 2 ] -E[ ν t+1 2 ]) , (5.62)
I 2 (T ) := T t=τ T 2C(τ (α t-τ + β t-τ + ξ t-τ ) + α t-τ (τ + 1) 2 + α T ) (5.63) I 3 (T ) := C T t=τ T α t β t + ξ t β t E[ ν t ] (5.64) I 4 (T ) := C T t=τ T α 2 t β t + ξ 2 t β t + β t . (5.65)
We derive estimates of each one of the terms I i (T ) for i = 1, 2, 3, 4.

(1) Since (ν t ) is a bounded sequence,

I 1 (T ) = T t=τ T 1 β t - 1 β t-1 E[ ν t 2 ] + 1 β τ T -1 E[ ν τ T 2 ] - 1 β τ T E[ ν T +1 2 ] ≤ C   T t=τ T 1 β t - 1 β t-1 + 1 β τ T -1   = C β T = O(T β ) . (5.66) Then, since τ T = O(log T ), it follows that 1 1 + T -τ T I 1 (T ) ≤ 1 1 + T -τ T C β T = 1 T ( 1 T + 1 -τ T T ) C β T = O(T β-1 ) . (2) Using the inequality b k=a k -β ≤ b 1-β 1-β for 1 ≤ a < b and the fact that τ T = O(log T ), we have I 2 (T ) ≤ C   τ T T -τ t=0 (α t + β t + ξ t ) + (τ + 1) 2 T -τ t=0 α t + (1 + T -τ )α T   (5.67) ≤ C(τ (1 + T ) 1-β + (τ + 1) 2 (1 + T ) 1-α ) (5.68) = O((log T )T 1-β ) + O(log 2 (T )T 1-α ) = O((log T )T 1-β ) , (5.69)
where we recall for the second inequality that 0 < β < ξ < α < 1 . As a consequence,

1 1 + T -τ T I 2 (T ) = O((log T )T -β ) .
(3) Using the Cauchy-Schwartz inequality, we can write: .70) Then, observing that the sequences ( αt βt ) and ( ξt βt ) are nonincreasing, we have:

I 3 (T ) = T t=τ T C α t β t + ξ t β t E[ ν t ] ≤ C 2 T t=τ T α t β t + ξ t β t 2 T t=τ T E[ ν t 2 ] . ( 5
1 1 + T -τ T T t=τ T α t β t + ξ t β t 2 ≤ 2 1 + T -τ T T t=τ T    α t β t 2 + ( ξ t β t 2    = 2 1 + T -τ T T -τ T t=0    α t+τ T β t+τ T 2 + ξ t+τ T β t+τ T 2    ≤ 2 T -τ T + 1 T -τ T t=0    α t β t 2 + ξ t β t 2    ≤ (T -τ T + 1) -2(α-β) 1 -2(α -β) + (T -τ T + 1) -2(ξ-β) 1 -2(ξ -β) = O(T -2(α-β) + T -2(ξ-β) ) . (5.71)
(4) Similarly to item (3), to control the fourth term, we write:

1 1 + T -τ T T t=τ T α 2 t β t + ξ 2 t β t + β t ≤ 1 1 + T -τ T T -τ T t=0 α 2 t β t + ξ 2 t β t + β t ≤ (1 + T -τ T ) -(2α-β) 1 -(2α -β) + (1 + T -τ T ) -(2ξ-β) 1 -(2ξ -β) + (1 + T -τ T ) -β 1 -β = O(T -(2α-β) + T -(2ξ-β) + T -β ) . (5.72) Hence, 1 1 + T -τ T I 4 (T ) = O(T -(2α-β) + T -(2ξ-β) + T -β ) .
(5.73) Define: 5.74) Collecting the bounds from items (a) to (d) and incorporating them into Eq. ( 5.79), we obtain:

N (T ) := 1 1 + T -τ T T t=τ T E[ ν t 2 ] , (
F (T ) := 1 1 + T -τ T T t=τ T    α t β t 2 + ξ t β t 2    , (5.75) G(T ) := 1 1 + T -τ T (I 1 (T ) + I 2 (T ) + I 4 (T )) . ( 5
νt+1 2 ≤ (1 -2ζξ t ) νt 2 + C(ξ t β t + α t ) νt + 2ξ t νt • ν t + C(ξ 2 t β 2 t + ξ 2 t + α 2 t ) . (5.80)
Rearranging Ineq. (5.80) leads to:

2ζ νt 2 ≤ 1 ξ t ( νt 2 -νt+1 2 ) + C β t + α t ξ t νt + 2 νt • ν t + C ξ t β 2 t + ξ t + α 2 t ξ t .
(5.81) Summing this inequality for t between 1 and T and taking total expectation yield:

2ζ T T t=1 E[ νt 2 ] ≤ Σ 1 (T ) + Σ 2 (T ) + Σ 3 (T ) + Σ 4 (T ) , (5.82)
where

Σ 1 (T ) := 1 T T t=1 1 ξ t (E[ νt 2 ] -E[ νt+1 2 ]) , (5.83)
Σ 2 (T ) := C T T t=1 β t + α t ξ t E[ νt ] , (5.84)
Σ 3 (T ) := 2 T T t=1 E[ νt • ν t ] , (5.85) Σ 4 (T ) := C T T t=1 ξ t β 2 t + ξ t + α 2 t ξ t .
(5.86)

Similarly to Section 5.8.1.1, we control each one of the terms Σ i , i = 1, 2, 3, 4 successively.

(i) First, using the boundedness of (ν t ), we estimate Σ 1 as follows:

Σ 1 (T ) = 1 T   T t=1 1 ξ t - 1 ξ t-1 E[ νt 2 ] + 1 ξ 0 E[ ν1 2 ] - 1 ξ T E[ νT +1 2 ]   ≤ C T ξ T = O(T ξ-1 ) .
(ii) Cauchy-Schwarz inequality implies:

Σ 2 (T ) ≤ C T T t=1 β t + α t ξ t 2 T t=1 E[ νt 2 ] ≤ C 1 T T t=1   β 2 t + α t ξ t 2    1 T T t=1 E[ νt 2 ] .
Moreover, 2(α-ξ) ) .

1 T T t=1   β 2 t + α t ξ t 2    ≤ 1 T (T + 1) 1-2β 1 -2β + (T + 1) 1-2(α-ξ) 1 -2(α -ξ) = O(T -2β )+O(T -
(iii) Invoking the Cauchy-Schwarz inequality again yields:

Σ 3 (T ) ≤ 2 1 T T t=1 E[ νt 2 ] 1 T T t=1 E[ ν t 2 ] .
(iv) Similarly to item (ii), we obtain

Σ 4 (T ) = O(T -ξ-2β ) + O(T -ξ ) + O(T ξ-2α ) .
Define for every T > 0 the following quantities:

W (T ) := 1 T T t=1 E[ ν t 2 ] , (5.87) X(T ) := 1 T T t=1 E[ νt 2 ] , (5.88)
Y (T ) := 1 T T t=1   β 2 t + α t ξ t 2    , (5.89)
Z(T ) := Σ 1 (T ) + Σ 4 (T ) .

(5.90) It follows from items (i) to (iv) and Section 5.8.1.1 (for the last estimate) that 2(β-ξ) ) .

Y (T ) = O(T -2β ) + O(T -2(α-ξ) ) , (5.91)
Z(T ) = O(T ξ-1 ) + O(T -ξ-2β ) + O(T -ξ ) + O(T ξ-2α ) , (5.92)
W (T ) = O(T β-1 ) + O((log T )T -β ) + O(T 2(β-α) ) + O(T
(5.93)

Eq. ( 5.82) can be written:

2ζX(T ) ≤ C Y (T ) + W (T ) X(T ) + Z(T ) .
Solving this inequality implies:

X(T ) = O(Y (T ) + W (T ) + Z(T )) .
(5.94) Since 0 < β < ξ < α < 1, we obtain: 2(ξ-β) ) .

X(T ) = O(T ξ-1 ) + O((log T )T -β ) + O(T -2(α-ξ) ) + O(T -
(5.95) 5.8.1.3 End of Proof of Th. 5.5

We conclude our finite-time analysis of the critic by combining both previous sections (5.8.1.1 and 5.8.1.2): 5.96) where the second equality follows from using the identity w * (θ, ω * (θ)) = ω * (θ) for every θ ∈ R d , the inequality stems from using the classical inequality a + b 2 ≤ 2( a 2 + b 2 ) together with the fact that ω * is Lipschitz continuous, the penultimate equality is a consequence of Eq. ( 5.94) and the last equality is the result of the previous section (see Eq. ( 5.95)). (5.98)

1 T T t=1 E[ ω t -ω * (θ t ) 2 ] = 1 T T t=1 E[ ν t + ω * (θ t , ωt ) -ω * (θ t ) 2 ] = 1 T T t=1 E[ ν t + ω * (θ t , ωt ) -ω * (θ t , ω * (θ t )) 2 ] ≤ 2W (T ) + CX(T ) = O(X(T )) = O(T ξ-1 ) + O((log T )T -β ) + O(T -2(α-ξ) ) + O(T -2(ξ-β) ) , (
Note that the TD error δ t+1 used in Algorithm 5.1 coincides with δ(x t , ω t ).

Since the function ∇J is L J -Lipschitz continuous, a classical Taylor inequality combined with the update rule of (θ t ) yields:

J(θ t+1 ) ≥ J(θ t ) + α t ∇J(θ t ), δ(x t , ω t )ψ θt ( St , Ãt ) -L J 2 α 2 t δ(x t , ω t )ψ θt ( St , Ãt ) 2 .

(5.99) Recalling that θ → ψ θ (s, a) is bounded by , (R t ) and (ω t ) are bounded (see Assumption 5.5.3) and S, A are finite, we obtain from Eq. (5.99) that there exists a constant C s.t.: J(θ t+1 ) ≥ J(θ t ) + α t ∇J(θ t ), δ(x t , ω t )ψ θt ( St , Ãt ) -CL J α 2 t .

( In Eq. ( 5.102), the first inner product corresponds to the bias introduced by the critic.

The second one represents the linear FA error and the third translates the Markovian noise. Our task now is to control each one of these error terms in Eq. (5.102).

For the first term, observing that δ(x t , ω t ) -δ(x t , ω * (θ t )) = (γφ(S t+1 )φ( St )) Then, we control each one of the second and third terms in Eq. ( 5.102) in the following sections successively.

Control of the Markovian bias term

We introduce a specific convenient notation for the second term:

Γ(x t , θ t ) := ∇J(θ t ), δ(x t , θ t )ψ θt ( St , Ãt ) -∇J(θ t ) .

Recall from Section 5.8.1.1 the auxiliary Markov chain (x t ), the Markov chain (x t ) induced by the stationary distribution and the mixing time τ defined in Eq. (5.51).

Similarly to Section 5. We address each term of this decomposition successively.

(a) For this first term, we write:

Γ(x t , θ t ) -Γ(x t , θ t-τ ) = ∇J(θ t ) -∇J(θ t-τ ), δ(x t , θ t )ψ θt ( St , Ãt ) -∇J(θ t )

+ ∇J(θ t-τ ), (δ(x t , θ t )δ(x t , θ t-τ ))ψ θt ( St , Ãt )

+ ∇J(θ t-τ ), δ(x t , θ t-τ )(ψ θt ( St , Ãt )ψ θ t-τ ( St , Ãt ))

+ ∇J(θ t-τ ), ∇J(θ t-τ ) -∇J(θ t ) .

Moreover, note that: δ(x t , θ t )δ(x t , θ t-τ ) = γ(V π θ t (S t+1 ) -V π θ t-τ (S t+1 )) + V π θ t-τ ( St ) -V π θ t ( St ) .

Remark that ∇J, θ → ψ θ and θ → V π θ are bounded functions under Assumption 5.3.1 . Since ∇J, V π θ , ψ θ are in addition Lipschitz continuous as functions of θ (see, for e.g., (Shen et al., 2020, Lem. 3) for a proof for V π θ ) under Assumption 5. Here, the first inequality is a consequence of the definition of the total variation distance whereas the second inequality follows from applying (Wu et al., 2020, Lem. B.2). Indeed, using this last lemma, to show the last inequality, it is sufficient to write:

d T V (P(x t ∈ •| St-τ+1 , θ t-τ ), P(x t ∈ •| St-τ+1 , θ t-τ )) = d T V (P(( St , Ãt ) ∈ •| St-τ+1 , θ t-τ ), P(( Št , Ǎt ) ∈ •| St-τ+1 , θ t-τ )) ≤ d T V (P( St ∈ •| St-τ+1 , θ t-τ ), P( Št ∈ •| St-τ+1 , θ t-τ )) + 1 2 |A|L π E[ θ t -θ t-τ ] .
Iterating this inequality gives the desired result of Eq. (5.106) . We conclude from this item that: We conlude this section from Eq. ( 5.104) by collecting Eqs. (5.105) to (5.107) (items (a) to (d)) to obtain: (5.108) where the last inequality uses the definition of the mixing time τ and the fact that the sequence (α t ) is nonincreasing. Let us introduce the shorthand notation:

E[Γ(x t , θ t-τ ) -Γ(x t , θ t-τ )] ≥ -C t i=t-τ E[ θ i -θ t-τ ] . (
E[Γ(x t , θ t )] ≥ -CE[ θ t -θ t-τ ] -C t i=t-τ +1 E[ θ i -θ t-τ ] -Cσ τ -1 ≥ -C t i=t-τ +1 E[ θ i -θ i-1 ] -C t i=t-τ +1 i j=t-τ +1 E[ θ j -θ j-1 ] -Cσ τ -1 ≥ -C t i=t-τ +1 E[ θ i -θ i-1 ] -C t i=t-τ +1 t j=t-τ +1 E[ θ j -θ j-1 ] -Cσ τ -1 ≥ -C(τ + 1) t i=t-τ +1 E[ θ i -θ i-1 ] -Cσ τ -1 ≥ -C(τ + 1) 2 α t-τ -Cα T ,
∆(x, θ) := ∇J(θ), ( δ(x, ω * (θ))δ(x, θ))ψ θ t-τ ( S, Ã)

Note here that θ t-τ is fixed for ψ θ t-τ in the above notation. The following decomposition holds:

∆(x t , θ t ) = (∆(x t , θ t ) -∆(x t , θ t-τ )) + (∆(x t , θ t-τ ) -∆(x t , θ t-τ ))

+ (∆(x t , θ t-τ ) -∆(x t , θ t-τ )) + ∆(x t , θ t-τ ) . (5.110) Similar derivations to the previous section allow us to control each one of the error terms.

(i) Using that θ → ∇J, θ → V π θ and θ → ω * (θ) are Lipschitz continuous, we obtain:

∆(x t , θ t ) -∆(x t , θ t-τ ) ≥ -C θ tθ t-τ .

( 5.111) Using similar manipulations to the previous section, we get:

(ii) (5.114) Then, recall that xt = ( St , Āt , S t+1 ) where S t+1 ∼ p(•| St , Āt ) and observe that: δ(x t , ω * (θ t-τ ))δ(x t , θ t-τ ) = γ(φ(S t+1 ) T ω * (θ t-τ ) -V π t-τ (S t+1 )) + (V π t-τ ( St )φ( St ) T ω * (θ t-τ )) . (5.115) Recalling that p = γp + (1γ)ρ and using Assumption 5.6.3 , one can then easily show that:

E[| δ(x t , ω * (θ t-τ )) -δ(x t , θ t-τ )||θ t-τ ] ≤ C FA .
As a consequence, noticing again that ∇J is Lipschitz continuous, we obtain from Eq. (5.114):

E[∆(x t , θ t-τ )] ≥ -C FA E[ ∇J(θ t-τ ) ] ≥ -C FA E[ θ t -θ t-τ ]-C FA E[ ∇J(θ t ) ] .
Combining items (i) to (iv) with the boundedness of the function ∇J, we conclude from this section that: E[ ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))ψ θt ( St , Ãt ) ] 5.116) where the last inequality has already been established in Section 5.8.1 with the choice of the mixing time τ = τ T . Rearranging and summing this inequality for t = τ T to T lead to: (5.121)

≥ -CE[ θ t -θ t-τ ] -C t i=t-τ E[ θ i -θ t-τ ] -Cσ τ -1 -C FA ≥ -C((τ + 1) 2 α t-τ + α T + FA ) , (
1 T -τ T + 1
Let us now provide estimates of each one of the quantities U i (T ) for i = 1, 2, 3 .

1. Noticing that the function J is bounded and the sequence (α t ) is nonincreasing, the first term can be controlled as follows:

U 1 (T ) = 1 T -τ T + 1   1 α T E[J(θ T +1 )] - 1 α τ T -1 E[J(θ τ T )] + T t=τ T 1 α t-1 - 1 α t E[J(θ t )]   ≤ C T -τ T + 1 1 α T + 1 α τ T -1 + 1 α T - 1 α τ T -1 ≤ C T -τ T + 1 2 α T = O(T α-1 ) .
(5.122)

2. Recalling that the sequence of stepsizes (α t ) is nonincreasing and that τ T = O(log T ), the second term can be estimated by the following derivations: (5.123)

3. Using the Cauchy-Schwarz inequality, we have:

U 3 (T ) ≤ C T -τ T + 1 T t=τ T E[ ∇J(θ t ) 2 ] T t=τ T E[ ω t -ω * (θ t ) 2 ] .
(5.124)

Define the quantities:

F (T ) := 1 T -τ T + 1 T t=τ T E[ ∇J(θ t ) 2 ] ,
(5.125)

E(T ) := 1 T -τ T + 1 T t=τ T E[ ω t -ω * (θ t ) 2 ] ,
(5.126) K(T ) := U 1 (T ) + U 2 (T ) + C FA .

(5.127)

Using these definitions, we can rewrite Eq. ( 5.118) as follows:

F (T ) ≤ C F (T ) E(T ) + K(T ) .
Solving this inequality yields:

1 T T t=1 E[ ∇J(θ t ) 2 ] = 1 T   τ T -1 t=1 E[ ∇J(θ t ) 2 ] + T t=τ T E[ ∇J(θ t ) 2 ]   ≤ C log T T + O   1 T -τ T + 1 T t=τ T E[ ∇J(θ t ) 2 ]   .
This completes the proof given Eq. (5.130). 5.8.2.4 Proof of Cor. 5.7 The result is a consequence of combining Ths. 5.5 and 5.6 and simplifying the obtained rate using the fact that 0 < β < ξ < α < 1 .

Conclusion and Perspectives

About non-convex stochastic optimization

The major part of this thesis was devoted to the study of momentum and adaptive gradient methods for non-convex stochastic optimization. Our analysis started with the popular Adam algorithm for which we considered two different stepsizes regimes. In the constant stepsize regime, we established a long run convergence result under a stability assumption. In the vanishing stepsizes regime, we proposed a novel variant of Adam for which we showed the almost sure convergence of the iterates towards the set of critical points of the objective function and a central limit theorem characterizing the fluctuations. In our study, the ODE method was pivotal to understand the dynamical behavior of Adam and derive convergence results in discrete time. In the second part of this thesis, beyond Adam and starting from a more general continuous-time dynamical system, we analyzed a general stochastic optimization algorithm in the decreasing stepsizes regime. This algorithm offers degrees of freedom and thereby encompasses several adaptive gradient methods including Adam and momentum methods such as S-NAG. In this part, similarly to our previous Adam analysis, we established stability, almost sure convergence and convergence rates results. A major issue we tackled in this thesis is that of avoidance of traps, showing that the stochastic algorithms we considered in this thesis cannot converge towards undesirable critical points such as local maxima and saddle points. The linchpin of our proof is a general avoidance of traps result extending the seminal works of [START_REF] Pemantle | Nonconvergence to unstable points in urn models and stochastic approximations[END_REF]; [START_REF] Brandière | Les algorithmes stochastiques contournent-ils les pièges?[END_REF] to a non-autonomous setting, and which we believe is of independent interest. Finally, we proved some quantitative results complementing our asymptotic analysis. These results consists of bounds controlling the expected gradient norm of the objective function along the iterations in the same stochastic setting and function value gap convergence rates in the deterministic setting.

Concerning these last contributions, we bring to the attention of the reader that the literature has witnessed many developments after the publication on which Chapter 4 is based. In particular, the recent works of [START_REF] Défossez | A simple convergence proof of adam and adagrad[END_REF] and (Gadat and Gavra, 2020, Th. 2) provide some interesting quantitative results in the flavor of the results we have presented about bounding the expected gradient of the objective function along the iterates of the algorithm. We also mention that it would be interesting to extend our convergence rates based on the Kurdyka-Łojasiewicz property to the stochastic setting. [START_REF] Benaïm | On strict convergence of stochastic gradients[END_REF] provides some results in this flavor for stochastic gradient algorithms.

Our analysis in the main part of this thesis opens the way for several directions of future research.

Constrained optimization. First, it can be interesting to explore constrained stochastic optimization problems and design proximal variants of momentum and adaptive gradient methods such as Adam.

Nonsmooth optimization. Second, this thesis was focused on the case where the objective function is differentiable. However, neural networks may involve points of non-differentiability because of the use of some popular activation functions such as ReLU (Rectified Linear Unit which is the positive part function). Recent results [START_REF] Davis | Stochastic subgradient method converges on tame functions[END_REF][START_REF] Majewski | Analysis of nonsmooth stochastic approximation: the differential inclusion approach[END_REF] study the stochastic subgradient method for a class of nonsmooth non-convex functions via the differential inclusion approach [START_REF] Benaïm | Stochastic approximations and differential inclusions[END_REF]. The mathematical artillery to tackle this nonsmoothness problem was recently set up in the literature. [START_REF] Bolte | Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning[END_REF] introduced a notion of generalized derivatives leading to a class of locally Lipschitz functions called path differentiable functions encompassing convex, Clarke regular and Whitney stratifiable functions (and many others). Authors then developed a generalized differential calculus which allows to analyze modern algorithms based on automatic differentiation in a nonsmooth context. In particular, following the differential inclusion approach of [START_REF] Benaïm | Stochastic approximations and differential inclusions[END_REF], discrete stochastic algorithms used for training nonsmooth deep neural networks were analyzed within this framework to obtain almost sure subsequential convergence to steady states. This framework can be applied to the popular algorithms we have studied in this thesis. Another approach based on the use of closed measures from the calculus of variations and dynamical systems was proposed by [START_REF] Bolte | Long term dynamics of the subgradient method for lipschitz path differentiable functions[END_REF] to study the subgradient method for Lipschitz path differentiable functions and could prove useful for more complex dynamics such as the ones induced by the algorithms considered in the present thesis.

Regarding avoidance of traps, more challenging problems arise in the nonsmooth setting. Very recent research efforts have been undertaken to address the case of the stochastic subgradient descent method in nonsmooth stochastic optimization [START_REF] Bianchi | Stochastic subgradient descent escapes active strict saddles[END_REF][START_REF] Davis | Subgradient methods near active manifolds: saddle point avoidance, local convergence, and asymptotic normality[END_REF][START_REF] Schechtman | Stochastic subgradient descent on a generic definable function converges to a minimizer[END_REF]. The question remains open for more complex algorithms.

Other applications. The success of Adam in deep learning and stochastic optimization inspired novel algorithms for other problems such as min-max optimization (see for instance Extra-Adam in [START_REF] Gidel | A variational inequality perspective on generative adversarial networks[END_REF] mixing Adam with the extragradient method of [START_REF] Korpelevich | The extragradient method for finding saddle points and other problems[END_REF]) for which no theoretical guarantees exist to the best of our knowledge. Algorithms based on Adam could also prove useful in estimating optimal transport distances as this problem was cast as a stochastic optimization problem by Genevay et al. (2016) and further explored by Bercu and Bigot (2021); Bercu et al. (2021).

About actor-critic methods with target networks

In the RL part of this thesis, we studied an actor-critic algorithm incorporating a target network inspired from several state of the art algorithms used in deep RL. Our algorithm uses three different timescales: one for the actor and two for the critic. Instead of using the single timescale TD learning algorithm as a critic, we use a two timescales target-based version of TD learning closely inspired from practical actor-critic algorithms implementing target networks. More precisely, we proved both asymptotic and nonasymptotic convergence results. First, we showed that the critic which uses a target variable tracks a slowly moving target corresponding to the well-known TD solution. Then, we proved that the actor parameter visits infinitely often a region of the parameter space where the norm of the policy gradient is dominated by a bias induced by linear function approximation. Second, we established a bound controlling the average expected gradient (of the performance function) norm evaluated at the actor iterates generated by our target-based algorithm showing the quantitative impact of using a target network.

There are several interesting directions for future research regarding our actor-critic method.

Nonlinear function approximation. Our analysis addresses the linear function approximation setting. Many state of the art actor-critic algorithms in deep RL make use of target networks to stabilize the training [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF][START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF][START_REF] Fujimoto | Addressing function approximation error in actorcritic methods[END_REF]. A theoretical justification of the use of a target network in the nonlinear function approximation setting beyond linear function approximation is a challenging problem that merit further investigation. In particular, as practical algorithms in deep RL seem to indicate, it would be interesting to see if such a mechanism can be a theoretically grounded alternative to the failure of temporal difference learning with nonlinear function approximation as was noticed in (Tsitsiklis and Van Roy, 1997, Section 10). A significant gap still persists between theory and practice. Actor-critic methods are usually analyzed in the two timescales stochastic approximation framework where two different stepsizes decreasing at different rates are used for the actor and the critic. However, recent general two timescales stochastic approximation results established by [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF] do not permit to show the convergence of actor-critic methods with target networks in the nonlinear function approximation setting as the assumptions made almost amount to consider linear function approximation and lead to suspect the existence of undesirable accumulation points (see Assumption (A6'), p. 7 and Remark 8, p. 12 in [START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF]). This challenging problem would probably need an approach tailored to actor-critic methods with target networks. Besides, we recall that the target network idea was originally proposed in [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] which introduced the DQN algorithm. Recently, some theoretical issues were pointed out in [START_REF] Avrachenkov | Full gradient dqn reinforcement learning: a provably convergent scheme[END_REF] for DQN (with single timescale) and authors proposed a modification to fix these issues with some additional computation. This work gives interesting insights on this target network mechanism and prompts us to be cautious about using such an appealing scheme, at least when considering a single timescale (see also [START_REF] Wang | A convergent and efficient deep q network algorithm[END_REF]).

Experience replay. This consists in storing past transitions (i.e., (state, action, reward, next state) tuples) in memory and reusing them throughout learning instead of only using transitions as they occur during simulation or actual experience. This feature has proven useful in RL thanks to its numerous advantages such as variance reduction, robustness to anomalous transitions, preventing overfitting, allowing data re-use and handling delayed rewards as this had been reported and discussed for instance in (Avrachenkov et al., 2021, Section 2.3) which deals with the specific case of DQN. The insights provided in the aforementioned work may prove useful for the analysis of actor-critic algorithms with experience replay.

Nonsmoothness. As previously discussed, some nonsmoothness issues related to the use of nonsmooth activation functions (such as ReLU) also arise and need further investigation.

Off-policy learning. Another possible avenue of future work is to extend our proposed actor-critic algorithm to the off-policy setting which is pervasive in several successful deep RL actor-critic algorithms [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF][START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF][START_REF] Fujimoto | Addressing function approximation error in actorcritic methods[END_REF].. In this setting, the policy followed to generate samples called the behavior policy differs from the so-called target policy which is the policy of interest to be evaluated for example in policy evaluation. Off-policy learning is paramount to manage the celebrated exploration-exploitation trade-off and to learn efficiently in large-scale problems in general. An algorithm combining off-policy learning, function approximation and bootstrapping simultaneously is usually not guaranteed to be well behaved, leading to the famous deadly triad (Sutton and Barto, 2018, Chapter 11, Section 3). If the discounted reward setting has been solved more than ten years ago (Sutton et al., 2009b,a), provably convergent value-based algorithms for the average-reward case have only been recently introduced by Zhang et al. (2021a). Breaking the deadly triad is still a current research question (Zhang et al., 2021b) and designing off-policy RL algorithms remains a notoriously challenging task which is the subject of active research [START_REF] Xu | Doubly robust off-policy actor-critic: Convergence and optimality[END_REF]. Furthermore, very recent finite-time analysis of off-policy actor-critic RL algorithms with linear function approximation [START_REF] Chen | Cada: Communication-adaptive distributed adam[END_REF] can be an additional starting point to investigate the case of off-policy actor-critic algorithms with target networks.

Importing momentum and adaptive methods into RL

We would like to end this thesis by discussing some future directions we actively began to explore. These opportunities of future work would allow us to bridge together the seemingly independent parts of this thesis which we summarized in the previous sections of this conclusive chapter.

Momentum and adaptive gradient methods such as RMSProp and Adam are widely used (if not ubiquitous) as optimizers to train neural networks approximators in deep reinforcement learning (see [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]; [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF]; [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]; [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]; [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] to name a few famous examples). During the last couple of years, a flurry of non-asymptotic theoretical analysis has been conducted around standard policy gradient, value-based and actor-critic methods to determine their sample complexity. However, in this recent growing line of research, only few works touch upon momentum and adaptive methods in RL and they are almost all restricted to the linear function approximation case excluding deep neural networks. We also highlight that theoretical guarantees for (two-layer) neural network approximators are very recent even for standard algorithms such as TD learning (see for e.g., [START_REF] Cayci | Sample complexity and overparameterization bounds for projection-free neural td learning[END_REF]) and Q-learning (see for instance [START_REF] Xu | A finite-time analysis of q-learning with neural network function approximation[END_REF]). Overall, despite the popularity of momentum and adaptive methods in deep RL, theoretical understanding of their convergence behavior is still in its infancy. Moreover, it can be interesting to design novel algorithms by incorporating momentum and adaptive gradient methods ideas from optimization into the landmark principled classes of RL algorithms, namely, policy-based methods, value-based methods and actor-critic algorithms. Few recent preliminary works have followed this direction [START_REF] Vieillard | Momentum in reinforcement learning[END_REF][START_REF] Sun | Adaptive temporal difference learning with linear function approximation[END_REF][START_REF] Xiong | Non-asymptotic convergence of adam-type reinforcement learning algorithms under markovian sampling[END_REF][START_REF] Romoff | Tdprop: Does adaptive optimization with jacobi preconditioning help temporal difference learning[END_REF][START_REF] Weng | Finite-time theory for momentum q-learning[END_REF]. To the best of our knowledge, all the works incorporating momentum or adaptive methods into RL have no guarantees in the nonlinear function approximation case. As a first step, a better theoretical understanding of adaptive optimizers in TD learning is still needed to discover optimizers achieving a significant improvement over vanilla TD learning in practice.

Momentum and eligibility traces. Although different, the concept of momentum incorporated to TD learning shares some similarity with the eligibility traces mechanism which is widely known in the RL research community. As explained in (Sutton and Barto, 2018, Chapter 12), eligibility traces can be seen as a tool to remedy the problem of long-delayed rewards and can be suitable for the non-Markov setting, i.e., when the environment is not modeled as an MDP. Interestingly, the concept of momentum from optimization captures the idea of a memory of the gradients used all over the iterations. Therefore, transferring momentum to RL has the potential to address the issue of long-delayed rewards and could also be beneficial in non-Markov settings. The comparison of momentum to eligibility traces has not been conducted in the literature and the analysis of a momentum variant of TD learning is missing. The recent work of [START_REF] Bengio | Correcting momentum in temporal difference learning[END_REF] is concerned with momentum in TD learning but does not address the questions we have raised here.

Policy gradient methods with momentum and adaptive stepsizes. To reduce the large variance of vanilla policy gradient estimates and improve the sample complexity, [START_REF] Huang | Momentum-based policy gradient methods[END_REF] and [START_REF] Yuan | Stochastic recursive momentum for policy gradient methods[END_REF] proposed to augment the update rules with an exponential moving average. Nevertheless, the proposed algorithms are not truly adaptive because the learning rates are forced to be decreasing unlike in Adam. Moreover, global convergence rates for policy gradient methods incorporating adaptive stepsizes are still missing. Since adaptive gradient methods are "geometry-aware" algorithms, these methods should have interesting connections with the recently introduced geometryaware normalized policy gradient (GNPG) and its analysis combining the concept of non-uniform smoothness and the non-uniform Łojasiewicz inequality [START_REF] Mei | On the global convergence rates of softmax policy gradient methods[END_REF][START_REF] Mei | Leveraging non-uniformity in first-order non-convex optimization[END_REF]. It would also be interesting to compare the convergence rates of our method to the linear convergence of GNPG in the deterministic setting where exact gradients are supposed to be available.

Beyond the RL problem we tackle in this thesis, we can consider several other RL problems such as risk-sensitive RL or multi-agent RL.

Risk-sensitive RL setting. Beyond risk-neutral RL where the performance metric to be maximized by the agent is for instance the expectation of the expected total (possibly discounted) reward, the agent may also aim at minimizing at the same time a risk measure (e.g., variance or value at risk). Although risk-sensitive control dates back to the previous century (see for e.g., [START_REF] Whittle | Risk-sensitive optimal control[END_REF]) and have witnessed many developments (see for e.g., [START_REF] Borkar | A sensitivity formula for risk-sensitive cost and the actor-critic algorithm[END_REF][START_REF] Borkar | An actor-critic algorithm for constrained markov decision processes[END_REF]; Bhatnagar (2010) actor-critic algorithms in this setting, [START_REF] Borkar | Q-learning for risk-sensitive control[END_REF] for Q-learning, and Borkar (2010); [START_REF] García | A comprehensive survey on safe reinforcement learning[END_REF] for surveys, and the references therein), risk-sensitive RL algorithms are less advanced in comparison to the empirical success of risk-neutral RL that we discussed so far. This gap calls for the need to design new algorithms for this setting which is motivated by real-world applications requirements. Some of the most recent efforts in this direction include the works of [START_REF] Bisi | Risk-averse trust region optimization for reward-volatility reduction[END_REF] and [START_REF] Whiteson | Mean-variance policy iteration for risk-averse reinforcement learning[END_REF] (see also [START_REF] Karmakar | On tight bounds for function approximation error in risk-sensitive reinforcement learning[END_REF] for error bounds). We could leverage our ideas in risk-neutral RL with advanced optimization methods for risk-sensitive RL.

Multi-agent RL setting. Finally, using momentum and adaptive gradient methods could also prove useful in the multi-agent RL setting. In the distributed RL problem, a network of agents seeks to maximize a global return cooperatively via communication with local neighbors. Consider for instance the setting where rewards are decentralized and each agent has access to the full state and action information. In the last few years, distributed versions of TD learning, Q-learning and actor-critic algorithms have been proposed in the literature (see [START_REF] Lee | Optimization for reinforcement learning: From a single agent to cooperative agents[END_REF] for a recent survey). If the Adam algorithm has been recently extended to a distributed learning context by [START_REF] Chen | Cada: Communication-adaptive distributed adam[END_REF] for supervised learning, an adaptation of the algorithm to the multi-agent RL setting is yet to be proposed. Title : Contributions to non-convex stochastic optimization and reinforcement learning Keywords : Optimization, adaptive gradient methods with momentum, avoidance of traps, stochastic approximation, dynamical systems, reinforcement learning, actor-critic methods Abstract : This thesis is focused on the convergence analysis of some popular stochastic approximation methods in use in the machine learning community with applications to optimization and reinforcement learning. The first part of the thesis is devoted to a popular algorithm in deep learning called ADAM used for training neural networks. This variant of stochastic gradient descent is more generally useful for finding a local minimizer of a function. Assuming that the objective function is differentiable and non-convex, we establish the convergence of the iterates in the long run to the set of critical points under a stability condition in the constant stepsize regime. Then, we introduce a novel decreasing stepsize version of ADAM. Under mild assumptions, it is shown that the iterates are almost surely bounded and converge almost surely to critical points of the objective function. Finally, we analyze the fluctuations of the algorithm by means of a conditional central limit theorem. In the second part of the thesis, in the vanishing stepsizes regime, we generalize our convergence and fluctuations results to a stochastic optimization procedure unifying several variants of the stochastic gradient descent such as, among others, the stochastic heavy ball method, the Stochastic Nesterov Accelerated Gradient algorithm, and the widely used ADAM algorithm. We conclude this second part by an avoidance of traps result establishing the non-convergence of the general algorithm to undesired critical points, such as local maxima or saddle points. Here, the main ingredient is a new avoidance of traps result for non-autonomous settings, which is of independent interest. Finally, the last part of this thesis which is independent from the two previous parts, is concerned with the analysis of a stochastic approximation algorithm for reinforcement learning. In this last part, we propose an analysis of an online target-based actor-critic algorithm with linear function approximation in the discounted reward setting. Our algorithm uses three different timescales: one for the actor and two for the critic. Instead of using the standard single timescale temporal difference (TD) learning algorithm as a critic, we use a two timescales target-based version of TD learning closely inspired from practical actorcritic algorithms implementing target networks. First, we establish asymptotic convergence results for both the critic and the actor under Markovian sampling. Then, we provide a finite-time analysis showing the impact of incorporating a target network into actor-critic methods.

Institut Polytechnique de Paris 91120 Palaiseau, France

N 2

 2 Set of integers: {0, 1, 2, . . .} A ∪ B Set union between the sets A and B A ∩ B Set intersection between the sets A and B If u, v are vectors of R d where d ∈ N, u v Coordinatewise product vector with coordinates u i v i u Coordinatewise square vector with coordinates u 2 i u v

Assumption 2 .

 2 7.1. The function F : R d → R is s.t.: F is continuously differentiable and ∇F is locally Lipschitz continuous. Assumption 2.7.2. The map S : R d → [0, +∞) d is locally Lipschitz continuous.

Theorem 3 . 2 .

 32 Let Assumptions 3.2.1 and 3.2.2 hold true. Then, for each x 0 ∈ R d , there exists a unique bounded global solution (m, x) : R + → R d × R d to (ODE-N) with the initial condition (m(0), x(0)) = (0, x 0 ). As t → +∞, (m(t), x(t)) converges towards the set Ῡ := {(0, x ) : x ∈ zer ∇F }.(3.3)

Assumption 3 . 3 . 3 .

 333 In Assumption 3.3.1, Conditions ii) and iii) are respectively replaced with the stronger conditions

  3.8 are related to the function e defined in Eq.(3.11), which can be seen as a non-autonomous perturbation of the autonomous linear ODE ż(t) = D(z(t)z ). These assumptions guarantee the existence of a local (around the unstable equilibrium z ) non-autonomous invariant manifold of the non-autonomous ODE ż(t) = b(z(t), t) with enough regularity properties, as provided by Prop. 3.18 and Prop. 3.20 below.3.4.2 Application to the stochastic algorithms3.4.2.1 Trap avoidance of the general algorithm 3.1

3. 4 . 2 . 2

 422 Trap avoidance for S-NAG Assumption 3.4.4. Let x ∈ zer ∇F and let the following conditions hold.

2

 2

3. 5 . 1

 51 Proof of Th. 3.1

Proof.

  The map Φ : R 2d+1 × R + → R 2d+1 is continuous, hence uniformly continuous on K × [0, T ]. The result follows from the application of the Arzelà-Ascoli theorem to the family Φ(z, •) : [0, T ] → R 2d+1 z∈K .

  ), with u (t) = A + B t for some A and B in R d . We then have by uniqueness of the limit that A = A and B = B. As a consequence, u (T ) = A + BT > C and we obtain a contradiction. Hence B = 0.

  .88) A similar result holds for E v nv n+1 2 1 zn∈W following the same arguments. In view of Eqs. (3.87)-(3.88), it holds that E n ρn+1 2 1 zn∈W < +∞ given the assumption n γ 2 n+1 < +∞. Therefore, n ρn+1 2 1 zn∈W < +∞ a.s., which completes our verification of condition v) of Th. 3.8. Assumption vi) follows from condition 3.4.3-iii). Finally, let us make Assumption vii) of Th. 3.8 more explicit. Partitioning the matrix Q -1 as

  Lemma 4.4. (Properties of the limit point set) Let (z k ) k∈N be the sequence defined for all k ∈ N by z k = (x k , y k ) where y k = √ a k p k and (x k , p k ) is generated by Algorithm 5.1 from a starting point z 0 . Let Assumptions 4.2.1, 4.4.1, and 4.5.1 hold true. Assume that Condition (4.5) holds. Then,

  Definition 4.5.2. (KŁ property, Bolte et al. (2018, Appendix)) A proper and lower semicontinuous (l.s.c) function H : R 2d → (-∞, +∞] has the KŁ property locally at

Lemma 4. 5 .

 5 (Uniformized KŁ property, Bolte et al. (2014, Lemma 6, p 478))

Theorem 4. 6 .Figure 4

 64 Figure 4.1 -Illustration of KŁ rates for a simple objective function f (x) = x p . From left to right: (i) curves of f (x) = x p , (ii) clipped version of Adam (see Algorithm 5.1), (iii) Adam ,(iv) Gradient Descent. Best seen in color.

Corollary 4. 8 .

 8 Let (z k ) k∈N be the sequence defined for all k ∈ N by z k = (x k , y k ) where y k = √ a k p k and (x k , p k ) is generated by Algorithm 5.1 from a starting point z 0 . Let Assumptions 4.2.1, 4.4.1, 4.5.1 hold true. Assume that Condition (4.5) holds.

Figure 4 . 2 -

 42 Figure 4.2 -Illustration of the first step of the proof of Th. 4.6

  .28) (b) Suppose now the opposite inequation 2r 2θ-2 k-2 < r 2θ-2 k

  .8) that (ω t ) is associated to the ODE d ds ω(s) = h(θ(s), ω(s)) -Ḡ(θ(s)) ω(sR d × R m → R m and Ḡ : R d → R m×m are defined for every θ ∈ R d , ω ∈ R m by h(θ, ω) := Φ T D ρ,θ (R θ + γP θ Φ ω) and Ḡ(θ) := Φ T D ρ,θ Φ . (5.10)

Proposition 5 . 1 .

 51 Under Assumptions 5.3.1 and 5.5.1 to 5.5.4 , the linear equation Ḡ(θ)ω = h(θ, ω) has a unique solution ω * (θ, ω) for every θ ∈ R d , ω ∈ R m and it holds that lim t ω tω * (θ t , ωt ) = 0 w.p.1.

  d ds ω(s) = Ḡ(θ(s)) -1 (h(θ(s)) -G(θ(s))ω(s)) , d ds θ(s) = 0 , (ODE-ω)where h : R d → R n and G : R d → R m×m are defined for every θ ∈ R d by h(θ) := Φ T D ρ,θ R θ and G(θ) := Φ T D ρ,θ (I n -γP θ )Φ .

Proposition 5 . 2 .

 52 Under Assumptions 5.3.1 and 5.5.1 to 5.5.4 , for every θ ∈ R d , the linear equation G(θ)ω = h(θ) has a unique solution ω * (θ) and lim t ωtω * (θ t ) = 0 w.p.1 . Moreover, for every θ ∈ R d , Φ ω * (θ) is a fixed point of the projected Bellman operator, i.e., Π θ T θ (Φ ω * (θ)) = Φ ω * (θ), where Π θ = Φ(Φ T D ρ,θ Φ) -1 Φ T D ρ,θ is the projection matrix on the space {Φ ω : ω ∈ R m } of all vectors of the form Φ ω for ω ∈ R m w.r.t. the norm • D ρ,θ .

Theorem 5 . 3 .

 53 Let Assumptions 5.3.1, and 5.5.1 to 5.5.4 hold true . Then, we have lim t ω tω * (θ t ) = 0 w.p.1 .

  ω, ω ∈ R m , z = (s, a) ∈ S × A, w(ω, ω, z) :=   R(s, a) + γ s ∈S p(s |s, a)φ(s ) T ω  φ(s)φ(s)φ(s) T ω(5.13) and η

Lemma 5 .

 5 10. If Assumptions 5.5.1 and 5.5.4 hold, there exists κ > 0 s.t. for all θ ∈ R d and ω

( b )

 b The functions θ → R θ and θ → P θ are Lipschitz continuous as P θ (s |s) = p(s |s, a)π θ (a|s) , R θ (s) = a∈A R(s, a)π θ (a|s) and Assumption5.3.1-(b) guarantees that θ → π θ (a|s) is Lipschitz continuous for every (s, a) ∈ S × A .

5. 8 . 2

 82 Proof of Th. 5.6 Recall the notation xt := ( St , Ãt , S t+1 ). In this section, we overload this notation with the reward sequence (R t ), i.e., xt := ( St , Ãt , S t+1 , R t+1 ) . Let us fix some additional convenient notations. Define for every x = (s, ã, s, r)∈ S × A × S × [-U R , U R ], and every ω ∈ R m , θ ∈ R d : δ(x, ω) := r + γφ(s) T ωφ(s) T ω(5.97) δ(x, θ) = r + γV π θ (s) -V π θ (s) .

  c) Regarding the third term, similarly to item (b), we can write:E[Γ(x t , θ t-τ ) -Γ(x t , θ t-τ )] ≥ -CE[d T V (P(x t ∈ •| St-τ+1 , θ t-τ ), P(x t ∈ •| St-τ+1 , θ t-τ ))] = -CE[d T V (P(x t ∈ •| St-τ+1 , θ t-τ ), d ρ,θ t-τ ⊗ π θ t-τ ⊗ p)] = -CE[d T V (P( Št ∈ •| St-τ+1 , θ t-τ ), d ρ,θ t-τ )] ≥ -Cσ τ -1 , (5.107) where the equalities follow from the definitions of xt , xt and the last inequality stems from Assumption 5.6.1 . (d) Since the Markov chain xt is built s.t. St ∼ d ρ,θ t-τ , Āt ∼ π θ t-τ , S t+1 ∼ p(•| St , Āt ), one can see that E[Γ(x t , θ t-τ )] = 0 .

  (θ t ) 2 ] ≤ U 1 (T ) + U 2 (T ) + U 3 (T ) + C FA , 1) 2 α t-τ T + α T + α t ) , (θ t ) 2 ] E[ ω tω * (θ t ) 2 ] .

U 2 (

 2 τ T + (Tτ T + 1)α T + τ T + 1)α T = O(log 2 (T )T -α ) .

F

  (T ) = O(E(T )) + O(K(T )) .(5.128) We conclude the proof by remarking that items (1) to (3) above imply:K(T ) = O(T α-1 ) + O(log 2 (T )T -α ) + O( FA ) .(5.129)Eqs.(5.128) and (5.129) combined can be explicitely written as follows:1 Tτ T + 1 T t=τ T E[ ∇J(θ t ) 2 ] = O(T α-1 ) + O(log 2 (T )T -α ) + O( FA )

Titre:

  Contributions à l'optimisation stochastique non convexe et à l'apprentissage par renforcementMots cl és : Optimisation, m éthodes à gradient adaptatives avec momentum, évitement de pi èges, approximation stochastique, syst èmes dynamiques, apprentissage par renforcement, m éthodes acteur-critique R ésum é : Cette th èse est centr ée autour de l'analyse de convergence de certains algorithmes d'approximation stochastiques utilis és en machine learning appliqu és à l'optimisation et à l'apprentissage par renforcement. La premi ère partie de la th èse est d édi ée à un c él èbre algorithme en apprentissage profond appel é ADAM, utilis é pour entraîner des r éseaux de neurones. Cette c él èbre variante de la descente de gradient stochastique est plus g én éralement utilis ée pour la recherche d'un minimiseur local d'une fonction. En supposant que la fonction objective est diff érentiable et non convexe, nous établissons la convergence des it ér ées au temps long vers l'ensemble des points critiques sous une hypoth èse de stabilit é dans le r égime des pas constants. Ensuite, nous introduisons une nouvelle variante de l'algorithme ADAM à pas d écroissants. Nous montrons alors sous certaines hypoth èses r éalistes que les it ér ées sont presque s ûrement born ées et convergent presque s ûrement vers des points critiques de la fonction objective. Enfin, nous analysons les fluctuations de l'algorithme par le truchement d'un th éor ème central limite conditionnel. Dans la deuxi ème partie de cette th èse, dans le r égime des pas d écroissants, nous g én éralisons nos r ésultats de convergence et de fluctuations à une proc édure d'optimisation stochastique unifiant plusieurs variantes de descente de gradient stochastique comme la m éthode de la boule pesante, l'algorithme stochastique de Nesterov acc él ér é ou encore le c él èbre al-gorithme ADAM, parmi d'autres. Nous concluons cette partie par un r ésultat d' évitement de pi èges qui établit la non convergence de l'algorithme g én éral vers des points critiques ind ésirables comme les maxima locaux ou les pointsselles. Ici, le principal ingr édient est un nouveau r ésultat ind épendant d' évitement de pi èges pour un contexte nonautonome. Enfin, la derni ère partie de cette th èse qui est ind épendante des deux premi ères parties est d édi ée à l'analyse d'un algorithme d'approximation stochastique pour l'apprentissage par renforcement. Dans cette derni ère partie, dans le cadre des processus d écisionnels de Markov avec crit ère de r écompense γ-pond ér é, nous proposons une analyse d'un algorithme acteur-critique en ligne int égrant un r éseau cible et avec approximation de fonction lin éraire. Notre algorithme utilise trois échelles de temps distinctes: une échelle pour l'acteur et deux autres pour la critique. Au lieu d'utiliser l'algorithme de diff érence temporelle (TD) standard à une échelle de temps, nous utilisons une version de l'algorithme TD à deux échelles de temps int égrant un r éseau cible inspir é des algorithmes acteurcritique utilis és en pratique. Tout d'abord, nous établissons des r ésultats de convergence pour la critique et l'acteur sous échantillonage Markovien. Ensuite, nous menons une analyse à temps fini montrant l'impact de l'utilisation d'un r éseau cible sur les m éthodes acteur-critique.

  2)are well defined; F is continuously differentiable and by Lebesgue's dominated convergence theorem, ∇F (x) = E(∇f (x, ξ)) for all x. Moreover, ∇F and S are locally Lipschitz continuous.

	Assumption 2.2.2. F is coercive.

Assumption 2.2.3. For every x ∈ R d , S(x) > 0.

  Adam-decreasing stepsize (((γ n , α n , β n

	Algorithm 2.2
	). Let Assumptions 2.2.1
	to 2.2.4, 2.4.1 and 2.4.3 hold. Let Assumption 2.4.2 ii) hold with p = 2. Then, for

  Assume that F (S) has an empty interior and that the random sequence ((x n , m n , v n ) : n ∈ N) given by Algorithm 2.2 is bounded, with probability one. Then, w.p.1, lim n→∞ d(x n , S) = 0, lim n→∞ m n = 0 and lim n→∞ (S(x n )v n ) = 0. If moreover S is finite or countable, then w.p.1, there exists x * ∈ S s.t. lim n→∞ (x n , m n , v n ) = (x * , 0, S(x * )).

	6. Let Assumptions 2.2.1 to 2.2.3, 2.4.1 and 2.5.1 hold. Let Assump-
	tion 2.4.2 i) hold with p = 4. Th. 2.6 establishes the almost sure convergence of x n to the set of critical points of F ,
	under the assumption that the sequence ((x n , m n , v n )) is a.s. bounded. The next result
	provides a sufficient condition under which almost sure boundedness holds.
	Assumption 2.5.2. The following holds.
	i) ∇F is Lipschitz continuous. ii) There exists C > 0 s.t. for all x

  d}}.

	Lemma 2.14. Under Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2, the following holds true.
	i) For every compact set K ⊂ Z + , there exists c > 0, s.t. for every

  2.13.

	Lemma 2.16. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0

  2 .

	Proposition 2.19. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that
	0

  .7.3.2 Asymptotic behavior of the solution to (ODE) Proposition 2.23 (APT). Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold true.

	2

). Corollary 2.22. Let Assumptions 2.2.2, 2.2.3, 2.7.1 and 2.7.2 hold. Assume that 0 < b ≤ 4a. For every z ∈ Z + , lim t→∞ d(Φ(z, t), E) = 0 .

Proof. Use Prop. 3.16 with K := {z}. and

(Haraux, 1991, Th. 2.1.7

).

  Assumption 3.2.1. The function F belongs to C 1 (R d , R) and ∇F is locally Lipschitz continuous.Note that this assumption implies that the infimum F of F is finite, and the set zer ∇F of zeros of ∇F is nonempty. Assumption 3.2.3. The map S : R d → R d + is locally Lipschitz continuous.Assumption 3.2.4. The continuous functions h, r, p, q : (0, +∞) → R + satisfy:

Assumption 3.2.2. F is coercive, i.e., F (x) → +∞ as x → +∞.

  3.3, we provide noise conditions allowing to choose larger stepsizes (see Assumption 3.3.4 compared to Assumption 2.4.2 with p = 2 in Chapter 2). Concerning the stability result (Th.3.4), we relax Assumption 2.5.2-iii) of Chapter 2 which is no more needed in the present chapter (seeAssumption 3.3.6) thanks to a modification of the discretized Lyapunov function used in the proof (see the proof in Section 3.6.4 compared to Section 2.9.2 in Chapter 2).

  we easily obtain from our assumptions that the function g defined in (3.1) is continuous, and that g(•, t) is uniformly Lipschitz on t ∈ [t 0 , t 0 + A]. In these conditions, Picard's theorem asserts that (ODE-1) starting from z(t 0 ) = z 0 has a unique solution on a certain maximal interval [t 0 , T ). Lem. 3.12 shows that v(t) ≥ 0 on this interval.Let us show that T = ∞. Applying Ineq.(3.16) with (v, m, x) = (v(t), m(t), x(t)) and using Assumption 3.2.

4, we obtain that the function t → E(h(t), z(t)) is decreasing on [t 0 , T ). By the coercivity of F (Assumption 3.2.2) and Assumption 3.2.4-i), we get that the trajectory {x(t)} is bounded. Recall the equation ṁ(t) = h(t)∇F (x(t))r(t)m(t).

  .34) Proposition 3.16. Let t 0 > 0 and let Assumptions 3.2.1 to 3.2.4 and 3.3.5 hold true. Let S be the limit set defined in Eq.(3.31). Let Φ

  Recalling that L W δ = L E∞ + δL G + δL H and combining Eqs.(3.35),(3.36) and (3.37), we obtain for every t ≥ t 0 ,

2 . (3.37)

  3.3. Finally, Assumption 3.3.5 implies that the set W δ (Λ Φ ∞ ∩ S) is of empty interior. Recall that Assumptions 3.2.1 and 3.2.3 both follow from Assumption 3.3.3 made in Th.3.3. Given Prop. 3.16, the proof is concluded by applying(Benaïm, 1999, Prop. 6.4) to the restricted semiflow Φ∞ (with (M, Λ) = (S, ΛΦ∞)). Note that a Lyapunov function for ΛΦ∞ is what is called a strict Lyapunov function. Such a function is provided by Prop.3.16. We obtain as a conclusion of

  and Eq.(3.64) hold true with m = 2. Assume in addition that ∂ n 2 ∂ k 1 ε exists and is continuous for 0 ≤ n < 2 and 0 ≤ k + n ≤ 2, and furthermore, that there exists a bounded neighborhood V ⊂ R d of zero such that sup

		∂ 2 ε(y, t) < +∞.	(3.69)
	(y,t)∈V×I
	Then, there exists a neighborhood V -⊂ R d -of zero such that
	sup	∂ 1 ∂ 2 w(y -, t) < +∞ ,	(3.70)
	(y -,t)∈V -×I
	sup (y -,t)∈V -×I	∂ 2 2 w(y -, t) < +∞ .	(3.71)
	Proof. By Prop. 3.18, the general solution φ(s, t, y) of the ODE (3.62) can be written
	as φ(s, t, y) = (φ

-(s, t, y), φ + (s, t, y)) with φ + (s, t, y) = w(φ -(s, t, y), s) for each s ∈ I.

  3.21 on the event A p = E p ∩ [y n → 0]. In the remainder, C will indicate some positive constant which can change from an inequality to another one. First, we verify that n ρn 2 < ∞ on A p by controlling each one of the terms of ρn . Combining the boundedness of ∂ 1 w(•, •) with the summability assumption n ρn+12 1 zn∈W < +∞ a.s., we immediately obtain on A p that, given our choice

	of σ, n C γ 2 n+1 by invoking Prop. 3.20. In addition, using the boundedness of ∂ 2 ρ+ n+1 -∂ 1 w(y -n , τ n )ρ -n+1 2 < +∞. Moreover, it holds that γ n+1 /γ n+1 1 w(•, •), we can 2 ≤ write

Table 4 . 2 -

 42 Theoretical guarantees of variants of Adam. The gradient is supposed L-lipschitz continuous in all the convergence results. g 1:T

Table 5 .

 5 1 -Comparison to closest related works.

	Discounted Actor Markovian Target Asymptotic Finite-time Timescales
	reward	critic	sampling 1 variable	results	analysis
	Lillicrap et al. (2016)					1
	Lee and He (2019)				2	1
	Wu et al. (2020)					3
	Shen et al. (2020)					2
	Zhang et al. (2021b)					2
	Ours					3

  This is because αt βt → 0, ξt βt → 0 by Assumption 5.5.2 , (ω t ) and (hence) (ω t ) are a.s. bounded by Assumption 5.5.3 , (R t ) is bounded by U R , θ → ψ θ (s, a) is bounded by Assumption 5.3.1 and S, A are finite.

	Let

  iv) (A4): The stepsizes (β t ) satisfy t β t = +∞ and t β 2 t < ∞ as formulated in Assumption 5.5.2.(v) (A5): The transition kernel associated to the controlled Markov process(Z t ) is continuous w.r.t. the variables z ∈ S × A, χ ∈ R d+m , ω ∈ R m . Continuity (w.r.t.to the metric of the weak convergence of probability measures) is a consequence of the fact that we have a finite-state MDP.

  As for item (c), we first observe that H(θ t ) is bounded since θ → ψ θ (s, a) is bounded for every (s, a) ∈ S × A thanks again to. Then, item (c) stems from the fact that ω tω * (θ t ) → 0 as shown in Th.5.3 . (d) Similarly to H(θ t ), upon noticing that the reward sequence (R t ) is bounded by U R and the sequence (ω t ) is a.s. bounded by Assumption 5.5.3 , the quantity δ t+1 ψ θt ( St , Ãt ) is also a.s. bounded. Then, item (d) is a consequence of the square summability of the stepsizes α t ( t α 2 t < ∞) as guaranteed by Assumption 5.5.2.

	(1) t	< ∞ w.p.1 ,
	(b)

Lemma 5.14

. (a) ∞ t=0 α t ∇J(θ t ), e ∞ t=0 α t ∇J(θ t ), ηt+1 < ∞ w.p.1 ,

(c)

  leads to ν t+1 2 ≤ (1-2κβ t ) ν t 2 +2β t Λ(θ t , ωt , ω t , xt )+C(α t +ξ t ) ν t +C(α 2 Control of the Markov noise term Λ(θ t , ωt , ω t , xt ) . We decompose the noise term using a similar technique to Zou et al. (2019b) which was then used in Wu et al. (2020); Shen et al. (2020). Let T > 0. Define the mixing time τ

	t +ξ 2 t +β 2 t ) . (5.50)

  .54) (b) Control of Λ(θ t-τ , z t-τ , xt ) -Λ(θ t-τ , z t-τ , xt ): following similar arguments toWu et al. (2020);[START_REF] Shen | Asynchronous advantage actor critic: Non-asymptotic analysis and linear speedup[END_REF], we upperbound the conditional expectation of this error term w.r.t. St-τ+1 , ωt-τ , ω t-τ and θ t-τ . Note that our definition of xt is slightly different from the ones used in the two aforementioned references because of the third component of xt (and also xt ) which is generated according to the original kernel p instead of the artificial kernel p. We have

  .76) (d) Since ω * is Lipschitz continuous, we can write:2 νt , ω * (θ t )ω * (θ t+1 ) ≤ Cα t νt .

  5.100)Now, we decompose the TD error by introducing both the moving target ω * (θ t ) and the TD error δ(x t , θ t ) associated to the true value function V π θ t : Incorporating this decomposition (5.101) into Eq. (5.100) gives:J(θ t+1 ) ≥ J(θ t ) + α t ∇J(θ t ), ( δ(x t , ω t ) -δ(x t , ω * (θ t )))ψ θt ( St , Ãt ) + α t ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))ψ θt ( St , Ãt ) + α t ∇J(θ t ), δ(x t , θ t )ψ θt ( St , Ãt ) -∇J(θ t ) + α t ∇J(θ t ) 2 -CL J α 2 t .(5.102)

δ(x t , ω t ) = [ δ(x t , ω t ) -δ(x t , ω * (θ t ))] + [ δ(x t , ω * (θ t ))δ(x t , θ t )] + δ(x t , θ t ) .

(5.101)

  T (ω tω * (θ t )), the Cauchy-Schwarz inequality leads to:E[ ∇J(θ t ), δ(x t , ω t )-δ(x t , ω * (θ t ))ψ θt ( St , Ãt ) ] ≤ -C E[ ∇J(θ t ) 2 ] E[ ω tω * (θ t ) 2 ] .(5.103)

  For the second term, we have:|E[Γ(x t , θ t-τ ) -Γ(x t , θ t-τ )]| = |E[ ∇J(θ t-τ ), δ(x t , θ t-τ )ψ θ t-τ ( St , Ãt )δ(x t , θ t-τ )ψ θ t-τ ( Št , Ǎt ) ]| = |E[ ∇J(θ t-τ ), δ(x t , θ t-τ )ψ θ t-τ ( St , Ãt )δ(x t , θ t-τ )ψ θ t-τ ( Št , Ǎt ) | St-τ+1 , θ t-τ ]| ≤ CE[d T V (P(x t ∈ •| St-τ+1 , θ t-τ ), P(x t ∈ •| St-τ+1 , θ t-τ ))]

		t
	≤ C	i=t-τ	E[ θ i -θ t-τ ] .	(5.106)

3.1 , one can show after tedious inequalities that:

|Γ(x t , θ t ) -Γ(x t , θ t-τ )| ≤ C θ tθ t-τ .

(5.105)

(b)

  5.8.2.2 Control of the linear FA error termRecall that θ → ψ θ is Lipschitz continuous, ∇J is bounded and remark that the quantity δ(x t , ω * (θ t ))δ(x t , θ t ) is bounded. Therefore, using the Cauchy-Schwarz inequality, we have:

E[ ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))ψ θt ( St , Ãt ) ] = E[ ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))(ψ θt ( St , Ãt )ψ θ t-τ ( St , Ãt )) ] + E[ ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))ψ θ t-τ ( St , Ãt ) ] ≥ -CE[ θ tθ t-τ ] + E[ ∇J(θ t ), ( δ(x t , ω * (θ t ))δ(x t , θ t ))ψ θ t-τ ( St , Ãt ) ] .

(5.109)

  E[∆(x t , θ t-τ ) -∆(x t , θ t-τ )] ≥ -C E[∆(x t , θ t-τ ) -∆(x t , θ t-τ )] ≥ -Cσ τ -1 .(5.113)(iv) For the last term, we can write:E[∆(x t , θ t-τ )|θ t-τ ] ≥ -C ∇J(θ t-τ ) • E[| δ(x t , ω * (θ t-τ ))δ(x t , θ t-τ )||θ t-τ ] .

	t
	i=t-τ	E[ θ i -θ t-τ ] .	(5.112)
	(iii)

  5.8.2.3 End of the proof of Th. 5.6Combining Eq. (5.102) with Eqs. (5.103),(5.108) and(5.116) yields:E[J(θ t+1 )] ≥ E[J(θ t )] + α t E[ ∇J(θ t ) 2 ] -Cα t E[ ∇J(θ t ) 2 ] E[ ω tω * (θ t ) 2 ] -Cα t ((τ + 1) 2 α t-τ + α T + FA ) -Cα 2 t .(5.117)

We follow their terminology for the word "adaptive" in this thesis, we bring to the attention of the reader that this same word has been used in the literature in different contexts for different purposes.

see though[START_REF] Jain | Accelerating stochastic gradient descent for least squares regression[END_REF] for an improvement over SGD for least squares regression (in stochastic convex optimization) with their Accelerated SGD algorithm which differs from the stochastic counterparts of the heavy ball method and Nesterov's accelerated gradient method.

sometimes dubbed inertial

Following the standard terminology of[START_REF] Sutton | Reinforcement learning: An introduction[END_REF] and the RL community, we also use the the terms agent, environment, action instead of controller, controlled system and control signal as one may encounter in control theory.

We use reward as in the RL literature instead of cost.

Other performance criteria such as the average reward and the total reward exist in the literature (see, for e.g.,[START_REF] Puterman | Markov decision processes: discrete stochastic dynamic programming[END_REF]). In this thesis, we focus on the expected discounted return.

An older version of the policy gradient theorem leading to the so-called REINFORCE algorithm was proposed by[START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF]. Here, we focus our exposition on this version which is more relevant for our purposes.

We use here the notation ∆π instead of the more common Aπ to avoid notational collision with the sequence of actions.

Note here that in its original form, the theorem uses Qπ instead of ∆π in Eq. (1.4). This replacement can be straightforwardly shown using the identity "∇ ln π θ = ∇π θ /π θ " and noticing that the function Vπ does not depend on actions.

B(R k ) is the Borel σ-algebra on R k .

Note here the slight difference with the algorithm in (1.15e) for which is outside the square root. This slight simplification considered throughout Chapter 3 allows us to avoid some differentiability issues which we have tackled in Chapter 2. Furthermore, debiasing steps are handled via the sequences (hn), (rn), (pn) and (qn).

 13 this notation means that the functions coincide for every t ∈ R+.

Please note here that these constants a, b in this chapter do not coincide with the constants a, b of Assumption

2.2.4 in Chapter 2.

α := 1 -θ is also defined as the KŁ exponent in other papers[START_REF] Li | Calculus of the exponent of kurdyka-łojasiewicz inequality and its applications to linear convergence of first-order methods[END_REF].

Note that the sup in the assumption is useful for more general (nondiscrete or infinite countable) state spaces.

Remerciements

Note that the classical condition a n ≤ 2/L shows up consequently. Moreover, the condition on (δ n ) is equivalent to

Note that we get rid of condition (4.21) while allowing adaptive stepsizes a n (see Prop. 4.9).

4.8.6 Performance of gradient descent in the non-convex setting.

In the non-convex setting, for a smooth function f , we cannot say anything about the convergence rate of the sequences (f (x k )) and (x k ). Nevertheless, as exposed in (Nesterov, 2004, p.28), we can control the minimum of the gradients norms. We prove this result in the following for completeness.

Consider the gradient descent algorithm defined by: x k+1 = x k -γ∇f (x k ). Assume that γ > 0 and 1 -γL 2 > 0. Supposing that ∇f is L-Lipschitz, using Taylor's expansion and regrouping the terms, we obtain the following inequality:

Then, we sum the inequalities for 0 ≤ k ≤ n -1, lower bound the gradients norms in the sum by their minimum and we obtain for n ≥ 1 :

.

4.9 Proofs for Section 4.5

Three abstract conditions

Inspired from the abstract convergence mechanism of Bolte et al. (2018, Appendix), we show that similar conditions hold in our case. We highlight that these conditions are slightly different here, since we do not deal with gradient-like descent sequences (for which the objective function is nonincreasing over the iterations

Policy Gradient framework

From now on, we restrict the policy search to the set of policies π parameterized by a vector θ ∈ R d for some integer d > 0 and optimize the performance criterion J over this family of parameterized policies {π θ : θ ∈ R d }. The policy dependent function J can also be seen as a function of the parameter θ. We use the notation J(θ) for J(π θ ) by abuse of notation. The problem that we are concerned with can be written as: max θ∈R d J(θ) .Whenever it exists, define for every θ ∈ R d the function

where ∇ denotes the gradient w.r.t. θ. We introduce an assumption on the regularity of the parameterized family of policies which is a standard requirement in policy gradients (see, for eg., (Zhang et al., 2020a, Assumption 3.1) [START_REF] Konda | On actor-critic algorithms[END_REF], Assumption 2.1)). In particular, it ensures that ψ θ is well defined .

Assumption 5.3.1. The following conditions hold true for every (s, a) ∈ S × A .

(a) For every θ ∈ R d , π θ (a|s) > 0 .

(b) The function θ → π θ (a|s) is continuously differentiable and L π -Lipschitz continuous.

(c) The function θ → ψ θ (s, a) is bounded and L ψ -Lipschitz.

Assumption 5.3.1 is satisfied for instance by the Gibbs (or softmax) policy and the Gaussian policy (see (Zhang et al., 2020a, Section 3) and the references therein for details). Under Assumption 5.3.1 , the policy gradient theorem [START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] (Konda, 2002, Th. 2.13) with the state-value function as a baseline provides an expression for the gradient of the performance metric J w.r.t. the policy parameter θ given by:

Here, the couple of random variables ( S, Ã) follows the discounted state-action occupancy measure µ ρ,θ ∈ P(S × A) defined for all (s, a) ∈ S × A by:

is a probability measure over the state space S known as the discounted state-occupancy measure. Note that under Assumption 5.3.1 , the policy gradient ∇J is Lipschitz continuous (see (Zhang et al., 2020a, Lem. 4.2)).

Target-based actor-critic algorithm

In this section, we gradually present our actor-critic algorithm.

Actor update

First, we need an estimate of the policy gradient ∇J(θ) of Eq. ( 5.1) in view of using stochastic gradient ascent to solve the maximization problem . Given Eq. ( 5.1) and following previous works, we recall how to sample according to the distribution µ ρ,θ . As described in (Konda, 2002, Section 2.4), the distribution µ ρ,θ is the stationary distribution of a Markov chain ( St , Ãt ) t∈N issued from the artificial MDP whose transition kernel p : S × A → P(S) is defined for every (s, a) ∈ S × A by

and which is controlled by the policy π θ generating the action sequence ( Ãt ) . We will later state conditions to ensure its existence and uniqueness. Therefore, under suitable conditions, the distribution of the Markov chain ( St , Ãt ) t∈N will converge geometrically towards its stationary distribution µ ρ,θ . This justifies the following sampling procedure. Given a state St and an action Ãt , we sample a state St+1 according to this artificial MDP by sampling from p(•| St , Ãt ) with probability γ and from ρ otherwise. For this purpose, at each time step t, we draw a Bernoulli random variable B t ∈ {0, 1} with parameter γ which is independent of all the past random variables generated until time t.

Then, using the definition of the advantage function, Eq. ( 5.1) becomes:

From this equation, it is natural to define for every

where S t+1 is drawn from the distribution p(•| St , Ãt ) and ( St , Ãt ) t∈N is the Markov chain induced by the artificial MDP described in Eq. ( 5.3) and controlled by the policy π θ . Notice here from Eq. ( 5.4) that we need two different sequences (S t ) and ( St ) respectively sampled from the kernels p and p. In our discounted reward setting, using only the sequence ( St ) issued from the artificial kernel p would result in a bias with a sampling error of the order 1γ (see (Shen et al., 2020, Eq. (14) and Lem. 7)).

Supposing for now that the value function V π θ is known, it stems from Eq. ( 5.4) that a natural estimator of the gradient ∇J(θ) is δ

This estimator is only biased because the distribution of our sampled Markov chain ( St , Ãt ) t is not exactly µ ρ,θ but converges geometrically to this one. However, the state-value function V π θ is unknown. Given an estimate V ωt ∈ R n of V π θ t and a positive stepsize α t , the actor updates its parameter as follows:

(5.6)

Critic update

The state-value function V π θ is approximated for every state s ∈ S by a linear function of carefully chosen feature vectors as follows:

T is the feature vector of the state s ∈ S. We compactly represent the feature vectors as a matrix of features Φ of size n × m whose ith row corresponds to the row vector φ(s) T for some s ∈ S . Now, before completing the presentation of our algorithm, we motivate the use of a target variable for the critic. As previously mentioned, instead of a standard TD learning algorithm [START_REF] Sutton | Learning to predict by the methods of temporal differences[END_REF] for the critic, we use a target-based TD learning algorithm. We follow a similar exposition to (Lee and He, 2019, Secs. 2.3, 2.4 and 3) to introduce the target variable for the critic. Let us introduce some additional notations for this purpose.

Algorithm 5.1 Actor-critic algorithm (γ, (α t ), (β t ), (ξ t )) .

end for Output: Policy and value function parameters θ T and ω T .

In particular, it ensures the existence of a unique invariant distribution µ ρ,θ for the kernel Kθ for every θ ∈ R d . Note that we can replace p by p in Assumption 5.5.1 . Algorithm 5.1 involves three different timescales. The actor parameter θ t is updated on a slower timescale (i.e., with smaller stepsizes) than the target variable ωt which itself uses smaller stepsizes than the main critic parameter ω t . This is guaranteed by a specific choice of the three stepsize schedules. The following assumption is a three timescales version of the standard assumption used for two timescales stochastic approximation (Borkar, 2008, Chap. 6) and plays a pivotal role in our analysis. Assumption 5.5.2 (stepsizes). The sequences of positive stepsizes (α t ), (β t ) and (ξ t ) satisfy:

We also need the following stability assumption. Assumption 5.5.3. sup t ω t < +∞ and sup t θ t < +∞ w.p.1 .

The almost sure boundedness assumption is classical [START_REF] Konda | Actor-critic-type learning algorithms for markov decision processes[END_REF][START_REF] Borkar | Stochastic approximation. A dynamical systems viewpoint[END_REF][START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF][START_REF] Karmakar | Two time-scale stochastic approximation with controlled Markov noise and off-policy temporal-difference learning[END_REF]. The stability question could be addressed in a look-up table representation setting (for e.g., m = n). Nevertheless, this question seems out of reach in the FA setting without any modification of the algorithm. Indeed, as discussed in [START_REF] Bhatnagar | Natural actor-critic algorithms[END_REF](Bhatnagar et al., , p. 2478(Bhatnagar et al., -2479)), FA makes it hard to find a Lyapunov function to apply the stochastic Lyapunov function method [START_REF] Kushner | Stochastic approximation and recursive algorithms and applications[END_REF] whereas the function J can be readily used in the tabular case. We highlight however that Assumption 5.5.3 is indeed strong. The almost sure boundedness of the sequence (ω t ) could be probably relaxed by using a generalization to three timescales of the rescaling technique of [START_REF] Borkar | The ode method for convergence of stochastic approximation and reinforcement learning[END_REF] which was extended by [START_REF] Lakshminarayanan | A stability criterion for two timescale stochastic approximation schemes[END_REF] to two timescales stochastic approximation in the case of i.i.d. samples. Relaxing this assumption via using the latter result would also require to handle the Markov noise. We leave this technical question to future work. Concerning the sequence (θ t ), as previously mentioned, it seems out of reach without modifying the algorithm, Lakshminarayanan and Bhatnagar (2017) (see 5.7.2 Proof of Th. 5.4 In this subsection, we present a proof of Th. 5.4 which is similar in spirit to the proof in (Konda and Tsitsiklis, 2003, Section 6). Recall the notation Z t = ( St , Ãt ). Note that (Z t ) is a Markov chain. The actor parameter θ t iterates as follows:

where for every

(5.28)

We now introduce the steady-state expectation of the main term

Recall that µ ρ,θ is the stationary distribution of the Markov chain (Z t ). Define the functions H : R d → R d×m and u : R d → R d for every θ ∈ R d by

(5.29)

where Z = ( S, Ã) is a random variable following the distribution µ ρ,θ .

Then, we introduce the quantity ω * (θ t ) which approximates well ω t for large t (in the sense of Th. 5.3) and only depends on the actor parameter θ t . We obtain the following decomposition

t + e

(2)

where the function f : R d → R d and the error terms e

t and e

t are defined as follows

(1)

(5.34)

The bias induced by the approximation of ∇J(θ) by our actor-critic algorithm is defined for every θ ∈ R d by b(θ) := f (θ) -∇J(θ) .

( 5.35) This bias is due to the linear FA of the true state-value function. It is defined as the difference between the steady-state expectation of the actor update given by the function f defined in Eq. ( 5.32) and the gradient ∇J(θ) we are interested in . The following lemma provides a more explicit and interpretable expression for the bias b(θ).

The state-value function V π θ will be seen as a vector of R |S| .

(c) lim t→∞ e

(2)

Proof.

(a) The proof is based on the classical decomposition of the Markov noise term e

(1) t using the Poisson equation (Benveniste et al., 1990, p. 222-229). We refer to (Zhang et al., 2020b, Lem. 7 and Section A.8.3) for a detailed proof using this technique. The proof of our result here follows the same line. For conciseness, we only describe the necessary tools, pointing out the differences with (Zhang et al., 2020b, Lem. 7 and Section A.8.3) which is concerned with a different algorithm.

Let Z := S × A. First, define the functions g * θ : Z → R d and ḡ : R d → R d by: 5.40) for every z = (s, a) ∈ Z, θ ∈ R d . Observe in particular that e

(1) t = g * θt ( St , Ãt )ḡ(θ t ) . Recall that for every θ ∈ R d , the kernel transition Kθ is defined for every (s, a), (s , a ) ∈ S × A by Kθ (s , a ) = p(s |s, a)π θ (a |s ) (see Assumption 5.5.1). The idea of the proof is to introduce for each integer i = 1, • • • , d a Markov Reward Process (MRP) (Puterman, 2014, Section 8.2) on the space Z induced by the transition kernel Kθ and the reward function g * θ,i (ith coordinate of the function g * θ ). As a consequence, the corresponding average reward is given by ḡi (θ) (ith coordinate of ḡ(θ)). Then, the differential value function of the MRP is provided by v θ,i := (I -Kθ + 1µ T ρ,θ ) -1 (I -1µ T ρ,θ )g * θ,i as shown for instance in (Puterman, 2014, Section 8.2). The functions v θ,i for i = 1, • • • , d define together a vector valued function v θ : Z → R d . Under Assumption 5.5.1, using similar arguments to the proof of Lem. 5.9 (see also (Zhang et al., 2021b, Proof of Lem. 4, p. 26)), we can show that the function K

i is uniformly bounded w.r.t. θ under our assumptions . Moreover, the differential value function satisfies the crucial Bellman equation:

for every z ∈ Z . We use the above Poisson equation to express e

(1) t = g * θt ( St , Ãt )ḡ(θ t ) using v θ . The rest of the proof follows the same line as (Zhang et al., 2020b, Lem. 7 and Section A.8.3).

(b) First, recall that (η t ) is a martingale difference sequence adapted to F t and so is ( ∇J(θ t ), ηt+1 ). Using the boundedness of the function θ → ψ θ (s, a) guaranteed by Assumption 5. Using items (1) to (4), we have:

(5.77)

(5.78) From Eq. ( 5.61) and items (1) to (4) above, we have:

Solving this inequality yields:

Remarking that 0 < 2(αβ) < 2αβ and 0 < 2(ξβ) < 2ξβ, we obtain: -2(ξ-β) ) .

Then, we conclude that:

1 T Using the update rules of (ω t ), (ω t ) and developing the squared norm gives: ) . We now control the scalar product in Eq. (5.79). We decompose this term into four different terms:

(a) Using Assumption 5.6.2, it holds that: 2ξ t νt , ω * (θ t , ωt )ωt = -2ξ t νt , Ḡ(θ t ) -1 G(θ t )ν t ≤ -2ζξ t νt 2 .

(b) The boundedness of the function g implies that:

2ξ t β t νt , g(x t , ωt , ω t ) ≤ 2ξ t β t c νt .

(c) Applying the Cauchy-Schwarz inequality gives: